Education and Skills Committee

Meeting date: Wednesday, September 13, 2017

Official Report 424KB pdf

---

Scottish Qualifications Authority

Agenda item 3 is an evidence session with the Scottish Qualifications Authority. In the past year, the committee has done a fair amount of work on the SQA’s performance and we consider it to be a good time to get an update from it on its work. I welcome to the meeting Dr Janet Brown, the chief executive; Linda Ellison, the director of finance; and Robert Quinn, the head of qualifications development, with responsibility for English, languages, business and core skills. Good morning.

I understand that Dr Brown wants to make a short statement.

Yes, please. Good morning, everyone, and thank you for the opportunity to provide an update on the SQA’s activities.

I will focus on actions that we have taken in response to the committee’s report “Performance and Role of Key Education and Skills Bodies”. We have taken action by reviewing our approach to engagement and communication with teachers and lecturers and using the opportunity afforded by the revision of the assessments of national qualifications to streamline documentation and the accessibility of the material on our website.

A significant feature of the changes is closer engagement with those who need to use the information, namely teachers. That engagement has been to ensure that we more fully understand how we can best structure the essential and support materials, so that they can be easily found and are clearly worded.

The documents and the structure of the new web pages have been user tested and feedback from those activities has been used to make further improvements. All documents for a national 5 subject can be accessed through a single web page. The documents are more precise and clearly worded with, for example, the core specification for national 5 maths being reduced by almost 60 per cent.

Those involved in the development of the qualifications assessment and the materials and events that support their delivery are predominantly teachers, whether they are the principal assessors, members of question paper setting and marking teams, subject implementation managers who are involved in supporting teachers, or those on secondment from schools to work with the SQA as the revisions to assessment are undertaken. Those people have recent and direct teaching experience in delivering qualifications.

A strong engagement with and response to the feedback from teachers, parents and learners remain a key focus for us. The SQA receives a significant amount of feedback on all our work, which we carefully consider. The feedback is often positive about the nature and content of the qualifications and the changes that we are making to the assessment of national qualifications. However, as is the case with the submissions to the committee, some of the feedback that we receive raises issues and concerns. The points raised are carefully reviewed and discussed and actions taken.

We also commission independent surveys of our customers and the findings are used to improve how we work. In May 2016, we published the results of our fieldwork on how the qualifications were working on the ground in schools. That identified several areas that needed to be addressed by the wider education system and highlighted some of the workload for both learners and teachers associated with unit assessments. Although the SQA instituted revisions to address that issue for the 2016-17 session, further planned changes were superseded by the decision recommended by the assessment and national qualifications group to remove the units from national courses. That has been completed for national 5 and work is on-going for the higher during the current session. A follow-up review has been undertaken and the findings were published earlier this month.

The feedback from senior management in schools, teachers, parents and learners themselves provides valuable insight into how the senior phase and qualifications are perceived. The fieldwork report will inform the discussions that are taking place at the assessment and national qualifications group, particularly around national 4.

We hold webinars on specific subjects: teachers can participate in live sessions or watch on catch-up television at a time that is suitable for them. So far, we have held 18 webinars, and a further 11 are scheduled. We continue with the understanding standards support programme. The SQA has a dedicated team that works directly with every school in Scotland, visiting schools regularly to address concerns or arrange for specific subject support at local authority or individual school level.

I reassure the committee that, although significant progress has been made, the SQA will continue to find ways to improve how we communicate and engage with teachers and other stakeholders. We have a programme that is focused on supporting our customers, which is bringing together groups of teachers, lecturers, parents and other stakeholders to help us to develop new approaches to ensure effective, timely, efficient and valuable engagement with the SQA.

Thank you, Dr Brown. I will kick off with a question that touches on the first two themes that we want to talk about: the development of the national qualifications and communication with teachers.

The new national 5 qualifications will be taken in the current academic year. Given that the issue of communication and relationships with teachers was raised in our previous meeting on the matter, how confident are you that teachers understand the changes? More important, what work have you done to take the profession with you on the changes and to ensure that the information gets to teachers?

Given the timeframe that was available for the development of the revisions, to a significant extent, we engaged with our national qualifications support teams on that. The support teams are made up of practising teachers, representatives of the professional associations and other stakeholders. We have worked actively to ensure that there is appropriate representation on the teams. That is how we engaged to ensure that we got feedback on the changes.

On communication of the changes, straight after Christmas, at the start of the year, we put out a high-level notification on what the changes were likely to be. Subsequently, in April, the detailed changes were published. During the summer, we have been publishing further information, and we have been engaging at school level through our liaison managers and at subject-specific level with key organisations and other teachers. We have also engaged with headteachers across the country on the work that we are doing and the nature of the changes.

Robert Quinn might want to add to that.

The only thing that I would add is that we now have, integral to our qualification development teams, subject implementation managers. They are usually secondees from colleges and schools who are experienced in delivering the existing national courses and have a strong insight into the changes that are being taken forward with national 5. The subject implementation managers do a lot of work around the country, sometimes with groups of teachers from within a local authority and sometimes in individual colleges or schools.

Just in the past day or two, I had feedback from a subject implementation manager who has been speaking to principal teachers in the Highland Council, Glasgow City Council and Edinburgh College. That is part of a slightly more informal mechanism for rolling out the changes and ensuring that people are happy with them.

We follow up such work with events and webinars. For example, we run question-and-answer webinars, which do not have a fixed agenda. As a result of feedback from centres, for example, we decided to have a Q and A webinar on modern languages that everyone could log into, with an open agenda. We use informal and formal activity to get closer to teachers and ensure that they are comfortable with the changes.

We will come on to communication, and I am sure that there will be lots of questions from the committee on that. Let us focus on the development of the new national qualifications. I ask the panel not to veer into other areas, about which other members will want to ask.

Dr Brown, when you were at the committee in November last year, you said that Scotland needed to have a national conversation about national 4. You are in the middle of that conversation just now. What specific concerns have teachers presented to you about national 4 and how is the SQA addressing those concerns?

The national conversation is taking place under the auspices of the assessment and national qualifications group. The SQA undertook research in the second portion of the fieldwork that we undertook towards the end of last year and the beginning of this year, which was published earlier this month. That was done through talking to senior management teams and teachers in schools and, importantly, to learners and parents about their perceptions of national 4.

The report highlights that there was not only a difference of opinion across the country about how national 4 is operated but a difference of opinion within local authorities and schools across subjects. There is therefore a variety of views on the nature of national 4 as it stands today and what it might need to be if it were to be revised. The opinions vary. Learners and some teachers felt that not having an exam for national 4 is appropriate because it prepares people for a different pathway towards potentially going to college or into other vocational qualifications that are internally assessed. Others felt that the lack of an examination is an issue that needs to be addressed.

There was definitely a consensus that there needs to be some form of differentiation at national 4. Currently, there is a pass and that is the way in which national 4 is certificated, but the consensus is that there needs to be some differentiation. There was not a consensus view on what the nature of that differentiation should be.

All that work and the detailed feedback that we got from those different groups will be fed into the assessment and national qualifications group and, the next time it meets, it will discuss that along with other input to decide on the nature of the future for national 4.

Is what has been reflected in the press and to members of this committee accurate that a substantial number of teachers feel strongly that national 4 is not in the best educational interests of a large number of children? Is that an accurate reflection of the feedback that you have had?

The feedback that we got showed that a significant proportion of teachers thought that there should be some form of external assessment—exams, in some cases—at the end of national 4.

Is that likely to change and will there be some form of examination in national 4?

It is for the ANQ group to discuss how it wishes to approach the changes to national 4, if there are to be any. It is a topic that will be of great debate in that discussion.

On a related issue, some of the changes that have been made were made on the basis of feedback that teachers gave you that they felt that their workloads were extensive, although John Swinney had made a commitment to unburden them. That aside, do you believe that the national 4 and national 5 structure is educationally sound and in the best interests of our pupils in secondary 4 and 5?

National 4 is set at Scottish credit and qualifications framework level 4 and there are candidates who will achieve that and move from that into other types of education or directly into the workforce. There is absolutely a requirement for a qualification at SCQF level 4.

The nature of national 4 and its content has been built on the broad general education and curriculum levels that were associated with the experiences and outcomes. The discussion about progression from national 4 to national 5 and from national 4 to other forms of qualifications at either SCQF level 5 or SCQF level 4 needs the teachers to think about customisation for the individual child and student, and what is the best direction of travel for them.

It is important to recognise that part of curriculum for excellence is about personalisation and making sure that there are appropriate pathways. We spend a lot of time discussing national 4 and national 5. There are a large number of qualifications at SCQF levels 4 and 5 that might be a better pathway for many learners. That is something that we should be thinking about. Robert Quinn might want to add to that.

10:15  

We see a lot of innovative practice at that level of SCQF, in not just national 4 but other qualifications. Sometimes, national 4 and other qualifications are combined. For example, we have modern languages for life and work awards at SCQF levels 3 and 4. Some teachers combine those awards in an innovative way with the work that they do at national 4. They involve local employers such as Halcrow or Holiday Inn to let the youngsters see the absolute relevance of what they are doing. That type of qualification gives the youngster freedom through personalisation and choice. We need to be careful that we do not become overfocused on national courses, because they are not the only show in town. At all levels, we want the senior phase to be a mixed economy of provision of qualifications and other experiences with which people can engage. National 4 is part of that mix.

I understand that, but one of the great concerns that the committee was presented with was about who makes the decisions and where the accountability lies in the management board for the curriculum for excellence and the qualifications that feed into it. Over the summer, there was a great deal of discussion about accountability for the national 4 qualification and who has the power to make decisions about whether it will change. That is an urgent point as far as parents and teachers are concerned. There is quite a general feeling among teachers that national 4 does not appear to be working in the best interests of quite a number of pupils. I am keen to ascertain where the SQA is on that pressing issue for schools.

As I said, the decision as to any changes that would be required for national 4 would be taken through the ANQ group and the SQA would be charged with implementing those changes. That is the point that we are at just now. We are providing information and there will be a detailed discussion at the ANQ group, the membership of which the committee is familiar with—it includes the professional associations, School Leaders Scotland, the Association of Directors of Education in Scotland, Education Scotland, the Scottish Government et cetera. A variety of voices around the table would come to a decision as to what the nature of the changes would be.

Is the SQA providing advice on national 4? Do you have an opinion as to what should happen with it?

The reason why we did the fieldwork was that there is such a variety of opinion about what should happen with national 4. One of the things that we will be doing is setting out the pros and cons of the different options that are available.

Do you not regard yourself as having a role in coming to a view, given the fieldwork that you have done? This is a big issue, but you are just talking about it dispassionately and saying that, on the one hand, there is one view and on the other hand, there is another view. There are a lot of people with a lot of views on it, but we need somebody to be driving this. While no decision has been taken, young people in the system are being failed by not having a qualification that will be regarded externally as reflecting their abilities.

The SQA is a key part of the ANQ group and we will be providing advice and guidance to it.

You said that there are pros and cons. Do you have a view? If so, will you share it with us?

It really is a decision as to the nature and requirement of what national 4 is to be. There are a variety of positives about internal assessment and a lot of positives around not having an examination but having some other form of external assessment. Having externally assessed course work is seen as a way for candidates to demonstrate what they can do in a different way. You will have seen in the press the variety of opinions as to whether an exam is good or bad for candidates. That needs to be thought through carefully.

With respect, if we thought that, we would carry it out across the qualifications. We are saying somehow that people working at the level of national 4 do not need external assessment, but nobody is suggesting that there does not need to be external assessment at higher or advanced higher level, although the same pressures exist at those levels.

I agree that lots of people have lots of opinions on this, but what is the SQA’s opinion? Originally, it was not intended that curriculum for excellence would remove the external assessment. Way back in the day, the Munn and Dunning reports said that certification for all meant that we could ensure that resources were spread fairly across the cohort of young people. If there is no external examination, the chances are that the resources will be directed elsewhere.

In simple terms, before going to the meeting to make a decision, will the SQA come to a conclusion about what should happen? I am not asking whether you will reflect what everyone has said to you; I am asking whether, on the basis of what everyone has said to you, the SQA will take a view into that meeting.

I would add that, with regard to SCQF levels 6 to 10, there are qualifications that are assessed purely through course work and qualifications that are internally assessed and externally verified. It is only higher, advanced higher and national 5 that have exams, so there are high-credibility qualifications that are not associated with examinations.

So, logically, you would argue that higher and advanced higher should be the same. If there is a problem with external examination, surely it applies to all levels.

There are advantages to examinations, especially in relation to certain subjects. Some subjects, such as mathematics, are appropriately dealt with through an examination. However, it is difficult to do an examination in certain other subjects, such as dance, where the activity is performance related. It is a subject-specific issue, and we have reflected that in the nature of the assessments that are undertaken in the national 5, higher and advanced higher courses.

I have a couple of supplementary questions. Dr Brown, does the fieldwork that you mentioned include an assessment of where young people go after taking national 4s?

The fieldwork was undertaken by going to schools and trying to find out how the qualifications were operating within them. It did not go into the destinations of candidates.

Has anyone in our education world worked out where young people are going once they have passed or not passed national 4s?

Part of the Insight programme, which the Scottish Government runs in the measurement system for the curriculum for excellence, is focused on destinations of students.

I presume, however, that when the assessment and national qualifications group meets to assess the qualifications it will want to know where young people are going.

That is part of what we expect to hear from schools’ representatives, local authorities and colleges.

On the accountability point that my colleagues are raising, who is doing that work and will it be presented to the ANQ group at the same time as you present your evidence on whether national 4s are working?

That information will come from others around that table.

Okay—let me try again. To whom do we speak in order to ensure that that is happening?

It is not a matter for the SQA. It is for local authorities, schools and the Insight group to consider the destinations of the students.

You have just been asked by Johann Lamont to explain your assessment whether the examinations are working. Do you agree that a big part of that involves where pupils are going after they have done national 4s?

Yes, absolutely.

So why is that not part of your fieldwork? Would that not be an essential component of the assessment that you will present to the ANQ group that you have been describing this morning?

We are part of the system that runs curriculum for excellence—

We get that—yes.

What you ask for is not work that we have undertaken. We would be happy to do it if it were regarded as a focus for the SQA.

Is your advice to us that we should ask local authorities and schools?

Yes.

So there is no national picture whatsoever being built up of where—

The Insight tool is there to help you to understand the destinations.

I do not know what the Insight tool is; I am just an MSP. What is it?

I am sorry. It is the measurement system that the Scottish Government has put in place to measure the effectiveness of curriculum for excellence. It takes account not only of national qualifications attainment, but of other types of qualifications, positive destinations and a variety of other measures.

If I put “Insight” into Google, will it tell me where all the people who have gone through national 4s have gone after they have completed their national 4s?

I do not think that it will do that, at this point in time.

So we do not know what the destinations are, on a national scale.

We are currently in the second year—

Sure.

It is early days yet, but that is the plan.

I have two more questions. Who currently chairs the assessment and national qualifications group?

The Deputy First Minister chairs the group.

Is the decision about the changes that Liz Smith asked about ultimately for the Deputy First Minister?

Yes.

Finally, when is it likely that a decision will be made on the timescale? I have read various things in the past month that suggest that it could be a three-year process. Is that fair? If there are to be changes to national 4s in terms of the points that have been raised about external examinations, when do you think those changes might happen?

Again, the ANQ group will decide when the changes will come into force.

When will the group meet next?

I do not know what date has been set.

Will it be this year?

Yes.

Do schools know who has left school before the end of term and moved on to employment or, more likely, college? Is that information there but not centrally collated? Would that be what Tavish Scott was getting at?

Schools will know that students have left. Information on where they have gone needs to be actively collected. That is part of the plan to understand positive destinations. The information is not collected at national level, but schools will have their own information.

Do schools pass the information to councils?

I know that there is normally a survey of people who leave at the transition points in order to try to discern their destinations. That information is collected and reviewed at either local authority or college level. That is the process.

The SQA has information on people who do national 4 qualifications then progress to other qualifications; we have a part to play. It is important to recognise that people are not just national 4 candidates or national 5 candidates. Quite often, candidates do a mixture of national 4s and 5s. The candidate voice is an interesting factor in this—on their perception of their load, the worth of the national 4 and progression.

The SQA has a part to play in providing information on qualifications progression and the mix of qualifications that the youngsters do afterwards. Schools, local authorities and colleges should be collecting destination figures at the various transition points. There should be more focus on destinations in Scottish education and less focus on absolute attainment. I do not mean in relation just to national 4, but in relation to highers, too.

We are only looking for part of the information because the SQA has the information on people who have gone through national 4s and moved on to another education destination.

If the people have other qualifications, the information is potentially available—certainly within subjects. There might also be information in relation to other qualifications.

I will follow on from that line of questioning. The committee has focused on concerns about national 4, and about assessment and progression, but concerns are not limited to those areas. One of the key concerns that I have heard from teachers is about deliverability, especially from teachers who are charged with teaching both national 4 and 5 in a single class. The example that sticks in my mind is the physics teacher who told me that waves form part of the syllabus for both those qualifications, but for national 4 it is sound waves that are taught whereas for national 5 it is the electromagnetic spectrum, and the two are just not compatible. Is compatibility an issue that the SQA is looking at and seeking to resolve?

As we consider revision of national 4, and understand the requirements and how it is operating in schools, some of the content will be looked at. There are different levels of complexity in national 4 and national 5. If significant numbers of schools are delivering multilevel teaching, we have to start looking at content. The challenge would be that we would change again after looking at the content: we would have to change the content of either national 4 or national 5 so that blended learning could take place.

Multilevel teaching was explicitly contained within the old regime of standard grades. It sounds as if the current national 4 and national 5 were designed in isolation. Is that correct?

They were not designed along the lines of standard grade; they were designed as a progression route, either to national 5 or to college.

So was no consideration given to how the two levels might be taught by a single teacher in a classroom?

Curriculum for excellence required that the qualifications be designed very specifically for progression from national 4 to national 5. We did not do the same as was done in the previous system. One of the issues that we need to look at is that we see teachers using the current qualifications as they used standard grades in the past.

10:30  

My interpretation of that is that deliverability was not taken into account.

The qualifications were organised into what were called units—modules, if you like—which were set up hierarchically. The intention was to create, as far as possible, a situation in which there was a hierarchical progression from one SCQF level to another, but I understand that there were challenges in some of the science subjects. Although the unit titles were similar and covered the same broad area, there were differences in content. The intention was to organise the courses into organisers called units, but they have been removed and we are re-evaluating that. We now have other organisers in the courses and are taking into account the deliverability issues.

I know how I interpret that answer.

Another key concern is breadth. Since 2014, the number of presentations for modern language subjects has declined by about a quarter. Compared with the old standard grade regime, there has been a 60 per cent decline in the number of presentations at national 4 and national 5. Are you concerned about the breadth of qualifications that are being taken? Are you looking at whether the examination regime might be contributing to that trend?

We have discussed previously with the committee the number of qualifications that candidates are taking and the associated changes in the pattern of qualifications. One of the things that we need to understand is whether that is a result of people bypassing a particular level and moving straight on to higher, for example, which was one of the aims of curriculum for excellence. It is also associated with the nature of the other types of qualification that are available. Robert Quinn has highlighted some of the other languages qualifications and awards that are available in schools. We monitor the presentation pattern for particular courses, and we give feedback on it to Education Scotland, the Scottish Government and local authorities.

It is also necessary to take into account the potential changes in the school roll and the philosophy of CFE, which is about not cramming three years of successive qualifications into the senior phase, but instead diffusing the process over the full three years of the senior phase. Taken together, those factors mean that there has been a change in the number of students who undertake qualifications in secondary 4, for instance.

Some of those factors might account for the change, but even if we take account of the effect of bypassing and so on, we are moving towards a situation in which, out of a cohort of 130,000 pupils, only about 20,000 are taking languages. A whole category of subjects is simply not being taken by people at school—people are leaving school without any languages qualifications. Is that not a concerning situation?

The SQA’s position on that is that we want all language qualifications to flourish—we want young people to engage with languages. Previously in the standard grade system, English, maths and a foreign language were compulsory, but that is not the case now. When choice is offered, statistically the only direction in which take-up of languages can go is down.

We want to retain the broadest possible provision of languages and to provide qualifications that pupils can engage in. For example, there is the modern languages for life and work award, for which there were 3,000 entries last year. Some people took that qualification in lieu of a national course. It is a much more flexible qualification, but it still gives people an insight into a language and, more importantly, how they can use the language in a real-life environment.

I agree with the sentiments that you have expressed. Obviously, we want more people to engage with languages, so our aim is to continue to provide as broad a base of provision as is viable to enable us to meet that objective.

Finally, are you taking a broad macro view of the total level of language attainment in schools that takes account of alternative qualifications and the bypassing of particular levels, and which results in an aggregate picture of the situation with modern languages in secondary schools?

We can certainly provide that information. When we review our language provision and look at its success or otherwise, we take into account the totality, not just the national courses.

We have talked a lot today about moving on from N4s to N5s and then to highers, but all those stepping stones ultimately lead to the world of work. What engagement have you had with employers, particularly those in small to medium-sized enterprises, many of whom will be the employers of the youngsters who leave school at N4 level? There is a lot of confusion at the moment, and a lot of employers have told me that, because they do not really understand the system any more, they are going to devise their own assessments when recruiting people. What are you doing about that?

In the course of developing the original curriculum for excellence qualifications and the revisions, we have had a programme of engagement with businesses and have communicated to them the nature of the qualifications. We also work with parents associations to ensure that parents are aware of the nature of the qualifications and the changes, the aim of the qualifications and the potential progression pathways from them.

Our work with employers is absolutely critical in developing the content of the qualification, particularly with regard to the developing the young workforce programme and ensuring that what is in the qualification is relevant to employers. In that respect, though, I am talking not necessarily about the national qualifications but about the other qualifications.

The issue of core skills, which is Robert Quinn’s particular focus, is about making sure that candidates who take our qualifications have that basis to enable them to be successful in work. One of the challenges in that respect is to ensure that we have strong engagement with employers. As far as the SME population is concerned, we work closely with the Federation of Small Businesses, the Confederation of British Industry and so on.

Are you confident that you can meet what employers are asking for? Obviously, there has been a false start, and you are now revising the system. Was what they told you akin to what you had decided to do the first time round, or are you now moving towards something that they had already asked for?

Employers are relatively familiar with internal assessment and units, and with taking their current and new employees through skills development, not only in a form that can be found in colleges but in a framework that is very similar to that for national 4. They are familiar with that and are comfortable with the approach that is being taken. Over the past couple of years, there has been an increasing concern about the perception of national 4 across the board, so we need to go back to employers and discuss the matter further.

Thank you. We will now move on to communication.

Good morning, panel, and thank you for coming. The theme of communication came up at your last appearance before the committee, and quite a lot of reference was made to jargon and language. In your opening statement, you mentioned “closer engagement” and “strong engagement” with teachers, so I would like to ask you quite simply: what does that engagement actually look like? Who are you speaking to? How are you speaking to them? How are you acting on what you are hearing, and how will you know that it is working?

Do you want to start off on that, Robert?

Yes.

With regard to qualifications, our mantra is constant engagement with teaching. After all, teachers and lecturers are the people with whom we work to develop and maintain qualifications, and we have taken a lot of steps to try to reach out more to the wider group of teachers who are not directly involved. As well as the customer liaison people, the new subject implementation managers out in the field are teachers, which means that we have teachers talking to teachers.

That is a wee bit different from the work of the liaison team, who are excellent but are primarily the voice of the SQA. Those people have a two-way communication channel and we can use them as a benchmark. If we are thinking about how we want to frame a piece of guidance on a particular subject or a piece of advice, or if we are getting from teachers a lot of feedback that they want more clarity around something, we can road test any communications that we have with those teachers. That has been a valuable resource over the past year or so. Indeed, we are trying to strengthen that further in areas where we need to provide even more support and clarity. That is one example.

How many of those posts are there? To be honest, the title sounds quite jargony.

There could be a qualifications manager who is a member of SQA staff, usually with an education and teaching background, and within that person’s team we would second a subject specialist. For example, in modern languages, we have taken two subject specialists into the SQA on secondment who can go out and work with and speak to teachers. They bring us closer to the teaching profession.

What sort of reach can two members of staff have? How much ground are they covering?

Quite a lot. They are free to go out and engage with local authorities, and they can meet colleges. It is a significant investment. In some areas, there might be a qualifications manager who looks after three or four high-profile subjects, but they will have individual subject implementation managers attached to those. That can be quite a rich resource if they use the time wisely.

We have also held workshops with teachers to tell them how we are thinking about restructuring our documents and ensuring that our website is more accessible. We are basically engaging them and getting their feedback about whether what we are proposing is a valuable way forward or whether we should stop and completely rethink it. We have had some very strong feedback on our website, as you can imagine, and we are taking short, medium and longer-term actions to address that.

We engage with teachers in a variety of different ways. Robert Quinn is highlighting the fact that, when we write the documents, we now not only write them from the perspective of what needs to be in them but get teachers to read them and tell us whether they are logical and easily understandable. We have removed a lot of the jargon from a lot of subjects. The new national 5 documentation is a lot more streamlined, and that will happen again this year for higher. It is a real focus.

I wanted to ask you specifically about that. Your submission gives the example of a course specification that has been reduced by 60 per cent. How do you know that that has worked for teachers? What feedback have you had and what further measures have you taken if other changes have had to be made?

We road tested the revised specifications with our subject implementation managers. We also spoke to teachers via our national qualifications support team and wider teacher networks. The key thing in achieving that reduction was to learn lessons. There were a number of standard statements—jargon, if you like—that were repeated across a number of documents. They were educationally appropriate statements to make but, when they kept being repeated, teachers had to wade through them to access the information.

Having taken a step back and taken feedback, we have stripped a lot of that out so that, when a teacher reads a course specification now, they can get to the heart of the subject right away without reading through a few pages on the broader educational aims of the development. Those were all laudable, but they did not get to the meat of the matter. That was the strategy that we used to achieve that.

How do you know that that has worked?

That is the feedback from teachers. When we spoke to Highland Council, Glasgow City Council and Edinburgh College, we asked them for feedback. The proof of the pudding will be when we move it forward. People are telling us that they like the reduction in jargon, the increased brevity and the instructive nature of the document where it needs to be instructive. There are aspects of guidance and autonomy that are important—we value those—but for some things we have to say, “This is how you do it” and try to be a wee bit more instructive.

Thank you.

10:45  

Dr Brown will not thank me for this but, last November, as the convener will remember, we had a submission from a physics teacher who said with regard to the higher physics unit and assessment that there were 81 pages of guidance across five different documents, three of which were accessible on the main SQA website and two of which were on the SQA’s secure website. I take it that that situation is rather better now.

It is better for national 5 and it will be better for higher. We have redesigned the pages, so all the links are now on one page. When we do the revisions of the assessments for higher, we will address that, too.

So the situation is better with regard to national 5—it is pretty well sorted. However, for teachers who are teaching higher physics, there is still a bit of a challenge.

There is still a bit of a challenge, but everything is in one place now.

So there is one website.

There is one set of links. Our longer-term approach is to develop a single sign-on and everything else that will make the process easier for teachers.

I take it that there are no longer 81 pages of guidance. I assume that the guidance has been stripped down to what really matters—

For national 5.

And for highers?

That will be done this year. We are taking advantage of the revisions to assessment in order to refine our documentation. This is a good and timely opportunity to do that.

You said that you get feedback from teachers. Do you have any plans to make an evaluation of your communications strategy after the changes to see how effective it is?

Yes. We commission a company to do regular independent surveys for us, and we will continue to do that. I think that we have highlighted that to the committee in the past.

I am grateful for the submission that we have received. However, I am slightly concerned that, in a 15-page document, there are only two mentions of parents, one of which is about improving the search experience on your web pages and the other of which is about materials that schools can adapt for the use of pupils and parents. I am not hearing from the panel this morning about engagement with parents, even though parents are key to what we are trying to do. How does the SQA engage and communicate with parents?

Part of the fieldwork whose results we have just published and the fieldwork that we conducted last year involved talking to parents and hearing how they thought that their children were doing in terms of their qualifications. That was a significant portion of that work.

There is no evidence of that in your submission, and I have not heard you talk about it this morning.

That is because we were focusing on certain specific questions. Perhaps that was an omission and we should have included more information about our parental engagement. I can reassure you that we are involved in such engagement. If you look at the fieldwork report that was published earlier this month, you will see a specific section on parents and carers. We work with both major parental organisations to ensure that we provide materials that are appropriate and meaningful to parents, because we recognise that some of the documentation on our website that is valuable to teachers is not necessarily as valuable to parents. We are undertaking a significant programme of work in that regard. We also meet the parental bodies regularly so that we can understand what they are doing. Also, they sit on the SQA’s advisory council.

The part of your submission that talks about means of engagement says that you made visits to schools in 2015-16 and 2016-17 to gain insight into teacher and pupil opinion. Where is the parents’ opinion in that?

That is what I was talking about. Obviously, we omitted to add a reference to parents and carers at that point. However, part of the second portion of the fieldwork that was done in 2016-17 involved deep discussions with parents and carers about how they felt that the qualifications were operating. The report highlights the feedback from parents and carers on the same sort of questions that we were asking teachers, members of the senior management team and candidates.

Do you not think that, since you are here to tell the committee about how well you are communicating, it is a glaring omission in your evidence that you are not telling the committee about your engagement with parents?

We could have done a lot better on that, yes.

We will move on to marking and invigilation of scripts and examinations.

For this year’s examination diet, the SQA decided that the papers would be withheld for 24 hours after the examinations and then released. How did you consult teachers and students before making that decision?

We made that decision in response to a series of events that had occurred in previous sessions. We had received complaints associated with specific posts being made during the course of an examination. We took the action to address the point.

We have reviewed that action and decided to revert to the original policy, which had not been fully implemented. The examination papers will be made available at the end of the school day after all the candidates have undertaken the examination.

I am glad to hear that the decision has been reversed. The feedback I received was that it was increasing rather than decreasing anxiety.

Given where we were last year, when the committee went through with you the issues of teacher trust in the SQA and the breakdown in communication, why did you not consult before making the decision? The Educational Institute of Scotland said that it was another example of an apparent lack of trust that the SQA had in teachers that further damaged your reputation in their eyes. Why did you not ask them before making that decision?

There are definitely lessons for us to learn. We also need to have an appropriate mechanism that ensures that all teachers and all candidates are given a fair experience on an examination day.

One of the issues that we had heard about and had evidence of was inappropriate use of examination papers during the course of an examination. We needed to address that and we needed to address it quickly. We did not consult and we probably should have done. We have subsequently consulted and reviewed how question papers are handled in schools. We have gone into many schools and done a lot of audits to understand how we can best change the policy and that has resulted in our change of approach for the coming year.

We learned the lesson about consultation and went back out and talked to teachers and went into schools during the examination period to understand the implications of any changes we might make to the availability of examination papers.

That is positive to hear. Part of the issue was the language that was used in your communications in response to media enquiries explaining why you had made the decision. Many teachers felt that the phrases used around inappropriate postings were accusatory. Have you reviewed your communication strategy in terms of the language the SQA uses?

Yes. We have talked a lot about the SQA taking stock of how we communicate and how we engage with teachers and the broader stakeholder community, including parents and carers. We have tried to understand when we need to be directive, when we need to be supportive and when we need to explain things in a much better way.

There are occasions, as Robert Quinn highlighted, when we need to be clear and directive on how things have to happen in order for secure certification to occur. We also need to be very cognisant of the nature of how we communicate and we have taken action on that.

I would like to ask a question that relates to our discussions of November last year and the recommendations that the committee gave to the SQA arising from them. Point 28 of our recommendations, which I am sure you are familiar with, was:

“The SQA’s core business is producing and marking exams. Errors in these areas are unacceptable. The committee is concerned that Dr Brown suggests that errors occur because of excessive workload and has presented the solution as being more work on quality assurance.”

We urged the SQA to

“consider how to re-prioritise resources to address such issues”.

How have you done that?

This year, we introduced additional quality assurance processes for the question papers. The committee will have noticed that we withdrew and replaced one examination paper, but there were no other errors in the system that gave rise to any concerns. The previous year, we discussed the fact that we had introduced additional quality control mechanisms associated specifically with the sciences because of the complexity of those papers. We have rolled out those changes across the board and we have introduced additional quality assurance procedures for both the development and the printing of the question papers.

You said that excessive workload was a reason behind the errors in previous years. How has that issue been addressed?

We are in the middle of revising the assessments, and we have employed additional staff to take some of that work and spread it across the piece. Robert Quinn might want to cover that.

A practical example of that has been narrowing the number of examination papers or assessments that each qualifications manager is responsible for, so that there is less pressure on that individual. We have broadened the qualifications staff teams so that each qualifications manager now has a narrower range of subjects or exam papers, which allows us collectively to focus more on quality.

We have increased the size of the teams and decreased the number of specific subjects that each team is responsible for. That allows them the time and space to deal with those subjects.

One of the criticisms that people made, anecdotally, after your comments on the matter was about how the SQA is operating internationally, which might have had an impact on your core business of overseeing the qualifications regime in Scotland. How would you answer that criticism?

Can I ask you to go into the international stuff later? Johann Lamont has a specific question about that.

Yes. I will do that, convener.

I have in front of me the Official Report of our meeting on 23 November 2016, and I return to a question that I asked about the recruitment of markers. That had been a problem in the past. You had not been able to recruit enough markers and there was criticism that you had not been able to recruit markers with enough experience. I believe that you require three years’ teaching experience before someone can be a marker for the SQA—is that correct?

Yes, it is three years and we do not count the probationary year.

Can you confirm to the committee that every person who is now marking for the SQA has had at least three years’ experience?

Yes. When appointees register with us, they tell us how much experience they have. We accept only teachers who have that amount of experience to become markers.

Another issue that was mentioned outwith the committee by teachers whom I know, who had been markers, was that the computer system that they had used to mark online had been quite difficult to negotiate. They often got locked out and had to phone a helpline. Has that system been streamlined while you have been looking at your other computer issues?

Yes. Every year, we review how our marking system has operated and make revisions for the subsequent session. One issue this year was associated with markers using Apple computers. The issue was associated with the supply and it was dealt with very quickly. We have in place a system that allows markers to flag up to team leaders any issues that they find, which are then addressed as quickly as possible. We also make incremental improvements in the course of the programme.

Going back to the number of markers that you had, did you have any issues this year? You had issues with getting enough markers the previous year. Was there an improvement?

There was an improvement. We put in place additional recruitment mechanisms to make sure that we had the right number of markers, and we were not short of markers this year. We are currently beginning recruitment for the next session.

Last year, I asked about appeals and you corrected me—they are not called appeals any more; they are called post-results services, I think. Ruth Maguire will laugh at the jargon. Forgive me if I lapse into calling them appeals. As a result of the measures that you took to improve the marking and to have more markers recruited, was there a decrease in the number of schools that appealed the results or went to the post-results services?

11:00  

We are in the process of finalising that information. We have seen a slight reduction, but I do not have the final figures with me.

On a point of clarification, Dr Brown, it is my understanding that, if there is some form of dispute over a request for a marking review, you do not let the candidates see their papers. What is the justification for that? In other parts of the United Kingdom and other countries, people can see their papers. Given the correspondence to MSPs on the issue, it might be helpful if there were that transparency in Scotland too.

The current practice is that we do not return scripts. One of the challenges historically has been that there is one copy of the script. We are moving to a different environment in which we have electronic copies of scripts, and we will consult this year on changing that policy if the system requires it. Again, there is a variety of opinions across the country as to whether that is a good thing to do.

From an integrity point of view, it helps transparency, because the candidate can see the reason.

It has been a historical procedure in Scotland.

It is good to hear that there will be a review.

I have a couple of questions about how the SQA sets the pass rates, principally for highers, and how much those alter from year to year. I was told that the figure for higher physical education went up this year, and by more than the normal variance of 2 or 3 per cent, although please correct me if I have got the numbers way wrong. Was it a statistically significant change in that subject and, if so, why?

As the committee is aware, in the grade boundary meetings, we review the nature of the assessment and how the assessment has performed. If the assessment has been too challenging or too accessible, we adjust the grade boundary appropriately. The adjustment is not associated with the pass rate. The pass rate falls out of the discussion around where the grade boundary should be set.

In the case of PE, I am not sure of the exact criteria that were in place. However, that is the mechanism that we have. There is generally more than one assessment, and we look at the nature of the assessments that combine to make the final pass mark. We would adjust the grade boundary based on how each component of that final assessment has operated.

Would it be fair to say that, normally, the adjustments are fairly small in terms of a percentage change, such as 51 changing to 49 rather than a change of 10 per cent?

Yes, that is normally the case. There are exceptional cases when something happens that needs to be addressed and the grade boundaries are there to allow it to be addressed.

We will move on to the capital investment programme and international activities.

I have one or two questions about resources, but I will start with a more general question. Section 1.3 of the SQA’s written submission says:

“This year, conferences were held in Inverness, Dundee, Stirling, Edinburgh and Glasgow, attracting 357 delegates.”

That does not sound like many delegates for five events.

Are you talking about the SQA co-ordinator events?

Yes.

There is a co-ordinator for every school and centre in Scotland, so we are not inviting every teacher to the events.

So you expect one from each school.

Yes, and there are around 430 secondary schools in Scotland.

The events are for the people who help to co-ordinate the data that is transferred between the SQA and centres. They are for people in the co-ordination role as opposed to those in a qualifications role. The events are about managing requests for information from the SQA, marking reviews and so on.

Okay—let us get back to money. I note from your submission that you intend to invest in your information technology systems. Given my experience on the Public Audit and Post-legislative Scrutiny Committee of hearing about other public IT programmes, that makes me break out in a cold sweat. What exactly do you intend to do?

What we are doing—and have been doing for the past two or three years—is evolving our IT systems, rather than radically moving from one box to another. The current system in the SQA has been in place since the late 1990s, and we need to bring our systems up to the speed and efficiency that will enable us to do some of the things that we have been talking about, in relation to engagement, transparency and giving people access to our systems.

We are taking some of the functionality of the current system and putting it on other, smaller systems, building up a modular system. We are not buying a gigantic brand-new system. It is important that we do not do that because, like you, we are aware of the need to minimise the risks to anything that we do in the SQA. We are in the process of moving some of the things that our legacy system does on to a corporate business system, to de-risk the main system. That is what Linda Ellison is currently working on. We are slowly moving things off, in a controlled and managed way.

Who is managing the project? Is there input from the Scottish Government?

Linda Ellison, who is director of finance, the director of business systems and I have met the Scottish Government chief information officer, and we are in regular communication with the Government. We have used the templates for programme management for major IT changes that the Scottish Government advises people to use.

Linda Ellison wants to add to that.

We have spoken to the Scottish Government, and we have taken a proposal to it on how we will make the shift in relation to our IT change programme strategy. In essence, it is about de-risking and moving transactional data from the big main system.

All the information that we hold about candidates—on their units, exams and qualifications—sits on a single system that is made up of a number of systems, which is called our awards processing system, or APS. We are considering how we de-risk that system. Our new corporate business system will run all our back-office services and do all the transactional activity to do with the 15,000 appointees with whom we work—the teachers who get payments and draw expenses from us. That activity is being moved to the new back-office service system.

Is a contractor involved in that, or is all the work being done in-house?

The system is a new enterprise resource planning—ERP—software system called Agresso, which is supplied by Unit4 Business World and is used by a number of public sector bodies. We are using a contractor to help us with implementation.

We have been careful to ensure that we have good programme management of all that. We have had discussions with the Scottish Government’s information department, and we regularly look at its recommendations about what we should be tracking and make sure that we are tracking those things. We have a monthly meeting to monitor progress, and we have a project manager who deals specifically with the matter.

This is not something that an organisation whose business is data takes lightly. The SQA is a data organisation—that is what we do—so we have to do this carefully.

What is the cash value of the project over its lifetime?

We made a spend-to-save submission to the Scottish Government. In total, we will spend around £3 million to £4 million over five years, but we will secure savings as we progress through the programme, which will be demonstrated. We have moved some of our office systems on to a new system—we have moved to Microsoft—and we are securing substantial savings through that move, which are helping to fund some of the other changes that we are making.

If I may just labour the point, I take it that you are aware of the successive failures of various IT projects. I know that your project is smaller than some of those projects, but I hope that you have learned the lessons from them.

That is one of the reasons why we are working to the Scottish Government guidelines. The Government has looked at those projects, as have we. We also have a very experienced business systems director, who has implemented such changes successfully in other organisations. We are monitoring carefully how we do that.

Do you have many more questions, Colin?

Just a couple, convener.

You have referred to volumes of data being safely, securely and efficiently held. Are you confident that there is no chance of hacking?

Everybody remembers what happened to the national health service earlier this year. We took the matter seriously and the security department in our business systems directorate is actively looking at any on-going penetration challenges. For instance, this week, the United States Government announced that there was a potential hack out there, and that was immediately sprung upon by the SQA. We have a continuing programme of looking at the security of our systems. We also carry out penetration testing to ensure that we are as safe as we possibly can be.

My final question is on resources. In section 2 of your submission, you state:

“The Revision of the National Qualifications Programme announced by the Deputy First Minister will mean further change over the next three years and will also demand significant staff resource and focus.”

Given the scope of what you have talked about having done, that must be taking up considerable staff resources. Can you quantify what you mean by “significant staff resource”? Will you need additional resources?

As we have said, we have completed national 5 and we are moving on to higher, and the resources that were taken up by national 5 will now move on to higher. We anticipate the workload to be roughly similar, but we cannot be absolutely sure that we will not have to add more resource. At the moment, we think that it is roughly similar.

How big is the roaming team that is handling that?

Some of the changes that we have made are associated with improvements in quality assurance and some are associated with the additions that we need to undertake the revisions. We can give you the financial figure for that.

With CFE, we took a lot of additional people on over the three years of developing the new qualifications. With the change to the assessments for the new qualifications, we have retained some of those people and they have come on to our payroll as opposed to having fixed-term contracts or working as secondees.

Might they be surplus to requirements down the line?

They are the same people who will work through the assessments, whether the qualification is national 5, higher or advanced higher. We made the decision to move some of them on to our payroll on the basis of our human resources department’s advice on the legislation around the status of those individuals.

What was the cost of that?

That is not what we are asking about, Colin.

I am sorry, convener—I am just trying to understand how it is affecting what the SQA is going to do.

Old habits die hard.

We started this evidence session talking about national 4. We currently do not have a timeline for national 4, so we anticipate that we will need those people for a period yet.

You will recall that, in last year’s discussions, one of the issues that was flagged up was the question of your international work. The charge is that the organisation’s ability to concentrate on its day job is affected by its desire to do international work or work across the rest of the United Kingdom. In a letter to the convener, you say that such work

“generates contribution to SQA’s finances (thereby reducing dependency on the public purse).”

Can that be quantified?

Yes. It is in our financial accounts.

I am interested to know how your international work reduces

“dependency on the public purse”.

Roughly by how much does it do that?

We undertake international work for multiple reasons—

11:15  

Our time is limited. I am aware that your submission lists reasons why that international work is important, including leadership, being respected across the world and all the rest of it, but the core charge against the SQA is that it cannot do its day job because of its work to get contracts elsewhere. Will you clarify how the process works? You have a business department section and a contracts section.

Yes.

Those are two discrete sections that do not draw on the rest of the work of the SQA.

No, they do not.

Staff are not taken from anywhere else in the organisation to support business development or contracts.

No. When we win a contract, we bring on contractors to deliver it for us.

When you develop plans or bids for a contract, do you draw on expertise beyond your business development and contracts sections?

No.

Not at all.

No. The only time that would happen would be if we were to have a meeting associated with that work, but no other resources are used for international work outwith the business development team from that perspective.

What is the size of the business development team?

I am sorry, but I do not have that detail.

It would be useful to know what the size of the team is and whether it has grown or reduced, as well as what the total number of SQA staff is and whether it has grown or reduced.

It is important that we know whether there is a means of auditing the benefit. If the charge is that people have been drawn to that other work, you would need to balance that against knowing what contracts you have secured, the cost of securing them and the cost benefit of securing them, which are two different things. Can that information be made available to the committee?

Yes, it would be appropriate to do that. I remind the committee that we also develop qualifications for colleges and training providers for modern apprenticeships and foundation apprenticeships. A challenge is balancing the national qualifications developments alongside all the vocational and other qualifications that we are required to deliver for Scotland. Robert Quinn’s team has, for example, the responsibility for both vocational and national qualifications. That blend is probably more of a challenge than the international component.

You accept that that is core business.

Yes.

Looking for contracts elsewhere would have to be balanced against the benefits of securing them and the amount of finance that comes in, and would not draw on the time and energy of other people in the organisation.

Yes.

Yes. That forms part of our decision-making criteria about whether we would undertake a contract.

Sorry, but what would?

When we decide whether we are going to undertake a contract, we look at the cost benefit analysis and whether there would be any benefit to the SQA in doing it. We would not take on something that would not be beneficial.

There is a process for that. Presumably, there is also a commercial assessment subsequent to the contract being completed on the benefits of the contract.

Yes.

Is that publicly available? Can it be made available to the committee?

We can make it available to the committee. The information is not publicly available. We are bidding for contracts, so some of it is commercial.

You confirm, though, that staff are not used from elsewhere in the organisation to develop proposals for a bid or in delivering the contract, and that those are discrete to the business development and contracts departments.

Yes.

Yes.

On international awarding, which is qualifications undertaken by overseas centres, the normal approach is that people can pick up and use what we develop for Scotland, so the units and group awards can be used internationally. The focus is on the work that we do for the Scottish market but, if other centres want to use it across the world, they can do so. The primary source is the work that we develop for the Scottish market, for the benefit of Scottish candidates.

That is a slightly different point, is it not? That is about other folk noticing the good work that the SQA is doing and using it, which is quite different from a focus on finding a means to contribute to the SQA’s finances, thereby reducing dependency on the public purse. One might argue that it should not necessarily be your job to do that. If your focus is on delivering qualifications to Scotland, you should not have to find the commercial means for funding that. Do you not have a view on that? Would it be better if the SQA could simply rely on public resource to do its job, so it did not have to do commercial work?

There is commercial work and there is international work. The value that we get from working internationally and, to an extent, from undertaking some contracts, allows us to learn how to improve what we do in Scotland, too. There are multiple benefits; it allows us to think of different ways of doing things and to be a bit more innovative.

It would be interesting to get information on where that has happened. A lot of practitioners in Scotland have a sceptical view of the SQA, so it would be interesting to know where there is evidence that the international work has improved its reputation in Scotland. That is an issue that we can perhaps explore further.

We would welcome any information that you can give us about the process and the numbers that are involved in that side of the SQA’s work.

We will provide that.

Thank you.

You mentioned the value of working internationally. One of the Scottish Government’s goals is for the economy to develop more internationally. Last night, someone from the University of Edinburgh told me that education in Scotland has incredibly good links internationally, which could be used to improve the economy. Do you see that as an advantage to the work that the SQA does internationally?

Some of the work that we do internationally is about supporting universities. For example, the University of Stirling has a follow-on programme in Oman, which follows on from our higher national diploma programme in Oman, where Omani students study at the University of Stirling. Our work supports that university.

Before we finish, I request that you write with the details of the national 4 students who go on to further education. That relates to Tavish Scott’s point about continuing education. You said that you know where the students go once they complete their national 4s. If you could send that information, we would know what gap we need to fill and we can write to local authorities, the Deputy First Minister or whoever is the most appropriate person to respond on that matter.

Thank you very much for attending.

11:22 Meeting continued in private until 12:09.