Agenda item 2 is the third of four sessions for the committee’s pre-budget scrutiny. Earlier this month, we heard from Skills Development Scotland and the Scottish Further and Higher Education Funding Council. This week we have the Scottish Qualifications Authority and next week we will hear from Education Scotland.
I welcome to the meeting Dr Janet Brown, chief executive, and Linda Ellison, director of finance of the Scottish Qualifications Authority. Before we start, I record the committee’s thanks to the SQA for hosting a visit last week by Fulton MacGregor and Ross Thomson.
I understand that Dr Brown wishes to make a short opening statement.
Good morning, everyone. As the national awarding and accreditation body for Scotland, the SQA is responsible for the quality, validity and maintenance of the credibility of qualifications that are offered to learners. Given the size of the accreditation function, compared with that of the awarding body, we fully understand that the committee’s focus will be on the SQA’s role as an awarding body. In that capacity, the SQA develops qualifications to support the education and skills system. Our remit is to ensure that learners’ are able to progress successfully through learning, with qualifications, by building on their previous learning and by preparing the individual to be successful in the next phase of learning or going into work. SQA qualifications must therefore be set at the correct level, and course content must reflect the knowledge, understanding and skills that are necessary for achievement of a successful destination for that learner.
In the case of curriculum for excellence qualifications, the SQA was asked to develop a suite of qualifications that built on the learning level that candidates would have achieved during their broad general education.
The CFE management board approved the design and structure of those qualifications. The course content and the associated guidance were developed in consultation with stakeholders from across the sector, including teachers, colleges, universities and employers, in addition to professional associations and, in particular, subject specialists.
It should be noted that the assessments reflected the desire to provide through CFE opportunities for personalisation and choice for candidates, and for teachers to set assessments in the associated personalisation areas.
The SQA plays a significant role, and it is important to understand that role in the education system as a whole. The structure of curriculum models and the nature and number of subjects that are taken by an individual learner or group of learners are determined in a school or a college, and the qualifications that individual candidates undertake is a matter for those centres, in consultation with learners, parents and carers, to support the best interests of that young person. That does not fall within the SQA’s remit.
Development and delivery of qualifications is a complex process, and the SQA’s approach is to try to ensure the inclusion of teaching and learning professionals, subject specialists and those who will receive the learners once they have undertaken and achieved the qualification—namely, colleges, universities and employers.
We have a strong working relationship with teachers and lecturers across Scotland. Indeed, more than 15,000 of them work with us in partnership every year to develop and deliver the qualifications system as a whole.
As the national awarding body, the SQA is responsible for ensuring the standards, credibility and sustainability of the education system over time and has, in doing so, to balance the needs of a variety of stakeholders.
The introduction of CFE has been one of the biggest educational changes in Scotland, and that change continues with the Deputy First Minister and Cabinet Secretary for Education and Skills’s recently announced decision to ask the SQA to redesign the assessments that are associated with national qualifications.
As the new nationals have been implemented, the SQA has provided additional support and has listened to and responded to teachers’ needs. In the past few years, it has taken significant time through implementation and adjustment for that significant change to become embedded.
During development, it has been essential that the SQA communicate as much as possible with those who have been involved in the delivery, and in the learning and teaching of candidates who complete qualifications. That engagement has continued as the qualifications have been put in place over the past three to four years.
Our responsibility to understand and to address issues is well understood. We solicit and receive regular feedback from teachers and other interested stakeholders. However, the nature of our work means that we sometimes receive different advice from different sectors and people, which reflects the approaches of respondents to their subjects, their different opinions of course content and the approach that they would like us to take.
The SQA sees as being its responsibility the need to understand fully how our qualifications and assessments operate in schools and to identify issues that need to be addressed. It was in that vein that, at the end of 2015, the SQA undertook a review of qualifications design and detailed field studies in order to develop a research and evidence base that allows us to understand what needs to be strengthened and what has worked well as the qualifications were implemented. The field study involved talking to teachers, senior management teams in schools and learners, so that we could get a full understanding of how the qualifications were operating.
It is our responsibility to publish our findings, so that others in the system can understand the results. The committee may be aware that in May, the SQA published two research reports—one on the fieldwork and one on the detailed study of the nature and design of assessments. As a result of the evidence, the SQA made changes—again, in May 2016—to unit assessments for the current session and we communicated those and planned future changes through the subject review reports for each subject.
Currently, the SQA is undertaking further study and fieldwork—again, that involves senior management teams, teachers and pupils—not only to understand how the changes have worked to improve aspects of assessment, but to take further the discussion in order that we can understand the nature and experience of implementation of, particularly, national 4 and national 5 in the school and college sectors.
We are fully aware of our responsibility to provide value for money for the public purse, and we are focused on safe and secure delivery of our remit within a decreasing public purse. We regularly review our processes and procedures in order to identify how we can improve and how we can provide value for the public purse. We welcome the opportunity today to discuss our activities with the committee.
Thank you very much for that short but full statement. You mentioned that you have a “strong working relationship” with teachers. Our online survey and a meeting that I had with teachers suggest that that is not quite the case. Why have there been so many negative comments? I would not usually take anonymous online comments seriously, but pretty much everything that those sources said was backed up by the teachers to whom we spoke face to face, who were very strong in their criticism of aspects of the SQA. What has been done over the past year to build the “strong working relationship” with teachers? Why do teachers not seem to feel that they have that strong working relationship?
There are two issues there. The education and qualifications system is run in partnership between the SQA and teachers: we have 15,000 teachers—a significant number—working with us every year on developing the qualifications and assessments that are undertaken every summer. They are very supportive and engaged in ensuring that we are trying to do the right thing for the learners of Scotland.
As with the introduction of any major change, such as curriculum for excellence, there are things that people agree with and things that people do not agree with. The approach that was taken with CFE was very much to move away from a prescriptive national assessment bank based assessment to something that is much more teacher driven. That has proved to be a challenge for some teachers; it is not something that all teachers wish to do.
As we move into the new situation in which there will be no unit assessments, that aspect of the discomfort that teachers have found with the approach that was agreed with CFE management board will be removed.
We also regularly get feedback from teachers about what they like and do not like in the courses and about how the assessment has worked. That was partly why we went out last year to do fieldwork: we wanted to understand better what the feedback meant so that we could address it.
Subject review reports for every single subject were published to tell everyone what we were planning to change and to address issues that we identified.
The group of teachers whom Ross Greer and I met covered primary and secondary schools, although they were mainly secondary teachers. They also had varying degrees of experience. They all, without exception, had the same complaints about the relationship between the SQA and the teaching profession. They do not seem to think that it is working.
Dr Brown’s response touched on communication: the teachers seem to think that communication between SQA and the teaching profession is non-existent or negative, rather than positive. They are not getting anything from it and it seems to obstruct more than to help them. Many things that the SQA has put in place for teachers seem to be done for no apparent purpose, as far as the teaching profession is concerned. Either the SQA is doing it wrong or it is communicating badly. Those are the only reasons why the teachers could be unanimously telling us that they do not feel that the SQA is getting it right for them.
Communication, as we all know, is an extremely complicated and challenging area, and it could always be done better. I will not—
Communication should not be like that, if you use simple language to describe complex reasons. I accept that the SQA is talking to lots of different audiences, but communication should be fairly straightforward.
We are considering the feedback that we got on the complexity of our documentation as one of the actions that we took from the assessment and national qualifications group. That is why it is really important—
I am sorry, Dr Brown. I know that Liz Smith is going to come in shortly on this, but I saw that the convener of last session’s Education and Culture Committee, Stewart Maxwell, had this discussion with you last year. We are having it again with you this year. What has happened in the interim?
If speaking simply is complicated, surely a lot of work should have been done between last year and this year?
A lot of work has been done between last year and this year. We have a specific liaison team that is targeted at every part of Scotland. We are getting very strong positive feedback on that. The team goes into individual schools and works with them to understand their concerns. We then act on that. We are getting strong and positive feedback from schools on that.
We have also focused on simplifying what we are doing and on trying to make sure that teachers are aware of the changes. Unfortunately, there have been significant changes: because of our findings, we made changes again to the nature of the assessment. That needs to be conveyed to teachers.
A lot of the complaints that we have had—I recognise that we have complaints—have been about the number of changes during introduction of the qualifications. We are about to enter another phase that will not be comfortable for some teachers. It is key that we communicate not only to teachers but to parents, employers and learners themselves.
It is interesting that you say that teachers will not be comfortable with the changes. There was a wide range of experience among the teachers to whom we spoke; I am sure that many of them have been through a number of changes over the years and have very quickly adapted to them.
I am glad that you mentioned parents. The national parent forum of Scotland was quite scathing about communication, and said that it could not take part in the survey because people did not understand it. So it is not just teachers who are affected; parents are, too. They are the most important adults in this process because it is their children for whom the SQA’s responsibility is to get them through their exams.
We meet on a regular basis with NPFS and with the Scottish Parent Teacher Council. We make sure that we try to understand the concerns of parents and how to make things simple. We worked with the NPFS to make sure that we supported it in putting out “Nationals in a Nutshell”, which was focused on language that parents would understand.
We accept feedback and try to address it as much as possible. We have a parent representative on our advisory council; that voice is there to give us feedback on what is and is not working. The advisory council is the key component of the feedback mechanism.
10:15
This is my final comment before I pass over to my colleague Liz Smith. Parents say that the submissions from you and Education Scotland were totally inaccessible to the average parent. Even if there is a parent on the advisory group, that is clearly not acceptable. There is a lot more work to be done, and I hope that, when we come back next year, we do not need to have the same discussion about the lack of communication or the inability to communicate with others.
Before I let Liz Smith in—now that I have had my turn—I ask everyone to keep their questions and answers as brief as possible, please, because we have a lot to get through.
In my 10 years in Parliament, I have seldom come across evidence that is so compelling in its concerns about the SQA—and I say that with regret. The message that we have received in the submissions is deeply worrying, and I am sure that it is a worry to pupils, staff and parents. You cannot be satisfied with what we are being told. Is the system fit for purpose? Is the delivery of curriculum for excellence fit for purpose?
The SQA needs to take seriously any feedback that we get that expresses the level of concern that we have seen in the submissions. That is why we are engaging across the country to understand what is going on. I fully accept that.
On curriculum for excellence, as I have said when I have been at the committee before, we have discussed the findings of the research that we did last year and have identified multiple reasons why there are issues with qualifications. One component of that is the responsibility of the SQA, and that is what we are trying to address. That is part of the reason why the units have been removed and we are redesigning the assessment. The way in which the qualifications are perceived by those who receive candidates who have SQA qualifications is testimony to the value that is put on them. We will continue to guard that value and work to make sure that it is maintained.
Thank you for that answer, although I am not sure that I can take much reassurance from it. One of the consistent themes in the submissions is the fact that people are questioning the process of setting exams and marking them. Teachers are making the strong comment that they feel that there is a lack of effective scrutiny and transparency.
I read the Official Report of the committee meeting of 22 November 2015, when you were asked about the production of the minutes relating to grade boundaries and the way in which the assessment takes place. You said that you would produce those minutes. The committee was not asking what the decisions were; it was asking about how they were made. In other words, where is the transparency in and the scrutiny of the setting and marking of exams? Those minutes have not been produced, as far as I can make out. I have visited the SQA website and it is clear what some of the decisions have been, but people are asking whether we need much greater transparency. What is your reaction to that? Would an independent scrutiny body be helpful?
We have changed how we report the way that we establish the grade boundaries in the grade boundary meetings. We have added more information on the background and the reasons behind the decisions that were taken. We have published that information and it is available for every subject at every level.
On the scrutiny that the SQA is put under, the organisation’s governance structure contains the qualifications committee, whose role is to ensure that the SQA’s approach to qualifications development, assessment development and the establishment and maintenance of standards is appropriate. That group is not only made up of SQA board members but contains representatives of teachers, colleges and professional bodies as well as assessment experts from universities north and south of the border. The group has to approve all aspects of the way in which the SQA undertakes its assessment work. Its scrutiny is very challenging, and it is there to ensure that we do the right things.
Thank you. Let me develop the point. In reply to a question from me, John Swinney said:
“It is intolerable if there are errors ... in exam papers.”—[Official Report, Education and Skills Committee, 2 November 2016; c 19.]
He said that, if those mistakes persist, that draws the SQA into question. Do you agree with the cabinet secretary?
We should not have errors in our exam papers. I have said that before in this committee.
Why are those errors happening, Dr Brown?
They are happening because people are working extremely hard. As a result of the errors that have occurred, we have taken additional quality assurance steps so that we have fresh eyes looking at the nature of the assessments. We also need to ensure that we control everything that is required to make the questions that are in the question papers valid and appropriate and that we have appropriate engagements with institutions to develop those particular aspects.
In your letter to me on the additional scrutiny for science, technology, engineering and maths subjects, you explain that the reason for the additional scrutiny is that some of the questions are increasingly technical. Which subjects are included in the definition of STEM for that purpose, and why do other subjects not receive the same degree of additional scrutiny?
Historically, the issue that we have had with STEM and computing science subjects has been the technical aspects of the questions, and it is important that we have a separate set of eyes look at them. In national 5 computing, a particular programming language is used that was written specifically for the SQA, and we need to ensure that we are engaged with that. Given the complex nature of a question in that programming language—as opposed to a verbal question that might be set in a social science subject—it is important that that level of technical detail is looked at by more than one person with the expertise. That is why we focus specifically on those areas.
Is geography—which has been much in the news—included in that group of subjects? Two geographers suggested in their submission to the committee that their discipline is more akin to science than social science.
I am talking about complicated questions such as statistical questions or questions on equations being put in papers.
That happens in economics, for example.
Yes.
Is that subject included?
There are some very specific actions that vary by subject, depending on the nature of the questions. However, there is an overarching quality assurance process in place.
I am not sure whether you answered Liz Smith’s question, Dr Brown. Are geography and economics included in that extra quality assurance?
Economics is definitely not, and I am not aware that geography is at the moment.
Could you get back to us on that?
I will get back to you.
That is extremely important from the point of view of scrutiny, and it is a question that parents want to know the answer to. As you have admitted this morning, we have had issues with specific exams and we cannot go on like this. We cannot have an on-going issue with the nature of some presentations. It matters which subjects are being scrutinised, and parents have a right to ask whether some subjects have an additional level of scrutiny. They want to know why and they want to know exactly which subjects are involved. Do you accept that?
That is a fair point. We develop a significant number of question papers every year, but I agree that no question paper should have any errors in it.
Thank you for having us over to the SQA last week. It was a very worthwhile and interesting visit. It was good to see the scale of the work that you are involved in. I reflected after the meeting that I had learned a lot about what you do, which is a lot more than I had initially thought.
The convener and Liz Smith have picked up the line of questioning that I was going to follow. There is no escaping the fact that the submissions and whatnot are very damning for you—indeed, you have reflected that view—but, instead of going over the facts again, I would like to hear something more emotional from you. Can you convince me and this committee that you will seek to change the nature of the relationship between the SQA and teachers? I would like to get an answer that would make me think that, when you come back next year, things will have changed. I think that you are capable of doing that. Indeed, the team whom we met last week are fantastic. I am appealing more to the emotional side of things. Your opening statement and your previous two responses have covered the facts, but I want to feel convinced.
That is what the SQA is trying to do. Our challenge is to reach every teacher.
We are focusing on running sessions for teachers and trying to provide specific requested support. If a local authority feels that its teachers need support on aspects of particular qualifications, we will engage with that. We regularly meet the teachers unions and have very productive—and sometimes very challenging—conversations. There is definitely strong communication and strong engagement, and we have the same thing with the Association of Headteachers and Deputes in Scotland and School Leaders Scotland.
We are revising and reviewing how our messages get out. You will have seen that the responses to your call for evidence contain feedback about our website, and we are aware of that. The challenge is in how we can take advantage of new technology. Instead of providing long documents that, by their nature, include all aspects of a particular subject, we should break them down and customise the responses that a particular teacher will get, depending on the nature of their inquiry. That is neither simple nor easy, but we are absolutely focused and working on it.
The nature of our engagement and communication with teachers is critical. One of the pieces of feedback in the submissions that you have received is that a lot of information goes through a school’s SQA co-ordinators. That is a challenging role for a teacher, and we need to make it possible for every teacher to be communicated with. That is not possible at the moment, not only for reasons of our own but for reasons in schools. It is about customising the information that an individual receives and ensuring that they get what they need in the format in which they need it instead of—as happens at the moment—getting a very long PDF document. That is a major focus for us, but it is a complex thing to do.
I am glad that you have picked up on that. After reading the submissions and hearing about the group of teachers who met the convener and Mr Greer, I took the opportunity last night to send a quick text to some teachers I know—like everybody, I have many friends who are teachers—and the responses that I got back, although they were perhaps not as critical, went down the same line.
I might be wrong, but my impression is that teachers feel that the SQA is something that is done to them and something that they do not have a real say in or are not part of the process of. I am encouraged by what you have just said and what I have heard. If you are able to drive that forward in the individualised way that you have described, you can really change the perception out there. I am, therefore, quite happy with your answer. Do you think that local authorities and teachers in general could play a bigger part in governance and accountability to ensure that they feel more involved in the process?
10:30
We currently have teachers on our board, and we have representatives of teachers and teachers unions on our advisory council. We very much focus on that and, when we make changes, we take those changes to the advisory council and get its feedback on them. We regularly meet the unions individually to get their feedback on what we are trying to do.
The SQA also meets the Association of Directors of Education in Scotland and engages with it, which gives us strong feedback on what works and what does not work and the different approaches that different local authorities across the country have taken in how they utilise the people whom the SQA has trained. The SQA has trained a significant number of people—over and above our requirement—in standards and internal assessment, and the expectation is that the local authorities will use them. We need to look at how we can improve that across the piece, which means closer engagement with the directors of education to understand what they specifically need from us.
Needs may differ in different regions of the country. For instance, Shetland finds it challenging to send teachers to the events that we run. Therefore, we have started to hold webinars and twilight sessions so that people who cannot travel or cannot get out of school for other reasons are able to engage with us, give us their feedback, learn and respond when we need them to respond. We do not just assume that everyone can come to an event, even if it is in Aberdeen.
I remind the witnesses that we have a lot to get through, so there should be shorter responses.
I would like to drag you back to the SQA’s performance if I may, Dr Brown. A submission from a physics teacher—please correct the record if this is wrong—says of a higher physics unit and assignment:
“Several of these documents are already on their third version despite it only being the third occasion the course has run.”
The submission goes to say that there are
“A total of 81 pages of guidance across five different documents, three accessible on the main SQA website but two on the SQA Secure website”.
Is that all true?
I cannot tell you whether those are the exact numbers. A significant amount of documentation is associated with this. That is what I was trying to say.
Is it true that
“Several of these documents are already on their third version despite it only being the third occasion the course has run”?
I do not know the answer to that specific question, but that is not unlikely, because we have tried to respond to the feedback that teachers have given us and, in responding to that feedback, we have had to modify our documentation. It is a double-edged sword for a teacher. They want us to change the way in which we approach things, so we have to change the documentation.
Sure. You have made some good points about the format and simplification. Is it simplified to have five different documents, three of which are accessible on one website and two of which are available on another website? Is that information accessible to teachers?
We try to make information accessible not just to teachers. Part of the challenge is the way in which it is worded, which is a significant issue. We try to make the information available to other interested stakeholders who wish to see it.
My son is doing physics, and I want his physics teacher to know what this stuff is.
Absolutely.
Your primary responsibility is to get that information easily to teachers, is it not?
Yes, it is.
Are three different websites and
“81 pages of guidance across five different documents”
the easiest way to do that?
No—absolutely not.
What are you doing to fix it?
We are improving our systems. It is a real challenge for a public sector body to renew its systems.
Do you mean information technology systems?
Yes.
Oh, my gosh. Please do not scare me any more.
We are not doing that in a very large, big-bang way.
We have heard that before.
I am quite happy to go through our approach if you would like me to.
No. I am not too bothered about the systems. Maybe you could write to the committee about the systems.
Okay.
Some of us have previously been on the Public Audit Committee and have heard about the systems of different organisations in the public sector. I have seen scare story after scare story. Maybe you could write to us about that.
May I answer your point about the different websites that are being accessed?
Sure.
We are in the process of giving individual teachers individual access. As I said, that will not happen tomorrow, but our approach is that a teacher will have associated with them what they can see. They will go through one website, which will give them access to the secure information that they need to see and the generic information. If a parent wishes to go on to the SQA website, they will not be allowed to go on to the secure website, but where the documentation is held will be transparent to the teacher.
When will that be available?
We are currently undertaking that process but it will not be available in the next session, because it is not an easy thing to do.
Forgive me, but when will it be available?
Right now, I do not know when we will be able to deliver that, but we are planning it and we have a detailed planning process in place to understand when we can deliver it, given the complexity of the other activities that we are undertaking.
Okay. I have another supplementary question that I wanted to ask further to the convener’s questions. You have obviously seen all the written submissions that the committee has received. The one from the Scottish Association of Geography Teachers claimed that this year’s higher geography exam was the “worst ever”. Why is that?
We continue to have regular dialogue with the Royal Scottish Geographical Society; in fact, its people were in to see us last week and we have another meeting set up in January. Other than what we have seen in the submissions to the committee, we have not received significant feedback on the nature of the exam paper. On the day, there was very little feedback and we got very little feedback afterwards.
We have had conversations about the content of the exam. If you were to look at our subject review reports on geography, you would see that part of the action was to address some of the findings associated with our fieldwork and our conversations with and input from the likes of the Royal Scottish Geographical Society. Some of the changes that it is looking for are already documented in the paper that went out in May.
But the committee has been told that more than half the respondents to a teachers’ survey said that the higher paper was
“poor/shocking/terrible/worst ever and nothing like specimen or previous paper”.
If we put aside all the adjectives, is it true that it was
“nothing like”
the
“specimen or previous paper”?
No.
That is not true.
That is not true.
You will be able to furnish the committee with the evidence that counters that claim.
Yes, we will.
Okay. Can you furnish the committee with a real, detailed answer as to why so many geography teachers think that the paper is the “worst ever”?
Yes.
Thank you.
Seventy-five per cent of the SQA’s costs are payroll costs, a significant part of which are for appointees such as markers and invigilators. I am concerned by a report in The Guardian that says that, at a recent meeting with the Poverty Alliance, the SQA could not confirm what hourly rate it paid to invigilators. Clearly, that has an impact on your ability to comply with the living wage and national minimum wage legislation. Given that invigilators are not paid hourly, can you confirm that the SQA has currently no way of knowing the effective hourly rate that each invigilator is paid? In other words, you do not know whether you are paying the living wage to your invigilators, do you?
Thank you for your question on that, because it is something that we have been working on both internally and recently with the Poverty Alliance. We have a process in place that takes account of the way in which invigilators and chief invigilators wish to work, but particularly of the way in which chief invigilators manage invigilators over the exam diet period. We pay a session fee of £27.15, which is either a half-day—
Which can be four hours or more.
I am sorry, but no. About 47 per cent of our diet sessions are under two and a half hours.
But that does not include extra time for preparation, does it?
I accept that half an hour of preparation time is on top of that.
So that should take it to over three and a half hours.
No, it does not.
Your own guidance says that it is half an hour on either side.
I am sorry, but no. It is 15 minutes either side.
I have seen it written down that it is up to 45 minutes.
The bottom line is that we have looked at the diet and the number of diet sessions, and at least half of them are under two and a half hours. Where those happen, what we are paying is obviously significantly more than the minimum wage and the living wage. Yes, other sessions might be longer, but the chief invigilators are responsible for making sure that there is a fair allocation of sessions across the payment period for the diet to ensure that we are satisfying the living wage.
The chief invigilator is able to look at the workload that an invigilator has taken on in a particular centre, so if they think that the invigilator has worked more hours, they have the right to add on another session fee for that invigilator. Equally, chief invigilators or invigilators can speak to us if they think that they have done more time than we have allocated. That has been the process.
Going forward, the process is that we will record the hourly rate. We have added to the forms that we use to collect information on the payment for invigilators so that we will now know how many hours have been worked and can be absolutely sure—and demonstrate the evidence—that we are paying the living wage.
In effect, you are telling me that you do not know the effective hourly rate that invigilators are being paid. That is the long and the short of what you have just said.
At the moment, we ask the chief invigilator to highlight the situation to us. What we are doing is formalising that mechanism.
Based on the information that you have just provided, the fee is £27.15 for a session that can last as long as four hours. If a session is four hours, that makes the effective hourly rate below £7 an hour. The living wage is £8.45 an hour. That is correct, is it not?
Yes.
It may be, in that one session, but when you calculate the minimum wage or the living wage, you look over the payment period at how much work has been done—
But you do not know whether that is happening, do you?
We pay the chief invigilators to manage the invigilators and keep those records. They have the right and the ability to add an additional session fee.
I have statements from chief invigilators who say that the only way that they can ensure that their invigilators are paid not only above the living wage but above the national minimum wage is by adding their invigilators on to sessions in which they know that those invigilators will be surplus to requirements and will be sent home. Surely that is a wholly inadequate way of paying people. Indeed, those are the sort of practices that, if we saw them in the private sector—for example, as extensions to people’s working day through additional requirements and duties—we would condemn them as poverty pay practices. We would condemn that in the private sector and it should not be taking place in the public sector.
If you look over the pay period, some of those invigilators will be working less than the full session time but will be paid for the full session time. That is the point of paying them over the entire pay period. It is as if you are working a working week and you get paid at the end of that week. On some days you work three hours and on other days you work eight hours. The pay period is over the week. That is the conversation that we have been having with the Poverty Alliance.
What is the pay period?
The diet period.
We pay monthly but the diet period is six weeks. There is essentially a payment probably at the end of June and then July, based on the fact that the exams are in May and June.
Why are you not paying an hourly rate? Surely that would be the straightforward, transparent approach and what good employment practice would dictate. Indeed, that is what happens in other exam boards in the United Kingdom.
In 2014, we reviewed the whole approach to invigilation payment and talked to chief invigilators about it. There were two reasons why we continued with the session fee. The first was that chief invigilators felt that it worked better for them in managing the process, and the other was that if we paid only the hourly rate, many of our invigilators would be paid significantly less than the session fee.
You have already admitted that you do not know what the effective hourly rate is.
But we know the session times and we know that 3 per cent of our session times in 2015 were three or three and a half hours, before additional admin. The four hours that you are talking about is about 3 per cent of the diet; the rest of the time, people are being paid either £9.05 an hour or significantly more than that.
But do you accept that, if invigilators are working for four hours—or more, once you have taken into account prep time and extra time—the effective hourly rate for that session will be well below £7 and therefore significantly less than £8.45?
In very limited circumstances.
It is not—
It is, because it is those parts—
You only have to look at the exam timetable to see that there are a significant number of exams that would take you past that four-hour period.
We have done an analysis of the 2015 diet, which I can share with you. As I have said, 3 per cent of sessions are in the three to three-and-a-half-hour band, and adding the half hour for administration would take them over the four hours.
10:45There are special circumstances in which there is extra time, but that applies to a small part of the diet. A lot of the exams are half an hour long; some are an hour and a half, some are two hours and some are two and a half hours. When we add in the admin element, which is 30 minutes, most people—the vast majority—are being paid £9.05 an hour or more.
And over the full pay period they are all paid the living wage.
Yes.
But, with respect, you do not know that, and you have no way of knowing that. You have admitted that the problem is that you have no mechanism for measuring or tracking that, so I struggle to understand how you can say that with such confidence.
I say that because one of the responsibilities of a chief invigilator is to highlight to us when they need to pay extra for people going over their time. You have highlighted the fact that we do not have that reporting mechanism, and that is what we are putting in place. I agree with you that a reporting structure needs to be put in place to address that.
And you will send us the information that you have to show that that is going to happen.
Yes.
Yes.
Okay—thank you.
I want to return to the previous theme of complexity and some of the issues that we have heard about from teachers. No one wants to undermine the SQA or call into question your credibility or the impact that you have had on Scottish education, but some of the feedback that we have received from teachers causes concern. Responses have referred to “Unclear marking instructions” and “Unclear, complex course documentation”, which previous questions have touched on. I want to know how we have reached a stage where teachers find that what the SQA has issued is very complex, vague and difficult to understand.
The introduction of any new qualification is associated with significant change, and we are required to provide information. One of the challenges that we have faced—and this has been discussed at previous committee meetings with me and with others at the table—is that teachers have rightly requested more and more information. As we put more information out into the system, it becomes more complex, especially when we put out individual documents instead of, as we have described, using a different format, which technology might allow us to do in the future.
The issue that was raised earlier around which version of a document we are talking about presents a real challenge. We have tried to respond when teachers have told us that they would prefer things to be presented differently. We have done that, and in some cases, it has increased the level of complexity because it means that there is another document out there.
We have committed to reviewing and streamlining our documentation. As we move into the revised assessment process for national qualifications, which we are just starting, that documentation will be completely revised and will be much simpler. The process is an opportunity for us to address that issue. We have got to that point because of the timing. The qualifications were introduced in the same session in which they were implemented, and that produced an additional challenge.
I have two more brief questions. First, are we in danger of sinking in a sea of jargon, and is there anything that the SQA can do about that? In your opening remarks, you used the phrase “associated personalisation areas”—whatever that means. I know that the leaders of the SQA are wrapped up in their day-to-day jobs, dealing with education and all the professional aspects, but how on earth do we demystify Scottish education? Perhaps if we addressed that issue, communication could be improved.
I think that there is a danger of us all sinking in jargon. In fact, when the committee visited the SQA, I kept reminding people to spell things out because, as in any business or organisation, we use our own shorthand. That is a key point. Another key point is about communicating with parents in a different way, which is about all demystifying things and going back to look at the original full set of documentation associated with CFE—not just the qualifications—and considering what to say and ensuring that we communicate that clearly and concisely.
I should explain that the issue of personalisation and choice is about allowing teachers to teach a particular topic, like angular momentum, in a context that is of interest to the class in front of them, as opposed to a particular context defined by the SQA. That is a long-winded way of explaining personalisation, but if personalisation is not understood, it is not valuable.
My final question is about one of the big themes in education at the moment: teacher workload. How do you measure teacher workload and the impact of the SQA issues on it? Do you feel that teacher workload has increased or decreased in Scotland since you have been in post?
Part of our reasoning behind going out and doing fieldwork was to try to understand the nature and causes behind what we had heard from teachers unions and teachers across the country about their workload and, specifically, to understand what part the SQA had played in that. As part of that research, we identified other factors playing a part in the workload that was manifesting itself in assessments. After all, it is our responsibility to address not just the component for which we are responsible; we should also continuously raise the issues that need to be addressed in order to reduce the workload associated with assessment as a result of other aspects from the system. I take that role very seriously. It is not just for us to say that we have done our bit—it is our responsibility to keep that in the forefront.
Curriculum for excellence qualifications and teachers developing their own assessments to allow them to teach a particular subject in a particular context resulted in a significant workload. That is our contribution, and it is balanced with what we are trying to achieve with curriculum for excellence. The way that the old qualifications were developed, with off-the-shelf assessments, off-the-shelf books and off-the-shelf everything else, probably resulted in less of a workload for teachers. The challenge for Scotland is to understand whether that is the type of education that we want or whether we want the ability not to teach in that straitjacket. The negative to that is providing the detailed assessments and the detailed information.
Maybe I should declare an interest, as an ex-schoolteacher. Indeed, I do not know whether it ever leaves you.
I was a schoolteacher for 20 years when standard grades were introduced, so I know that when there is change, there are some in the profession who might be a bit reluctant, concerned and anxious. However, I have to say to Dr Brown that the evidence that we have received from a whole range of organisations and individual teachers is at a different level. Does she recognise that? Surely what they are saying is at a different level from simply saying, “We know what teachers are like—they do not like change. They would rather take a book off a shelf than do their job.” Frankly, that is the defence that Dr Brown is presenting today.
That is not what I meant. Teachers have a very difficult job; they feel passionately about educating the young people in front of them. What I was trying to indicate was that, because of the nature of what curriculum for excellence was trying to do, a decision was taken not to provide detailed off-the-shelf assessments to teachers. There was a move towards using evidence that was naturally occurring within the classroom. That has not worked for multiple reasons, and that is why we need to address that issue. As a result of that approach, things have been more complicated and challenging.
If in your mindset the problem is that teachers do not want to change, you end up coming to the committee—having, I presume, read the same papers that I have read—and saying that you have a strong working relationship with teachers. In what parallel universe do you have a strong working relationship with teachers?
I am not saying that teachers do not want to change. I am saying that, because of the philosophy behind curriculum for excellence, the nature of the information and structured provision by the SQA was different.
Have you looked in any detail at the themes that are coming out of the evidence? This is not really about curriculum for excellence; this is about not being able to access information, and getting different answers to the same questions at different times. It is about constant change. I would have thought that if your core business was to present a document, you should not wait for feedback and then go through seven different iterations of it. Perhaps the document should have been solid at the very beginning instead of your throwing something out and seeing what feedback came back. I am genuinely astonished at the number of criticisms, and what we are getting from you is all about process and how you use the technology—which has existed for quite a long time—in future. What, for example, motivates geography teachers not to tell you that there is a problem with the geography paper, only for the SAGT to present a document that is scathing about it?
I refer to what I said earlier about the subject review reports. We have actively engaged with each particular subject area. We have a national qualification support team, which is made up of teachers, teachers unions and other stakeholders who are involved in that particular subject. Every single one of those stakeholders was engaged in trying to understand what was working and what was not working on a subject-by-subject level. As a result of that conversation in geography, a set of actions was identified and published in the subject review report.
We actively engage with teachers, we continue to engage with the Royal Scottish Geographical Society and we will continue to move forward on making sure that we address the issues. As I said earlier, in some subjects, you will find two different teachers saying that a particular course should have two different sorts of content. It is the SQA’s job to try and mediate—
So it is their fault, then?
No. If you talk to two different academics in a university, each will have a different view of what is important in their subject.
Then whose job is it to make the decision?
That is why we pull together a group of people from across the piece to get a consensus around what should be in that particular qualification.
You have singularly failed to get that consensus.
We have a consensus but there will always be people who are not happy with a consensus.
So all the people who have written to the committee to express their concerns are people who, for whatever reason, are not prepared to fall in with the consensus. The consensus that I think exists is that people do not think that the SQA is working properly and that it is getting in the road of them doing their job. They are not people who are dragging their heels because they do not want change; they are clearly trying to navigate a system in the best interests of the subject that they care about and the young people whom they are teaching, and they regard the SQA as a block to that. What is your answer to that? It cannot simply be that you have built a consensus and the other people just do not get it.
Absolutely not. I have been trying to say that when we get feedback such as the feedback that you have seen, we look at it and make changes. Unfortunately, when we make those changes, we end up changing our documentation and that has a knock-on impact on—
But people are not just concerned about the documentation. If you had just put the wrong word in a document and there was a debate about what that word meant, that could be sorted. It took me three hours to read the submissions last night—I am sure that it took my colleagues the same—and it is not just people nit-picking. People are saying that there is a fundamental problem, and what I am hearing today is that you do not accept that there is a fundamental problem. When you were asked about the geography teachers, you said that you did not know that they had a problem with the paper. You were not aware of it—despite the fact that the RSGS produced a survey, the like of which I have never read in all my life.
We have had a follow-on conversation with the geographers and we fully understand the position. At the point of the examination’s delivery, we did not get feedback that there was an issue. That is what I was trying to indicate. The paper was to standard; it was aligned to the previous question papers that had been put out there.
11:00
So the geography teachers—all of them—are wrong.
What we are saying is—
Seventy five per cent of them said the paper was not adequate. That is not 75 per cent of a group on a committee, but 75 per cent of those surveyed—
Of those surveyed.
These people, who care deeply about their subject, are saying that there is a problem. Do you or do you not accept that there is a problem?
I have said that, as part of our research and the taking of information, we have published a subject review report on geography. That report indicates the changes that we are making as a result of the feedback, some of which has come from the Royal Scottish Geographical Society.
The issue that I will raise has been mentioned a number of times, so the response to my first question should be relatively short. There are clearly problems in how the assessments are put together—we saw that last year with the higher maths paper and this year with the national 5 computing paper.
I am confused. A lot of the discussion so far has revolved around quality control measures and resource allocation—you mentioned extra resources being allocated to STEM subjects—yet this year, the most high-profile issue was with the national 5 computing paper. Is it the case that it does not matter how much extra resource you put into the system, because your quality control structures are not working and need to be fundamentally re-evaluated?
First and foremost, we evaluate everything that we do every year after we finish a diet. We go through what has and has not worked and we try to identify ways in which to improve. That includes looking at the question papers and at the procedures and processes that we undertake to deliver the 140,000-candidate certification.
An aspect that was particularly associated with the national 5 computing was that the computing language involved is Haggis. We have had a subsequent conversation with the developer of that language and we have reached a process improvement that will ensure that the issue that arose will not reoccur.
Each problem is addressed at that point in time. However, as you pointed out, addressing each problem is not sufficient, and we need to look at our quality control procedures across the piece. We have introduced e-marking to improve the quality assurance of the marking procedures that we undertake. We are actively looking at how we will continue to improve the quality.
As I have said, SQA qualifications are seen as high-quality qualifications; they are regarded highly not only by the candidates who take them but by the receiving organisations that take the candidates.
When the issues with the national 5 computing paper came up, the SQA engaged in what I would describe as defensive public relations—indeed, it said that the anecdotal evidence that it had received from teachers was positive. I cannot understand how that was case. The anecdotal evidence that I received was from teachers who contacted me—I am sure that they contacted my colleagues, too—who were so concerned by the paper and by the SQA not taking their concerns seriously that they were asking me to raise the matter in the chamber, which I did, as I believe Liz Smith did, too. Why does the SQA believe that, at a time when there are clearly issues of teacher trust in the organisation, a defensive PR exercise that publicly dismisses concerns is helpful?
If you remember back to that time, I specifically said that we had made errors in the paper and that we needed to address that.
I have the SQA’s initial response here. It said that the paper
“met our course and assessment specifications ... allowed candidates to demonstrate ... knowledge and understanding ... anecdotal feedback”
was
“positive.”
You later altered that line and released a statement saying that the exam paper contained
“a small number of typographical errors.”
That was not in the initial statement, which was defensive.
I take that point. However, I remember, from sitting through the grade boundary discussion, that how the paper was set allowed candidates to demonstrate what they could do. We have subsequently had meetings with the British Computer Society in Scotland at which we have discussed whether there were any issues with candidates not being able to do the question paper. All of us felt that candidates were able to demonstrate what they could do. Although the errors were there, we were absolutely able to certificate appropriately with the examination.
If we leave to one side the issues with the paper, I fail to understand why you publicly dismissed concerns initially. In its submission to the committee, the SQA cites a survey that it conducted, the results of which entirely contradict the survey results that the committee received and the overwhelming weight of the evidence and submissions that the committee received. I do not understand that. There are clearly issues of teacher trust in the SQA. Your public statements are initially dismissive when such issues come up. I do not understand where the survey response that you submitted as part of your evidence comes from.
The survey that we undertake is an independent survey; it is not an SQA survey. It is an independent survey that randomly samples our customer base biannually.
Why is it radically contradictory to all—I stress all—the other evidence that we have received?
Because the sample is random, we get a random set of views. Rightly, people who have concerns see the committee as a way to address those concerns. That is an important role that the committee has.
We will go out again for our fieldwork, which we will get feedback from, and we will go out again with an independent survey to get feedback from our customers.
Is the random survey unrepresentative or is our evidence unrepresentative?
I do not know. I do know that there are teachers who have concerns about what we do, and I know that we continue to engage with as many of them as possible. We try to address their issues and change what we do to ensure that we continue to address those issues.
We will move on to questions on resource pressure from Ross Thomson. Thank you for your patience, Ross.
Thank you, convener. I, too, extend my thanks to our witnesses for welcoming us to their headquarters in Glasgow. I found the visit really helpful; thanks for your time last week.
During that visit, a key theme on resources that came through for me was that of how the SQA is again going through an intense period of assessment redesign. That is on top of business as usual, as well as the commercial side—which is on top of your programme of transformation, to touch on the IT issues. As has been clear to the committee and as you have acknowledged, when it comes to getting the new assessments in place for diet 18, it is crucial to get it right and have no mistakes. I am aware that you do not have the same resource as you did with the curriculum for excellence, when you had the development team, and that all the work is being done in house now. On the pre-budget scrutiny side of things, the questions are: what resources do you need, where should they come from and do you have enough to ensure that we get it right going into diet 18?
The decision was made in September to redesign the assessments. Because of the complexity of the job that we do, the SQA needs a detailed plan to understand exactly what we have to do and when to make the deadline. We are in the process of doing that planning, which includes understanding what we will do, how we will identify what we are going to do, what resources we will require and how long we will have those resources for when we need them. We are in the midst of that planning process, and we expect it to have finished by the end of November or the beginning of December.
We fully expect to require additional resources. The people who help to develop and deliver the qualifications are the teachers of Scotland. We will be asking teachers to engage with us on that, which will be a challenge. As I said, we engage with the Association of Directors of Education in Scotland, the Association of Headteachers and Deputes in Scotland and the unions. To develop qualifications, we have to have teachers’ participation. That is why we get so many of them to work with us.
The identification of the nature of the resources is being scoped. We are ensuring that we minimise that, but you are absolutely right that we have to get this right, because it is not appropriate to have any errors.
The timetable is hugely challenging because the session starts not in August, as many people believe, but straight after the exam cycle in May. However, the nature and content of courses is not changing, so teachers do not have to worry about that. What is changing is the method of assessment, which is different from where we were with curriculum for excellence. I cannot tell the committee exactly what we will need, but I can tell members that we will eventually know that. We will then work closely with our partners to identify where that will come from.
Other members have highlighted mistakes that have been made—particularly the typographical errors that were in the national 5 computing exam. We know that the SQA is responsible for such mistakes and that they ranged from grammatical mistakes to questions that simply could not be answered. How many of those mistakes related to the resource issues that you have identified? In addition, are you confident that the executive management team has the necessary skills and leadership?
The executive management team is well positioned and has a strong level of expertise for delivering what is required. With regard to what the SQA was asked to deliver on curriculum for excellence and the qualifications, the milestones might not have been what the teachers wanted but they were agreed by the curriculum for excellence management board and they were all met.
You asked whether the errors that have been referred to were associated with a resource issue. One of the things that we have done as a result of the work that we have identified through our lessons-learned exercise is to increase the resource that has gone into science, technology, engineering and mathematics subjects. That is an exam-heavy and question-paper-heavy area, so we have put increased resources into it. I will not hide behind the fact that a resource issue resulted in the errors, because it would not be appropriate to do so.
One theme that came out of the presentation that we had during our visit to the SQA was that giving significantly more support to the system for continuing professional development and teacher training creates a cost pressure. We were advised that that role had been carried out previously by local authorities, which we know have their own cost pressures and reducing budgets. I would like to get an idea from you of where you think the public money should come from for that role. Given that the role is a significant cost pressure for the SQA, would it be better for local authorities to take it on? Should the SQA continue to do it, although that would require a significant grant from the Scottish Government?
Some aspects of the support that the SQA provides must be provided by the SQA, given the assessment expertise that is required. However, other areas might well be delivered elsewhere. We need to discuss that issue; the assessment and national qualifications group is discussing it to try to understand where the best place for such support might be.
One reason why we trained more nominees than we needed on understanding standards was that we wanted to ensure that that resource was available in the local authorities. We are not precious about retaining the ability to give the level of support that we have given, and that is right, because teachers have a right to expect a lot of support during the early implementation of new qualifications. Where it is appropriate for the SQA to do that, we believe that we should continue to do it. For subject-specific support, we offer to go into a local authority to talk about a subject, if they wish us to do so, and we run sessions for the authority’s schools. We ask through our liaison officers, who visit every school in Scotland, whether there are particular issues in any school. If there are, we will go into that school and work with the staff on the issue. However, such work is not necessarily best done by the SQA, and we need to think about that aspect of the system.
11:15
I have a supplementary that follows on from the convener’s questioning. As you know, we conducted an online survey, to which 71 per cent of the respondents were teachers. Sixty-seven per cent of all the respondents expressed distrust in the SQA by disagreeing or strongly disagreeing with the statement:
“our customers and users trust us to ‘get it right’ for them.”
Why do your customers not trust you?
The feedback that we get varies significantly. We have customers who are frustrated by how we engage with them, but we have other customers who are very happy. You will have seen the submission from one college that indicates the level of support that we provide, while another submission cited the SQA as one of the best awarding bodies in the United Kingdom. There is a variety of views, and it is important that we do not focus only on the good news; we must focus on the issues that are identified. We try to do that.
I have a question on quality assurance. Questions have been raised about markers for geography and computing science. I was contacted by a constituent whose son was predicted to get five As but ended up getting four As and failing geography, despite achieving 92 per cent in his geography prelim. The appeal was rejected, and his parents are worried that some of that work might have been lost in transit to the marker.
What reassurances can you provide about what you do to ensure that you have the quality of markers that is needed, especially given that some markers have said in submissions that not enough information is provided and that the information that is provided is confusing? In relation to art and design, there was criticism that the SQA had not communicated well enough to teachers or pupils what was required.
Every year, we run markers meetings specifically so that we can meet markers face to face and to make sure that, if they have concerns or if there is confusion about the nature of the marking, they can ask questions.
We have team leaders for the e-marking group. If we see that a marker is having difficulty with marking, the team leader will contact that marker and have a conversation about what they are finding difficult and whether they need clarification on anything. When we originally set out with CFE and e-marking, it was expected that we would not have needed as many face-to-face events by this point. However, we have continued the face-to-face events because we believe that teachers get benefit from them and like them—the feedback that we get from markers meetings is hugely positive. Teachers run the system and it is extremely important that we have clarity on marking.
Given the variable factors that come into the budget, is the current financial model sustainable?
We discuss that regularly with our board and with our external auditors, and the view is that, from an accounting perspective, we are sustainable from the point of view of the going-concern principle, but it is quite difficult to predict accurately what our budgets need to be, because we have not had the stability that I thought that we would have coming into this year.
I had hoped that 2016-17 would be the start of the end of the implementation of CFE, which has been the biggest change in the education sector for a generation. I had expected 2016-17 to be the first year in which the system started to bed down and be business as usual, but the announcement about the revision of national qualifications has meant that it is harder to predict what our costs will be. We are working through that. We will have discussions with the Scottish Government and determine by December what our budget will need to be.
On sustainability, we are trying to reduce the pressure on the public purse. We want to minimise the grant that we need to deliver the business, but we are balancing that with ensuring the safe delivery of the diet each year while we make further changes. It is quite difficult to get to a sustainable position.
Another component that is in our submission is that we are aware that the fees that we charge for qualifications do not cover the cost of delivery, so a funding gap is associated with that. The fees have been in place since 2010, but a fixed rate has been charged that is based on 2013-14 candidate numbers—
It is 2012-13 candidate numbers.
Sorry—it is 2012-13 candidate numbers. That is to local authorities. More candidates than were expected are sitting our qualifications, so our costs are going up, but our income remains the same.
Let us look at one or two parts of your submission. It mentions
“the fixed charge arrangement for local authorities”.
I presume that that does not cover the cost at this time.
No.
So the cost is, in effect, subsidised.
Yes.
Local authorities have some certainty about what they will be charged for their budgeting, but the uncertainty is passed to you. The cost is just passed up the line.
Yes. The 2010 prices did not even cover costs at that point, and the 2010 prices have been held. The 2012-13 candidate entry levels were true for that year, but we have taken the risk of their varying. There have been more candidate entries.
Is that a fixed fee per head or a fixed fee as a global—
It is what the local authorities were charged for their 2012-13 candidate entries. We have charged them exactly the same for the past three years.
That is the full total.
The full total—so the total per local authority.
So the numbers of candidates can vary.
What has been the percentage increase in candidates?
If we had charged 2010 fees by candidate this year, we would have had additional income of £1.2 million.
I see that in your submission, but what has been the percentage increase in candidates?
This is not the SQA’s responsibility, but the approach of two-year qualification structures would have resulted in candidates bypassing the national 5. They were rolled into our assumptions. I think that there was a slight decrease in the number of candidates in this diet versus the previous diet, but it was not significant.
I want to move on to look at a specific area. Page 5 of your submission refers to
“non-commercial products ... to support niche sectors in the economy.”
Can you give us a little more information about that?
Yes. As the awarding body for Scotland and a non-departmental public body, we are asked, and are expected, to develop and maintain qualifications that are not commercially viable. We charge based on candidate entries, and some have very low uptake.
Can you give us an example?
We support aquaculture, which is quite a small niche sector and a niche part of the economy. There are other areas that we support, too.
Furniture making is another example—
—and stonemasonry.
That is business that we should be doing; we are not in any way saying that we should not be doing it. We are a national awarding body, and it is our responsibility to provide qualifications for the industry sectors in Scotland. Some of those industry sectors are quite small or require only a small number of candidates a year to go through a particular qualification. I am talking about things such as fish farming and furniture making. We have qualifications in kilt making and stone wall building, which are part of the fabric of Scotland. What it costs us to develop and deliver those qualifications is absolutely not covered by the small number of candidates who take those subjects every year.
Do you bear a significant cost there?
Yes.
How much?
The cost is significant. For us to break even on national qualifications—
That is a different piece—
Can you give estimates?
We can give you an estimate of what that means. It varies from topic to topic. We can tell you that, for example, advanced higher Latin probably does not wash its face—
Never!
—but English definitely does.
I am not looking at particular subjects; I am just looking to see what the cost to your budget is in supporting them.
Can we move on now?
Can I ask one final question?
Yes, of course.
Can we look again at how the SQA prepares its budget? If I recall correctly, the SQA’s submission says that it knows on 31 March how many students will come through. Is that correct?
Yes, that is the final number.
The final number?
Yes.
Of course, you have to prepare your budget in advance of that.
Yes.
How do you do that?
We base it on estimates. We get initial indications of candidate entries in November—we need those to ensure that we can start to plan. However, teachers or centres can continue to change the numbers of candidates going forward for final assessments—or not—right up until 31 March, so we do not know the final numbers until then. So it is essentially—
What is your margin of error?
Sorry?
What is your margin of error? You must have some margin that you work to.
We predict the number in November, and we have tended to be reasonably close. I do not have a margin of error, as such. We know how many candidates we expect to come through and then we confirm that at the end of March with the Scottish Government. There is a fluctuation but it is not hugely significant.
It is not significant and the more years of the new qualifications that we get under our belt, the clearer that will be.
Yes.
There has been a changing pattern of presentation. It is starting to stabilise, although we are not sure what will happen next year with the introduction of the new assessments. We are still not in a steady state. We have an estimate: we can look back at what happened the previous year and estimate what we think might happen in the March.
That is why, when we talk to the Scottish Government about our budget requirements, we present a draft as at January, when we submit it with our corporate plan. We estimate what we think we will need, but we continue to refine that after 31 March, as we see what the actual cost is going to be.
So you only firm up your budget with the Government after 31 March.
Yes. We continue to have discussions with the Government throughout the year. We tell it what we believe we will need for the full year. However, we firm that up with it in the course of the year. At the spring and autumn budget revision stages, it will allocate additional funding to us in relation to what we need.
From what you are saying, I would say that there is no prospect of the SQA becoming self-sustaining.
Given the focus that we absolutely have to have on curriculum for excellence, a lot of the work that we were doing in that space was put on the back burner. The issue for us now is to take advantage of the opportunity to bring in surplus that reduces our dependence on the public purse. It is the surplus—not the income—that we need to focus on.
Thank you very much. We are going to move on. I remind both members and witnesses to speak through the chair.
A lot of the questions that I was going to ask have already been asked. However, I will come in off the back of Colin Beattie’s questions about the SQA’s income. What is the income from appeals from schools?
Linda Ellison is looking for the number, but I can tell you that the post-results services fee absolutely does not cover the cost of those services.
If there is perhaps some inconsistency—which has been mentioned in some of the responses—around assessments and marking, that inevitably leads to more schools wanting to appeal decisions on behalf of their pupils. On the one hand, that affects the public purse quite significantly, because it affects schools’ budgets and they have limits on how much they can spend. On the other hand, you say that there is a shortfall in the fees that you get from schools, and that those do not cover your efforts. How can that be resolved?
11:30
The post-results services are there to address anything that teachers feel is inappropriate if a candidate has not got the qualification that they wanted. They are not an appeals process. They involve an examination of how that candidate has done in their assessments.
As we have said before, how the fees are paid is a matter for local authorities and schools, not the SQA.
As we increase our quality assurance processes, we see that the success rate under the process is declining. We have yet to publish this year’s figures—we will do so in the coming month. We need to ensure that we improve our quality assurance processes so that candidates get the right result the first time. It is not like the old appeals system.
Under the old appeals system, people’s preliminary exam marks were looked at. Is that still the case?
No.
Why not?
A review was undertaken, with extensive consultation, and it was found that teachers and many other stakeholders felt that the old appeals system was unfair. The new system is much fairer. The old appeals system was the only one of its type in the world—no one else did it that way.
The post-results services involve the idea of special circumstances. That is a huge benefit of the change from appeals. If a candidate is truly disadvantaged at the time of the exam—for instance, if they suffered a bereavement or were ill—we can now look at whatever evidence the school wishes to give us and are able to certificate that candidate on results day. Under the old system, we were unable to do that—we could look only at their prelim result and, if the prelim result was poor, we could do nothing for that candidate. We have focused the exceptional circumstances requirements on those candidates who truly need that specific support in relation to their personal circumstances.
So the system involves re-marking instead of appeals.
No, it is not re-marking.
Can you talk me through how it is done?
There is a debate across the world about whether the second marker is better than the first marker. We are talking about checking whether the marks are correct. That means that a qualified marker will go back and check whether the first marker has done a good job. If they have done a good job, there is no change, and the candidate will be charged at that point. If the marker has not done a good job and the candidate’s result is changed, there will be no charge. There is no charge for a change relating to exceptional circumstances, either.
You are looking ahead at your plan for the number of markers needed for next year, and there has been some criticism about the quality of markers, given the quantity needed in order to get the results out. What are you doing to ensure that you get the right quality of markers so that the results that are sent out are of the standard that you would expect them to be?
As I said, that relates to the way in which we train markers, and to the markers meetings that we have. Further, the recruitment of markers is critically important. We ask markers to tell us what their experience base is, and we have criteria that markers must meet, including the requirement that they must be a practising teacher or extremely recently retired and have been teaching at a certain level for a certain period of time.
We anticipate that we will need more markers in the coming session. As a result of the removal of units, we are extending the amount of external course assessment, whether that relates to course work or the exam itself.
Another aspect of quality assurance involves e-marking. We put markers through a set of scripts that they have to pass—effectively, they have to prove that they can mark to the required standard. During the course of the marking procedure, we seed the scripts that they see with marked scripts, which they do not know have been marked, and will use that process to monitor quality. That has happened over the past few years. There has been a significant increase in monitoring the quality of markers.
Finally, I will pick up on a comment made in a submission that
“SQA has an annoying habit of making changes to assessments and examinations mid-session”.
That will obviously have an impact on results. The submission adds that
“In Higher History important changes have recently been made to the way essays are marked in the final exam—for example ... essays were previously awarded 4 marks”—
and so on. Teachers had to advise students that they would lose marks for essays that they had written and which previously would have been a pass. That is a fairly bald criticism.
We learned very early on that we should absolutely limit the number of changes that we make during a session. It is about the balance between responding to what teachers say is an issue and making changes. We now try to make all changes well before the start of the session so that teachers know what is going on. As we move into next session, we anticipate that the information about what the assessments will look like will be available to teachers by the end of April. We have not finished the planning yet, but we hope to make the information available by then, which is just before the start of the next session. We very much focus on making sure that we do not make changes.
Is it guaranteed that changes will not be made beyond that—halfway through a teaching session, for example?
The only time that that would happen would be if an issue makes something completely and utterly invalid. We are not anticipating that because, as I said, the course is not changing.
The issue is not only to do with the quality of markers. Do you sometimes have a shortage of markers and have to seek markers nearer to when they are needed?
Yes. In fact, we saw that in the last session.
Given that you have a rough idea of how many students sit exams, why do you not have a pool in place earlier, so that that situation does not arise?
We have a pool. The issue is that we are seeing an increase in the requirement for markers. We have 15,000 markers, but we might need 16,000 or 16,500 next year. It is about recruiting and training new markers, and we are already in the process of recruiting markers for next year. It is a year-round activity, and a lot of people actively want to engage with the SQA.
You must not have been doing that last year because you were still struggling to get markers towards the end of the session. You must have had a rough idea of the complexity, and of the need for more markers.
Yes, we did. A small number of markers pulled out last year. That can be a challenge, depending on the subject. In certain places, we are struggling for markers in specific subjects—particularly the small subjects. In such cases, we work with schools and ADES to actively look at how we can make sure that everything is covered. That is what we did last year.
I was very relieved that you said in response to Ross Greer that you would not focus only on the good news. I ask that, following this meeting, you look again at the submissions that we have received and respond to some of the pretty serious questions that they raise.
I want to ask about equality in the current system. We have talked about the budget pressures on the SQA, and you have identified some of those. I presume that the SQA has made a number of its decisions partly for educational reasons and partly because of budget pressures. How many state schools have taken the opportunity to put in what is no longer called an appeal? I cannot remember the term that you used, but how many state schools have used that facility?
In 2013-14, we had 6,901 requests from state schools.
How many were there from independent schools?
There were 1,369. I think that we provided that information to the committee at the end of the last parliamentary session. We will again be very happy to provide that information when—
Have you tested the proposal against issues of equity and justice? Have you done an equality impact assessment?
It is for the schools and the local authorities to decide how they allocate the fees, so we have—
So it is not an issue for the SQA that local authority schools may not have the means to access justice or the confidence in the system that independent schools have.
The SQA is concerned that what we provide is an equitable, valid system. The way in which it is accessed is a matter for other people.
Then you have no view on whether there is a difference between an independent school and its access and resources and a local authority that has no money.
We regularly talk to local authorities about whether they believe that they are limited in relation to the post-results services. One thing that will be interesting to look at this session is the relative performance of the requests that have come in from private and independent schools and from the state sector.
On the decision that national 4 should be pass or fail, as I said to you earlier, I was a schoolteacher during implementation of standard grades. The most powerful thing that was decided then, in my view, was certification for all. Part of my working life was getting young people from foundation to general level. Do you accept that the new proposal undermines entirely the valuing of that group of young people?
As I said in my opening statement, the design and nature of the qualifications was agreed with the CFE management team. The philosophy of national 4 was very much that it would be a progression pathway for candidates who ultimately would leave and potentially go to college. The use of internal assessment and of pass/fail was associated with that philosophy.
At this point we are going to schools and actively soliciting feedback from senior management teams, teachers and pupils, as well as from employers, on the value of national 4 and whether it should be pass/fail, whether it should be internally assessed and so on. That is a conversation that absolutely needs to be had. The assessment and national qualifications group that the Deputy First Minister leads are also looking at it and will be discussing it at its meeting in January.
I totally agree with you that it is a very important area.
Would you agree with Dr Lindsay Paterson, for example, who gave evidence to the committee and expressed grave concerns about what is now expected from national 4 with regard to inclusion, opportunity and closing the attainment gap?
That is an area that we as a system—definitely not just the SQA—need to look at and discuss to decide what we want to do.
There are two other areas that I want to highlight, which presumably have to do with budget decisions.
As the Educational Institute of Scotland highlights, there has been a decision to remove human scribes who support people with additional support needs. I worked with young people who were exceptionally bright and who were able, but who needed support through scribing, by a person either writing what they said or reading a paper to them. You no longer do that.
We do that for all subjects with the exception of those that include a literacy component. The scribes are there for all other subjects. The issue was where a qualification included a literacy certification. What we did in the particular case of the literacy addition was to go round to each school to understand what their concerns were. Because the assessments are made internally, the candidates have a lot more flexibility in the nature and the timing of the assessment that they undertake. We worked very hard to make sure that we absolutely did not have issues associated with that for special needs candidates.
The EIS highlighted that issue, so it would be useful if you could provide us with more detail on the technicalities of it. We would be gravely concerned if young people were not able to access examinations because of a budgeting decision to remove—
Sorry—it absolutely was not a budgetary decision. The scribes are there for other subjects.
We would all agree that in the last period there has been a really important transition in terms of education. Can you explain why, according to EIS’s figures, there has been a
“500 per cent increase in certifications outside Scotland since 2010”?
As we talked about earlier, if we can generate a surplus associated with the work outside of Scotland, and if we are supporting Scottish Government’s international agenda with the positioning of education on the international stage, one of the things that we should be doing is international work. That figure is associated with that.
With respect, positioning the SQA on the international stage by diluting support and the organisation’s capacity to deliver in Scotland would not, I would have thought, be terribly good for your reputation. The EIS suggests—and I find this compelling—that there must be a dilution of your concentration on your main job of Scottish education if there is that level of certification externally. Would you accept that?
No, I would not. We look at this very specifically. The resources that we allocate associated with international work are not resources that could be put in place in Scotland.
The committee needs to remember that we deliver vocational qualifications in colleges and for training providers, industry sectors and private companies across Scotland for the benefit of the learners of Scotland. We use that expertise and knowledge to provide an income base or profit base that allows us to do more in Scotland. It is not about using resources that could be applied to Scottish activities.
11:45
I am well aware of the range of areas where the SQA has a role. However, to go back to my point, can you at least explain where the resource has come from to deliver a 500 per cent increase since 2010 in external qualifications if it has not come from your core provision?
The additional provision is paid for by the income that we generate. A profit is also generated.
Do you recognise that there is a concern among some people that the SQA has become an organisation that has a business model that creates an incentive to sell abroad or beyond Scotland and the danger is that you are not focusing on the day job?
All that I can tell you is that that is not the case. Our major focus and the major activity that we currently undertake is curriculum for excellence. The other thing that we absolutely need to ensure that we do not lose sight of is our requirement under statute to support the vocational space in Scotland. Those are the two major priorities. The other aspect is to try to ensure that we can continue to do that over the long term.
My final point is on the question of national 4 and what I think is an utter injustice towards a whole range of young people, who no longer have external certification. There is a real fear that we will go back to the days when I taught non-certificated classes, when no resources followed those because there was no external examination. When will there be a conclusion to the work that you talked about to assess how national 4 is playing out and the impact on the aspirations of young people?
As I said, that is part of the discussion that the Deputy First Minister is leading at the working group on assessment and national qualification. I am not aware of that timetable.
Okay—thank you.
I want to ask about comments in the submissions from Mark Priestley, Lindsay Paterson and the Royal Society of Edinburgh. To sum up, there are questions as to whether the current national qualification structure at S4 is compromising subject choice and quality and therefore compromising pupils’ ability to study the core subjects that are required for university entrance.
Professor Priestley says:
“There was a lack of dialogue about what is ... learned in schools, and why. Official documents have tended to focus on skills rather than knowledge. Our research suggests an absence of this sort of dialogue in schools”.
All those submissions draw the conclusion that there is a serious issue with the delivery of curriculum for excellence in that subject choice is being compromised, which is leading to further problems in colleges and universities. Do you agree with those comments?
There are two aspects to that. One is about the SQA’s role in that space. As I said at the beginning, our job is to ensure that we provide a course and a subsequent qualification that builds on what the learner has achieved and takes them to the point at which they can move to the next level. There is a fixed amount of content about knowledge and skill and about application and understanding, and that needs to go into the course. That creates a certain size. The nature of how that is applied within a school is a matter for that school, the local authority and consultation with parents and learners.
The way in which broad general education prepares a candidate for a qualification has an impact on the issue. As we talked about at the beginning, part of the philosophy of curriculum for excellence was to get away from the two-term dash. If a candidate was absolutely going to get a higher, they would work straight to a higher and if they were better suited to a national 5 or national 4, they would work towards those qualifications. That is still a topic for debate across the country.
Notwithstanding the fact that it is not your decision to say what courses are taught in schools—it is not for the SQA to decide that—have you had discussions about the concerns that you have raised not just today but previously that, as a result of curriculum for excellence, there is a squeeze on the number of subjects that pupils can opt for?
That is seen by many parents, as well as by some local authorities and schools, to be compromising the choices that pupils can make when they enter university. Do you share that concern, and have you had discussions about that with Education Scotland and the other education bodies?
That is one of the conversations that Scotland as a whole needs to have.
There have been many conversations about what we are trying to achieve through the outcomes at the end of the senior phase. One of the issues that we have tried to address through insight, the measurement tool that the Scottish Government has put in place, is that we should be thinking about the number of qualifications not at the end of a particular year but at the end of a candidate’s time at school. The practicalities of that need to be thought through.
Johann Lamont spoke about youngsters who might end up with nothing. On top of that, there seem to be concerns that the subject choice of those who are studying for the qualifications, which is crucial to what they are going to do after school, is being compromised because of the way in which the system is being run. You say that a conversation needs to be had about that, but I hope that it has already been had. Parents, pupils and staff are asking about that now because of the seriousness of it.
The seriousness of the issue has been brought home to us by our constituents, who tell us that fewer subjects are being offered in one school than are being offered in another school that might be only 10 miles away, and there seems to be no rhyme or reason to it. The SQA must surely be concerned about the situation even if you are not entirely responsible for it.
We should all be concerned—as we are—about positive destinations for students. However, I am not privy to all the conversations that occur in that space. That sounds like a cop-out, but it is not part of our remit, although we input into those conversations when we are available.
I accept what you are saying. It is true that it is not for you to dictate that. However, are you not concerned that there is a disjointed approach? We need a joined-up approach to ensure that youngsters are able to choose their courses and be examined in them—which obviously is within your remit—but professors in education, who are highly experienced people, are rightly making the point that the approach that is being taken to subject choices is not compatible with the statement about the need to enable students to get a broad, well-rounded education, particularly in what they go on to choose. That is hitting the strong tradition of what Scottish education has been able to deliver in the past. Is the subject choice issue becoming a big problem for curriculum for excellence?
The topic is increasingly being discussed.
Is it a problem, Dr Brown?
Some candidates need a lot of subjects; other candidates are ill served by being offered a lot of subjects. One of the philosophies behind curriculum for excellence was that schools should be the best place to discuss that. All that I can say is that the teachers should be the ones to decide. The challenge in the school sector is in whether the teachers are able to do that. Are they given the flexibility to do that, and what are the other constraints on the system?
On the issue that Liz Smith raises, the SQA recommends 160 hours of teaching and assessment time for its courses, but one of the submissions says that that is impossible to achieve in a single year. Is not your requirement of 160 hours of teaching in one year—forgive me if it is not your requirement—the building block of the answers that you have been giving us?
Yes. That is absolutely true. We are trying to take someone from the position that they are in at the end of their broad general education to the point at which they can get entry to the next level. That has a specific requirement associated with the amount of knowledge, understanding and skills development, and that is what defines the size of the course. We could have a half-size course—in England, there was the A level and the AS level or something halfway between—but, if we want people to get the full course, there must be that amount of knowledge, learning and understanding.
The notional learning hours reflected our expectation of approximately how long it would take an average candidate to do the course. It is therefore appropriate that schools allow for that amount of time to do the course. If schools try to do it all in one year, they are limiting the number of subjects that they can do.
But that is the reality of it. That is what is happening in every school that you have talked about.
Or schools are giving courses with as little as 90 hours of learning. That is a real challenge for teachers’ workload, because teachers are now teaching not just one 90-hour course but multiple 90-hour courses. That situation also has huge issues for learners in terms of the amount of information involved.
I agree with that. The other end of that is—you will tell me if this is not the case—that there is no school in Scotland that is teaching three sciences in one year, is there? That means that those kids who are good enough to do three sciences—I appreciate your earlier point that only a small percentage want to do that—because they want to go to university to do science, cannot take three sciences. Tell me whether I am wrong.
I think that there are some schools teaching three sciences in one year.
So that can still be done.
Yes.
Who keeps a record of those things? How do we find that out?
We do not collect that information.
No one seems to.
I do not know whether Education Scotland does.
It does not. We asked Education Scotland that question. We do not know who collects that information.
I think that the challenge here—what we need to be doing—is making sure that candidates are given the appropriate level and time for learning, teaching and the requirement for assessment that they need to be able to be successful in that course and qualification and successful in terms of the amount of learning that they have to make them successful in their destination. It is not just about passing the hurdle of getting the qualification.
Okay, but I am not sure that I have fully understood that.
I have a final question. In all your evidence today about workload, on which the committee has cited the submissions to the committee and the SQA, you said—I am sure that this is true—that the SQA meets Education Scotland and all the other organisations regularly. Do you get together and say “Look, together, this is what we are doing to schools”?
Yes.
Do you really?
Yes.
Well, why then have we had all the submissions that we have been referring to? Similarly, when we see Education Scotland, why do we find that we have a comparably vast number of submissions talking about having to go through 1,800 experiences and outcomes and the changes to benchmarks that have just come out again? Do you not get together, sit down and say “Right, you’re doing this, you’re doing that.”? How do we ease the pressure on schools and teachers?
Yes, and that is why we put out the subject review reports. Prior to being asked to remove the units, we recognised that our units were having a very detrimental effect on schools, so we tried to address that.
So in a year’s time, we will not be receiving 142 pages, or whatever Johann Lamont was citing for us, on all this again—and it will be similar with Education Scotland. It will all be smoothed out for next year.
No.
No, it will still be there.
No, because we do not know. We are now on a path of removing units, so we are not going to follow through with what the subject review reports requested, which focused heavily on units. We are no longer doing units. I think that we will have feedback on the changes that we are implementing.
Okay. Thank you.
Feedback—good. Thank you.
We had two other subject matters to consider—quality assurance and accountability and governance—but they seem to have been covered already in questions. Does anybody have a further contribution before I round up the session?
Dr Brown said in response to Ross Thomson’s question—Johann Lamont mentioned this as well—that it is important not to focus only on the positives, but she did not really provide any explanation of the negatives. However, as Tavish Scott has just pointed out, it is not usual for the committee to receive the volume of critical submissions that we have been referring to. Without reference to your current processes, Dr Brown—I think that in most of your answers you have talked about your feedback and current processes—and in just broad, simple terms, can you tell us why you think that a substantial and significant number of teachers, regardless of whether it is the majority of teachers, seem to have lost confidence in you? Again, can you tell us in broad, simple terms what you are going to do to fix that? I think that that is what this committee needs to hear today.
I think that a lot of the negative views are associated with the way in which the qualifications have been designed and implemented and the way in which they have worked. What we have done—and continue to do—is try to understand why they have not worked in the way that we anticipated they would work. We need to make sure that we remove the problems that we have created as a result of those designs.
12:00
Is it not a pretty damning statement to say that the problem is due to the way in which the exams are designed and delivered? Is that not what the SQA is for? If you have failed in that, it is surely pretty significant.
Again, in our research we tried to understand the root causes—I emphasise the word “causes”—of why they have not worked. There are aspects that are in the SQA’s remit and that are our responsibility to address. My responsibility to Scotland is to tell people when we have not got something right. That is how we can improve. Introducing any new qualification from any awarding body anywhere, you learn what is practical and how things operate in schools. We have had a lot of discussion about the nature of the curriculum and how things are operating in schools.
There were other reasons why the assessments were not working properly. I cannot address those, but I can highlight them. All that I can do is to address the issues that the assessments themselves have caused. That is something that the SQA will be sharing, because we should learn from the things that we had assumed would work. We had conversations with teachers and we thought that the feedback would be that it worked, but in certain cases it has not worked. That is not the situation across the board. The qualifications are not a problem in general. There were issues associated with aspects of the unit assessment, but the course assessments were all fine. That was the feedback from our fieldwork.
This has been a very challenging evidence session for our witnesses. I hope that you respect that it has to be that way because the committee’s job is to scrutinise and ensure that things are as good as they can be.
However, I wanted to come back in to say that, following last week’s visit, all is not lost. Other members should take the opportunity to visit the SQA and see the amount of work that is being done. Last week I saw a group of people who are dedicated and proud of Scotland’s educational position and who want to make a difference. We need to hold on to that. I ask you to take away what you have heard from around the table and from the submissions, and when you are carrying out your fieldwork and other assessments be honest about the situation and come back to the committee next year, or whenever, when I hope that we can have a more positive discussion.
Okay, thank you. Do you want to respond to that, Dr Brown?
Thank you for that. The one thing that you notice about the SQA when you walk in the door is that everybody cares passionately about the learners of Scotland. We do not like it when we do not get it right.
Thank you for your time and evidence—it has been more than two hours since we began. You are right that there will be feedback no matter what the changes are next year—as there should be—but it is clear from the evidence that we have heard today that we should not be getting the same response from people in relation to your relationship with teachers and the communication with teachers and parents. The SQA should at least be able to ensure that it is communicating with the organisations involved about whatever changes are required and the need for those changes, in a way that everyone can understand. That seems to be an on-going problem.
Thank you once again for coming.
12:03 Meeting continued in private until 12:48.