Education and Skills Committee

Meeting date: Wednesday, August 12, 2020

Official Report 610KB pdf

---

Scottish Qualifications Authority Results 2020

Item 3, which is our main item of business this morning, is an evidence session on the Scottish Qualifications Authority results for 2020. I welcome from the SQA Fiona Robertson, who is chief executive, and Dr Gill Stewart, who is director of qualifications. We are very tight for time this morning, but I invite Fiona Robertson to make a brief opening statement.

Thank you, convener—I would like to make an opening statement.

I thank the committee for the opportunity to discuss the 2020 results, which were published last Tuesday. Given the unprecedented nature of this year’s process and its impact on many thousands of young people across Scotland, I am very pleased to appear before the committee this morning to give you more detail and answer any questions that you may have.

Everyone at the SQA is keenly aware of the strength of feeling about last week’s results, from young people in particular. I have heard some of those concerns directly. The committee will, of course, want to discuss what we did and why. I hope that my opening statement and answers today will provide the clarification that you will be seeking.

As you know, exams in Scotland have been held every spring since 1888. On 19 March, the Deputy First Minister announced the closure of schools and the cancellation of exams in response to Covid-19. At the same time, he commissioned the SQA to develop an alternative certification model for 2020 to ensure that the hard work of young people across Scotland would be recognised through our qualifications system, as it would in any year. Maintaining standards over time was a key element of that, and it was central to the approach that we took. Indeed, since the outset, our approach in good faith has been based on three core principles that reflect both how we work and the circumstances of 2020: fairness to all learners; safe and secure certification of our qualifications while following the latest public health advice; and maintaining the integrity and credibility of our qualifications system, ensuring that standards are maintained over time in the interests of learners.

On 20 April, we set out the four key stages of our approach: estimates, awarding, results and certification, and appeals. I will talk you briefly through each element, as I think that it is important to do so. The key input to our alternative certification model has been based on estimated grades. Research tells us that school and college estimates are not always accurate; the accuracy of estimates varies across centres, subjects and courses. We receive estimates every year from schools and colleges. In 2019, we found—using matched data at candidate level, which is important—that 48 per cent of grades estimated at national 5 resulted in those grades; at higher, the figure was 44 per cent; and at advanced higher, it was 43 per cent. That is an important part of the case for moderation of grades.

We knew that evidence of the accuracy of teacher estimates was below 50 per cent and, given the commission from ministers, we had a responsibility to consider the moderation of teacher estimates where appropriate. I made that clear to the committee on 1 May. However, given the importance of the estimates this year, we made considerable efforts to assist teachers with estimation. We provided schools and colleges with detailed guidance and an online course on 20 April, and we also provided them with additional data, covering estimation over a number of years, so that they could look at it.

On 4 August, the SQA accepted almost three quarters of teacher and lecturer estimates, and 99 per cent of awards were awarded at or within one grade of the estimate. I will say more about that in a moment, but I can say now that there could have been no certification of any nature this year without those estimates, and we owe schools and colleges our sincere thanks. We treated those estimates with the utmost respect.

Professional judgment for assessment is at the heart of Scottish education. Every year, effective judgments take place in schools and colleges and are supported, validated and enhanced through moderation. This year, in the absence of external assessment, moderation was a key part of our approach, both within schools and colleges, using a range of data and discussion, and nationally by the SQA.

On 29 May, we receive estimates from schools and colleges across Scotland, representing 511,070 entries across 129 subjects and 467 centres. At a national level, the estimates that we received were above previous A to C attainment at national 5, higher and advanced higher. Attainment rates of course vary between subjects and over time, but estimated A to C attainment rates were 10.4 percentage points higher at national 5, 14 percentage points higher at higher and 13.4 percentage points higher at advanced higher, compared with results from the previous year. The level of estimation at grade A contributed most to those higher A to C estimated grades, particularly at higher and advanced higher.

On the basis of the commission that we received from the Scottish Government, there was a clear and unequivocal case for some moderation. My colleague Gill Stewart can say a little more about the detail of that process if the committee wishes. Full details of our awarding process are provided in a range of documents that were published last week, including our methodology report. We also published 129 reports relating to each course.

Starting point distributions were constructed for each course, setting out the parameters for each grade and at A to C, with tolerances around the mean to allow some variability in attainment this year. That included the input of teachers, our principal assessors and subject specialists to ensure integrity.

Historical attainment formed part of the process at centre level. If we accept the premise of moderation, it had to. However, we acknowledged the volatility of a centre’s attainment over time, and we made additional allowances for that. Historical attainment was not defined purely on the basis of past performance.

Every school and college was in scope for moderation, and all of them were treated in the same way. We did not know the name or location of the centre when it was being moderated; that is an important part of demonstrating fairness and consistency in our approach. Schools and colleges were not seen as postcodes at all.

We said that we would consider entering into a dialogue with schools and colleges about their estimates to explore the reasons why they were higher or lower than before, and we did consider that. I can understand why the committee and others may have seen that as a sensible step. Time constraints were an issue but, crucially, we were not confident that that could be done fairly and consistently. It needed to be evidence based, and we designed an appeals process to be evidence based.

We always acknowledged that the extraordinary circumstances of this year could lead to a number of anomalies in the results—and there were. I have received very reasonable and considered correspondence from schools, which has highlighted that.

The last part of our four-stage process, the appeals process, was a very important stage. It would have provided for further evidence-based review of individual candidates’ work if schools and colleges did not think that the awarded grades fairly reflected candidate performance. I can assure the committee that we were ready for a higher volume of appeals this year. There has been misreporting of that, which has not served disappointed young people well.

The committee was right to press me on our responsibilities in relation to equalities, but our commitment to that was never in doubt. Throughout the development of the model, we had equalities at the heart of our thinking, and we developed and refined an equalities impact assessment and a children’s rights and wellbeing impact assessment to inform our approaches and decisions and to demonstrate our compliance with our statutory obligations.

Our guidance to centres on equalities in the estimation process assisted them in fulfilling their equalities responsibilities. Our post-certification review and exceptional consideration arrangements were also designed to address any cases of discrimination or bias in the original estimates by centres. Our statistical analysis of available data from 2006 onwards demonstrates that, after moderation, learners in the most deprived Scottish index of multiple deprivation bandings saw attainment levels for A to C higher than both the 2019 results and the average result for 2016 to 2019, as well as a narrowing of the attainment gap between those in the most deprived and least deprived SIMD bandings over the same period.

What matters is outcomes and achievements. I have already highlighted in broad terms the moderation outcomes, although it is important to underline again the value that we placed on teacher estimates. Nearly 75 per cent of estimates were accepted, and 99 per cent of entries were awarded at or within one grade of the estimates. Using the alternative certification model, A to C attainment in 2020 was as follows: national 5, 81 per cent, compared to 78.2 per cent in 2019; higher, 78.9 per cent, compared to 74.8 per cent in 2019; and advanced higher, 84.9 per cent, compared to 79.4 per cent in 2019.

To conclude on the approach that we took, we were commissioned by ministers to develop an alternative certification model to maintain standards over time, and we did that in good faith. On the basis of the commission, the SQA felt that there was a clear and unequivocal case for moderation, and we developed an approach to do that. An enhanced, individualised and evidence-based appeals service was developed as the final stage, recognising the exceptional and unique circumstances of this year. However, I acknowledge that, although we can congratulate many thousands of young people who, in line with their expectations, achieved a strong result last week, the system also meant that some people did not receive the awards that they felt that they were capable of achieving. Joy and disappointment are a feature of results day every year, but this year disappointment was compounded by the cancellation of exams. Young people did not have the opportunity to demonstrate what they could achieve, and moderation felt impersonal to them.

Thank you.

If I may, I have a final comment on the latest developments of yesterday.

Yesterday, in his statement to Parliament, the Deputy First Minister outlined that he was using powers available to him under the Education (Scotland) Act 1996 to direct the SQA to change the previous approach to certification this year and, instead, award candidate results based solely on the estimates that were provided by schools and colleges. As members know, the moderation process was symmetric, and we moderated up around 9,000 entries; those awards will stand. It is the prerogative of ministers to issue a direction of that kind; it is our responsibility to comply fully. I am pleased to say that schools and colleges already have the estimates information that will form the basis of the amended results, but we will work to ensure that centres receive their confirmed results as soon as possible and no later than the end of next week. Any candidate who, as a result, has a grade change will receive a new certificate, and we will publish final headline results on Friday 21 August. We will also share confirmed results with the Universities and Colleges Admissions Service.

With the Scottish Government, we are considering in more detail any appeals process, which should support that direction, and we will announce further details shortly. Of course, we will co-operate fully and positively with the review of the alternative certification model for national qualifications 2020 and the Organisation for Economic Co-operation and Development review. We will shortly be announcing a consultation exercise on options for arrangements for qualifications in the 2020-21 session. Our focus there will be to assist the system in what could be a difficult year ahead.

To conclude, the SQA delivered on the Scottish Government’s initial request, and we believe that we moderated grades judiciously to maintain standards over time and ensure the credibility of qualifications for the benefit of learners. Through our equality impact assessment, we demonstrated that the process produced fair outcomes. An individual, evidence-based appeals process was in place to deal with any issues. Ministers have reflected on the results in this extraordinary year and have asked us to deliver a change to the process.

As chief executive of the SQA, I can assure the committee that we are an organisation that, above all else, is committed to delivering for learners. We are also an organisation of integrity, and we seek to undertake our work to the highest professional standards. We worked very hard this year to deliver, and I can assure you that any criticism of the past week has been felt most keenly by me and my colleagues, who I have the privilege to work alongside.

10:15  

As chief examiner, I have a responsibility to this year’s learners who entered for our qualifications, but I also have a responsibility to the learners of the past and of the future. Therefore, the focus on maintaining standards of our qualifications over time and between schools felt not only right but essential. However, the ask in this extraordinary year has changed. Given that change, we will meet our responsibilities and we will deliver on them.

Thank you for the opportunity to appear before the committee this morning. I am happy to answer your questions.

Thank you. I again urge brevity in questions and answers. That would be helpful, as there are a lot of areas to cover and a lot of members who want to come in.

Ms Robertson, it is unfortunate that you have taken 15 minutes to open with what was largely a defence of the decision that has just been overturned. I think that what we need is a reflection on what has happened and what needs to be altered in future arrangements.

I will begin by addressing the key reflection that needs to take place. Your defence of the methodology has been based largely on the aggregate outcomes of results. However, the issue is the impact on individual schools at the micro level. I will give you an example from my constituency.

Perhaps the most improved school in the city over the past five years is one that is in my constituency. However, it is also the school that was the worst impacted, with 76 per cent of higher results downgraded. In maths, over the past four years, the school has consistently shown a 3 per cent increase in attainment, and this year’s cohort outperformed last year’s cohort in national 5. However, despite that, not only did the pupils in that cohort do worse in higher maths than last year’s cohort, they received the second-worst set of results in that school in the past five years. Surely that demonstrates the problem that, when you take a mean average of attainment over the past five years and use that to, essentially, predict the results of that cohort, that completely removes the ability to take into account improving attainment levels from a centre. I ask you to reflect on whether that is a critical issue with the methodology that has been deployed. More importantly, did you test the methodology? Did you identify that as being a problem? Did you look at the range of downgrading? Although the headline figure for downgrades is around a quarter, to have any centre with 76 per cent of grades being downgraded surely should have rung alarm bells.

There are a number of issues there and I will seek to address them. I will also bring in Gill Stewart to talk a little bit more about the centre level moderation.

You are absolutely right to say that the national position in relation to estimates is not the full story, because every school and college across Scotland also provided estimates as part of that, and those estimates varied considerably between and across schools and between and across subjects. What we were seeking to do was to bring in a moderation process that recognised that and to ensure that, crucially, moderation was a function of the estimates that we received. For the most part, attainment in schools across Scotland went up this year—as I highlighted in my opening statement, attainment actually increased. There are some subjects and some schools that have shown anomalous results. We have had discussions with those schools, and the intention was that that issue would be considered as part of our appeals process. Those schools are relatively small in number, and I think that we recognised that, in the extraordinary circumstances of this year and the task that we were asked to undertake, it was important that that safety net was put in place.

With regard to the methodology around what we call centre level moderation, we were looking at the historical attainment by course in each school.

As I highlighted in my opening statement, we looked to ensure that there was an element of relative flexibility—a tolerance—at either end of that range of historical attainment. What we were seeking to do, but with some flexibility, was to ensure, in broad terms, that awards reflected that historical attainment.

Gill Stewart may want to comment on the centre level moderation. As well as looking at the national position, it was important that we understood and reflected on what centres told us. I have a responsibility not only to this year’s young people and young people across years but to ensuring that the standard of an A in the school that you describe is also the standard of an A—

Apologies, Ms Robertson, but we have only a limited number of questions this morning. I did not hear you say that you had tested that approach or looked at outliers, and you have insisted that you looked at a historical mean.

Moving on, I note that the process that you set out in your document confirms what you are saying this morning, which is, essentially, that you looked up a statistical range of results and applied that. When you appeared before us on 1 May, you repeated information that you had provided on 20 April, which was:

“We will use the information from these estimates, in addition to prior learner attainment, where this is available. For example, if learners achieved National 5 or Higher courses, in a previous year.

We will also look at schools’ and colleges’ previous history of estimating and attainment”.

However, those were not features that appeared in your methodology, so what you stated would happen in terms of moderation was not what happened.

Critically, in using only historical average attainment by centre, did you not, ultimately, break the link between individual learner performance and the grade that they finally received? It was not based on that at all; it was based purely on whether the predicted grade was in line with the historical mean. You did not check whether there was justification for applying that estimate; you just rounded the grade down if it fell outside the average that you had arrived at in your statistical process. Is that not correct?

That is not correct. Prior attainment formed a feature of the work that we did in setting the starting point distributions for each course.

The difficulty that we have in Scotland is that we have an integrated senior phase, with no prior attainment data for national 5s. That is set out in our methodology report. For highers, we have partial prior attainment data.

A number of young people and, indeed, a number of schools do not undertake national 5s at all—they go straight to highers. In addition, a number of young people will do a crash higher in a subject, so they will have no prior attainment.

The data set is imperfect. Because the data set was incomplete and therefore could not be applied consistently, we could not include it in the final model. The fact of the matter is that, even at quite high qualification levels, young people in Scotland do not always have a lower level of qualification.

In terms of previous estimates of attainment, what we sought to do this year was ensure that the estimates from schools and colleges were as good as they could be. To an extent, we effectively broke the previous relationship, and that is evident in the figures. We have gone from an average of 45 per cent of grades; on the basis of our moderation approach, we accepted 75 per cent this year. Estimating behaviour by schools was a little different this year. We thought that it was more accurate, because we accepted 75 per cent, when we knew that last year the figure was less than 50 per cent.

There are reasons for the approach that we took, and I hope that you understand the point about prior attainment. We did not have individualised prior attainment for young people across our qualifications.

You said that I was not correct in my assessment, but you then gave an explanation that seemed to reinforce it. However, I will move on.

I think that I highlighted, just to be clear, that prior attainment was part of our methodology. That was why I gave you the answer that I did.

Just to add another point, we tested our methodology on a number of occasions. There are 22,000 course and centre combinations, so we could not possibly look at every single one of those. However, for each refinement of our model, we looked at the impact at the national level and the impact on every single course, and we also sampled impacts on individual centres. However, we could not look at the impact on every single centre for every single course that was on offer, because we had a relatively short period of time.

I am going to bring in remotely Jamie Greene, who will be followed by Alasdair Allan.

Thank you, convener. Good morning to the committee and our guests.

I appreciate that this has been an extraordinary and difficult year for everyone in the education sector, and not least for those who work on its front line. Given that today is back-to-school day, it is important that we pay credit especially to those teachers who have worked hard under the instructions of ministers and agencies.

I want to get to the crux of the matter. I listened with intent and respectfully to Fiona Robertson’s opening statement, but there is one word that I did not hear anywhere in it: the word “sorry”. I say that because the Deputy First Minister and the First Minister have apologised this week for the handling of this year’s awards, and yesterday the Deputy First Minister pretty much overturned everything that you and your agency have been doing for the past four months. What do you say to him today? What do you say to the young people whom he had to apologise to?

What I sought to do in my opening statement was to explain what we had done and the basis for that. It was, of course, very difficult to see the reaction to last week’s results, but we were asked to fulfil a role, and part of that role was to maintain standards across Scotland.

Although I absolutely fully understand the feelings of young people and the strength of those feelings, it was important—I think that the Deputy First Minister reflected this in his statement to Parliament—that I should explain the basis on which we undertook that work. It was a commission from ministers after exams had been cancelled and in an extraordinary set of circumstances—I think that we would all acknowledge that. We did our very best to deliver, but I fully appreciate that, as I highlighted in my opening statement, young people felt that their achievements had been taken outwith their control. I absolutely get that and, of course, I regret how young people have felt about the process.

However, it is also important to highlight that I have had messages from many young people over the past week that have said how pleased they are with the outcome. It is therefore important to temper quite clearly the strong feelings of young people across Scotland who feel that they have been unfairly treated with the delight of others who feel that they have got awards that reflect their achievements. I absolutely regret the circumstances of the past week, but it is important that the committee and, indeed, anyone who is watching the committee understand the basis on which we undertook the work on a commission from the Government and did our very best to fulfil it.

It is also important to remember the last stage in the process, which is the appeals process to identify—

Sorry, but—

Sorry, Mr Greene, but Dr Stewart wants to come in, so just give her a few seconds. Thanks.

10:30  

Thank you, convener.

It was just to say that it is important to remember that the alternative certification model had four steps, including the appeals process, which was put in place so that any issues that resulted from the model that the SQA applied could be addressed through schools and colleges submitting appeals for individuals or groups of candidates with evidence, which would be looked at by teachers who work with us and who are trained to do that. It was always the intent that the process should include that last step, to allow any issues with the results to be addressed. Obviously, we did not get to the stage of implementing that stage of the alternative certification model.

I apologise for interrupting, Dr Stewart.

Does that not get to the nub of the issue? The appeals process is a fundamental part of the whole process, but that process no longer exists. It has been thrown out of the window by the ministers.

I think that there is still some confusion over this. Who made the decision to apply the methodology that downgraded those in our most deprived communities at a much higher rate than—almost double—the rate for those in our least deprived areas? At what point did somebody flag that and say, “That doesn’t look right. That doesn’t feel right”? Who signed off the methodology? Was it the SQA, the education secretary or the chief executive?

It is important to remember that all the data was anonymised during the moderation process. We did not know which centre or which candidate it came from—it was all anonymised to avoid any bias that might emerge through the methodology. At the end of the process, we did the analysis, working with the Scottish Government, to look at the impact and the data that we held and it held around the SIMD, age and sex. During the process, the data was completely anonymised, so there was no opportunity for any bias about particular schools.

I think that your substantial question, Mr Greene, was about the broader point of who was responsible for the methodology. I am sure that issues around equalities will come up during the course of this session. We ensured that we fulfilled our responsibilities. Before we concluded the process, we made sure that we had fulfilled our equality duties as an organisation, for all the right reasons.

As the Deputy First Minister said yesterday, I think, the SQA is an executive non-departmental public body. Members of the committee will be familiar with what that means. I report to a board that is appointed by ministers, and that is the link between the organisation and ministers.

I appreciate that the announcement yesterday and the direction that was given did not happen under normal circumstances but, in a normal year, it is for the SQA to discharge its statutory functions in awarding qualifications in Scotland. This year, despite its extraordinary nature, that has been no different. We were given a commission by the Deputy First Minister on 19 March, when he had announced school closures and the cancellation of exams—which is absolutely his prerogative, and understandable in the circumstances—and it was our responsibility to fulfil that function. The methodology was the responsibility of the SQA, and, as you would expect, it was considered by the SQA board, which is accountable to ministers.

It is important that I set out that broad piece. As the Deputy First Minister said yesterday in Parliament, I provided a briefing to him on 30 July, to set out the context and the detail of the approach that we had taken. That is quite a standard feature of the system around pre-release of the attainment statistics.

You have very helpfully explained the relationship between the agency and ministers, and I get that. However, my question is not about that. My question is: who takes ultimate responsibility for the way that this year’s exams were handled? The First Minister and the education secretary have very publicly absolved you and your agency of all responsibility for it. If you are not to blame, who is?

What I would say—I suppose that I am repeating a point that I have already made—is that we were asked to develop an alternative certification model to maintain standards, and we believed that we did that to the best of our ability. It was our responsibility to do that. That was done under a commission or a direction from ministers back in March.

Although I understand what you are saying, I think that that accountability relationship is quite important. Indeed, that was brought into sharp focus yesterday with the announcement of a ministerial direction. I think that Mr Swinney also highlighted that we had undertaken and fulfilled our role as it was expressed back in March. We have already talked about the circumstances that led to that change.

As chief examiner, I have responsibility for ensuring the credibility and integrity of our qualifications, but also—crucially—for maintaining standards over time. I genuinely believe that it was a reasonable thing—indeed, I made this clear at committee in advance of sight of the estimates, as we had not received them at that point, on 1 May—that we needed to reserve the right to moderate. The reason for that was simple. It was on the basis that we knew that teacher estimates did not always result in those grades, so it felt fair to candidates that we considered moderation.

There was no presumption that moderation would be downwards. Moderation was a symmetric process, and, indeed, we moderated about 9,000 entries up. If estimates came in low, we would have made that adjustment. That is an important point. This was not about taking grades off young people; it was about achieving fairness, albeit in the extraordinary circumstances of this year.

However, as I have said, I absolutely acknowledge the strength of feeling as a result of the results that we published last Tuesday.

There has been, as I do not need to tell you, a great deal of debate in Scotland and indeed today in England about the wisdom of linking the methodology to the whole issue of past performance.

You have indicated that the SQA was responsible for the methodology. Putting to one side the crucial issue of whether it was wise in itself, I want to ask you about the degree to which you felt that appropriate data was available to allow you to use that system. I am thinking specifically of small subjects, by which I mean subjects for which any one school has very little evidence from having candidates in the past, and also small schools, where the number of people taking any one higher in any one year is low. I can think of some examples in my constituency where the numbers might be one, two or three people.

We highlight some of the challenges in our methodology report, and I also mentioned them last week in my chief examining officer’s report. There is no doubt that, as you rightly say, we have quite small uptakes for some subjects, even at the national level.

As you will know, Dr Allan, we offer a lot of subjects across our curriculum, and we even offer low-uptake subjects. At the school or college level, the entry levels are sometimes really quite small. However, the centre level moderation allowed for that in the sense that it did not confine attainment this year to historical attainment.

The methodology report also highlights that, for schools or colleges that did not have a previous record of attainment, we accepted the estimates. That related to, I think, about 6,000 entries in total across a range of subjects. If a centre was presenting a course for the first time, the estimates were accepted after moderation so that it did not affect any other part of the approach that we took.

In committee previously, you mentioned the SQA’s view on how the methodology might be implemented. You said:

“I hope that that provides a bit of reassurance. The assertion that somehow we will fail a young person because of the school that they go to is an unfair one to make on the basis of what I have sought to say about the process that we are undertaking.”—[Official Report, Education and Skills Committee, 1 May 2020; c 15.]

The accusation that many young people have made is that the school that they go to did play a part. Even if we put that to one side, and although I understand that there will be situations in which people get two or three grades below or above what they were expecting to get, did the system flag up situations, of which we have heard many examples, in which people dropped two or three grades and failed—or failed in effect, from the point of view of university entrance—on the basis of the contentious algorithm that we are talking about?

I highlighted that the shape of the estimates and the attainment profile that came in resulted in moderation that was largely down. The moderation was done on the basis of a tolerance around historical attainment. I understand that historical attainment may feel contentious to the committee as it feels like a constraint and does not allow for variability, but I need to assure the committee that the approach that we took allowed for some flexibility. However, on the basis that quite a lot of the increase in attainment was at A grade, there was inevitably a fall down from A to B, B to C and so on. The reason for that is that we maintained the rank order of teacher estimates—we maintained the relative view of teachers and lecturers about their pupils. We got rank order information for each band that we received.

As I made clear in my chief examining officer’s report, I acknowledge that, as a result of moderation, a number of young people moved from A to C, and to D and E. However, the vast majority—in excess of 90 per cent of those young people—were estimated at a C grade.

For the young people who moved from an A to a C, or from a B to a D, was the system that you were using sufficiently sensitive to identify them and flag them up as potential outliers, and were such individual cases followed up and investigated to find out whether the system that you were using was robust?

Particularly in the early work that we did around the moderation process, we absolutely looked at what we called “multigrade changes” as a result of moderation. Without getting too technical about it, the approach that we took sought to minimise as far as possible multigrade changes. As I highlighted at the start of my statement, there was a relatively small number of them—99 per cent of entries were awarded at or within one grade of their estimate. There was very little multigrade movement.

There was some media reporting about young people moving from an A to an F, but we do not have an F grade in Scotland. Last week, we recommended to young people that they speak to their school about the estimate that they received because, in some cases, either schools had not shared the estimates with young people, or young people were just unaware of the estimates that had been presented for them. That set of expectations and that understanding have been an important part of the process. However, I absolutely accept that, this year, if you were a young person who was expecting an A last week and you got a C from us, that felt like a tough result. I understand that.

10:45  

Before I pursue a slightly separate line of questioning, I want to continue on the issue that Dr Allan raised. Ms Robertson, you will have received similar correspondence to that which members have received, but I want to pick out one example that has been raised with me and ask how the methodology resulted in this situation.

I was contacted by a parent whose child has a really exceptional aptitude for maths—he got 94 per cent in his national 5 maths, 90 per cent in his higher and he was working towards advanced higher. His parent believes that he was predicted to get an A although, as you said, his teacher may have estimated a B. If so, there would be questions to ask as to why but, whether he was predicted to get an A or a B by his teacher, he was assigned a D.

I am trying to figure out how the system that you used could result in that. Is it the case that, because no pupil in his school for the past three or four years had managed to achieve an A in advanced higher maths, the methodology made it impossible for him to get an A, or am I misunderstanding the methodology?

I do not have the background to that example but, taking it at face value, obviously, he was a strong candidate at national 5 and at higher, but it is important to highlight that an A at higher does not necessarily translate into an A at advanced higher. There are lots of reasons for that. At advanced higher in particular, attainment can be impacted by unconditional offers from university, for example, although I am not making any judgment on the case that you cite.

It is important to understand what the school estimated because, without that, it is difficult for me to be definitive about how the situation came about. Certainly, with a high number of people who moved from an A to a B, their teacher estimates were at A band 5, so they were at the lowest banding of an A—

Forgive me for interrupting. That is useful, but we are constrained for time. The specific question that I asked was this: if no pupils at his school had managed to achieve an A at advanced higher maths for the previous few years, was it ever possible under the methodology for him to be awarded an A?

Yes, it should have been possible because of the tolerances that I highlighted. It certainly was possible to have an attainment pattern in a school that was not fully reflected in historical attainment—that was an important part of the process. I understand that the committee might provide examples this morning and assert that they show that the process has not quite worked. It is important to highlight that I have also had correspondence about situations where there have been anomalies in particular courses in particular schools. A number of factors come into play in that.

I realise that this is going into the technical detail, but the results could be partly impacted by the shape and nature of the estimates that the school provided, and partly by the starting point distribution for the course. In effect, as part of the moderation, we were looking to assess all centres across all courses and bring the results within the starting point distributions. In some cases—

I am really sorry to cut you off again, but I feel that we are slightly repeating and going over the same ground here, and we need to move on, because we have a lot to get through.

I appreciate that. I hope that you appreciate that I am seeking to do the best that I can to answer your questions, and that it is important that I try to explain the technical detail on some of the issues.

I accept that—I get that it is hard to explain a complex system concisely and in a short space of time. However, I want to move on to the construction of the system. I understand that the SQA has internal statistical capacity and has a permanent statistician in its staff. Were additional statisticians or other equivalent postholders brought in, either directly from the Scottish Government or from external sources to assist in the construction of the process? If so, how much did that cost?

The short answer is yes. As I highlighted, it is an extraordinary year. We do not normally do awarding in this way. We needed to undertake some further work, and additional resources were required. We were able to second some statistician support from the Scottish Government, and we also secured the services of two external providers. That is detailed in the methodology report.

I do not have the sums to hand, but I can certainly provide that to the committee, as you wish. That is not a problem.

Thank you. Very briefly, to finish off: am I correct in saying that there was a post of permanent statistician in the SQA?

There are a number of statisticians in the SQA; there is a small statistics team.

Obviously, you cannot go into any specific individual details, but is it the case that, during the process, at least one of your own statisticians resigned?

Yes, I understand that one of our statisticians resigned. In any organisation of the size of the SQA, there will be people joining and people leaving during such a process.

Staff turnover is normal. However, did they resign because they were concerned about the process?

I am not privy to the full details of that individual, and, in fairness to them, it would probably not be appropriate for me to go into that. People leave organisations for a number of different reasons.

Thank you. I am conscious of time, convener, so we can move on.

Good morning to the witnesses.

My question is on behalf of several teachers who have been in touch with me during the past week.

Fiona Robertson, I think that you said in your opening remarks that about 9,000 pupils had had their marks revised upwards in the initial marking. A lot of pupils who have had their marks downgraded will be wondering why some people have had their marks upgraded.

What was the methodology for marking grades upwards from teacher estimates? If those pupils go on to be presented for highers or advanced highers that they are not ready for, has that not given a group of young people false hope?

On the specific question about the 9,000 entries that we moderated upwards, it is important to make the point that, in the moderation process, there was no presumption that grades would go up or down, or stay the same. There was symmetry to the moderation process. If we found that the estimates were such that we felt that grades could go up, it was important that they should. That was an important part of the process, and, through the direction from ministers, we confirmed yesterday that those upgrades would stand. Some 9,000 entries went up.

On your point about progression, are you interested in progression from national 5 to higher and advanced higher, and how this year’s set of results impacts on that?

Yes.

To be honest, I think that schools were very cognisant of that this year, given the disruption to schooling—it was really nice this morning to see young people going to school for the first time after so many months. They will be making those curriculum decisions about progression from national 5 to higher, and to advanced higher, in the normal way. That is important, and it sits alongside the awards that they have this year. Those are discussions that can quite rightly take place in a school or college, in the normal way.

But the point is that, if a teacher estimated that a pupil would get a B or a C or whatever—the grade is irrelevant—and the pupil was not expecting to get a grade that would lead to their being presented at higher or advanced higher level or, indeed, to their getting a place at college or university, the SQA’s sudden deeming of the teacher’s estimate to be incorrect and the awarding of a higher grade might mean that the pupil will not be ready for the course that they are presented for. They will be proceeding on the basis of a false assumption that goes against what the teacher recommended that they could go on to do.

Are you suggesting that, in the awards that we provide this year, we should not accept the higher grades?

No, that is not what I am suggesting at all. I am asking why the SQA would upgrade teachers’ estimates, with the result that pupils might go on to do courses that they are not ready for.

I go back to the reason for our undertaking moderation. We wanted to ensure that the standard of an A or a B in one school was, broadly speaking, the standard of an A or a B in another school. The estimates were undoubtedly very pertinent to that, but in a number of courses in a number of schools we felt that it was legitimate to move candidates up.

The crucial point here is that decisions about subject choice and curriculum decisions, including decisions about whether young people should move on to take a higher or an advanced higher, are always made in schools. I have highlighted the case for moderation. In moderating more broadly, we sought to ensure that there was fairness across centres. I fully appreciate that some members of the committee are contesting the premise of moderation—I accept that—but that is why we undertook it. It is important to recognise that we understood that estimated grades could go up as well as down through moderation. That is an important part of what we were doing. There was no presumption that estimates could go in only one direction. The symmetry of the process felt right.

You say that you wanted to have fairness across all centres. I can absolutely understand why you would want that, but how is it possible to have fairness across all centres if, as you have said, the results were all anonymous?

The reason for that anonymity in the moderation process was to ensure that we treated all centres fairly and consistently. It was an important part of ensuring fairness, as far as possible, that, in principle, we treated all centres in the same way, that they were within scope and that we looked at the data in the appropriate way. It would have been wrong to have made judgments about a school on the basis of its postcode, its location or its sector, so that felt like the right and the responsible thing to do as part of the process.

So all that you knew about centres was their past results; you had no idea where they were placed.

No, and I think that that was an important part of the process. Taking that argument through to its conclusion, it was important that we did not make a judgment about a school on the basis of anything other than the evidence that was presented to us. In any normal year, for the most part teachers will not know very much about the scripts that they are marking, and I think that that is right.

It is a central premise of what we do that we look at the evidence quite robustly but, as it were, without fear or favour. That is quite an important element in the process. Of course, all of that said, towards the end of the process, as we were doing the analysis and we undertook the equalities impact assessment, we needed to have an understanding of the impact of the process. It was at that point that the data was presented in a different way, so that we could have a look at it.

11:00  

We will move to Beatrice Wishart.

First, I pay tribute to the educationists across Scotland and especially to our young people, who have been at the heart of this process. They have carried themselves remarkably well and with dignity.

Ms Robertson, you indicated that the professional judgment of teachers was at the heart of the process each year. However, some teachers have had their professional judgment questioned—I am thinking of the SQA markers. Is there a risk that teachers will no longer have confidence in the SQA or want to take up SQA roles in future?

You have touched on an important part of what we do every year, which is quality assurance. Albeit that this year was very different in many respects, the principle of quality assurance stands. It is important to note that those checks and balances in the system are undertaken by teachers for teachers. We work with many thousands of teachers each year who do that work for us, and they understand that it is an important feature of maintaining standards. It was actually a pretty big ask for every teacher and lecturer in Scotland across all 467 centres to estimate entirely consistently. I accept that there is a debate about the method of quality assurance, but I do not think that the principle of quality assurance was lost in the system with regard to our thinking about the best way to proceed.

Principal assessors were involved in the final stages of the process, when we undertook national awarding meetings to look at all 129 courses, and the grade distributions were signed off by them. The teachers who you mention were still very much part of the process, and we engaged with them as part of the process. I think that they understood why we were doing what we were doing.

My question was actually about future confidence, but I will move on to a question about communication to students and parents.

There was certainly an expectation across the piece that teacher estimates were what young people would be awarded. Could the SQA have done more in communicating to our young people what the process was?

We spent a lot of time on communication. There will always be a debate about whether we should have done more or done something differently. We will reflect on that. Certainly, in the work that we have done since March, there has been quite a lot of engagement with teachers, learners and parents. We have tried to consider our communication channels and the way in which we communicate. We have met young people and brought them into our qualifications committee. We have done quite a lot in that regard. If the perception is that we have not done enough, I have to accept that and reflect on it. However, I think that we tried quite hard to keep the system informed.

During that period, I put out about 20 communications to the whole system, so we had regular communications with the system on the things that we were doing and the guidance that we were providing, and it felt important to do that. There was good material there; it was well received, and we got good feedback on it. In the time that we had, we did as much as we could to communicate the approach.

In the interests of time, I will stop there, convener.

Thank you. We are a little ahead of where I had thought so, once every member has contributed, I will give others a chance to come back in.

I will return to the point about the anonymised data and the anonymisation of the centres, because that is a red herring. Nobody has suggested that the SQA looked at the exam results, identified centres and said, “Oh, that is EH33, which is Tranent, so we will knock those grades down.” What is being suggested is that the SQA signed off a moderation system that was largely based on the centres’ past performance, in the sure and certain knowledge that that would mean that pupils in schools with a poorer past performance would be more heavily impacted. Is that fair?

I—

I think that the answer is yes.

No—my hesitation is because I do not accept that characterisation. It is important that I answer the question honestly and that I have the opportunity to do so.

The moderation process was rightly based on data; it was the evidence that we had. Again, because of the cancellation of exams, the extraordinary circumstances of the year meant that we were awarding on a basis that we would all accept was not ideal; it was certainly unprecedented. However, we sought to ensure a process that would not bind centres entirely to historical performance, albeit that, in the absence of historical attainment and the pattern of attainment, it would have been difficult to do any moderation at all. That process had been undertaken in other parts of the United Kingdom in their awarding decisions this year.

Our equality impact assessment work highlighted that there was a narrowing of the attainment gap between the least and most deprived areas. I accept that the estimates highlighted significant differences between historical attainment and estimates for this year, and increasingly so across the deciles of deprivation. That is a function of the estimates, and the outcome showed a modest narrowing of the attainment gap.

Surely, the outcome showed that young people in schools in more deprived communities were twice as likely to have their grades reduced as those in better off communities. It was the function of a moderation scheme that took account of historical performance in the centre, which we have previously discussed in the committee.

I will move on, because part of that defence—and of the defence that we heard from the Deputy First Minister yesterday—was that moderation takes place every year. In her opening remarks, Ms Robertson said that, if we accept moderation, we have to look at historical attainment by centre. However, those things are not entirely true, because moderation takes place every year on a national level. Grade boundaries in different subjects shift, which is sometimes controversial. Sometimes, the pass mark for maths is 34 per cent, and people say that that is disgraceful. However, compared to this year, the difference is that those grade boundaries are applied consistently to young people who sit those exams, wherever they sat them and wherever they went to school.

I am sure that you will be aware of the paper by Professor Nason from Imperial College London in which he makes an analogy with a driving test. He states:

“If you take a driving test in the UK, it is a national test set to national standards. Your result should not depend on what has been happening in your local town.”

He points out that

“the exams in each of the devolved nations are meant to be national exams”,

so they should be moderated at a national level, not at a local level against historical past performance. Is it not the case that we do not do that every year; we have done it only this year, which is why we have seen what has happened?

You are right that we have not moderated in that way in any previous year, which is because young people have been able to sit exams and have been assessed in their own right in that way. The difference this year was with what moderation sought to do. I think that you would be right in your characterisation if you assumed that every school and college in Scotland had had consistent estimation that perhaps would—

So the problem is that the SQA could not trust the estimations that were made by teachers. You were moderating teachers, not the pupils.

No, I do not accept that. As I highlighted, without teacher estimates, we could not have certificated; we accepted almost three quarters of those estimates. We felt that we had a responsibility to ensure, as far we could, that estimation practice across schools was consistent and therefore, given the point that you have made, that pupils’ results would not be judged by their schools’ estimation process but instead would be judged on the basis of the data that we had available to us, including historical data.

I realise that that is quite a subtle argument, but it is important because, in principle, some schools could have estimated significantly over or under their previous experience. Albeit that there are some changes in attainment year on year, it is a fairly stable system overall. If I had seen a school that had estimated in excess of say 20 or 30 points below or indeed above previous years for a particular subject, there would have been an issue of fairness there, too.

We sought to look at centres before we made national awarding decisions. If we had simply made national awarding decisions, I may have been sitting here today and you would have been suggesting that there was an unfairness to that, because schools had estimated in a different fashion. That was part of the reason why we undertook the moderation that we did.

I do not think that you should make hypothetical assumptions about what complaint we might make. Hypothetically, if I were still a teacher, I would feel that it was my estimates that had been moderated and that that was what the SQA was doing.

I have two more questions. Is that okay, convener?

You may ask one more, and I will bring you back in again at the end, if I can.

I will ask one more and then I will come back in with my other question later.

I am intrigued by the timeline that you have outlined, Ms Robertson, although it is fairly sparse. I have tried to note it down as you have gone along, but I really have only two dates: 19 March, when exams were cancelled and the Deputy First Minister commissioned the SQA to produce a moderation or alternative awards scheme on the basis that you have described; and 30 July, when you provided the Deputy First Minister with a report including the overall results and so on.

I have two questions about that. First, between those dates, when did you sign off the moderation scheme and when did you begin moderating the results? Secondly, are you saying to the committee that, between 19 March and 30 July, the SQA had no contact, direct or indirect, from the Deputy First Minister, his office or special advisers wanting to find out how that process was going, whether everything would be in place and what was happening? Was there any contact at all?

11:15  

The first question was on signing off the moderation scheme. If it would be helpful to follow up in writing on this, I can do so, because there were various stages to that process. When I appeared before the committee on 1 May, I described that we had a situation in which there was a degree of iteration, although it was an appropriate degree. Obviously, we did quite a lot of work before the estimates came in, but we needed to consider the estimates, so there was a process of iteration. However, it is fair to say that we were working on that well into July as we finalised our approach and undertook our national awarding meetings.

In relation to discussions with ministers, obviously I have on-going engagement with ministers and Scottish Government officials in the normal way. However, there were really no detailed discussions beyond a general discussion about how things were going in our work. Mr Swinney joined staff in a discussion in, I think, the middle of June to hear about some of the process issues that we were undertaking. As he said yesterday, I presented him with the outcome of the approach on 30 July.

In relation to Mr Greene’s questions earlier, it is important to highlight that, given the operational independence of the SQA, it was important that there was that separation between the SQA and ministers in this context. Although the Deputy First Minister has an interest in what we do, until yesterday, he has not had a role in awarding decisions or taking any decisions in relation to awarding decisions. It is important that the committee understands that the responsibility for the certification model that was presented last week was the SQA’s.

As I also said to Jamie Greene, the board is appointed by ministers and is accountable to them, and obviously the board were very close to the work that we were doing and had lots of discussions about it. That is effectively how the accountability arrangement works.

It is hard to believe that you had a discussion with the staff group on the process of this most important moderation in the history of exams without any of the issues that we have been presented with today coming up.

We did not discuss the estimates that had been received. At that point, it would not have been—

No, that is not what I mean; I mean the generalities of the moderation scheme.

Gill Stewart might want to say a little bit more about that, but we discussed in broad terms with a number of colleagues just some of the work that we had done—in quite short order, actually—to stand up new processes. We talked through the work that we were doing, but I do not think that it would constitute anything beyond a discussion with staff just to hear more about how we were getting on, which I would consider to be appropriate. We did not get into any areas of detail or intervention that would be inappropriate.

I hope that my questions will lead to short answers, because they are mainly factual. You said that you gave a briefing to the Deputy First Minister on 30 July. Was the full data available to the Deputy First Minister from 30 July, including the information that would have highlighted some of the issues that became so prominent? Did you raise any concerns with the Deputy First Minister about the impact of the results coming out? Did he raise any concerns with you?

On 30 July we shared the data with the Scottish Government, but that happens every year, so—

Do you share all the data, as it would be presented when the results come out?

No. We share the outcomes with Scottish Government officials, but as part of pre-release arrangements. That is not unusual with statistics of this kind. We shared the outcomes of that process, not the detail of it. In effect, the attainment statistics that were published last Tuesday were shared with Scottish Government officials on the Thursday before results day.

In addition, I took the chair of the board, Mr Swinney and a supporting official through the approach that we had taken. That was, in effect, a presentation: I ran through the detail of what we had done and the impact of that. It was a wide-ranging discussion, as you would expect.

So the downgrading of grades for schools in more deprived areas would have been available and clear to the Deputy First Minister on 30 July from the information that was provided to his office.

The focus was very much on the outcomes. Although equality considerations were included as part of the process, it is important for me to highlight that we did not take the Deputy First Minister through the full detail of all the documentation that we published on—

But that would have been available to his office.

No—not all the documentation would have been available on 30 July. As I said, I took him through the headlines of the work that we were doing in some detail, but the full documentation was not provided to the Scottish Government at that time.

Were any concerns raised by the Deputy First Minister in relation to the outcomes, or was there anything that you highlighted to him?

I took him through the detail of that—that was the purpose of that discussion. It was a fulsome conversation, as you would expect in the context.

It is important to know this. Given the U-turn that was made a number of days after the results came out, it would be interesting to know exactly what was known and how much time there was for a decision to be turned round.

I want to move on to ask about the change that has been made. You have talked about the accuracy of teacher estimates, which I think you said was below 50 per cent, or around that mark. You have also spoken about protecting standards and about having responsibility for students, past and future.

The Deputy First Minister and the First Minister have both said that the changes that were eventually made would not have been credible before they actually made them. Are you comfortable with the decision that the Deputy First Minister has taken to go purely with teacher estimates? Were you consulted? If you were, did you raise any concerns? Do you think that there will be issues, or do you think that standards have been maintained, given the decision that has been taken?

In his statement yesterday, the Deputy First Minister set out the reasons for taking the decision that he took. I understand that decision. Ministers have issued me a ministerial direction, which I will fully comply with. Under the Education (Scotland) Act 1996, we require to be consulted on a ministerial direction, and that consultation did take place, both formally and informally, so of course there was a discussion about the decision.

Can you tell us what your response to that consultation was?

There was obviously some discussion about the terms of the ministerial direction, in the normal way. As regards the outcome that we have, and as the Deputy First Minister and the First Minister have expressed, it reflects the extraordinary nature of the year. On that basis, it is very important that young people who receive awards this year feel that they have achieved their awards.

I absolutely accept that, but that was not the position at the beginning of the week. The position then was that it was not credible. The position that you have set out today is that you needed to maintain standards and there would be an issue.

Yes.

There has now been a volte-face on that—a complete turnaround. Are you comfortable, given the decision that has been taken, that standards will be maintained?

I have two points to make in response to that. First, my expression of the work that we have done has been on the basis that we were asked to undertake this task and, as part of that commission, to make sure that standards were maintained, and that is what I sought to do. That is an important part of why we did what we did.

Have standards been maintained?

The Deputy First Minister expressed yesterday that 2020 will be treated as an exceptional year. I think that it will be very difficult—

I get the impression that you are not comfortable in saying that. You are telling me what the Deputy First Minister has said already. As I said, we have had two positions from him. Is it your consistent position that you have concerns about whether standards will be maintained for the future?

I do not think that we will be able to compare this year’s results with previous years’ results or future results in the same way, because the basis on which the awards have been made is different. However, there is an important caveat to that: our focus and attention should—quite rightly—be on the young people who got the awards. We have accepted the teacher judgments—that has been a ministerial decision. I accept that decision, and we will fully comply with it.

I begin by highlighting the human cost of the episode last week. On Friday, I had a call from a mother whose daughter, since the age of five, has—in her words—“worked her guts out.” They live in a deprived area, and the whole family firmly believe that the way to get out of deprivation is through education. Her daughter was expected to go to college and was anticipating confirmation of her teachers’ recommendations on her grades. She did not get the grades that she had expected on Tuesday, and by Friday she had not come out of her room, or eaten, for three days. She was in a total state—she was distraught.

The human impact of this whole episode will be with us for some time to come. I very much welcome the First Minister’s statement yesterday, but what has worried me so far about this morning’s session is that I am not getting the impression that the SQA has learned any lessons, in particular—to follow up on the point that Iain Gray made—in respect of the use of local schools’ records as a factor in deciding individual pupils’ results.

Can I get a guarantee from you that, if we end up having to go through a similar process next year because we are still not in a position to hold exams, you would change the system?

On the human cost, absolutely—there is obviously a very significant human cost arising from the events that we have all experienced over the past few months. I understand that.

We all remember how it feels, as a young person, when the certificate drops through the door, and I am also a parent, so I understand the point that you make very powerfully. However, it is also important to say—as I already have—that joy and disappointment are a feature of results day. There were unique differences this year that served to magnify that sense of disappointment, and I fully appreciate and understand that.

With regard to learning lessons, many parts of the public sector will of course wish to learn lessons from the work that they have had to do over the past few months, and the SQA is no exception—it is very important that I say that to the committee. Schools were closed a matter of weeks before exams were due to be held. We were fully ready for that, and although we had contingency arrangements in place, I hope that all members of the committee accept that it would have been very difficult to fully predict the circumstances that we found ourselves in. I assure you that we have worked very hard to deliver for learners this year.

11:30  

With regard to lessons learned, a formal process is now under way. It has been initiated by the Deputy First Minister, and Professor Mark Priestley at the University of Stirling will undertake work on it. We will co-operate fully and positively with that process and, if there are lessons to be learned, we will learn them.

I have been the chief executive of the SQA for a year. There are always things to do and improvements to make in any organisation, but it is really important that I say to the committee—I say this genuinely and sincerely—that we have worked very hard to deliver on a commission from the Scottish Government this year. We have done our best to do that.

Mark Priestley’s review, which will be concluded within five weeks, will inform the process for next year. We will ensure that we learn from that in what could be a complicated year.

However, we remain on track for exams next year. That is the conclusion of the education recovery group, which is chaired by the Deputy First Minister. We are putting some additional arrangements in place to help teachers and young people maximise learning and teaching time and to ensure that we have contingency arrangements in place for a variety of circumstances in the coming year.

That is good to hear. However, you can understand my concern, because when you met the committee on 1 May, almost every member of the committee warned that to develop a methodology in secret, without proper and fairly wide consultation—on the understanding that such consultation would need to be very quick—would end in tears. We told you that, and the SQA absolutely refused to listen to the committee’s point about the need to consult on the methodology before it was approved.

Iain Gray is right; I think that everybody and their granny knew that, if you used the records of local schools, you would end up in the situation that we ended up in, in which the moderation process led to two and a half times the rate of downgrades in the poorest areas compared with the more affluent ones. That was built into the methodology from day 1. It was entirely predictable.

In today’s issue of The Herald, Larry Flanagan, the leader of the Educational Institute of Scotland, said that the “tarnished” SQA should have heeded warnings and that it needed

“to lose some of its hubris.”

I have to say that I do not always agree with Larry Flanagan, but I have some sympathy with his point.

Do you not think that one of the lessons that you need to learn as an organisation is that, as well as listening to this committee, you need to listen, in particular, to pupils, parents and teachers much more than you did during this exercise?

It is important that we are a listening organisation. I highlighted the reasons why we did not publish the methodology until results day in my correspondence to the committee, and the Deputy First Minister also spoke about that yesterday.

It has been a very challenging year for everyone. The whole notion of moderation was always going to be challenging. There are two parts to the story. The first question is whether the system and the committee accept that moderation should have been done at all, and the second is what the right method of doing that is.

I certainly made it clear way back at the beginning of this process that we needed to reserve the right to moderate to ensure fairness for learners, given the potential issues with estimation that I highlighted to you earlier, and that knowledge of that would inform the approach that we took. It was on that basis that we took that process forward in good faith. The context is quite important here. However, I understand Alex Neil’s point.

We have spent a lot of time over the past few months speaking to teachers—we have teachers working in our organisation—and engaging with young people and others. However, if some parts of the system feel that that was not enough, we need to hear that and learn from it.

I do not think that the issue was about the principle of moderation—it was that the devil is in the detail. As Iain Gray highlighted, if you agreed to rely so heavily on the fairly recent results in individual schools, you were bound to end up in an unfair situation in which the grades of pupils in the poorest areas were downgraded at two and a half times the rate of those in the most affluent areas.

My question is this: will you give an undertaking—I do not want civil service flannel—that, in the future, these things will be done out in the open and will be properly discussed and consulted on? I am talking about not just the principle, but the mechanics—the algorithms—of how these things are going to be done. At the end of the day, the SQA is charged with making the final decision on what methodology it is going to use, but surely that cannot be done in secrecy ever again.

I understand what you are saying but, without meaning to sound flippant, it is important that we do not publish our marking instructions before the exams have been sat. The United Kingdom statistics regulator has highlighted the need for exam regulators and awarding bodies to think carefully about the amount of information that they publish before results day. No other part of the UK has published its full approach in anticipation of results day, although I acknowledge that some material has been made available.

It is fair to say that we set out the parameters of the approach that we took. You are absolutely right that we did not publish the detail of that approach but, as I highlighted to you, we had in place a range of processes to ensure that we got to a position of being able to award on 4 August. There were limits on what we could publish before then.

Of course, as part of Professor Priestley’s review and the work that we are doing to prepare for 2020-21, we will consider those issues. You have my assurance on that.

Ms Robertson, you said today that you treated teachers’ judgment with the “utmost respect” and that you looked at historical attainment by course for each school. A school in my constituency contacted me with a pretty startling statistic. One department showed A passes at 64 per cent in one subject for the past three consecutive years, but the moderated grade was 24 per cent. Passes at D grade increased from an average of 1 per cent to 27 per cent this year—an astonishing 27-fold increase in the lowest grade. That teacher planned to appeal for 76 per cent of his students. I am not a statistician, but something tells me that the system is not working correctly.

You said that you regretted that young people were upset. In a way, this follows on from my colleague Alex Neil’s question. Do you think that the system that you implemented actually worked and achieved fair results for young people? Were you surprised at the fallout from the system? God forbid that we should ever be in this situation again, but if we were, would you use the same methodology?

On the outcomes, we tried very hard to understand and consider the estimates that came in, both from across schools and nationally, and what that meant. As part of the moderation, we worked really hard—I am looking at Gill Stewart, who was leading the technical work around this process—to understand the impact on individual centres and courses. For the most part, that was achieved.

There are some anomalies. Again, I am not seeking to give a technical answer, but the anomalies, in part, reflect the schools’ estimation process, the shape of the estimates across the bandings, which can impact on the kind of example that you have given, the volatility or, indeed, the stability of a centre’s attainment and, in some cases, the issues in relation to the starting point distributions for a subject.

It is fair to say that some anomalies have been thrown up. However, it is also important to say that this was a four-stage process and that we recognised that in the appeals process that we put in place. We recognised that there would be circumstances in which schools and colleges would want to appeal, given the moderation that we had to undertake, which could not be individualised for all the reasons that I have given and because we did not have material in front of us for individual candidates beyond the teacher estimate. The evidence base for that process was really important. That was literally looking at all the work that young people had done during the year, but it is important—

I appreciate what you are—

It is important that I highlight that end-to-end process, because if you look at last Tuesday in isolation and accept the upset that has been caused, that does not fully reflect the process that we designed across the year.

I must say that I am really struggling to understand how a statistic could go from 64 per cent to 24 per cent. No matter what you say, I do not think that you have explained how anomalies such as that could occur.

I have a quick final question. You said that you had taken a child-centred approach. The Children and Young People’s Commissioner Scotland has said that that is not the case and that there was no engagement when it came to the alternative grading process. Will you tell us what engagement you had?

We produced a children’s rights and wellbeing impact assessment. We had engagement, including a discussion with young people, as part of the commissioner’s work.

There was not detailed engagement on the precise model with young people, but we have had engagement with young people, the Scottish Youth Parliament and a number of groups of young people throughout the process, to reflect on the circumstances of this year and the work that we have done. We also brought young people into our qualifications committee, to ensure that we were mindful of that. In our appeals process, we made sure that there was an opportunity for exceptional consideration arrangements to be in place, to tackle any issues around the original estimation by centres. Obviously, that requires learner consent.

The children’s rights and wellbeing impact assessment highlights our approach. I am happy to have further discussions with the commissioner’s office, and I understand that his office has recently written to me and has made some comment on the back of Mr Swinney’s statement.

If there are additional measures that we can take, particularly in the year ahead, to make sure that there is an even greater focus around the engagement of young people on any processes and procedures that we need to put in place, we are happy to do that. I am clear that delivering for learners is what we are about—that is the heart and soul of the organisation.

Gill Stewart might want to come in on a point of detail on that, if that is okay.

Do we have time for that, convener?

It will need to be quick.

I will add one more point. Later this week, we will carry out a rapid consultation on modifications to courses for the coming session. Part of that work will be to run focus groups with young people around the principles of what we are doing, so that we can take into account their views. Obviously, yet again—as is the case with everything that is Covid related—we have a short window of time in which to do that. However, we have taken on board the issue, and we will be engaging with young people through focus groups.

I have a final question on this topic, and members want to cover a couple of other areas before we finish.

The committee and the wider public want to be assured that there has not been an inequalities bias in the SQA’s processes historically or in this instance. One thing that was mentioned in your previous session was that you had an understanding of how good, or not, centres were at estimating results. Given that estimation became crucial to the whole process, what has the SQA done in the past to address with centres the issues around estimation, and has it engaged Education Scotland in that process?

11:45  

Gill Stewart may want to provide some further detail. Estimates are provided each year, and form part of the grade boundary process, in particular in looking at a cohort for a particular subject in any year—those can vary year on year, due to entry patterns and other reasons. Estimates are a part of the process, but they are not fundamental to awarding—would that be fair to say, Gill?

Yes. Estimates have taken on an additional, prominent role this year, and we were very aware of that when we were commissioned to develop an alternative certification model that was based on teachers’ estimates. That is why we put a lot of work into developing additional guidance for teachers and lecturers on estimation. We also developed an online learning academy package for teachers, covering aspects of conscious or unconscious bias, raising their awareness about that, and exemplifying.

We also provided each centre with three years of data on their historical estimates, matched to the results that young people actually achieved through their exams and coursework, and we advised them to use that at a departmental level and to look at their previous historical estimation practice, to see whether they had got it just about right, or whether there was a tendency in their department to underestimate or overestimate. We provided that in order to help teachers, because we knew that estimates were the fundamental part of the alternative certification model.

Fewer than 50 per cent of estimates were accurate in previous years but the moderation process has shown that about 75 per cent were accurate so there has been an enhancement to estimation accuracy through this process, given that teachers and lecturers knew the central role that it played in awarding this year.

Knowing that there was previously a less than 50 per cent accuracy in estimation, are you content that, this year, that became the core and most important part of the system for awarding grades?

In the absence of coursework, which became very evident shortly after lockdown, I think that it was right that estimates were the core part of the approach. We sought to provide additional guidance and material to assist schools with that. I think that the integrity of using teacher estimates for the work that we did this year was right; that approach has been taken in every other part of the UK, in the absence of exam performance data. I think that it was right, and we put quite a lot of effort into seeking to work with teachers to improve that process, because it had a different status this year.

The second part of your question was about Education Scotland. It has done quite a lot of work, in particular in the context of the development of standardised assessments in Scotland, at primary 1, P3, P7 and secondary 3. That work has been an important part of its approach, and we have regular discussions with Education Scotland on some of those issues.

Another important aspect is that we have delivered a lot of events about things such as standards, so that teachers understand the context for our qualifications.

Estimation has not been perfect in the past, but it is very hard for it to be so, so we ensured that we were doing as much as we could to assist teachers in that process. Indeed, teachers, schools and colleges had moderation processes in place when they were looking across their different courses. That was a key feature of the process.

We are still tight for time, but the committee would like to investigate a few other areas.

Ms Robertson, could you confirm whether, by the conclusion of the process—not necessarily by 4 August, when the results were released, but certainly by the point at which you had finalised the results and were able to share them with the Deputy First Minister—you had satisfied the concerns of the Equality and Human Rights Commission? That is a line of questioning that the committee has pursued previously.

Sorry—are you asking whether we had satisfied the Equality and Human Rights Commission?

The committee became aware that the EHRC had become involved in the process and had expressed concerns to you. Were those concerns all resolved?

To my knowledge, yes. There have been a lot of discussions with the EHRC in relation to the equalities impact assessment, although I have not been involved in all those discussions. That process has been taken forward by my colleagues in parallel with the development of this work.

The committee will discuss that later, but it may be an area on which we will ask for a bit more information in writing.

I am happy to do that.

I understand why there were concerns, but it was never entirely clear to the committee what the concerns were and how they were being resolved.

The initial issue was the fact that we had not published an equalities impact assessment. As I said to the committee, we took the view that the EQIA should reflect the end-to-end process. However, we have had some very productive discussions with the EHRC, and I know that it wrote to the committee as part of the evidence for this meeting. I know that the committee has focused very keenly on the EQIA, and I hope that the material that we provided last week demonstrated that we have worked hard to consider the issues and to meet the letter and the spirit of our obligations.

There is further analysis to do. EQIAs are living documents, and we were able to undertake some analysis of 2019 in relation to protected characteristics. That information is held by the Scottish Government and further work will need to be done on that. I would acknowledge that the work is not complete in that sense, but the EHRC will have been aware of that from the discussions that we have been having.

I have a specific question about the combination of what was included in the EQIA, and the methodology report in relation to rankings. The committee has previously raised the concern that there is a danger of unconscious bias in the use of a ranking system. Resources were produced for teachers to tackle that.

Concerns have been raised with me about how the rankings were used in relation to the estimated grades. If I am reading a combination of the EQIA and the methodology report correctly, the rankings were not, in the end, used as absolute individual rankings but essentially as groups by those refined grade bands. The concern that has been raised with me is that those 19 groupings were used almost as an absolute—their infallibility was not really brought into doubt in the process, in the way that teacher estimates were. There were situations where a teacher may have estimated an A for a pupil, but it was decided to award them a C. Where the teacher had put the pupil in the refined ranking did not really seem to be in doubt. The SQA was not adjusting those rankings—you were not moving pupils from group to group.

That is right.

A lot of teachers have spoken to me about that, and one example summarises it best. A teacher with 90 kids in their cohort got in touch and said:

“The grades of my top 10 in the rankings were all left alone. Everyone beneath the top 10 was moved down one grade.”

Their concern was that, despite the huge amount of work that they had put in to issuing estimated grades, their professional judgment in doing so was brought into far more doubt than where they had put pupils in the ranking, which was just assumed to be correct. People were concerned about how those two things married up in the methodology. If a teacher’s estimated grade can be brought into so much doubt that it is moved two—or even, in a very small number of cases, three—grading points, how does that marry up with a ranking system that did not seem to doubt which refined grade group the teacher had put the young person in?

I think that I understand the point that you are making. You are absolutely right to say that we viewed rank order as sacrosanct. For example, if we moved some young people from an A to a B, it would not necessarily be the case that we moved all the Bs to Cs and so on. However, there absolutely was what was described in the report as a “waterfall effect”—although it is important to note that in some cases we were pushing water up the hill as well.

We absolutely treated the rank order as sacrosanct. Gill Stewart will be able to say more about the literature on teacher estimates, which highlights that rank order has quite a lot of integrity. Teachers are able to differentiate in that way, in some cases more easily than they can identify the correct grade. There is some evidence around that.

I suppose that the issue, for us, was this: on what basis would we interrupt the rank order? Doing so would have raised genuinely different issues of substance that would also have been subject to questioning. There absolutely are judgments in this regard, but we felt that the rank order from teachers needed to be held. That did not mean that, if some young people moved from a grade A to a B, everyone moved, but it meant that the rank order was treated as quite an important element.

Gill Stewart might want to say a little about the research evidence, if the committee is interested, because there is an important issue here about how we judge the relative performance of young people.

I understand the point that you are making about bias, but we really sought to ensure that teachers were aware of their responsibilities in that regard. Indeed, we made sure that a number of colleagues were involved in decisions around estimates, so that not just one teacher but a group of teachers was involved and there was some moderation at the local level.

The research evidence at the UK level backs that up. Studies tell us that teachers and lecturers are very good at relative rankings of individual pupils—how one person is doing relative to another—and are perhaps slightly less accurate when they make what we call an absolute judgment, which places an individual candidate on an individual grade or band, at a national scale.

That is what the national data from other UK awarding bodies says; it is also what the SQA’s historical data tells us when we compare estimates with the actual grades achieved in examinations. We relied on that research in our judgment. We made rank order sacrosanct because the evidence tells us that teachers are good at making those relative judgments—that is why we preserved them. Indeed, we had lots of dialogue with colleagues in other parts of the UK and the Office of Qualifications and Examinations Regulation, in particular, about those matters.

Convener, I have a question about appeals, but I am aware that other members want to come in.

Yes, and I wanted to be fair to other members. It is unfortunate, but we must draw things to a close at this point. The committee was keen to talk about appeals and the exam diet in 2021; we will probably look to have a conversation with the SQA about those areas in future.

I thank Ms Robertson and Dr Stewart for attending the meeting. I want to put on record that this has been the most difficult of circumstances; being an SQA employee cannot have been an easy situation for anyone to be in over the past week. We know that the SQA has been working under the same pressures as everyone else during the Covid crisis.

Before we conclude, I thank the committee clerk team leader, Roz Thomson, who moves on to a new committee today. Thank you, Roz, for all your support to the committee.

Meeting closed at 11:59.