Education and Skills Committee

Meeting date: Wednesday, February 20, 2019

Official Report 635KB pdf

---

Scottish National Standardised Assessments Inquiry

Agenda item 3 is the final evidence session of the Scottish national standardised assessments inquiry. I welcome to the committee John Swinney, Deputy First Minister and Cabinet Secretary for Education and Skills; and Andrew Bruce, deputy director, strategy and performance, and David Leng, professional adviser on improvement, both from the Scottish Government. I invite the cabinet secretary to make an opening statement.

Thank you, convener. I welcome the opportunity to contribute to the committee’s inquiry into the Scottish national standardised assessments.

When we published the draft national improvement framework in 2015, the Scottish Government set out how the SNSA results would help to inform teachers’ professional judgments of whether children had achieved curriculum for excellence levels, which remain our key and annually published measure in the national improvement framework. Such judgments take into consideration the full range of assessment evidence that is available to teachers. We have been clear throughout that the SNSAs were being introduced for diagnostic purposes, to provide teachers with objective, nationally consistent information about individual learners’ strengths and areas in which they might benefit from further support.

The national improvement framework in its entirety is the means by which we wish to review performance at every level in Scottish education—national, local authority and school. In response to the widely recognised need to improve the data on Scottish education, the national improvement framework takes an holistic look at the education system and brings together evidence and information from all levels of the system and across all aspects that impact on performance. The idea is that no one aspect takes precedence; rather, different parts of the system interact and connect with one another to drive improvement.

The assessment of children’s progress is one driver. Children’s progress is considered in its widest sense—from their development in the early years right through to their destination on leaving school—and the primacy of health and wellbeing is recognised throughout.

I highlight to committee members the value that SNSAs bring to Scottish education, to classroom practitioners and, most importantly, to children and young people. Assessment has long been recognised in Scottish education as a core aspect of daily learning. The SNSAs were designed specifically for the Scottish curriculum and, as such, should complement, rather than distract from, core learning. The information that the assessments generate is used by teachers expressly to direct to best effect the next steps in learning for individual learners.

We know that the overwhelming majority of local authorities have been using some form of standardised assessments for many years. The SNSAs remove the need for local authorities to buy in those various assessments. At zero cost to local authorities, they provide a set of nationally consistent assessments that are, for the first time, aligned to the curriculum for excellence and linked to the literacy and numeracy benchmarks.

On the comments that have been advanced about the SNSAs telling teachers nothing new, one perspective is that that is a strength, rather than a weakness, as it indicates that the assessments are correctly pitched and that teachers’ judgments of individual learners’ progress are predominantly sound. Equally, comments have been made about the value that SNSAs provide in ensuring moderation of performance across the system, resulting in greater confidence in the profession and vital diagnostic information when aspects of assessment outcomes suggest greater or lesser capacity than a teacher has expected.

It is certainly the case that SNSAs do not cover all aspects of literacy and numeracy in the curriculum, nor have they been designed to do so. However, attainment in literacy and numeracy has been identified as one of the key priorities in the national improvement framework. Literacy and numeracy are the core building blocks for learning across the curriculum and, as such, it is imperative that we identify early on in a child’s learning journey any obstacles to their progress in those areas.

Committee members have expressed great interest in returning to the Scottish survey of literacy and numeracy, either as it was, an enhanced version or run in parallel with the SNSAs. I understand the attraction of that objective; it was, after all, the Government’s own starting point. As is outlined in the Scottish Government’s evidence to the committee, we undertook a review of the SSLN in 2014. That review concluded that scaling up the survey model to produce local authority-level results was not a viable or realistic option—essentially, because that was not the purpose of the survey at its inception and the design of the survey did not lend itself to that kind of upscaling. In short, it cannot address the deficit in our data that the Organisation for Economic Co-operation and Development identified and which the CFE levels data is designed to address.

It remains the strong belief of the Scottish Government that the SNSAs are the right thing to do, that they will bring value to broad general education in Scotland and, fundamentally, that they will bring value to children and young people, enabling all learners to reach their potential. That is not to suggest that the assessments, at this early stage in their implementation, are perfect. The need for continuous improvement is built into our contract with the assessment providers.

The Scottish Government is alive to the need to continue to work with practitioners to seek feedback and to identify means of enhancing the system. The primary 1 practitioners forum, the independent inquiry being undertaken by David Reedy and feedback from schools, teachers and—importantly—children themselves will all help to inform our next steps in that regard. We will strive to improve the operation of and communication around SNSAs, and I look forward to dialogue with the committee as part of that process.

Thank you, cabinet secretary. Iain Gray will ask the first question.

Good morning, cabinet secretary. In taking evidence, the committee has asked most witnesses the same question: what is the most important thing in designing a scheme of assessment? They have all, in one form or another, given the same answer, perhaps most clearly expressed by Professor Merrell of Durham university, who said that the first thing is to be absolutely sure what the purpose of the assessment is.

Do you feel that you have always been clear what the purpose of the SNSAs is?

I think that that has been the case. As I set out in my opening remarks, the purpose of the SNSAs is to provide a diagnostic tool to members of the teaching profession, which will enhance their confidence about the moderation that they are applying in relation to an individual child’s performance and help them to identify any particular performance issues that emerge from the SNSA diagnosis.

The purpose of the SNSAs, in my view—and I think that this has been the Government’s clearly expressed approach—is as a diagnostic tool within the education system.

So these are formative rather than summative assessments.

They are formative, yes.

Some of our witnesses, including Dr Keir Bloomer and those from the Educational Institute of Scotland, were strongly of the view that the purpose of the national standardised assessments has changed—that they began as summative assessments, which would give information about the performance of the education system, but then changed into formative diagnostic tests.

I do not think that that is the case. I read the EIS evidence on that and I see the point that it makes. Essentially, the EIS is saying that the assessments ended up being formative assessments because of EIS intervention. Although I disagree with its starting point—that the SNSAs started off as summative assessments—that reinforces my point that, whatever they are now, the EIS and I are agreed that they are formative assessments.

I do not think that the Government at any stage suggested that the assessments should be summative, but the SNSAs contribute to—and again, I covered this in my opening statement—the formulation by teachers of information on the achievement of CFE levels, which of course is aggregated at a national level, although that is a separate piece of data. The SNSAs inform that assessment, but they are only one aspect of the information that is drawn together to formulate it.

The EIS, I think, makes a fair point in that, in constructing the national improvement framework, it was influential in ensuring that we ended up with a framework that was based fundamentally on teachers’ professional judgment—which is what we have, because we have an aggregation of the achievement-of-levels data that is contributed in June each year by individual teachers around the country.

That puzzles us, though, because this morning I was looking back at an exchange at First Minister’s question time between Liz Smith and the First Minister, and when Ms Smith asked about the removal, or abolition, of the SSLN and how we would have national-level information about the performance of our education system, the First Minister said that the SSLN had been replaced by the SNSAs, which she argued are superior because they are not a sample but a census of all pupils. That completely contradicts what you have just said.

The argument that I am advancing is that that picture is constructed by the levels data—

I am sorry, but the argument that the First Minister was advancing was a different one.

If I go back to what the First Minister said when she launched the programme for government in 2015, she was crystal clear about the purpose of all of this. If Mr Gray gives me a second, I will find the quote.

The First Minister said:

“The assessments will inform, not replace, teacher judgment. They will provide robust and consistent evidence to help teachers judge whether a child is achieving the required level of curriculum for excellence.

The new assessments will introduce greater consistency to curriculum for excellence. They will provide reliable evidence of a child’s performance or progress, but they will not be the sole measurement.”—[Official Report, 1 September 2015; c 18-19.]

That is what the First Minister said on 1 September 2015 when she launched the programme for government, which included, for the first time, the commitment to SNSAs.

I read that quote this morning, but I also read the quote from her from June 2018, where she is very clear that the SNSAs are a replacement for the SSLN. What the committee has found puzzling is how they can be both of those things. We have asked some of the witnesses about that and they have been quite clear that they cannot be both of those things.

I think that what is important is that we vest our analysis in the quote that I have just read out from the First Minister in September 2015, because what she was saying there was that the SNSAs will contribute to the formulation of teacher judgments, those teacher judgments will then be aggregated into the achievement-of-levels data, which will be aggregated at a national level, and we will have a more comprehensive picture of the performance of young people within the education system and of their achievement of the levels than the SSLN, as a limited survey, could ever have given us. That is the squaring of the argument that Mr Gray is wrestling with.

Okay. I am not sure how we are supposed to know which quotes from the First Minister we are supposed to accept as correct and which ones we are supposed to discount, but let us leave that for another time.

Does that not mean that we now have no statistically rigorous information about the performance of the school system as a whole at a national level—information that was previously provided in a statistically rigorous way by the SSLN but that has now gone?

I think that we have a statistically rigorous analysis at national level of the performance of the education system because of the comprehensive data that we now collect on the achievement of CFE levels at the early, first, second and third levels, which is reported nationally and openly on an annual basis. That is a superior and more comprehensive analysis than the SSLN, which would, at most, identify the contributions of two primary 4 and two primary 7 pupils in each primary school in the country and, if my memory serves me right, 12 pupils in each secondary school in the country.

10:15  

Sorry, did you say that we have national level information through summing the SNSA results?

No—it is through presentation of the CFE levels data, which comes from teacher judgment.

That is right. However, our witnesses have said repeatedly that that introduces some questions about moderation and statistical rigour.

The SNSAs provide the statistical consistency and rigour to inform moderation across the country.

However, they cannot be summed nationally.

I am not attempting to sum them nationally; as I said, they are diagnostic. They provide the moderation of performance across the system, which many teaching professionals have said to me has been a challenge with curriculum for excellence. The committee has taken evidence that substantiates that point. Professor Paterson gave evidence to the committee that the SNSAs are statistically strong and robust enough to provide that consistency of moderation across the system, which then underpins the teacher judgment that informs the performance data that we collect and aggregate nationally.

Another aspect of the difference between the SSLN and the SNSAs—and, indeed, the CFE levels—that some witnesses brought up in the course of evidence was that the SSLN was undertaken across state and private sector schools, whereas the SNSAs are done only in state sector schools. Given that the Scottish Government’s main objective in education is to close the attainment gap, could you explain how removing from the data that is collected a cohort that, it is fair to assume, comes largely from the more privileged end of the pupil cohort will help to do that?

The purpose of SNSAs is to help to inform the diagnostic analysis of pupils’ performance. There is increasing evidence that that enables schools to focus better on the individual challenges that young people face in achieving the CFE levels that we expect them to achieve. That puts a rigour into the education system, which is of benefit to individual teachers in assisting pupils to improve their performance. Improving pupils’ performance is our best means of closing the attainment gap.

How will we know whether you are achieving that if a significant cohort at the privileged end has been removed from the data that you collect and we can no longer make that comparison?

The data will be able to demonstrate what levels are being achieved by young people in the Scottish education system. That will demonstrate openly what improvements in capacity and performance they have made, which, in turn, will help to demonstrate whether we are seeing the type of educational performance that we would expect to see among young people in the education system.

It will not do that, because the performance of part of the education system—the independent sector—has been removed from that data.

Essentially, the information that we present demonstrates the extent of the attainment gap within Scottish education. We then demonstrate year on year how we are closing that gap, using comparable data on a whole series of different indicators on which the Government has consulted. I think that we have reached a broad consensus that the measures that we tabulate indicate the framework within which the closure of the attainment gap should be undertaken.

But you are excluding from that data a group that, it is fair to say, we would expect to be at the higher end of the attainment range, because of its privileged position in the private sector. The data from private sector schools is no longer part of the data that is collected that is necessary to make the comparison that you have just described.

No, because the data that we provide looks at the performance of pupils across the education system, where the data on CFE levels will be captured and monitored year on year.

When you say “across the education system”, you mean the public education system.

We will demonstrate what progress is made in that respect.

You are talking about the public part of the education system.

Yes.

That is fine.

Would the process not have been simpler if the Government had maintained the run of data provided by the SSLN, even as it introduced the SNSA, which—as you have explained at some length this morning—is for an entirely different purpose?

As the committee hears frequently, we must be mindful of the burdens on the education system. The SSLN involves an additional burden on the education system. We judged that we had to put in place a system that would enable us to provide the core diagnostic information for teachers. The analysis and guidance that we have looked at from the OECD’s review of Scottish education in 2015 make it crystal clear that we did not have sufficient effective data at our disposal in the Scottish education system to be focused on how we were improving performance. We had to put in place arrangements to enable that to happen.

Our judgment was that that purpose would best be served by introducing standardised assessments and moving to the capturing of CFE-level data that would be valid and available in relation to all pupils in the education system. If we had continued to capture data for the SSLN, we would have been asking the education system to generate ever more information, when we had judged what information would be useful and reliable to meet the tests that were applied by the OECD in its report on Scottish education. That report had identified deficiencies in the information that we gathered and collected on the performance of pupils in the education system below the senior phase.

No witness in any of the evidence that we have heard has ever described the SSLN as a burden on the system or teachers. Indeed, the EIS made it clear that it felt that the SSLN was proportionate and relatively easy to administer.

We had to address the fact that the OECD report indicated to us that there were significant deficiencies in the information that was available about the performance of individuals in the education system. Professor Sue Ellis was pretty emphatic with the committee about the point that the SSLN did not result in any scrutiny at local authority level of the issue of how to improve performance. That is not a criticism of local authorities. It is just that they would not know where to start, because they would not know to what extent schools in their area were contributing to the challenges that might be highlighted by the SSLN.

We realised that we had to do something about that, because the mechanisms to assess performance were not in place. We put in place a performance framework that enabled us to do that, but we did not want to add it on to existing arrangements in the education system, which is why we discontinued the SSLN.

The SSLN provided a degree of accountability for Government on the performance of the education system that had statistical rigour. Have you not thrown that baby out with the bath water?

No, we have enhanced that, because we have put in place a framework that requires the aggregation by teachers around the country, based on professional judgment and supported by the moderation that is offered by SNSAs, of a nationwide picture—constructed individual by individual—of the performance of the education system, which is much more comprehensive than anything that the SSLN ever delivered for us.

None of the witnesses from whom we have taken evidence believed that to be the case at a national level, even though some of them were very supportive of a lot of the other points that you have made about the SNSAs.

My contention is that, through the collection of the CFE-level data, we have available to us a much wider cross-section of information about the performance of the education system. Crucially, the system is underpinned by the ability to deliver enhancements in young people’s performance, because of the data being driven by the judgment of individual teachers on individual pupils. Whatever picture the SSLN presented, it did not provide an analysis or a route by which performance could be improved.

The Scottish Government’s evidence to the committee says that the SNSAs are specifically “Not for accountability purposes”. However, when I ask you about the Government’s accountability in delivering success and improvement in our education system, you say that that is provided by data that is underpinned by the SNSAs. Do you understand why there is ambiguity about the purpose of the new assessments?

There is no ambiguity, because I resolved the ambiguity for Mr Gray quite a while ago when I said that the SNSAs provide information that is statistically reliable, consistent across the system and linked to the curriculum and the benchmarks. All those points were put on the record by Professor Paterson. That information helps to inform teacher judgment in the creation of performance levels, which give us a much more comprehensive—

The tests either contribute to accountability or they do not.

Let me finish my point. The information from the SNSAs gives us a much more comprehensive picture than was ever the case under the SSLN, and the ability to interact to resolve those questions.

I am trying to understand your points about the collection of national data. You made the point about the levels under curriculum for excellence, but the committee received evidence that ACEL is “badged as experimental”, and the report on the statistics stated that

“over a third reported that they were generally confident in the robustness of their ACEL data”.

Therefore, by definition, two thirds were not confident, so how can you be so confident that you are building up that picture?

As with all data, we need to go through a period to get to the point at which statisticians will give us the appropriate standard, and we are on course with that work. It is underpinned by the robustness of the assistance on the moderation of performance that I believe the SNSAs provide us with in contributing—along with other interventions relating to the training and development that is undertaken by Education Scotland and local authorities—to enhancing teacher judgment.

When will the ACEL information be comparable? In which academic year do you expect that to be the case?

I give the caveat that I cannot make that judgment, because it needs to be made by statisticians independently, but my expectation is that the information will probably be comparable in the next academic year.

The 2017-18 report cautions against comparing the data across years, not across year, so you expect the data to be comparable in 2018-19 or 2019-20.

I expect the data to be comparable in 2019-20, but I caveat that by saying that it is not my decision to make.

I entirely accept your caveat, but 2019-20 will be the first year for which we will have a set of figures for the performance of Scotland’s national education system that we will be able to compare with performance in 2020-21 or 2021-22. Am I correct in making that assumption?

That is correct.

So it will be three, four or five years before we can assess what has changed following all the interventions that have been made since national testing was introduced.

10:30  

As Mr Scott will know, we publish a range of information through the national improvement framework—indeed, we did so just in December—that tabulates different indicators of the education system’s performance. We have consulted widely on that, and we believe that it gives us an assessment of the actual attainment gap in Scottish education. We went to considerable trouble to consult on those questions and to ensure that we had a comprehensive picture that would command confidence in the education system, and there is a lot of data in there that enables us to form a picture of the progress that is being made.

I am sure that that is entirely true, but the fact is that, in your answers to Iain Gray and the committee, you have laid considerable stress on the ACEL data. My judgment, therefore, is that you consider that to be the pre-eminent data in your ability as education secretary to assess what is happening in Scottish education.

With respect, I point out that the committee’s inquiry is about standardised assessments and the curriculum for excellence data, which is why much of the conversation has concentrated on that point. If we had started off with my being asked what I would use to judge the closure of the attainment gap, I would have vested my argument in a host of different factors that have already been published, which give a very good and broad set of measurements on how we are working to close the poverty and attainment gaps.

I take the point entirely. I am not going to go back to the discussion about what the First Minister said or did not say, but I point out that the arguments that are being made are all about our—that is, the Government and the Parliament—not knowing enough about what is going on. I am just suggesting that you have—understandably, I think—laid considerable stress on ACEL as the measure that will produce enough data to tell us what is happening.

Yes, as part of the wider assessment of the closure of the attainment gap that the national improvement framework is designed to provide information on.

I have a couple of questions on the testing itself. It is fair to say from the evidence that we have received from a variety of witnesses that, particularly at P1 but presumably at other levels, testing takes place at different times of the year with different levels of support. Some children are taken out of the classroom, while some children take the test while in class, and teachers judge when it is appropriate to do these things. Is it fair to say, therefore, that the approach is not at all standard across Scotland? Indeed, I think that you have frequently made that point.

Obviously, the same assessment is being carried out; it does not change over the year. It changes from year to year—in moving from the first to the second year, we have replenished a third of the questions—but the same assessment is undertaken through the year. It is designed to provide consistency to support the moderation process in the teaching profession, so it represents an opportunity to provide an assessment of individual young people’s performance. In my view, that approach gives the consistency that makes it a standardised assessment.

I genuinely cannot take the point about it being the same assessment. If children are taking it at different times of the year and are getting different levels of support, you cannot possibly consider that to be the same, either statistically or in any other way.

It will be the same, because we are confident, through the norming studies that we have carried out, that the assessment is a true reflection of the level that individual children are expected to achieve. It gives a teacher an insight into how close to or far away from achieving a particular CFE level a child is and the relevant issues that they are facing. Ultimately, that will inform the data that is recorded and which we emphasise with regard to assessing the performance of individual young people in the system.

I accept that it is all about where a child is and how they are doing in reaching a certain level—I get that—but if child A takes the test in November with all the other children in the classroom and child B takes it the following spring outside the classroom with the help of a support teacher, that is not the same, is it? That is what is happening—that is the reality of Scottish education.

That is an argument for relying on teacher judgment about how best to utilise the resource to identify the performance of young people and the diagnostic issues that need to be addressed. In my view, it is best to leave individual teachers to make that judgment, because the purpose of the assessment is not summative; we have been very clear about that. It is not there to identify who has passed or failed, but to provide an analysis at the point during the year at which, in the teacher’s judgment, it is best to assess whether the child is in command of the level that is being assessed.

In some circumstances, a teacher may judge that to do the assessment early in the year will provide them with good diagnostic information about what they will have to do in the remainder of the year to satisfy themselves that the child is achieving the required level. In other circumstances, a teacher may feel that they are better to leave it until later because they have a fair sense of what the individual child requires. They will concentrate on that and then undertake the assessment to see whether those issues have been addressed and what else flows from it. It is ultimately a matter of teacher judgment.

I agree with that, but my argument is that it is not standardised. I do not see how you can possibly say that it is.

It is standardised, because the same assessment is being undertaken.

I feel that I am dancing on the head of a pin, so I will stop.

I have one last question. I agreed with your comments about Renfrewshire Council and closing the attainment gap, which were published on Monday. The BBC report might not have reflected everything that you said, but nowhere in that report is standardised school testing mentioned. Lots of things that I recognise as being extremely sensible are being done there to help young people to develop and grow, but there is no mention of standardised testing in that three-page report, which includes quotes from lots of different people.

What is the link? If we are to be convinced that standardised testing is so important to the future of education, why does it not even get a mention in a report about an area that has obviously made considerable progress in closing the attainment gap?

The data that is collected provides comparability of performance over time on the progress that we are making, which informs the wider performance information in the national improvement framework. The fact that news reports about Renfrewshire do not refer to that is a matter for those who compile news reports—

You did not refer to standardised testing in what you were quoted as saying. I agreed with your comments, but there was no mention of standardised testing.

Fundamentally, we are looking at the focused contribution that reports by individual local authorities and schools make to meeting the expectations of the Scottish attainment challenge, which is about closing the poverty-related attainment gap.

Thank you.

I want to return to Tavish Scott’s argument that the SNSAs are not standardised. What happened prior to the SNSAs? I think that 28 out of the 32 local authorities used some form of assessment. Was any of that standardised? Do we know, for example, how many of them were benchmarked against curriculum for excellence?

Twenty-nine local authorities ran some form of standardised assessments, but they were not standardised across local authorities and they were not made bespoke for the curriculum in Scotland. They were invariably products that local authorities bought in to provide, within their area, some form of moderation on the performance of young people.

Iain Gray mentioned that the EIS said that the SSLN was not a burden. If I am honest, when I was in the classroom it was a pain in the neck to have a group of children removed during a lesson, because I then had to revisit the content that had been covered in the lesson. It was quite disruptive to teaching and learning.

Sue Ellis told the committee that the only people she heard talking about the SSLN

“were politicians and the odd academic.”—[Official Report, Education and Skills Committee, 9 January 2019; c 23.]

Does the SNSA offer an opportunity to track an individual pupil’s progress, which the SSLN could not do?

Yes. The fundamental point about SNSAs is that they are related to the performance of individual young people. They are for diagnostic purposes, to assist teachers in supporting young people to achieve curriculum for excellence levels. That improvement, in itself, helps to improve the performance of the Scottish education system.

Is there a cultural challenge in Scottish education in terms of there being a view that data is for academics and politicians, and not for teachers?

If that cultural problem exists, I think that it is changing. I hear increasingly about greater utilisation of data in the classroom in order to identify particular challenges that children face, and about use of that data to assist in a variety of purposes, not least of which is closure of the poverty-related attainment gap. That that is happening is obvious from the feedback from Her Majesty’s Inspectorate of Education. There is clear evidence that the system is becoming much more focused on utilisation of data for the purpose of improving the performance of individual young people. As a consequence of that improvement, we will see an improvement in the performance of the whole education system.

I will return to evidence that was submitted by the EIS. It made the case that some of the concerns that have been raised about SNSAs have resulted from local authorities not following the guidance that was agreed and issued at the start of the process, and many local authorities have not followed the guidance that has been issued since then. Is that because the local authorities disagree with the guidance, or is there still confusion about the purpose of the SNSAs or the guidance itself?

I say with respect that neither is the case. To be honest, I think that the reason is that we are in the early days. This is the first year that we have had SNSAs, and some established practices that were used under the old standardised assessment arrangements have continued. For example, one of the principal concerns that the EIS has raised is the window of time for undertaking assessments. The guidance makes it clear that no window should be prescribed, and that the matter should be down to the judgment of teachers. However, some local authorities have continued with their previous arrangements, in which a window was used.

As I have made clear in Parliament, we have reinforced the guidance to local authorities, so I expect changes and differences in performance and approach when the assessments are rolled out in the coming years.

The window issue is an interesting example of what I am talking about. You will be aware that, in the course of our enquiry, the committee has written to local authorities. You have probably seen a lot of the responses that we have received. Inverclyde Council had something notable to say about the issue that you raise. It said:

“The authority does not support the idea of ‘testing a pupil when they are ready’ as the tests are not designed to be used in this way. We are also wary of individual teachers choosing to test pupils late in the academic year because this may give the best perceived results.”

Inverclyde Council sets a window of time from January to March. Do you agree that that contributes to a culture of high-stakes testing?

In my view, a window of time from January to March is pretty big. If the window was the first two weeks in March, that would be different. If we are talking about a three-month period, we are getting into the territory of perhaps stretching some of the definitional issues. To be honest, I do not think that Inverclyde Council’s position on the three-month window is particularly unreasonable. However, teachers should be making judgments about when to undertake the assessment within that period.

10:45  

Part of the issue that has been raised—I am sure that teachers will have mentioned it to you and to other members—comes back to the fundamental tension between formative and summative assessments. Teachers are concerned about how standardised the information is and how comparable the data might be.

The example that Tavish Scott gave is highly relevant. Some pupils who take the SNSA at the start of primary 1 will be four and a half years old, and others who take it at the end of primary 1 will be six years old. There is obviously a significant difference between the development of a four-and-a-half-year-old and that of a six-year-old. If the purpose of the SNSAs was entirely to inform a teacher’s judgment, and to allow them to glean information about individual pupils, that difference would not be an issue. However, it is clear that there is confusion, at least about the use of the data from the SNSAs at any level beyond that of the individual pupil—at classroom, school, local authority or national level. Do you acknowledge the concern about the validity of data that is aggregated at any level beyond that of the individual pupil, when there are such inconsistencies?

I accept that there is a need to get the messages clearly understood. That is why I said in my opening remarks that, as we are one year into the process, there might well be lessons that we need to learn in terms of strengthening communication about the points that Ross Grier raises.

I hope to explain to the committee where the SNSAs fit into the wider assessment of performance of the Scottish education system. Fundamentally, the SNSAs are to assist diagnostic assessment by individual teachers, which will help to inform practice and performance in relation to individual children. Ultimately, that will contribute to it being defined whether a child has reached a particular CFE level. That information will be aggregated.

I appreciate the sensitivities of all those matters. I have absolutely no interest in aggregation of data for the kinds of purposes that bring many such systems into disrepute. That is why I am heartened by judgments that have been made about what we are doing by people including Andy Hargreaves and Allison Skerrett, who say that we are trying to do something different. Fundamentally, we are trying to provide, in the classroom, an approach that gives teachers greater advice and support on moderation of the performance of young people in curriculum for excellence, so that they can diagnose pupils’ performance and support them in overcoming challenges that arise. That is the means by which we are trying to drive improved performance in the education system. We will not drive improvements in the performance of the education system if we do not drive improvements in the performance of individuals.

I get that that is the case at the level of the individual, but you have said that there is a step in the process at which data is aggregated. The question that has been asked is how comparable the data is. How valid is the aggregated data when there are inconsistencies such as the possible 18-month developmental gap among children who take the SNSA in primary 1?

At the various stages in our curriculum—early level, first level, second level and third level—we make judgments about whether young people have reached the level. I appreciate that young people might be at different ages at those points in that process. That is just a fact of life. We must make a judgment about whether the needs of young people are being met. That information will be publicly reported as part of the annual reporting.

I am sorry, but I am still confused about the validity of the aggregate data. There could be an issue with the age gap or there could be other issues—for example, the inconsistencies that Tavish Scott mentioned about the circumstances in which the test is taken.

I absolutely accept what you say about use of the data that is gleaned on an individual pupil by the individual teacher in helping them to adjust the pupil’s learning experience. However, at any level of aggregation, the data is not entirely standardised, so there will be issues about its validity. You have explained that the data is aggregated, but you have not explained how such issues are resolved.

Those issues are resolved in the reporting because, essentially, reporting crude SNSA data at national level will provide a profile of how performance is being delivered across a variety of indicators and assessment elements. Ultimately, however, the assessment that really matters to us is the level that young people have reached within curriculum for excellence, which is informed not just by SNSAs, but by a variety of other mechanisms.

Absolutely—but SNSAs are one of the data points. I am questioning the validity of that data once it is aggregated.

The reporting demonstrates the range of performance against the elements of assessment within the SNSA, but fundamentally it flows into the aggregation of data on the levels within curriculum for excellence. The assessment that matters to us is the assessment whether young people are in command of the details of the curriculum.

If the data is being aggregated, how is it not summative?

That part of it is ultimately a summative—

So, are the SNSAs both formative and summative?

In the aggregation of data, the SNSA is summative, but its purpose is to be a formative assessment.

I think that that is a fundamental tension. I see how much time I have taken up: I am sure that other members will want to follow up on the issue.

The character of the SNSA is that it is a formative assessment, because it leads to a diagnostic assessment of—

It is a formative assessment that leads to summative assessment data.

That is only because of the ultimate aggregation of the data.

Exactly.

I want to pick up Ross Greer’s point about the tests being high-stakes tests. The Government has consistently said that the tests are not high-stakes tests, and Professor Hargreaves agreed with that in his evidence. Do you think that the narrative that has played out around the tests has skewed their purpose and meaning?

There is an active debate, which we have looked at very carefully, about the nature of standardised assessment. We have learned from the Organisation for Economic Co-operation and Development’s 2011 report, “Student Standardised Testing: Current Practices in OECD Countries and a Literature Review”, which carefully maps out the benefits and dangers of standardised assessment and identifies themes and key lessons for countries to be mindful of when they go down that route. It does not say that standardised assessments are not appropriate, but that there is a danger that they can create the impression of a high-stakes approach.

Our view is founded on the OECD’s 2015 analysis, which states:

“Standardised assessment tools can be used formatively in all parts of the system if they are referenced to the curriculum”,

as, I contend, ours is,

“flexible in their use”,

as ours is,

“and provide high quality just-in-time information for teaching and learning”—

which I contend is the case—

“while at the same time having efficient ways to aggregate the results through the system.”

That was the OECD’s advice to us in 2015, and I contend that we have followed it in every characteristic. The advice derives from an OECD report that says that there is merit in standardised assessment, but the details need to be implemented carefully. We have taken great care to ensure that that is the case.

You have rightly said at previous committee meetings that you have ultimate responsibility for what goes on in schools across Scotland. I think that every parent sees you as the person who has responsibility for raising attainment. I think that we accept what you say about the prime focus being on formative and diagnostic assessment, but you have just admitted that there is a summative purpose as well.

What evidence of underperformance by schools and local authorities will the Scottish Government look for in the SNSAs? What evidence will you look for to support comments that a school or local authority has to improve?

I want to create a constantly improving education system. That is my mantra, and everything that I do tries to support that objective. I welcome the focus of Her Majesty’s chief inspector of education on ensuring that inspection is about driving improvement in our education system. The work that we are undertaking through the regional improvement collaboratives is about improving professional learning and standards, and improving pedagogy. The work that is being taken forward through the attainment challenge is focused on creating an improving system.

I want to ensure that we have a relentlessly improving system. In previous appearances before the committee, I have said that I have a “relentless focus on improvement”, and that is exactly what drives the agenda that we are pursuing.

You commented that the SNSAs will provide an enhanced assessment of the youngsters in our schools, and you have made it clear that, because of that enhancement, you expect that there will be a better focus on the levels of curriculum for excellence and, we hope, better attainment at those levels. In that context, how will the Scottish Government use the information and what is your expectation of how local authorities will use it?

In answering that, it is important that I go right back to the individual children. We will not have an improving education system in Scotland if the education and performance of children is not improving. The improvement has to be felt in individual classrooms. That is why, in my answer to Jenny Gilruth a moment ago, I talked about greater utilisation of data in the education system to identify where improvement needs to be made, child by child. Improvement in the system is not an amorphous thing that floats about in the ether; it is driven by the performance of individual children. That is why we have opted to ask for teachers’ judgment about satisfactory achievement at the CFE levels. To paraphrase the OECD, it said to us that we were pretty blind on individual performance across the system until the senior phase, and that is the gap that we have remedied. We have done so by reliance on teacher judgment about whether CFE levels have been achieved, and one part of the information base that informs that is from the SNSAs.

Fundamentally, it is not just about me or local authorities being focused on improvement; it is about individual class teachers having data available to help them to improve the performance of young people’s education. Obviously, my job is to ensure that the education system is well supported and well resourced to enable that work to be undertaken at a classroom level.

To pursue that point about your role, which you have rightly identified, if, as a result of the inspection reports that you receive, you felt that one or two local authorities were not performing as well as you would hope and there were clearly issues, would you look for information on the curriculum for excellence levels in that local authority to try to support that local authority to improve, or would it be other aspects of the SNSAs?

I would look at a variety of information.

Could you spell that out?

11:00  

For example, I have looked very carefully at the specific performance of one local authority—HMIE has published on this—that was causing concern, which was Argyll and Bute Council. There was a general concern about the performance of the local authority, which predated the availability of the ACEL data. The inspectors looked at the local authority’s performance and published requirements for its improvement. To the credit of Argyll and Bute Council, it has responded positively to that call. Obviously, I have statutory powers that enable me to look at local authorities’ performance, if required. I would look at a range of data and information and, undoubtedly, ACEL data would be part of that judgment.

I will put that in a wider context. In trying to create a constantly improving education system, I want to see local authorities buying into and participating in that journey of improvement, and leading that journey, supported by the Scottish Government. It is about the willingness of local authorities to work with us on the standardised assessments, for example. There has been tremendous local authority collaboration in introducing standardised assessments. Why? Some of the reasons that were discussed in my exchange with Jenny Gilruth are relevant. Some local authorities might have been undertaking some form of assessment, but it was not related to curriculum for excellence and it could not give them any read-across on whether what local authority A was doing was at a standard that was sufficiently comparable with what was going on in local authority B or across the whole country. Of course, standardised assessments now help to inform that judgment by local authorities.

My final question goes back to the point that you raised in your introductory remarks to the committee, which is about what the OECD said. Despite the many strengths of Scottish education, the OECD highlighted the issue about the richness of data and made some suggestions about what we should do to enhance that richness. You said that that is one of the reasons for introducing the new standardised assessments. At the same time, however, the Scottish Government took Scotland out of other measurements, principally the progress in international reading literacy study measurements and the trends in international mathematics and science study measurements. In the light of what the OECD said when we did not have enough evidence and data, can you explain what the Scottish Government’s thinking was in taking Scotland out of those other measurements?

If I remember correctly, some of those decisions are quite historical. However, to answer your question, a quote from page 155 of the OECD 2015 report is relevant:

“The light sampling of literacy and numeracy at the national level has not provided sufficient evidence for other stakeholders to use in their own evaluative activities or for national agencies to identify with confidence the areas of strength in the years of the Broad General Education across the four capacities of CfE. Nor has it allowed identification of those aspects or localities where intervention might be needed.”

That is essentially a commentary on what SSLN delivered for us, and that would be my assessment of some of the international work as well. Those measurements do not give us an insight into what needs to be done to improve the system, which is what the framework that we have put in place enables us to do.

But you would accept that some of the experts who have been in front of this committee, such as Professor Hargreaves and Lindsay Paterson, have made it clear that those international measurements are actually important.

We subscribe to some international assessments, as Liz Smith will know. However, our judgment was that we had to focus on the international judgments that were relevant and appropriate for us in balance with the other changes that we had to make to enhance the data in our own system.

You have talked a lot about formative assessment and the importance of standardised assessment. Most people would agree that Professor Dylan Wiliam is an expert on formative assessment and he says that the tests

“will have little, if any, formative use.”

Do you have a view on why he might say that?

I cannot speak for Professor Wiliam.

Do you accept that he is an expert in this field?

I acknowledge his support for formative assessments.

He disagrees with what you are saying about the purpose of these assessments.

Lots of people disagree with lots of people on education.

So we get to pick our experts, do we?

If I can—

Are you saying that you do not agree with him?

I think it is a good system. I am not sure about the extent to which Professor Wiliam has studied the particular system that we have introduced. I do not know about that. However, I refer Johann Lamont to the expert opinion that informed our judgments. It can be found on page 157 of the OECD report:

“Standardised assessment tools can be used formatively in all parts of the system if they are referenced to the curriculum, flexible in their use, and provide high quality just-in-time information for teaching and learning, while at the same time having efficient ways to aggregate the results through the system.”

That is the expert opinion from the OECD and I contend that we have followed it to the letter in making the judgments that we have implemented.

I suppose that we are now probing whether what you have put in place meets that purpose. I go back to the question of standardised testing and the fact that young people can do the tests under different conditions at different times. A child could be four and a half when they start primary one and another child could be six by the time they finish primary 1. The child could be in a class of two or three young people being supported individually to take the test or they could be sitting in a bank of 30 children with a laptop or iPad in front of them. In what way is that standardised?

If there is a four-and-a-half-year-old doing the test on their own, without any support, and a six-year-old who is fully supported—your officials told me that the young people are allowed to practise beforehand—can you tell me what is standardised about the results of those tests?

They are undertaking the same assessment and they have been supported appropriately within the context of their learning, which is one of the important parts of the core—

Could you explain what that means?

I mean that assessment is an integral part of learning within the education system; anyone would acknowledge that it is a key part or characteristic of the system. Therefore, the test will be undertaken under recognisable classroom conditions, with the support that any young person would ordinarily get for their education. The results provide consistent reporting and diagnostic information on individual young people.

Can you explain how the classroom conditions are consistent when there is a child of four and a half doing this test in a group of 30 children and a child of six doing it with one individual supporting them? How are those the same classroom conditions? Theoretically, it is the same test. However, if, for example, I do a driving test in a car that does not work, on my own, with no support whatsoever and no lessons and another person does the same test and has been supported all the way through, I would say that those were very different circumstances.

I venture to suggest that the example that Johann Lamont has just presented is ridiculous in the context of the performance of a classroom teacher.

Fair point.

A classroom teacher would never, ever conduct an assessment in that fashion.

Well, you explain to me how a four-and-a-half-year-old in a class of 30 doing that test, without practice, can possibly be the same as a child of six doing it with one person supporting them, having had a lot of practice?

Fundamentally, the same assessment is being undertaken by every child, in proper classroom conditions, with the appropriate support—not the support in the example that Johann Lamont suggested—that a child would have in undertaking their learning, so that it would not be something that seemed different or unusual. If the committee members read the practitioner feedback, they will see that the practitioners wanted to deploy the assessments as part of the learning process of children in the classroom, so that one day a child might work with a piece of technology, with support or independently, and on another day, they might be invited to do the assessment using the self-same piece of technology, with—I would imagine—a degree of supervision, because it would be unusual for a five-year-old to undertake such a process without appropriate supervision. Obviously, when you get to P4, P7 or, indeed, S3—which the inquiry is looking at, too—the circumstances are very different, because those young people have, as they should, greater capacity to do these things independently.

You would reflect, however, on the fact that even those who came before the committee to advocate for standardised testing said that such complete flexibility made getting standardised information a bit of a challenge.

I will go back to the example that I gave with regard to the benefit of the assessment as a diagnostic test. I think that we know the challenges in that respect. In one question that we were shown, a child had to identify the word that rhymed with another word. There was a little button that, when you pressed it, allowed you to hear the word. At the presentation that you put on for us—and which we were very grateful for—I asked whether, in the reporting, any distinction was made between the child who could identify the word without hearing it and the child who could not, and I was told no. However, when I raised the issue with experts who appeared before the committee, they said that that was problematic, because that did not give information that was of any benefit. How can the test possibly be valid when, not even in a theoretical, diagnostic and formative way, you are not making a distinction between a child who can read without hearing the word and a child who needs to hear the word to understand whether it rhymes?

I will invite David Leng, a former director of education in Stirling and Clackmannanshire and the professional adviser to this programme, to respond, because the question is at a level of detail that it would be appropriate for him to give a view on.

I recognise the question—I was part of the demonstration of that question to the committee. If I remember correctly, I think that it was about a word that rhymed with “pie”.

The point is that the standardised assessment is an adaptive one, which means that it questions progress through a range of skills—in this case, the literacy skills of primary 1 pupils. That particular question assessed phonological awareness—basically, whether the child could identify one sound with another. Whether the child was able to read the word or had to listen to it was not key. What was key was their ability to make the phonological identification, therefore children were given a choice in what was a supported process. If they were able to read the word, that was fine, and if they wished to listen to it, that was fine, too. That was not the significant part of the assessment.

The issue that you have raised—that is, the decoding of a question, which is the ability to read a question and understand what it means—comes later in the assessment. The teacher gets feedback on the particular skill that the question seeks to assess, so a range of things are taken into account. As many of you will remember, there were some questions later in the assessment in which children were asked to read without any audio support; some found that quite challenging, while others found it straightforward, and that enabled us to identify the children who were able to decode. There are, therefore, a range of questions that children go through and in which they are asked to display certain key skills, and the assessment builds on that and adapts to their ability level.

You have taken one question and asked whether the child needed to know the audio, but that is not the point. The question is whether the whole assessment tells the teacher how well—or not—the child is able to deal with an audio or visual stimulus, and that is what they get. If you have seen the report—and I know that you have seen examples of it—you will see that that is made very clear in the outcomes. It is not about the specific support to answer the question, but about the skill that the child is being assessed on, the level of difficulty of the question and whether they were able to answer it.

Have you factored into the assessment the extent to which a child is comfortable with navigating themselves around the system? I think that we were told at the presentation that there were many opportunities for children to practise, but does the test reveal the extent to which support was available to ensure that the child understood how to get around the system? A teacher in an individual school will know how much support was given, but if we are talking about standardised assessments and if we are to ensure that the comparison is a reasonable one, surely we will need to know whether a lot of support was given or a lot of practice happened beforehand—or does that not distort the results in any way?

We go back to the purposes of the assessment. We have said all along that they are diagnostic and that their primary purpose is at a school level. We have tried to build in flexibility and support at that school level, so the teacher makes the decision about the level of support that they are going to provide. Therefore, the rich information is at the individual level.

11:15  

I have heard a lot of discussion this morning about aggregation and at what point it is made. We have been very clear about supporting the child and ensuring that the child gets through the assessments as easily and as comfortably as they can, and is given the support that they require, because we are assessing their ability on elements of literacy or numeracy, not on the manipulation of the assessment tool. When they look at the reports, the teacher and the school understand the level of support that was provided. Therefore, any aggregation that they may make is based on an understanding of the support that the child was given. The standardisation is therefore about the assessment—the questions, the matching to the curriculum, the common reporting and the common scaling that they are on. It is not about having exactly the same conditions or timing—that is a different understanding of standardisation.

It certainly is. I get the point that it makes sense at a local level. It is a question of whether the assessments can do the two things at once. That is one of the challenges and I have not heard anything convincing here that says that there is a rigour round both. You say that the teacher knows exactly what support was provided and therefore they can contextualise it, but I am not sure how that can possibly feed through to Government level. That has come out in the evidence.

My final questions are on the SSLN. You said that it did not give sufficient information and I think that people could accept that, but is that a ground for getting rid of it? You also said that the Government got rid of the SSLN because it was deemed to be a burden but, at the same time, there are significant reports about the burden that is created by standardised testing, because senior management staff are being taken from their work in order to support the classroom and learning support staff are being taken out in order to ensure that the tests are done. There is a lot of anecdotal evidence that there is a consequence to running the tests, which has created a burden.

On the question whether the SSLN provided sufficient information, can you accept that that is not, in itself, a ground for getting rid of it? If the reason was because of the burden that was created, have you been able to quantify what the burden was? It sounds to me that it would be the equivalent of a group of kids being taken out of the class for an hour, which causes the same amount of annoyance and inconvenience that teachers face all the time when there is a school show or whatever. Have you been able to quantify the burden that led you to decide to drop the SSLN?

On the first point, the judgment in relation to the SSLN was that we were essentially moving to a more comprehensive collection of data about performance in the education system—

Sorry, I do not want to interrupt, but we have heard from Tavish Scott that you will not be able to look at the ACEL until 2019-20. Even if the SSLN is not sufficient—it is not perfect—it at least provides information in the meantime. Did you at any point consider a transition period in which you would continue the SSLN until the other process was in place, or was the burden of the SSLN so overwhelming that you decided that it was necessary to get rid of it? If so, how did you quantify that burden?

We considered the potential to expand the SSLN to provide us with some of the information that would address the gaps that were clearly identified by the OECD. We gave very close consideration to expanding the SSLN—indeed, that was the initial option that the Government looked at, because it has a degree of longevity about it—to try to ensure that we got a more comprehensive picture. However, in order to do that, our judgment was that we would be creating an approach in the education system that would have been even more significant than the measures that we have opted for. One of the reasons why we opted for those measures is that many local authorities—29 out of 32—were operating some form of standardised assessments.

However, as I explained, those local authorities carried with them a frustration that they were operating the assessments individually, that they were not related to the curriculum and that they did not provide moderation of standards across a wider canvas. The local authorities—to the best of their abilities—might have been making an assessment in relation to the curriculum, but they were doing so without a bespoke assessment mechanism and they had no basis of moderation across local authority boundaries. We were, in essence, replacing all that. However, I accept that, in a limited number of cases, some local authorities—for good reason—are continuing with their previous systems, because of the line of sight that they are interested in establishing.

The final point is that, ultimately, we are trying to fulfil the objective of enhancing the available information to ensure that we support young people in the improvement of their performance. That is why we have taken those decisions.

What was the burden of taking a sample group of young people out, across the system, for one SSLN test?

I constantly wrestle with the issues of workload.

You quantified the burden as what?

I did not quantify it.

So, was it not really about the burden?

Of course it is about the burden. I did not quantify it by doing a sum. I made a judgment that the SSLN was not necessary, because we were opting for a more comprehensive reporting of the performance of young people that would address the key challenges that were set out in the OECD report. I am mindful at all times—I am under constant pressure about this—to remove things from schools when we ask them to do different things. On balance, I felt that it was the right thing to discontinue the SSLN.

There was no serious suggestion that the SSLN was creating a burden, but its removal created a gap in the information that is currently available while the new system goes into place.

Johann Lamont is being cavalier with the question about burdens in schools.

No, not at all.

Well, I am constantly—

Forgive me. One of the serious questions around this is the displacement of people from supporting individual young people into running a series of tests. It is legitimate for me to ask you what burden the SSLN created that was so great. There is now a gap in the information and your own experts have said that the SSLN could continue in an amended form. Of all the issues that I am concerned about, burden is one of them. You must always balance burden against purpose. What was the burden of the SSLN in comparison with the level of burden that the standardised testing has created for many folk who are in the education system?

Johann Lamont said that we must balance burden with purpose. I made that judgment between burden and purpose about the SSLN. With the pressure that I am constantly under to remove burdens from schools, I judged that it was right to remove the SSLN, because we were moving to a more comprehensive performance measurement system. I am constantly focused on how I can remove burdens from individual schools, and I cannot afford to be cavalier about that.

No one is cavalier about it. With respect, of all the arguments that you have put forward today, that is the least credible—that the reason why we got rid of the SSLN is because of burdens. Everybody on this committee is conscious of the level of burden on teaching and support staff in our schools.

Cabinet secretary, you mentioned today that the new assessments align with the curriculum, and the Government’s submission to the committee says:

“Alignment to the Scottish curriculum is also key for Scottish teachers and sets the SNSA apart from other standardised assessments previously used by schools and local authorities.”

Could you say a bit more about the process of deciding how to make that alignment? Are you satisfied that the assessments are aligned?

In the implementation of curriculum for excellence, we went through a number of steps to ensure that the system was well informed about what the different staging posts and levels were envisaged to be and the progress that children and young people would make through the curriculum. That discussion has been informed over time by the production of experiences and outcomes, which provide a shape and a framework of the curriculum. That has developed and it remains a major part of the judgments that teachers make about the content of the curriculum.

When I came to office, I detected concern in the teaching profession that the profession did not have sufficient clarity about what individual levels would be represented by, so I asked Education Scotland to design benchmarks that would give clarity about what individual levels look like, as opposed to prescribing the content of those levels.

The feedback that I get from the profession is that benchmarks have now made a significant impact in stabilising that understanding of what the levels are and, as a consequence, the profession is more confident about what the levels look like. Against that background, the SNSAs then had to be set within the context of those benchmarks, and the material and the contents of them judged by our education specialists to be appropriate. It is the blending of that education advice with the content of the curriculum for excellence benchmarks that has given us the approach that we are taking but, obviously, that is kept under review with the replenishment of the questions that are undertaken in each of the assessments.

You said something about the levels and the structure of curriculum for excellence but, obviously, it is about more than that, as you well know. It is a philosophy and a pedagogy. How do you respond to the comments that have been made across the spectrum of views about how the assessments fit in with the philosophy behind play-based learning in primary 1?

It is important that we highlight the title of what Dr Allan refers to: play-based learning. The play-based approach in the early level is designed to ensure that young people have command of the learning that they would be expected to acquire as part of the early level. The play-based approach is the medium through which they undertake that learning, so the assessments have to operate in a fashion that is consistent with that approach. One of the questions that I have asked David Reedy to tackle and challenge in his independent assessment of the P1 assessments is whether we have got that approach correct. Fundamentally, however, the assessments are there to consider whether young people have command of the learning that they are expected to undertake as part of the early level, acquired through that play-based approach.

In putting those things together—the play-based learning and the approach to the assessment—how do you ensure that the breadth of what is being tested in primary 1 is relevant in terms of producing information?

We have to be clear that the standardised assessments capture or address only a part of the literacy and numeracy elements of the curriculum, and of course the curriculum is much broader than that. They consider and assist in the diagnosis of challenges that young people will face in that respect but, ultimately, teachers have to make a more comprehensive, holistic judgment about whether young people have command of the CFE levels, which will be done through individual teachers applying judgment in the reporting that is undertaken on CFE levels.

Thank you.

I have a couple of questions about teacher training. Professor Hargreaves told us about developing a collaborative culture in teaching, and we have spoken today about moderation and having a better understanding of shared standards. He also said that the profession needs to have a role in informing the continuing development of the SNSAs. Is that the Government’s intention?

Yes. That fits into the wider work that we are undertaking to create the more collaborative climate in Scottish education that is crucial to how we advance our education agenda. The regional improvement collaboratives are gaining more significant momentum and, as a consequence, are beginning to influence and enhance classroom practice.

11:30  

We engage the profession on our learning from the assessments in a number of ways. In a very direct way, the P1 practitioner forum that I established is doing good work in engaging with practitioners on the experience of undertaking assessments in P1 and helping us to identify the challenges that have to be addressed. We are capturing the opinions and views of members of the teaching profession on how they are implementing the assessments, more generally, and extensively surveying teachers to gauge their experience. We will reflect on that as we develop the assessments further.

On the teacher training front, SCHOLAR has developed a programme with Heriot-Watt University that has a 95 per cent satisfactory rating, but that is an online training programme. There is also the General Teaching Council for Scotland training package. Are you looking to develop a more consistent approach to training teachers in the use of SNSAs at a national level, perhaps through the collaboratives?

I envisage that that will happen within regional improvement collaboratives, some of which have done work on that. The west partnership has undertaken work on moderation, in particular, which perhaps will always be a challenge in the system, particularly given the nature of the curriculum for excellence. I certainly want to see steps taken to ensure that those programmes have an effect in the classroom that allows the profession to feel strengthened by their access to that type of training, and that it is considered as part of the professional learning of the teaching profession.

I want to return to the points that David Leng made in response to Johann Lamont’s specific question on the button that a pupil can press in order to hear a word. I am 29 now. I have dyslexia and dyspraxia and I know that I have a problem with reading and that I am prone to making stupid mistakes. If you had asked me about that at four or five years old, I would not have known, and I would have been quite confident in my own reading ability and my ability to move along a line and read a simple short question. Do you recognise that the result that the test shows would vary depending on whether or not I knew to press the button to listen, check and confirm that my reading was correct?

I think that we have tried to stress all along that, assuming that your teacher knew you and your level of ability in the classroom well, the assessment—

They did not.

I apologise for saying that.

I got a formal diagnosis only when I was further on in school. I would not have known that I would have found that test difficult, and nor would the teacher at that point.

I want to say categorically that the assessments are not about assessing whether children have additional support needs. That is not their purpose or the extent of the precision of the tool. However, they will always indicate the particular developmental needs that you may or may not have to the teacher. That is the diagnostic process. That will then be added to the teacher’s knowledge of you and of all the other work that you are doing. That is why we have stressed all along that the assessment is indicative of the ability that a child has at a particular moment in time. It indicates areas of further exploration to the teacher. It does not definitively say that a child can or cannot do something.

You accept that the variability and adaptive nature of the test comes with the risk that certain specific learning difficulties or particular learning styles or patterns could be disguised by adaptive features in the test. There is a risk.

The assessments try to establish the developmental capabilities and challenges that an individual may have in their education.

I want to pull you up on “developmental”, as I find that it is a confusing term to use. When people talk about a developmental test, they are not looking at whether someone is achieving certain points in the curriculum. In the context of this conversation, a different type of developmental test would be needed to assess those things, as is used in other countries and as we have heard about from witnesses.

Forgive me if I have used the wrong terminology. Essentially, the assessments try to find out the areas for further development that will arise out of the diagnostic assessments that are made. In my other remarks, I talked about areas where a teacher may identify greater or lesser capacity within a young person, given the evidence that is thrown up by the undertaking of the assessment.

I accept that. My question is whether the adaptive nature of the test in some cases comes with a risk that, because of the design of the test and questions, certain difficulties that a child is facing in their learning might be disguised or might not be as obvious as they would be if the test or questions were done in a different way.

Where I was trying to get to in my answer was to say—and I would rather not consider this in a specific fashion—

That is why I asked the question of your officials rather than you.

The point that I am trying to make is that the diagnostic assessment that arises out of the assessment may give rise to a number of points of inquiry by education professionals that may beg questions as to why a young person is not able to handle elements of the assessment as well as he or she might have been expected to. It will not provide a diagnosis of a particular condition, but it may give rise to a series of questions about how to configure the learning of the young person to ensure that they can achieve their full potential. That may open up a whole series of other questions.

I understand that argument. By the same logic, is it not entirely possible that, through the design of the test, there could be false positives that would close off lines of inquiry that might otherwise have been picked up through the teacher’s individual judgment? The adaptive and time-related nature of the test and the different possibilities for how the questions are completed could present a false positive on how well the pupil performs. That might be particularly so for bright pupils with a specific learning difficulty. The teacher might be inclined to think that they were middling students and doing okay, closing off lines of inquiry that, if they had had a diagnostic test, as was suggested by some witnesses, particularly at the early stage, might have picked up the needs of the child better.

Fundamentally, that relates to the ultimate question that is being answered, which is what the teacher’s professional judgment is about the child. The SNSA informs the process but does not dictate it. It is determined by the professional judgment of the teacher. In that context, a teacher would have to take into account a range of different information and experience about the educational performance and contribution of a child. Given the focus that we have in our system on getting it right for every child, that could beg several questions about the needs of the child and how they could best be met. That is fundamentally the judgment that we look to educators to make in our system.

If the test is poorly designed and features in the test make it more difficult for teachers to exercise judgment, is that not a problem?

Obviously, nobody wants that to be the case. I do not think that it is the case for the assessments that we have undertaken. I am not going to say that we are impervious to the idea. We would want to make sure—

What specific analysis was done to ensure that the tests are suitable for young people with additional support needs? What assessment was made of them and what evidence did you solicit from experts ahead of introducing them and suggesting that they would be a good thing for all children to take between the ages of four and a half and six.

One of our key principles at the beginning was to be as inclusive as possible, so we took steps to involve experts in the field. We used CALL Scotland and a reference group of additional support needs experts. We did trialling in special schools and with children in mainstream schools who had additional support needs and we had one-to-one cognitive labs, which looked at how children were interacting with the assessments. We made every effort to ensure that we included as many children as possible and were able to get useful information across the spectrum of abilities that there are in our classrooms.

We accept that it is a very big ask for one standardised assessment to cover every single child in Scotland. It is an ambition and an aspiration, and we continue to take feedback and to improve the process. By calling in expert reference groups and testing and trialling questions prior to them being released, we made a very serious attempt to be as inclusive as possible. The feedback was that it is significantly more inclusive than any other current standardised assessment in the Scottish system. We have made progress; I am not going to claim 100 per cent success, but I think that we are making the right moves in that direction.

Would you be willing to share that expert analysis of the questions and assessments with the committee?

I am happy to share what has been shared with us. Most of it was very practical; it involved looking at assessments and comments and Scottish Government accessibility people also looked at it. I am happy to make that available.

My final question is for the cabinet secretary. Can you understand why I and other colleagues in Parliament might be angry and irritated that, during the debate on this topic, some members of Parliament suggested that the tests could be used for the diagnosis of additional support needs? Can you see why that would cause confusion and annoyance?

I understand that point, yes.

I will finish by asking about the way forward for standardised assessments. One area that has been discussed in detail is the time constraints on schools, the technology that is available to them and how they have been able to implement the assessments. We were lucky enough to have a focus group that included some teachers, and we asked them about previous standardised assessments that had been bought in by local authorities or schools and how they managed the time for those tests. The answer was that it was exactly the same; teachers had to accommodate that kind of test in the curriculum, so it was something that they were used to doing.

In its submission, Moray Council stated that technology had not been an issue because most of the schools were following procedures that were in place for previous assessments, and North Lanarkshire, which is my area, stated that

“it has been universally accepted that there are real benefits to the SNSAs and that any issues encountered in the first year of implementation can be overcome in the subsequent session.”

Are you content that the assessment is being embedded in schools, that there is no issue in terms of resource and the time that is needed by teachers, and that lessons have been learned from the first set of assessments that have taken place?

We are just in year 2 of the process. In year 1, there was a theoretical maximum number of 613,000 assessments that could be undertaken; in fact, 578,000 were undertaken, which was a completion rate in excess of 94 per cent. Having taken the view that it is up to individual schools to judge whether it is appropriate and suitable for pupils to undertake the assessment, which means some pupils will not be given it, I feel that 94 per cent in the first year of implementation of a new system is a pretty high level of participation. As of yesterday, 144,941 assessments have been taken during this academic year. At the same date last year, the figure was 119,616. The significant increase in the proportion of assessments that were undertaken by 19 February this year is indicative of the system adjusting to the tests and taking them forward.

11:45  

In all the work that I do, I am not always able to reach agreement readily with local government on many questions. Local government was very supportive and participative in taking forward the SNSAs. That is because, as I have mentioned, their assessments—through no fault of theirs—were not curriculum related and did not support moderation effectively. Significantly, local authorities have embraced the new assessments. Obviously, in order to get a 94 per cent participation rate, schools have had to embrace them readily as well.

We are making progress. As with any new approach, we have to be open to doing things better or differently. I am open to that. That is why we have set up the P1 practitioner forum, why we have taken extensive feedback from teachers and pupils about their experience and why we will continue to do so.

Professor Louise Hayward of the University of Glasgow said:

“The idea of having information from tests that supports teachers’ professional judgment is an entirely appropriate approach.”—[Official Report, Education and Skills Committee, 23 January 2019; c 3.]

On Oliver Mundell’s point about tests, friends have told me that they wished that the tests had been available when their children were young. Do you agree that the test is an assessment that will enable parents to know at a general level how their child is doing, that it will confirm teachers’ professional judgment and that we are in danger of overcomplicating the meaning of the tests?

Fundamentally, it will be up to individual teachers to judge what information to share with parents about pupils’ SNSA performances. If a parent is interested in information about how their child has performed, I see no reason why that could not be shared with them. That is particularly true in relation to what issues arise from the test, such as strengths or challenges. Ultimately, if a diagnostic assessment identifies that a young person has a challenge, parents could be motivated to ensure that, through school and parental support, the challenge could be overcome for that child.

Ultimately, I think that schools will report on whether children are achieving levels and where they are on the journey to achieving levels. The levels are the milestones of the broad general education. Our objective is to enhance the quality of that information by virtue of the robustness of the SNSAs.

It has been a long session. I thank the cabinet secretary and his officials for their attendance.

11:48 Meeting continued in private until 12:27.