The Committee agreed to undertake an inquiry on the evidence base for the recently introduced Scottish National Standardised Assessments (SNSAs). These are assessments in literacy and numeracy completed by school children in P1, P4, P7 and S3. The inquiry specifically focussed on:

-the evidence base for moving away from the Scottish Survey of Literacy and Numeracy;

- the evidence base for introducing SNSAs at P1, P4, P7 and S3;

-international comparisons to understand similar and differing approaches used elsewhere; and

-what information the Scottish Government's assessments can provide that contribute to improving the educational outcomes of children and young people.

Following public debate around the issue of P1 assessments, in September 2018 S5M-13945 was agreed to (63 for; 61 against) in the Scottish Parliament:

Primary 1 Tests: That the Parliament believes that good-quality pupil assessment is an essential component of the drive to raise educational standards in Scotland's schools, but notes the level of concern that has been raised by teachers and other education professionals regarding the introduction and delivery of new testing arrangements for Primary 1 (P1) pupils; considers that this concern questions whether the new P1 tests are in line with the play-based learning philosophy of early years provision in the curriculum for excellence, and, in light of this concern, calls on the Scottish Government to halt the tests in P1 and to reconsider the evidence and the whole approach to evaluating the progress of P1 pupils.

Meeting of Parliament minutes, 19 September 2018

The Scottish Government responded by commencing an independent review of P1 assessments. Given the general issues regarding SNSAs, the Committee agreed to undertake an inquiry into SNSAs across P1, P4, P7 and S3 to consider these issues. The Committee agreed that its inquiry should not duplicate the work of the Scottish Government's independent review.

The Committee has evaluated the SNSAs using the key considerations:

• what is assessment for?

• what features do effective assessments possess and how does the SNSA perform against these criteria?

The OECD produced a review of large scale no-stakes standardised tests across OECD countries in 2011, Student Standardised Testing: Current Practices in OECD Countries and a Literature Review (“The OECD 2011 review”). The OECD 2011 review highlighted six “lessons” for policy makers to ensure that standardised testing is “more conducive to improving school outcomes”. The Committee has framed its inquiry around these lessons which, in summary, were:

• Clearly establish the purpose of the test and allow this to lead all following test design, implementation and use decisions;

• Be cautious in employing large-scale, standardised tests that serve multiple purposes;

• The assessment system should be developed around established learning objectives and educational standards;

• Develop assessment literacy of teachers and administrators;

• Reduce distortion and strategic behaviour by increasing teacher involvement and buy-in from an early stage; and

• Incorporate multiple measures of achievement especially in systems where standardised tests may be perceived as ‘high-stakes’ for teachers and school administrators.

Beyond assessing the evidence base as set out in the remit, the Committee has also collated evidence on areas where the SNSAs potentially could be improved. The SNSAs are at an early stage in their implementation. They have only been in operation for one full academic year and so there are understandably areas where practical suggestions for improvements to the policy have been identified. Work is underway by the Scottish Government and a number of Government agencies and other organisations as part of the continuous improvement of the implementation of the policy. This includes engagement by the designers of the assessments, Australian Council for Educational Research International United Kingdom (ACER UK), with teachers on their experiences of the first academic year of the assessments.

The Committee's evidence taking involved five formal evidence sessions and a small focus group of teachers and teaching union representatives. The Committee also wrote to all local authorities for their perspective. It received numerous written submissions including from members of the International Educational Assessment Network (IEAN) who provided detail on experiences in their countries. The Committee offers its sincere thanks to all those who submitted evidence to this inquiry.

Standardised assessments take a number of forms in education systems in different countries to serve different purposes. Formative assessments, which are sometimes called assessments for learning, are where assessments are used to inform teaching practice. The Northern Ireland Council for the Curriculum, Examinations and Assessment describes diagnostic assessments as being linked to formative assessment, and “often takes place at the beginning of a learning programme and can be used to identify pupils’ strengths and areas for improvement”ⁱNICCEA webpage, What is formative assessment?. The Scottish Government’s submission stated that the SNSA is a diagnostic assessment¹Scottish Government. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181221Scottish_Government.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181221Scottish_Government.pdf</a> [accessed 17 April 2019]. The Cabinet Secretary for Education and Skills, John Swinney MSP, also acknowledged that, "In the aggregation of data, the SNSA is summative".ⁱⁱEducation and Skills Committee 20 February 2019, John Swinney, col. 24 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11948&mode=pdf The University of Glasgow states that the SNSAs are diagnostic, “in that they are linked to proposals for action based on each young person’s performance".²University of Glasgow School of Education. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181221Louise_Hayward_University_of_Glasgow.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181221Louise_Hayward_University_of_Glasgow.pdf</a> [accessed 17 April 2019]

Before the introduction of SNSAs, standardised assessments of some form took place within twenty nine of the thirty two local authorities across Scotland.³Education and Skills Committee 20 February 2019, John Swinney, contrib. 112, <a href="http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&amp;c=2152998" target="_blank">http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&amp;c=2152998</a> The Committee was told that the vast majority of these were not benchmarked against Curriculum for Excellence levels and distinct approaches were taken by local authorities creating a complex landscape which the OECD described as "fragmented".⁴OECD. (2015). Improving Schools in Scotland. Retrieved from <a href="http://www.oecd.org/education/school/Improving-Schools-in-Scotland-An-OECD-Perspective.pdf" target="_blank">http://www.oecd.org/education/school/Improving-Schools-in-Scotland-An-OECD-Perspective.pdf</a> [accessed 17 April 2019] East Renfrewshire's assessments were the only ones cited in oral evidence that had been aligned with Curriculum for Excellence.

Standardised assessments have been used at a local level in Scottish schools for many years, the submission from ADES suggested that many of these assessments had been put in place in the absence of local level standardised information that could inform local authorities in relation to attainment.⁵ADES. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181213ADES.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181213ADES.pdf</a> [accessed 17 April 2019] Some local authorities run them across all their schools, others run them specifically in primary schools or secondary schools. Some local authorities have standardised assessments focussed on one particular year group. In other local authorities standardised assessments are selected and run by individual schools as opposed to on a local authority wide basis. In some schools standardised assessments are used to assess particular subsets of pupils as opposed to assessing everyone in a particular school year.ⁱSubmissions from Professor Christine Merrell, formerly of CEM assessments, and GL assessments set out the forms of assessment used at local authority and school level that CEM and GL provide. Submissions from local authorities on the inquiry webpage set out the variation in approach taken by different local authorities and schools across Scotland. Standardised assessments of all these kinds are funded at local authority and/or school level.

In general these assessments involve commercially produced, externally marked packages. The Centre for Evaluation and Monitoring (CEM) and GL assessments are two main providers of the various forms of assessment available including on: cognitive function, motor development, non-verbal ability, reading, spelling, mathematics and English vocabulary. The range of assessments used in schools and local authorities, designed around local need, therefore do not have equivalence with the functions performed by the SNSAs.

The SNSAs are a form of large scale standardised assessment intended to be taken nationally in all state schools. The Scottish Government introduced SNSAs in the 2017-18 academic year. The assessments are taken by students in state primary schools and secondary schools once in P1, P4, P7 and S3. The results of the assessments are intended to assist teachers, as a diagnostic and formative assessment, in supporting children and young people in their learning. The information from the assessments can then be used at school, education authority and national level. The information can be interrogated to provide an analysis broken down by different subsets. For example a local authority could look at SNSA information broken down by gender. This information can be used to identify particular areas for improvement.

The Scottish Government's intention is that the SNSAs, which are specifically aligned to the Curriculum for Excellence, will replace standardised assessments run at a local authority level. The assessments are completed by children online and are focussed on elements of the literacy and numeracy part of the curriculum. They are not time bound meaning a student does not have to complete an assessment in one sitting.

Teachers can access and analyse the results of the assessment straight away. The assessments are adaptive, meaning a child is presented with different questions as they move through the assessment, based on the answers they have given earlier in the assessment. The assessment is intended to adapt to the ability of each child. Teacher training on the assessments has been introduced simultaneously with the implementation of the assessments.

Norming studies, which are based on samples of pupils who take SNSAs, provide national scales of ability against Curriculum for Excellence levels. Teacher judgments on the CfE levels their pupils have achieved, based on SNSA results and numerous other factors, are submitted by teachers and collated at a national level. This national level information is called the Achievement of Curriculum for Excellence Levels (ACEL). It is published annually and is a key source of information on education within the Scottish Government's National Improvement Framework. ACEL is a means of assessing the performance of the education system and to identify potential areas for improvement. Using ACEL, teacher judgments are a primary means of assessing performance at a national level.

The Scottish Survey of Literacy and Numeracy (SSLN) was a national survey that produced national level data. It ran using a sample of pupils and teachers from primary and secondary state schools and also independent schools. The Scottish Government discontinued it in 2017.

In 2017, the Cabinet Secretary stated in relation to the merits of data collection including SNSA:

The problem with the SSLN is that it does not enable us, when we see declining performance, to identify from the survey where that is happening. However, the data that we have requested to be put in place, which will be substantially reinforced by standardised assessments, will give us the ability to do that and to support young people to fulfil their potential as a result.

Meeting of the Parliament 9 May 2017, John Swinney, col. 26

The Scottish Government submission to the inquiry ascribed a number of advantages to the ACEL, which SNSA contributes to, in contrast to the SSLN, namely:

• It empowers teachers and makes their professional judgement the key indicator of children’s progress prior to national qualifications;

• It looks across the full CfE level (e.g. it is broader than just literacy and numeracy);

• It is based within the curriculum and uses a broad range of sources;

• It is aligned to systems that schools and local authorities already have in place;

• It provides annual data at school and local authority level and data which is broken down by pupil characteristics, allowing school and local authority staff to analyse their own data for improvement purposes. National level data also contributes to national improvement planning;

• It provides annual data on both literacy and numeracy and includes an additional stage, Primary 1, that was not covered by the SSLN;

• The results can be published and used for improvement purposes more quickly than the SSLN – within 6 months of the data being collected;

• Reflects the OECD’s view that “an assessment system that encompasses a variety of assessment evidence, that includes rich tasks and a clear indication of expected benchmarks referenced to the breadth and depth of the curriculum, can enhance teachers’ assessment skills and learners’ progress.”[1]

[1] Improving Schools in Scotland: An OECD Perspective (2015): page 157

The submission and the Cabinet Secretary's oral evidence also places emphasis on the findings of a 2014 review of the SSLN, that sets out the barriers to the SSLN being scaled up to allow it to provide information at local authority or school level.

...we undertook a review of the SSLN in 2014. That review concluded that scaling up the survey model to produce local authority-level results was not a viable or realistic option—essentially, because that was not the purpose of the survey at its inception and the design of the survey did not lend itself to that kind of upscaling.

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

The Cabinet Secretary highlighted that the 2015 OECD report Improving Schools in Scotland: An OECD Perspective (p 155) suggested limitations in the 'light sampling' of the SSLN and stated:

“The light sampling of literacy and numeracy at the national level has not provided sufficient evidence for other stakeholders to use in their own evaluative activities or for national agencies to identify with confidence the areas of strength in the years of the Broad General Education across the four capacities of CfE. Nor has it allowed identification of those aspects or localities where intervention might be needed.”

Education and Skills Committee 20 February 2019, John Swinney, contrib. 149, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153035

The Cabinet Secretary described ACEL as:

...our key and annually published measure in the national improvement framework. [emphasis added]

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

ACEL data has been published with a caveat that comparisons should not be made across local authorities and schools “without knowledge of the underlying approach to assessment and the context of the authority or school”.⁴Scottish Government. (2018, December 11). ACEL 2017/8. Retrieved from <a href="https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/" target="_blank">https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/</a> [accessed 17 April 2019]

ACEL data is currently badged as experimental and "over a third [of those contributing data] reported that they were generally confident in the robustness of their ACEL data".⁴Scottish Government. (2018, December 11). ACEL 2017/8. Retrieved from <a href="https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/" target="_blank">https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/</a> [accessed 17 April 2019] This would indicate that close to two thirds of those contributing data were not generally confident in the robustness of the data.

The Committee has heard some evidence on the statistical rigour of the ACEL. Professor Lindsay Paterson from the University of Edinburgh stated:

I do not think that it is an adequate substitute for the SSLN, for two major reasons. First, the assessment of where children have reached is made according to teacher judgments. We have already talked about the unreliability of those. Secondly, it is not an adequate substitute for a completely different reason, which is to do with the measurement of social circumstances. Actually, the SSLN suffered from that, too. We need much better measures of social circumstances. I think that the committee has addressed that before, but it comes up over and over again.

We know that two thirds of children who are living in poverty are not in the 20 per cent most deprived neighbourhoods. Your constituency probably has no deprived neighbourhoods, but that does not mean that it has no deprived families. There are other ways in which the annual December [ACEL] report is inadequate, but those are the two major ways.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 97, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147730

The Cabinet Secretary described the ACEL data as comprehensive a number of times in evidence, which suggests there is a strength in a system based on data on almost all pupils (including SNSA results) contributing towards ACEL. This is compared with a sample survey approach based on fewer pupils:

...the SNSAs will contribute to the formulation of teacher judgments, those teacher judgments will then be aggregated into the achievement-of-levels data, which will be aggregated at a national level, and we will have a more comprehensive picture of the performance of young people within the education system and of their achievement of the levels than the SSLN, as a limited survey, could ever have given us.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 51, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152937

Professor Sue Ellis from the University of Strathclyde suggested that a weakness of the SSLN was that it was not used by teachers:

The only people I heard talking about [the SSLN] were politicians and the odd academic. I did not hear directors of education saying that they would look at their system because obviously something that they were doing was not working and that they would reassess their teaching; I did not hear class teachers or headteachers talking in that way.

If we want something that works and benefits the children of Scotland, we need something that has purchase with the practitioners who can make a difference. It has to speak to the teaching and learning that goes on in classrooms and how teachers think about the children sitting in front of them.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 118, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2146249

Dr Keir Bloomer from the Royal Society of Edinburgh Education Committee suggested that the link between sample survey data and improvement in learning may be more remote but the link remains:

In my view, it is a mistake to assume that a survey—or, come to that, a system of universal assessment—that says something about how the system as a whole is performing has nothing to do with learning. Learning in the system will improve if we know more about how we are doing and whether we are progressing or moving backwards. Although the connection is less direct than it is in the case of the feedback that is given to teachers about the individual’s performance, survey information of that kind is still a valuable contribution towards improvement.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 50, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147683

Jackie Brock from Children in Scotland, Professor Louise Hayward from the School of Education at the University of Glasgow and Susan Quinn from the EIS suggested that the issue was not the data that was produced previously, but that its potential was not fulfilled because it was not used enough:

In Scotland, we do not have a wealth of assessment data and information on follow-through at individual, school, local authority and Government levels.

I will say two things about that. No local authority chose to enhance the sample of the SSLN. What does that say? Also, the committee has lost an opportunity for consistent tracking at national level using the SSLN or other national information. You could have had an annual report based on evidence of improvement, and you could have been homing in on where we need to go to improve our education, based on real data, by addressing the individual literacy and numeracy needs of children. Scotland has lost that.

Education and Skills Committee 30 January 2019, Jackie Brock (Children in Scotland), contrib. 74, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150095

We are not using the information that we already have.

Education and Skills Committee 30 January 2019, Jackie Brock, contrib. 101, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150122

Jenny Gilruth has put her finger on the crucial issue, which is that some people had access to the [SSLN] information and others did not. That was simply not good enough.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 48, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147681

The SSLN provided something nationally that the SNSAs will not provide. I understand your point: I was the headteacher who removed young people to administer the survey, and I was the class teacher who had pupils removed from my class. Just because something did not happen does not mean that it could not happen. There were ways and means of using the information from the SSLN and, as James McEnaney said, there were potentially ways of developing it to get a genuine picture of Scottish education at a single point in time, as well as some of that really rich information about young people and the targeted interventions that we could make.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 77, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151568

Jackie Brock, Children in Scotland, also set out how the SSLN data could be used to reach down to teacher level:

The first year of SSLN’s reporting nationally on numeracy showed us that we were doing really well in terms of the ability of children in the early years of primary to add, subtract and do basic multiplication and division—essentially, teachers were doing really well in teaching basic numeracy to children. What was appalling was that children in primary 4 and beyond were not able to apply that knowledge to more sophisticated concepts. The situation with fractions was the evidence for that view.

That finding enabled us to understand teachers’ needs for development in numeracy: they were good at teaching basic numeracy, but their ability to transfer that to enable children to apply it in more sophisticated ways needed more attention. That was the case across Scotland; it is not the case that some pockets of the country were doing better and some were doing more poorly.

The evidence on what was happening in primary 4 also helped us to unpack what was going wrong in the later years, and the implications for Parliament’s and our aspiration that pupils do well in science, technology, engineering and mathematics subjects.

Education and Skills Committee 30 January 2019, Jackie Brock (Children in Scotland), contrib. 74, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150095

Professor Paterson gave a specific example of the process of the SSLN influencing teaching and learning:

the people who were running the SSLN would pick out those test items that children were not doing well with and would use them as the basis for professional development sessions for teachers. That was extremely good practice. For example, if they found that children were not good at telling the time, they would use the mistakes that children made in their answers to the questions about telling the time to advise teachers on how they could teach that better. That was a great idea, and it shows how a survey can be used.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 49, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147682

The Committee explored the extent to which the SSLN provided information of use in assessing the extent and the nature of the attainment gap. Dr Bloomer explained that:

The SSLN had information about family background and surveyed teacher views, so there was a richness to the information, although I accept Lindsay Paterson’s point that the SSLN’s predecessor—the Scottish survey of achievement—was probably a better test still than the SSLN. We have lost quite a lot of contextual information, which is valuable in trying to narrow the attainment gap.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 28, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147661

The SSLN collected data from independent schools as well as state schools, whereas the SNSA and ACEL only collates data from state schools. The RSE submission made the point that without the independent sector, which can be assumed reflects a higher average socio-economic background, it is not possible to measure the extent of the poverty-related attainment gap.¹⁷Royal Society of Edinburgh Education Committee. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181221RSE.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181221RSE.pdf</a> [accessed 17 April 2019] The Cabinet Secretary responded on this point:

The data will be able to demonstrate what levels are being achieved by young people in the Scottish education system. That will demonstrate openly what improvements in capacity and performance they have made...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 63, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152949

This response does not entirely address the issue raised by the RSE. The data will be able to reflect improvements in state schools. At a national level, without comparable assessments in independent schools, the data will not be able to reflect the attainment of all children in Scottish education and on that basis it cannot accurately reflect the overall size of the attainment gap. However, the Cabinet Secretary emphasised in relation to the attainment gap:

...the information that we present [in the National Improvement Framework] demonstrates the extent of the attainment gap within Scottish education. We then demonstrate year on year how we are closing that gap, using comparable data on a whole series of different indicators on which the Government has consulted. I think that we have reached a broad consensus that the measures that we tabulate indicate the framework within which the closure of the attainment gap should be undertaken.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 65, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152951

Dr Bloomer stated that Scotland is "relatively data poor"ⁱEducation and Skills Committee 23 January 2019, Dr Bloomer Official Report col. 26 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11910&mode=pdf at present and suggested that SNSAs and the SSLN could have been run in parallel to produce both universal and sample data:

We are puzzled by the abandonment of the Scottish survey of literacy and numeracy. It is unfortunate that there has been no continuity in the information that has been made available in the past. We had a previous assessment called the Scottish survey of achievement, which ran for five years; a short interval and then the SSLN ran for six years; abandonment; and now we have a third system. A sample survey, which is what the SSLN was, is not incompatible with universal assessment of the kind that is provided by the new SNSAs. We do not see what the rationale was for abandoning the SSLN, because it would be perfectly possible to run both systems in parallel.

Education and Skills Committee 23 January 2019, Dr Keir Bloomer (Royal Society of Edinburgh), contrib. 3, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147636

Professor Paterson supported this position:

Secondly, I agree completely with what has been said about the abandonment of the SSLN: the two surveys could have run in parallel. The great advantage of a survey is that it can ask for a much wider range and much deeper kinds of information. Incidentally, I agree that the design of the Scottish survey of literacy and numeracy was not adequate for some of those purposes. The older one—the Scottish survey of achievement—was better. One way in which it was better was that it could provide a national picture, as has been said. The cabinet secretary’s legitimate complaint was that the SSLN could not tell us where things were happening—where they were getting better or worse—which was a design feature of the SSLN but not of the SSA. The SSA’s design allowed us to say, anonymously, that a particular school was doing better for certain reasons, such as homework practice, discipline or school uniform. In other words, it is possible to design a survey that gives us a national picture, and also council-level and school-level ones. Both could be done.

Education and Skills Committee 23 January 2019, Professor Lindsay Paterson (University of Edinburgh), contrib. 5, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147638

The Committee has received differing evidence as to whether the SSLN could have been expanded, including a larger survey size, to enable it to produce data at a local level, for example for local authorities. Professor Sue Ellis²²University of Strathclyde. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181218University_of_Strathclyde.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181218University_of_Strathclyde.pdf</a> [accessed 17 April 2019] and the Cabinet Secretary¹Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, <a href="http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&amp;c=2152924" target="_blank">http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&amp;c=2152924</a> were of the view that this was not a viable option. The 2014 review assesses in detail the pros and cons of scaling up, including the impact of increasing the sample size and the potential margins for error in the information produced. James McEnaney, a writer and college lecturerⁱⁱEducation and Skills Committee 6 February 2019, James McEnaney, Official Report col. 9 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11938&mode=pdf and Professor HaywardⁱⁱⁱEducation and Skills Committee 23 January 2019, Professor Hayward, Official Report col. 29 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11910&mode=pdf suggested that an enhanced SSLN was possible and would be beneficial. Assessment at Transition, a research report co-authored by Professor Hayward recommended that this option be explored.

ACER suggested in evidence that the SNSA could be expanded to gather qualitative information based on a sample approach:

Another important area that could be developed alongside or in the SNSA is qualitative explanatory information about, for example, how children engage with their learning, what their attitudes are to learning and what the school atmosphere is like. Currently, there is no instrument or survey mechanism in the SNSA, or alongside the SNSA, that captures such information. If ways of managing that could be integrated with the SNSA, that would be really helpful in trying to work out why things are happening in the way that they are.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 29, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143341

Should the Scottish Government consider the option of expanding the SNSA, such work could look at the potential to factor in features of the SSLN's predecessor, the SSA. Professor Hayward, Professor Paterson and Dr Bloomer all suggested that in some respects the SSA was superior to the SSLN:

Another interesting thing about the SSA was that local authorities, in addition to having access to information from the national survey, had the opportunity to ask for a boosted sample within a particular local authority, which would give them information at a local authority level. Technically, there is nothing to suggest that a headteacher, for example, did not take that information from a survey to use in a school or for a teacher to use in a classroom. With the SNSA, the norming studies provide the opportunity to develop a survey approach that could build some of the advantages that I have described into our system if that is our purpose.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 48, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147681

ADES supported the idea of building features into the SNSA, stating:

...one of the good things about the Scottish survey of literacy and numeracy was that there were both pupil and teacher questionnaires that measured confidence levels and gave good information to local and national authorities about how confidence levels could be improved for particular aspects of the curriculum. It was very valuable information, and we should look at building that into the SNSA.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 51, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143363

On the specific option of running the SSLN alongside SNSAs, the Committee notes that the EIS described the SSLN at different points in evidence taking as low stakes, proportionate and relatively easy to administer, and a good assessment tool. When the option of running SSLN alongside newer approaches was put to the Cabinet Secretary, and whether this option was considered in 2016, he responded:

The SSLN involves an additional burden on the education system.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 73, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152959

We put in place a performance framework that enabled us to do that, but we did not want to add it on to existing arrangements in the education system, which is why we discontinued the SSLN.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 75, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152961

In terms of the policy basis for SNSAs, the Cabinet Secretary highlighted the value of information from SNSAs for teachers. In particular, he highlighted the value of the granular information in identifying a child's strengths and specific areas where they may have challenges that require specific support or further assessment.

At local authority level, the Cabinet Secretary highlighted that under the SSLN there was no means of local authorities being able to compare performance between authorities, whereas the intention under SNSAs was that data aggregated at local authority level would enable this to happen. He explained:

There has been tremendous local authority collaboration in introducing standardised assessments. Why? Some of the reasons that were discussed in my exchange with Jenny Gilruth are relevant. Some local authorities might have been undertaking some form of assessment, but it was not related to curriculum for excellence and it could not give them any read-across on whether what local authority A was doing was at a standard that was sufficiently comparable with what was going on in local authority B...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 147, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153033

Darren Northcott, representing the NASUWT, stated in evidence that "The SNSA has the potential to be a better assessment than those that it is replacing in 28 local authorities."ⁱEducation and Skills Committee 6 February 2019, Darren Northcott, col. 29 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11938&mode=pdf He added-

An important part of the story is that, before the SNSAs, 28 local authorities imposed standardised assessments on their schools. One element to consider is that some—I will not name anyone—of those standardised assessments were incredibly narrow.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 90, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151581

in the longer term, if we persist with SNSAs, those local authorities will need to be challenged because, as has been said, one disadvantage of those systems is that they are not only relatively expensive but not at all aligned with curriculum for excellence. That brings into question the value of local authorities spending public money on standardised tests that bear no relation to the curriculum that schools should be pursuing. Perhaps we can cut them a bit of slack early on and say that they have needed a period of support or transition to SNSAs...

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 125, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151616

The Cabinet Secretary also highlighted that smaller scale standardised assessments run at local authority or school level were not assessing based on the Curriculum for Excellence. He added that SNSAs are not an overall picture of a child's attainment, rather they are focussed on elements of the curriculum within literacy and numeracy that have been identified as priorities for improvement:

It is certainly the case that SNSAs do not cover all aspects of literacy and numeracy in the curriculum, nor have they been designed to do so. However, attainment in literacy and numeracy has been identified as one of the key priorities in the national improvement framework.

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

Elements of literacy that are not covered by the assessment include writing ability, as they are based on multiple choice questions. Juliette Mendelovits, representing ACER before the Committee, set out in more detail the specification for the SNSA in relation to literacy and numeracy:

The brief for the SNSA covers literacy and numeracy only; it does not cover the whole of curriculum for excellence, which has many other facets. Even within literacy and numeracy, there is no attempt to cover every aspect of curriculum for excellence. We must be perfectly frank and acknowledge that. For example, we cannot hope to assess engagement in reading through the kind of assessment that the SNSA is.

...we have shaped the assessment around key organisers in numeracy and in reading and writing, and every item in the assessment has been aligned with one of the [Curriculum for Excellence] benchmark statements.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 155, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143467

In terms of information at a national level, the Cabinet Secretary highlighted what he perceived to be the limitations of the SSLN in enabling national agencies and the Scottish Government to identify strengths and weaknesses in particular areas of the system. In a series of statements the Cabinet Secretary explained that the OECD findings in its 2015 report had a key influence in the Government's decision to introduce ACEL [and SNSAs] and to discontinue the SSLN.ⁱThe 2015 report did not cover the senior phase of the Scottish education system.

Our view is founded on the OECD’s 2015 analysis, which states:

“Standardised assessment tools can be used formatively in all parts of the system if they are referenced to the curriculum”,

as, I contend, ours is,

“flexible in their use”,

as ours is,

“and provide high quality just-in-time information for teaching and learning”—

which I contend is the case—

“while at the same time having efficient ways to aggregate the results through the system.”

Education and Skills Committee 20 February 2019, John Swinney, contrib. 139, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153025

page 155 of the OECD 2015 report is relevant:

“The light sampling of literacy and numeracy at the national level has not provided sufficient evidence for other stakeholders to use in their own evaluative activities or for national agencies to identify with confidence the areas of strength in the years of the Broad General Education across the four capacities of CfE. Nor has it allowed identification of those aspects or localities where intervention might be needed."

That is essentially a commentary on what SSLN delivered for us, and that would be my assessment of some of the international work as well. Those measurements do not give us an insight into what needs to be done to improve the system, which is what the framework that we have put in place enables us to do.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 149, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153035

It is worth noting at this stage, given the emphasis placed on the OECD's 2015 report in the Scottish Government's decision to introduce SNSAs and the ACEL, that the Committee received evidence from the University of Glasgow that highlighted that the OECD did not specifically recommend a universal form of standardised assessment as the only possible means of addressing its findings.

The final paragraph on p.161 is clear:

‘Currently, however, the way national assessment is constructed in Scotland does not provide sufficiently robust information at all levels of the system, including LAs or an individual school or across important domains of CfE for learners and their teachers. This problem does not mean that everyone must be tested at particular year levels in order to have this information. An alternative, for example, could involve sample testing of a range of learners within each school on rich tasks which can then be used to benchmark the achievement of other learners on the curriculum.'

University of Glasgow School of Education. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181221Louise_Hayward_University_of_Glasgow.pdf [accessed 17 April 2019]

When asked about this position, Professor Hayward told the Committee:

The OECD is clear that it does not mean that, by necessity, one particular path must be followed. It was open to a wider debate to think around the issues.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 34, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147667

The OECD also argued that we should look at the range of sources of evidence that we had available and relate them back to the purposes that we intended them to serve.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 36, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147669

The Committee explored with witnesses the extent to which the Scottish Government consulted stakeholders in reaching the decision to introduce a large-scale national standardised assessment, and associated decisions to discontinue the SSLN and introduce ACEL. Susan Quinn from the EIS set out her recollection of the policy development process for SNSAs:

...we had a meeting where...It was generally agreed that there was a wealth of information in the system and that we needed something that would allow us to talk to each other so that there was a national understanding. We discussed whether there was a need for a national standardised test and the general viewpoint was that there was no need...We discussed how we needed to find a way to gather [existing] information together.

Two weeks later, however, we attended an event at which the First Minister announced that she would introduce standardised tests. I was at that meeting—I had been at the one before—and I sat there, wondering when, between the previous meeting and that one, any of the people round the table had been spoken to again about what was decided and what was to become policy...

We then had to have negotiations about what the tests would look like and everything else. The manner in which the tests were introduced was a real problem...

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 129, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151620

This evidence raised questions about the extent to which the Scottish Government's consultation process had been meaningful, and whether there was scope for those present to influence the Scottish Government's thinking. The contributions from parent representative organisation Connect suggested that consultation can sometimes feel combative as opposed to collaborative. Lindsay Law told the Committee:

There needs to be more transparency. That is a general point about policy formation, and this policy is part of that. In our experience of consultations, what usually happens is that an idea is created and then stakeholders from Scottish society are brought in as part of the consultation process. However, because the idea already exists, that sets up a naturally combative response between the people who have had the idea and those who say, “Have you thought about this and that?” We do not give much time to those people and it becomes an exercise in someone defending an idea, others knocking it down and the idea still going through, leaving a load of stakeholders feeling disempowered and disenfranchised.

Education and Skills Committee 06 February 2019, Lindsay Law, contrib. 128, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151619

James McEnaney is a writer and college lecturer who has submitted FOI requests on the policy development of SNSAs. The FOI responses have reflected that a series of meetings with stakeholders had taken place in relation to the National Improvement Framework in 2015. These meetings were unminuted so it is unclear as to the extent of discussions on the idea of establishing national standardised assessments during the meetings. Detail of the dates of the meetings and the organisations involved in a number of them are published on the Scottish Government website. James McEnaney added in evidence that he had been able to find little information on the views expressed by stakeholders or a substantial written evidence base for SNSAs:

Susan Quinn’s evidence says that, in the various meetings, nobody said that they wanted a new set of tests, and that aligns with the information that I found. It is difficult to be sure, though, because all those meetings were unminuted...

If we look at the available information and material, we can see the scarcity of written advice that led to the implementation of the system.

Education and Skills Committee 06 February 2019, James McEnaney, contrib. 94, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151585

The lack of transparency on the development of the standardised testing system fed into the way in which it had to be defended as the process went on. Part of the reason why it has been difficult to be clear about what the assessments are doing and what they are for is that all of that has become bound together.

Education and Skills Committee 06 February 2019, James McEnaney, contrib. 127, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151618

Sue Palmer from Upstart Scotland recalled, in relation to a developmental assessment, that there were alternative forms of standardised assessment at P1 considered:

The [Early Development Instrument] measure that I described earlier, which is used in Ontario and across Canada and Australia, has been piloted in East Lothian and validated for Scotland on the basis of that pilot. However, it never reached parliamentary level but stopped at civil service level. It was done roughly around the time that the idea of introducing standardised tests of literacy and numeracy came in.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 96, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150117

In relation to the pace of the policy decision making processes, Professor Paterson commented that:

I think that evidence should be much more a part of the policy formation cycle.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 84, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147717

Dr Bloomer set out the tension between taking time to develop an evidence base and progressing to implementation:

There is always a tension, in policy making and implementation, between taking time to get it right and getting on with the job. If anything, the tendency in recent years has been to accelerate timescales, which means that less is done to perfect the instrument before starting off.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 54, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147687

In terms of decision on when to rollout the policy, an NASUWT representative suggested during the teacher focus group:

...the policy implementation felt rushed, for example technology and training was not in place in advance of implementation.

Education and Skills Committee 6 February 2019, teacher focus group note

The National Parents Forum for Scotland also highlighted the negative impact of deciding not to publicise the new SNSAs to parents:

We acknowledge, but still dispute, Scottish Government’s reason for not highlighting the SNSA rollout to parents: they believed it would result in unnecessary profile raising; but, as NPFS made clear at the time and we then saw earlier this year, when there is a vacuum of information it allows fear and anxiety to spread. This issue is wider than the standardised assessments, it is about a repeated lack of good, direct communication from schools, local authorities and Scottish Government to parents. Major initiatives, both local and national, are undermined; entire policies tarnished, often before they have the opportunity to get off the ground. We are now in a situation with SNSAs that is entirely unacceptable: our children’s education is being used to score political points.

National Parent Forum of Scotland. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181220NPFS_letter.pdf [accessed 17 April 2019]

The Cabinet Secretary was not specifically asked about stakeholder engagement in policy formulation processes. The Scottish Government submission referenced stakeholder feedback in relation to the SSLN and highlighted a related relevant OECD recommendation:

Based on stakeholder feedback, a recognised limitation of the SSLN was that it didn’t provide a breakdown below national level (the sample size was too small to provide reliable school or local authority level data) so the data it provided was therefore of limited value to schools and local authorities in determining where to target improvement activity.

In its 2015 review of education in Scotland, the OECD made the following recommendation: “Strike a more even balance between the formative focus of assessment and developing a robust evidence base on learning outcomes and progression. While learner outcomes should not be the only focus of a standards or appraisal system, stronger reference to learners’ progress will create improvement...

Local authorities have sought to fill this space with their own assessments but this is fragmented. The challenge now is to improve the quality of information on those aspects of CfE that are valued by stakeholders including all the capacities of CfE, while retaining the strongly formative focus.

Scottish Government. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181221Scottish_Government.pdf [accessed 17 April 2019]

The importance of establishing clear objectives for a standardised assessment, before designing the specifics of the assessment, was a theme in evidence taking. Professor Hayward put it concisely:

Our central focus should be purpose, and then we should decide how best to collect information.

Education and Skills Committee 23 January 2019, Professor Louise Hayward (University of Glasgow), contrib. 4, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147637

The Cabinet Secretary was asked whether the Scottish Government's position on the purpose of the assessments had changed from the announcement of the policy. He responded:

We have been clear throughout that the SNSAs were being introduced for diagnostic purposes, to provide teachers with objective, nationally consistent information about individual learners’ strengths and areas in which they might benefit from further support.

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

The purpose of the SNSAs, in my view—and I think that this has been the Government’s clearly expressed approach—is as a diagnostic tool within the education system.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 41, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152927

I do not think that the Government at any stage suggested that the assessments should be summative...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 45, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152931

A number of witnesses suggested that the initial Scottish Government intention was to produce national-level performance data from standardised assessments as opposed to having a formative diagnostic assessment designed for teachers. Professor Andy Hargreaves, a member of the Scottish Government International Council of Education Advisers, stated:

...it is public knowledge that the initial position was to have a high-stakes standardised test in the Scottish education system. As advisers, the advice that we offered—whether you like it or not; the nature of advice is that you can ignore it—was that having a high-stakes, large-scale standardised test would have all kinds of negative impacts on teaching and learning. However, your Government feels that, in an unequal society, large-scale information is needed to guide it on where best to provide support and intervention.

There is now meant to be a lower-stakes assessment that is one of the things that informs teacher judgment. The main way in which we will figure out how the system is moving is through the aggregated data on teacher judgments.

That is the art and science of how we are trying to get beyond, on the one hand, a high-stakes, large-scale standardised test with utterly predictable and pervasive negative consequences and, on the other hand, no standardised testing at all, which leaves us unsure and unclear about the consistency of teacher judgment across schools and local authorities.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 68, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150089

Dr Bloomer concurred that the intention of the policy had changed since the outset:

The emphasis has clearly changed. It was on national monitoring at the outset, and it is now more on the diagnostic capability. One has to recognise that the diagnostic value of the tests is limited. They have some strengths. They can monitor the same pupils over time, which we were not able to do through the sample surveys, because the same pupils did not figure in successive runs of the survey. We now have what are described as long scales, which stretch through from primary 1 to secondary 3, and it is possible to monitor how the individual pupil has progressed up the scale. That is valuable information, and researchers will be able to make something of it in the future.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 15, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147648

James McEnenay also considered that national-level data was the initial focus:

When the testing system was put forward initially, it was clearly conceived to be about national-level data. It was going to be a national measurement because that was what we needed. Ultimately, that is why it has incorrectly been seen as having replaced the SSLN.

Education and Skills Committee 06 February 2019, James McEnaney, contrib. 12, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151503

When asked why the Scottish Government's position may have changed, Susan Quinn from EIS responded:

I would argue that one of the key drivers was that the EIS indicated that we would ballot our members to boycott any system of tests being put in that was as the system was described at the outset.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 48, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151539

To frame the discussion around the data provided by the ACEL compared to the data generated by the SSLN, the Committee heard evidence from Professor Paterson on the amount of national data available to monitor the performance of schools in general. In advance of giving evidence to the Committee he had stated that “Scotland has no reliable method of monitoring the performance of schools in literacy and numeracy for the first time in almost 60 years”.¹The Herald. (2018, March 31). Monitoring of Schools "woefully inadequate" warns top academic. Retrieved from <a href="https://www.heraldscotland.com/news/16129147.monitoring-of-scottish-schools-woefully-inadequate-warns-top-academic/" target="_blank">https://www.heraldscotland.com/news/16129147.monitoring-of-scottish-schools-woefully-inadequate-warns-top-academic/</a> [accessed 17 April 2019] Professor Paterson provided context when he appeared before the Committee:

...I was discussing the demise of almost all surveys of school students, including those of leavers or any other group. The only survey that remains is the programme for international student assessment, which is inadequate for most purposes; it is only for pupils who are aged 15 and so on.

I referred to 60 years, but we could even say that we need to go back nearly 80 years, because Scotland pioneered the use of good-quality surveys to understand the progress of people through education systems. From that came a series of things, including the Scottish school leavers survey, various surveys of primary school children, the SSLN, the SSA, predecessors to that, and the assessment of achievement programme. All of them have gone and are no longer used.

We do not now have the kinds of information that we had 20 years ago, for example, when the Parliament was established.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147671

Dr Bloomer commented on a similar theme:

We are now involved in only one international survey. In my view, it was a mistake to abandon the other two, and I hope that at some point that will be reversed, because we need more information about how we compare with other countries. Although PISA is an excellent survey, it operates at age 15, so it tells us nothing about what is happening at the stages of the education system that we are already most ignorant about.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 94, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147727

The ACEL data is not yet considered statistically rigorous, it is described as "experimental".⁴Scottish Government. (2018, December 11). ACEL 2017/8. Retrieved from <a href="https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/" target="_blank">https://www.gov.scot/publications/achievement-curriculum-excellence-cfe-levels-2017-18/</a> [accessed 17 April 2019] However, as mentioned above, the Cabinet Secretary placed emphasis on its importance as a source of data describing it as:

our key and annually published measure in the national improvement framework. [emphasis added]

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

The Committee explored with the Cabinet Secretary the extent to which, in the absence of certain international surveys and the SSLN, there would be a gap in the availability of reliable data on the performance of the education system in advance of progress being made on ACEL data:

I expect the [ACEL] data to be comparable in 2019-20, but I caveat that by saying that it is not my decision to make [referring to Scottish Government statistician judgment].

Education and Skills Committee 20 February 2019, John Swinney, contrib. 89, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152975

Should the ACEL data be comparable in 2019-2020, it will be towards the end of 2022 before there are three years worth of ACEL data available from which to identify trends in performance. The last results of the SSLN were published in 2017. This creates a minimum gap of five years between the last set of national statistically rigorous performance data from the SSLN that could identify trends in 2017 and the first set fulfilling this function from the ACEL in 2022. That assumes that 2019-20 ACEL data is deemed comparable.

The Committee sought to understand why the SSLN had not been continued including raising the option of running it alongside the SNSA. The Cabinet Secretary stated:

Our judgment was that that purpose would best be served by introducing standardised assessments and moving to the capturing of CFE-level data that would be valid and available in relation to all pupils in the education system. If we had continued to capture data for the SSLN, we would have been asking the education system to generate ever more information, when we had judged what information would be useful and reliable to meet the tests that were applied by the OECD in its report on Scottish education.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 73, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152959

Dr Bloomer commented on continuity of data stating:

It is unfortunate that there has been no continuity in the information that has been made available in the past. We had a previous assessment called the Scottish survey of achievement, which ran for five years; a short interval and then the SSLN ran for six years; abandonment; and now we have a third system. A sample survey, which is what the SSLN was, is not incompatible with universal assessment of the kind that is provided by the new SNSAs.

Education and Skills Committee 23 January 2019, Dr Keir Bloomer (Royal Society of Edinburgh), contrib. 3, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147636

The Scottish Government's decision making does not appear to have placed emphasis on the gap in national performance data that has been generated in between the ending of the use of the SSLN and the establishment of ACEL as a reliable data source. There has also been a loss in continuity of data as ACEL will not create data that is comparable with SSLN data. In other words, the data from 2017 and before will not be comparable with data generated by ACEL. Therefore it will be challenging to identify longer term trends on changes in performance in the absence of baseline data.

The Committee observes that the Scottish Government places emphasis on the findings of the OECD in informing its decision to move to SNSAs and ACEL. The Committee also observes that the Scottish Government has decided to take actions that reduce the amount of valuable statistically rigorous data on Scottish education, for example discontinuing TIMSS and PIRLS, when the OECD places emphasis on the importance of the richness of data.

The Scottish Government has chosen to focus on literacy and numeracy including large new policy initiatives such as the SNSAs. The Government has also set out its intention to close the attainment gap and has committed to £750 million, over the course of this Parliament (2016-2021), of targeted funding through mechanisms such as PEF and the Attainment Challenge. The Cabinet Secretary described ACEL as 'key' however the information it produces currently has 'experimental' status, as reflected in the third ACEL statistical publication. On that basis statistically rigorous data is not yet available. The Committee notes that until ACEL data is deemed comparable by statisticians, there is a lack of a statistically rigorous performance mechanism at a national level through which to measure overall progress contributed to by these substantial policy initiatives.

The Committee wrote to the Cabinet Secretary seeking information on the cost of establishing SNSAs and running costs.¹Education and Skills Committee. (2019, January 25). Letter to Scottish Government. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20190125Out_ltr_to_DFM_re_ccost_of_SNSA.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20190125Out_ltr_to_DFM_re_ccost_of_SNSA.pdf</a> [accessed 17 April 2019] The response states that:

ACER UK was awarded the contract for the SNSA in October 2016 for an initial period of three years ending in October 2019. The estimated contract completion cost is in the region of £9m. This figure includes the initial start-up costs of £1.1m (suppliers development and initial implementation fee) and the annual running costs (assessment fees and management fees) which cost around £3.4m per academic year. The assessment fees are estimated, assuming a 100% uptake of assessments for P1, P4, P7 and S3 in the academic year (the year 1 uptake was around 94%). The £3.4m per annum figure includes the cost of payments made by ACER to its sub-contractors, SCHOLAR and Twig Ltd.

Cabinet Secretary for Education and Skills 12 February 2019. Response to Committee correspondence.

The Committee requested supplementary detail from the Scottish Government on the estimated and actual cost of the contract.ⁱWritten answer S5W-17209 includes an estimated figure of £8,813,753 for the 3 year contract. The response states:

The estimated value in the invitation to tender was £10 million over 5 years (a 36 month contract term with the possibility of extension for a further 24 months)...

The contract with ACER UK was awarded in October 2016 for 3 years, with the potential to extend by up to a further two years (on a +1-year +1-year basis). The core contract has since been extended by 9 months (rather than taking up the first full plus 1 year option), to bring it into line with the school year. The anticipated annual running cost is in the region of £3.4m per annum up to and including the 2019/20 school session, with the potential to extend the contract for a further year for the school session 2020/21. The cost of providing assessments for the 2021/22 school session onwards will depend on the outcome of a future procurement exercise to determine who will be awarded the contract to deliver standardised assessments.

Scottish Government March 2019, supplementary written submission

As stated above, the contract from October 2016 to October 2019 will cost in the region of £9 million. Should the SNSAs continue to have running costs of £3.4 million for the next 2 years (October 2019 to October 2021), the cost over 5 years since the ACER contract commenced would be in the region of £16 million. This is set against an initial estimate in the tendering exercise of £10 million over a 5 year contract. As the Scottish Government makes clear, the running costs may change, including depending on which provider delivers SNSAs after the next contract is tendered (the next contract could commence in June 2020 or June 2021). Another variable is the percentage of pupils that sit the SNSAs each academic year. On that basis the figure of £16 million should be caveated with the possibility that the annual running costs could be higher or lower than £3.4 million per annum.

In relation to value for money, Darren Northcott, NASUWT, highlighted that there would be a cost associated with alternative approaches at a national level:

If you were to replace SNSAs with some form of moderated teacher assessment, to which there would be some point, that could be workload intensive, too, and it could detract from other parts of the system. If we want to think about the value-for-money element of SNSAs or whether they are an effective use of resource, we have to think about what the costs of the alternatives would be.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 112, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151603

Darren Northcott also highlighted the costs associated with 28 local authorities undertaking some form of standardised assessment:

The points that have been made about value for money and cost benefit are really important. We know that there are costs associated with SNSAs. It is difficult to say what the cost of the alternatives would be, but it is important that we try to bear them in mind. For example, if the alternative to having SNSAs was to go back to the system that we had before, we should note that one of the costs of that system was the—quite substantial—cost of each of the 28 local authorities purchasing tests from different test providers.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 112, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151603

The Committee sought information from the Scottish Government on the cost of standardised assessments at local authority level in advance of the introduction of the SNSAs. It also sought details of any projections as to how these costs will reduce following the introduction of the SNSAs. Supplementary evidence received from the Scottish Government stated:

The Scottish Government has not carried out an analysis of the cost to local authorities and/or schools of buying in standardised assessments. Local authority provision presented an extremely varied and complicated landscape and we did not consider it practical to attempt to estimate the likely costs...

The SG has not carried out any projections of the reduction in costs to local authorities at this point or in years to come - we do not believe it is possible or practical to generate an accurate estimate.

Scottish Government March 2019, supplementary written submission

In moving to SNSAs an assumption was made that at a local authority level SNSAs would replace local standardised assessments. The Cabinet Secretary set out the previous approach:

Twenty-nine local authorities ran some form of standardised assessments, but they were not standardised across local authorities and they were not made bespoke for the curriculum in Scotland. They were invariably products that local authorities bought in to provide, within their area, some form of moderation on the performance of young people.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 112, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152998

On the basis of introducing a national approach, an argument made to local authorities was that the cost of standardised assessments in Scotland would no longer be borne by local authorities. Mhairi Shaw from ADES recalled:

Certainly, the publicity and the advice around the introduction of SNSAs suggested that they would save local authorities money, as they would not need to continue with the other tests.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 189, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143501

The SPICe summary of local authority submissions showed that a number of local authorities are no longer using their own standardised assessments, meaning anticipated savings of the SNSA policy at local authority level are already being realised in some areas. Others have removed authority level assessments but will allow individual schools to continue using their own assessments:

Eight authorities [out of 21 responding] stated outright that they do not use any other form of assessment, just the SNSAs. A further five authorities replied that while other standardised assessments have not been retained at local authority level, individual schools were using (or deciding whether or not to continue to use) other assessments.

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

East Renfrewshire reported that they were working on bridging their assessment work over to the SNSA. A number of local authorities highlighted forms of local authority wide standardised assessment that they did not state would be discontinued. In some cases this was because these assessments perform distinct functions to those of the SNSA. For example, West Lothian CAT assessments in S1 are a "cognitive ability assessment for all pupils".⁶SPICe. (2019, February). Summary of local authority responses. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf</a> [accessed 17 April 2019]

Glasgow City Council stated they use the SNSA alongside other standardised assessments (covering maths, progress in reading, and spelling) as they: “complement formative assessment approaches by adding reliability and rigour”. The Council highlighted that they do not use standardised assessments for all pupils.

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

The Committee heard evidence from Professor Merrell from the University of Durham to suggest some standardised assessments were being used along with SNSAs in a blended approach:

Assessments can be done at time points that are different from those for the national assessments. I have examples of schools that do that. They collect information from multiple sources to inform their practice. Assessments from the CEM are one example. I have the example from the current year of a primary 1 teacher who assesses her children with CEM assessments at the start of primary 1 because she wants some information about what they know and what they can do to inform her practice. Later in the year, she uses the standardised national assessments to confirm her judgments about where the children are. That is a nice blend of both assessments.

Education and Skills Committee 09 January 2019, Professor Merrell, contrib. 25, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143337

The Committee also heard evidence from the EIS to suggest new assessments were being introduced at this stage:

Some councils use standardised assessments in every single year of a young person’s time at primary school, and so they have not given up the in-between years. Others have—dare I say it?—introduced assessments beyond that in relation to PEF; they have started to use assessments that they never used before, which are paid for out of PEF money, to give them a benchmark for the start of the PEF process.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 124, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151615

An example in relation to PEF was provided in Moray Council's submission. The Council had schools which had continued to use PIPs/INCAS: "predominately...to track small cohorts within PEF funding... This has been for small groups and not whole stages".⁶SPICe. (2019, February). Summary of local authority responses. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf</a> [accessed 17 April 2019]

The Cabinet Secretary understood the basis for the continuation of some local authority standardised assessments. It was unclear whether he anticipated that these authorities would keep their own assessments permanently or discontinue them in the longer term:

...I accept that, in a limited number of cases, some local authorities—for good reason—are continuing with their previous systems, because of the line of sight that they are interested in establishing.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 180, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153066

The evidence from local authorities shows that the SNSA policy could, as intended, replace the majority of the standardised assessments undertaken at local authority level and school level over time. However this remains to be seen. For local authorities that move from their own standardised assessments to solely using the SNSAs, this represents a financial saving for these authorities since the SNSA is funded at a national level. The Committee wishes to caveat this by highlighting that the Scottish Government independent review is currently considering how the SNSAs operate at P1. Should the review recommend that P1 assessments should stop, the extent of the saving at local authority level may be reduced if local authorities choose to retain their own assessments in the first years of primary school.

Formative assessments, which are sometimes called assessments for learning, are where assessments are used to inform teaching practice. Summative assessments are an assessment of learning. Northern Ireland’s Council for the Curriculum, Examinations and Assessment (CCEA) stated that whether an assessment is formative or summative is not intrinsic to the test itself but relies on how the assessment is used. It stated:

What makes an assessment formative is what the teacher does with the information it generates. Unless the result of the assessment is used to change something in the teaching or learning, there is nothing formative taking place ... Formative and summative are not labels for different types or forms of assessment but describe how assessments are used.

A theme throughout evidence taking has been the importance of clarity of purpose in designing and implementing standardised assessments effectively. Professor Merrell from Durham University stated:

First and foremost, we have to consider the needs of stakeholders and establish the primary purpose of the assessments, before we get into the technical details of reliability, validity, content and so on; those need to be very clear from the outset. Different stakeholders have different needs: the learner may want to know about their current level of understanding and the next steps to aim towards; parents and carers need information; teachers are looking for various levels of information; and headteachers need management information, as do authorities at a national level. We need to be really clear in the first place about what we are conducting the assessment for, and move on to the quality and how we might best assess to get the information that we want.

Education and Skills Committee 09 January 2019, Professor Merrell, contrib. 12, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143324

Dr Bloomer set out how the features of the SNSAs must be linked to its purpose for the policy to be effective:

If the assessments are primarily to generate information about the system, they need to be able to fulfil that purpose, which points us in the direction of greater standardisation of approaches. If their purpose is diagnostic, that will not be important. It is a question of clarity about objectives first of all, and the rest follows on from that.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 94, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147727

The evidence from ACER, who designed the assessment according to the Scottish Government's specification, was that:

There are dual purposes. I do not think that we would subscribe to the view that there is a single primary purpose.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 54, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143366

A very important purpose is to give teachers good information about where children are in their stage of learning, which allows them to reflect on where those children are and to find out something new about them to help them to take the next steps.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 58, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143370

Another important purpose is to help the Scottish Government and the education community to improve the overall capacity of children in literacy and numeracy and to close the attainment gap. In order to have information about what the gap is and whether it is being widened or narrowed, one needs national-level data as well as data at the individual school level.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 60, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143372

The first purpose described by Juliette Mendelovits is formative. The second purpose described involves the production of summative data in that it supports an understanding of what has been achieved. Her evidence suggests that these purposes have had equal weighting in the development of the SNSAs.

Darren Northcott, NASUWT, suggested that at Government agency / inspectorate level there was some confusion between the cited purpose of the policy and how it was operating in practice:

I was taken by the evidence that the committee received from Education Scotland that said on one page that the tests are designed to be used formatively and not as summative assessments, then on the very next page talked about how the assessments have been used to form judgments about the effectiveness of a particular department in a school.

If there is confusion at that strategic level, it is not surprising that teachers and parents question the purpose of the assessment and how it should be used.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 11, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151502

At local authority level, ADES also reflected the use of the data, reflecting that education authorities would act on the data produced by SNSAs:

it might show that, at a school level, particular components, such as addition and subtraction, are not being taught particularly well. If we looked at that as a local authority and found it to be an issue across the authority, it would be incumbent on us to do something to bring about improvement, including by helping teachers to improve the learning experiences of youngsters. That is how the data is used formatively, and that summative information that we will get will allow us to do that.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 126, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143438

The Cabinet Secretary told Parliament on 5 September 2018 that the purpose of the assessment was formative, and not summative:

Scottish national standardised assessments are formative assessments. That is what they are—they are designed to inform teacher judgment. If they were the other type of assessment, they would be summative. If they were summative, they would be high-stakes testing. That is not what they are.

The fundamental point is that the assessments contribute to teacher judgment, and teachers across Scotland have been supported to deploy the assessments effectively in the classroom.

Meeting of Parliament 5 September 2018, John Swinney col. 28

In evidence, the Cabinet Secretary did not support the suggestion that the SNSAs have dual purposes. He did acknowledge that the aggregation of data created a summative element to the assessments:

The reporting demonstrates the range of performance against the elements of assessment within the SNSA...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 128, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153014

In the aggregation of data, the SNSA is summative, but its purpose is to be a formative assessment.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 132, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153018

Darren Northcott from NASUWT highlighted the confusion over the purpose of the assessments a number of times during his evidence, adding that:

They are described as formative assessments and then they are described as summative assessments. There is a legitimate policy debate to be had about which we want, but we cannot get one assessment to fulfil both tasks.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 72, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151563

The Committee notes the variation of answers it has received as to what the purpose or purposes of the policy are three and a half years after the policy was developed and announced, and what differing importance is placed on the formative and summative elements of the assessments. In other words the emphasis on information to support teaching and learning or on information on achievements in schools.

This is a large scale new policy which requires collaboration between national government, agencies, local government, senior management in schools and teachers amongst others. Effective implementation requires a co-ordinated approach and the cornerstone of this needs to be clear strategic direction about what the assessments are trying to achieve and for what purpose. All work to implement the policy stems from there.

Whatever the weighting different stakeholders place on the relative importance of the formative and summative elements of SNSAs, a fundamental question is can a standardised assessment which performs two purposes be effective? The SNSAs seek to use the same data for more than one purpose.

The OECD 2011 review highlighted that combining different purposes in one assessment carries risks and that this is a debated topic. Many countries have more than one stated purpose for national standardised assessments; however, the researchers and academic networks quoted by the OECD warn that doing so is problematic. The OECD quoted the Eurydice Network’s 2009 report on education in Europe which said, “assessment experts have warned that the use of a single test for several purposes might be inappropriate where the information ideally required in each case is not the same”.¹¹OECD. (2011). Student Standardised Testing: Current Practices in OECD Countries and a Literature Review. Retrieved from <a href="https://www.oecd-ilibrary.org/education/student-standardised-testing_5kg3rp9qbnr6-en" target="_blank">https://www.oecd-ilibrary.org/education/student-standardised-testing_5kg3rp9qbnr6-en</a> [accessed 17 April 2019]

The GL assessments submission stated that:

In his paper ‘Clarifying the purposes of educational assessment[1]’, Dr Paul Newton outlined how assessment can be used for 22 different, distinct purposes – ranging from formative and diagnostic assessments, to assessments that enabled various types of monitoring and accounting. His conclusion was that you should only use one kind of assessment for one thing; the moment you try to do more than that, it will not work.

GL Assessments. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181220GL_Assessment.pdf [accessed 17 April 2019]

[1] Paul Newton: Clarifying the purposes of educational assessment 2007

The Committee has explored with many of the witnesses the extent to which having one assessment that seeks to perform both formative and summative functions is an issue. Professor Hargreaves responded in terms of multiple uses of the same data:

There is a general principle that many, but not all, people accept, whereby data that is collected for one purpose should not be used for another, but that does not mean that data should not be collected for two purposes.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 10, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150031

Darren Northcott, from NASUWT which has international experience in relation to the differing approaches taken to standardised assessments, reflected:

...you said that the purpose of the tests could be to help teachers to make effective teaching and learning decisions about the next stages in the learning journeys of pupils and to get information about where the system is at the national, local and school levels. In other words, the one assessment would have a formative purpose and a summative purpose. That has never been achieved anywhere.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 11, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151502

Professor Hargreaves highlighted the somewhat experimental nature of the approach being taken in Scotland whereby national data is based on teacher judgment informed by standardised assessments:

Scotland is at the leading edge in that regard. It is good that you are watching the world, but the world is watching you. Figuring out how to make this a success over the next three years—it is possible that it might not be—and to be a learning Government as much as an improving Government is the key challenge for the Scottish Government.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 8, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150029

Dr Bloomer considered the idea of a hierarchy of purposes:

It is of course possible to use a single assessment to generate information of more than one kind, although, in doing so, you have to be careful that one purpose does not compromise the other. Therefore, it may not be necessary to say that the assessments serve only one purpose, but it is necessary to be clear about the hierarchy of purposes.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 94, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147727

The Committee explored the extent to which the conditions that would be set for the SNSAs to fulfil their formative function were compatible with the conditions required for collection of accurate data that may be used for summative purposes. This sought to assess whether setting conditions to fulfil one function compromised the ability to fulfil the other function.

The most common example provided in evidence related to when the assessments should be taken. The intention of the policy is that SNSAs can be taken at any time during the school year. For P1 for example, assessments are taken throughout the year meaning a child will be aged anywhere between 4 1/2 and 6 when they sit the assessment.

The first national report on the SNSAs was published in December 2018 and stated (p.9):

Given the possibility of administering SNSA throughout the school year, results from all learners should be interpreted with some caution when making any comparative judgements about individuals or groups.

ACER UK. (2018, December 11). SNSAS: national report for academic year 2017-2018. Retrieved from https://www.gov.scot/publications/scottish-national-standardised-assessments-national-report-academic-year-2017-2018/ [accessed 17 April 2019]

When asked whether, to improve the accuracy of the aggregated data, ACER would prefer that assessments were taken by all pupils at the same point in the school year, Juliette Mendelovits responded:

If that were the sole purpose of the programme, yes. Given that there are other purposes that are at least as important—namely, providing formative information to schools, teachers and individual learners of the kind that Sue Ellis has outlined—combining those desiderata is the way to move forward.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 94, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143406

Professor Paterson, when asked about the reliability of the aggregated P1 data where pupils can be tested at any point in the year, responded:

Where it is a problem...is in trying to aggregate the results to make interpretations about the system as a whole, the local authority or the school. If that is happening to an extent that we do not know about, it comes close to invalidating the results when they are aggregated to those levels.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 122, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147755

[In relation to P1] I would say that it would invalidate the data—it is too big a variation at that age.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 134, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147767

Susan Quinn from EIS suggested that if the assessments were solely focussed on the formative purpose, they would be taken at any time of year and could be optional:

If it was not being used for national comparators, a school would choose what it wanted to use it for. I was a primary school headteacher, and I know that my local authority has indicated that it is up to the school to determine when it will use the test in the year. I say “when it will use the test”, but actually I would always say “if and when”, just to make it clear that the test could genuinely be used as a diagnostic tool. You would be looking at which individuals you were absolutely confident were achieving the level from the evidence that you already had, which means that you would not have to waste their time or your own time with an additional assessment tool. It would genuinely be used as part of the assessment bank to inform teachers with regard to what they were doing.

Indeed, I would argue that, in some cases, it would not need to be used at all, as the body of evidence would show that the person was achieving the level.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 106, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151597

Timing of the assessment is one variable in the ways in which the assessments are administered across Scotland. Deciding on the timing of the assessments highlights that assessments would be approached differently if they were solely formative tools for teachers or solely used to provide local authority and national level summative data.

The Committee considers that the evidence from the Scottish Government and the evidence from the designers of the assessment, based on the Government's specification, is contradictory. ACER's evidence points to dual purposes with equal weighting. The Government suggests the primary purpose is formative, initially reflecting a position that the assessments were purely formative. The Cabinet Secretary then acknowledged that the information aggregated from assessments can be regarded as summative. The Committee notes that its work did not identify international examples where assessments, such as the SNSAs, perform formative and summative functions effectively in tandem.

Assessments can be used to measure performance of teachers, schools and local authorities. In some countries these results can inform “league tables”, for example tables are produced in the United States and in England. Proponents of this approach argue that this creates an incentive to improve performance and for schools and teachers to internalise the norms expected of them by stakeholders. Objections to this approach include that education outcomes are co-produced between families, students, the school and others and therefore schools would be held accountable for outcomes over which they have little control.

Furthermore, if a school’s performance is measured solely or mainly on a particular assessment, the assessment becomes “high-stakes” for the school and therefore will likely influence behaviours of the school. For example, teaching to the test or narrowing the curriculum. Some argue that this behaviour would invalidate the assessment as it would no longer provide a true picture of the quality of education. The Scottish Government has stated that it will not publish data from SNSAs at a school level. Its submission stated:

The SNSA is a diagnostic, supportive assessment that is designed to improve children’s learning, giving teachers helpful feedback on children’s next steps in aspects of reading, writing and numeracy. This is fundamentally different to other models of standardised assessment which are about ‘proving’ learning, with results being published.

Scottish Government. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181221Scottish_Government.pdf [accessed 17 April 2019]

The submission from the University of Glasgow School of Education, p.3 and 4, sets out the means by which standardised assessment systems need to be carefully designed and monitored to ensure information generated is used for improvements in learning and the impact of unintended consequences are minimised:

Assessment systems provide information and influence what people do. A National Improvement Framework influences the actions of those who work within it, policy makers, practitioners and researchers. A current challenge internationally is how to make that influence positive, ie, enhance learning (Hayward, 2015). Too many current national performance frameworks have not had a positive influence (Mons, 2009); there is powerful, consistent evidence that high stakes test-based monitoring systems lead to undesirable effects. Washback effects commonly include teaching to narrowly defined tests, narrowing the curriculum, teaching test behaviours, demotivating more vulnerable pupils and reducing levels of teachers’ confidence in their professional judgement and in their wider professionalism. The decision of the Scottish Government not to collect data from standardised assessment separately from evidence from teachers’ professional judgement was a welcome attempt to reduce the stakes of standardised assessment in Scotland.

Any decision about how to collect evidence at a national level has to consider the potential for washback and those responsible should seek to avoid predictable undesirable consequences and design a system where washback is positive, ie, leads to improvement in learning.

University of Glasgow School of Education. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181221Louise_Hayward_University_of_Glasgow.pdf [accessed 17 April 2019]

The Committee explored in evidence the extent to which the conditions for the SNSAs make them feel low stakes, and what factors may increase the stakes. Elements of practical implementation cited in evidence that can make young people feel that the assessments are higher stakes than they would be in optimum conditions include:

• groups of young people being taken to a computer suite to take the assessments;

it is an important point that the assessments become more high stakes if children have to be taken to ICT suites to be able to undertake them, such as in schools where wi-fi is not available.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 187, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143499

• assessments being taken during a narrow window in the school year:

...if the test is taken in a particular week of the year, an atmosphere starts to develop around it and it starts to attract stakes that no one wants it to have.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 88, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147721

• using technology that children are unfamiliar with:

The secondary teacher said their school had dealt with the issue of access to computers by purchasing 30 ipads which S3 pupils were adept at using... Two primary teachers suggested they were having a different experience, one suggesting their school had 70% pupils from SIMD 1 and 2 who do not tend to have access to computers at home, so sitting the assessment on a computer at P1 does not feel normal and can be daunting.

Education and Skills Committee 6 February 2019, teacher focus group note

A common theme of evidence in relation to whether teachers viewed tests as low stakes or higher related to their perception of how data from the assessments could be used as an accountability measure. Examples of this evidence are reproduced below:

The EIS representative suggested teacher feedback through their survey work on the assessments reflected that whether it was perceived as low or high stakes was dependant on how people considered the data could be used...

Education and Skills Committee 6 February 2019, teacher focus group note

The secondary teacher echoed the point on how the data is used being important. They had some reservations on how senior management teams in schools would interpret them, and more concerns about local authority level interpretation and use of data. The possibility of data being published under FOI, the influence of concerns on big online forums such as Mumsnet was, in their view, having an unsettling impact on teaching colleagues at their school.

Education and Skills Committee 6 February 2019, teacher focus group note

If I were a teacher in a local authority that had imposed a narrow assessment window on my school, I would be sceptical about claims that the purpose of the assessment was to help me to make professional judgments about the children I was teaching. That does not help to provide the clarity that is lacking about what the assessments are supposed to be for.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 141, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151632

Only doing SNSAs at 4 year levels suggests to me that it will eventually be used by Scottish Government for other purposes.

Taylor, A. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181206Alison_Taylor_Scottish_Borders_Council.pdf [accessed 17 April 2019]

The influence of perceptions of the assessment was also a theme in evidence relating to parents. The Committee heard evidence to suggest there was an importance placed on assessments, including in the media and in politics, that has increased the stakes for many parents. The NPFS criticised the Scottish Government's decision not to publicise the new SNSAs when they were introduced, saying: "when there is a vacuum of information it allows fear and anxiety to spread".⁷National Parent Forum of Scotland. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20181220NPFS_letter.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20181220NPFS_letter.pdf</a> [accessed 17 April 2019] Sue Palmer from Upstart Scotland set out some of the consequences of this anxiety in relation to P1 assessments:

That point about the genie being out of the bottle is particularly significant when it comes to P1, because the ratcheting up of parental anxiety impacts on the children. Within a year of the announcement that we would be testing primary 1 children, workbooks on how to help your child with P1 literacy and P1 numeracy had already appeared in the bookshops. As soon as people get wind of what is in the tablet-based tests, I dare say that there will be apps. That makes what is happening in P1 very high stakes, which is why something like a developmental checklist that the teacher goes through is much less distorting than a process that is linked to testing throughout the school system and which is highly specific to particular literacy and numeracy skills.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 115, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150136

Professor Paterson suggested that:

When parents start getting report cards that incorporate the results of the tests, the misunderstanding will go away. In fact, it will then be difficult for the Government or anybody else to go back. Once parents start getting the scale that has already been published on the Education Scotland website and in the ACER submission, they will wonder why they did not get such detailed information previously. [emphasis added]

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 111, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147744

Scottish Government officials provided information in this area:

There are no plans in place to provide “report cards” to parents based on standardised assessment results. Nor are we aware of any plans for reports to parents to take a particular form in future. The online assessment system does produce detailed feedback for teachers about where, within the assessment, a child or young person did well and where assessment outcomes indicate further support is required. It is important to note, however, that teachers remain best placed to know how the children in their class are progressing, based on their daily and ongoing interaction with those learners. Our expectation is that teachers will use standardised assessment outcomes, alongside a wide range of other assessment information, to discuss with children and their parents how learning is progressing and to identify next steps.

The Cabinet Secretary suggested that, should a parent specifically seek details of SNSA results, then it would be reasonable to share information:

...it will be up to individual teachers to judge what information to share with parents about pupils’ SNSA performances. If a parent is interested in information about how their child has performed, I see no reason why that could not be shared with them.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 232, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153118

The Committee sought perspectives from local authorities on sharing information with parents. An extract of the SPICE summary of responses is set out below:

...Moray Council stated that schools share ‘high-level’ messages with parents as part of progress reporting and has followed national guidance on how SNSA information should be shared. North Lanarkshire stated that it was aware of parents asking Head Teachers for SNSA results but that schools: “explained that the purposes of the assessments are to complement and ratify the results of the teacher professional judgements.”

...Inverclyde Council offered the below text from the guidance it has issued to schools: “Should I let parent know that the assessments are being administered? The link to the SNSA has a page for parents and it is advisable to make them aware of the link through your normal communication processes. This is simply one part of an ongoing assessment process and should not be given undue focus. The assessment is to support teachers’ judgements and parents should be reminded that the results form only a part of the ongoing assessments to form a judgement. As with any information held on a pupil, whilst parents have a right to know about individual assessments if they so require, the totality of the judgements should be discussed rather than individual test results in isolation.”

Glasgow, Midlothian and West Dunbartonshire reported that they did not have a policy on sharing the results of SNSAs with parents. Orkney Islands Council said: “schools are working with the service to evaluate the information before sharing”.

East Renfrewshire, Shetland Islands and South Lanarkshire Councils stated that information from the SNSAs would be directly shared with parents if they explicitly requested it...

Only South Ayrshire was definitive that, in keeping with previous practice concerning standardised assessments, it did not share the outcomes of the SNSAs with either parents or children...

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

Evidence from parent, teacher and children's representative groups highlighted a number of issues at a national level that can impact on the perception of assessments. The potential for assessment results at a school or local authority level being published under FOI was raised a number of times, the concern being that local league tables could be produced and be published in the media and conclusions drawn from them that would create an unofficial accountability measure. A snapshot of views are reproduced below:

The stakes increase, because the results can be the subject of FOI requests or gathered by journalists in order to create school league tables and the like, which was never the intention.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 138, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151629

school-level information is bound to find its way into the public domain whether we want it to or not, because of freedom of information. It would be far better to prepare for that...

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 95, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147728

There are pressures on politicians, local authorities, individual teachers and children in relation to how high stakes the tests are and to the semantics around the issue. If freedom of information requests are used to measure individual schools and, therefore, individual teachers, and to shape performance, and if that gets into the press, we will have a huge problem in how we consider Scottish education...

There is a real danger that the information that will be formed, judged and used from SNSAs will become distorted, and it will be out of the Government’s hands.

Education and Skills Committee 30 January 2019, Jackie Brock, contrib. 82, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150103

I understand the bureaucratic definition of high stakes, mid stakes and low stakes, but when the genie is out of the bag and parents have information that can help them to say where their child is, which the local press, councillors, ministers and the committee can also use, we have reached a high-stakes position, have we not?

Education and Skills Committee 30 January 2019, Jackie Brock, contrib. 114, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150135

Another theme on the influence of national actions on perceptions of the assessments was that political actions at Scottish Government level and in the Scottish Parliament can increase the perceived importance of the assessments, including in the media. A snapshot of views on this theme is below:

Ultimately, if a politician [the First Minister] stands up and gives a speech asking to be judged on their record and trying to tie educational improvement to election cycles, that will be the starting point for how things show up in the media.

Education and Skills Committee 06 February 2019, James McEnaney, contrib. 29, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151520

...SNSA is politically “a high-stakes policy”. That will affect public perceptions of it, which will affect what goes on in schools. If people feel under pressure to improve results, it is more likely that there will be the unintended consequences and behaviour that are often described as being related to testing.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 77, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150098

They [EIS representative] said that the political focus on SNSAs meant that their significance, given the small amount of the curriculum they cover, had been forgotten.

Education and Skills Committee 6 February 2019, teacher focus group note

Having worked in the Scottish Government and seen the maelstrom of panic and concern that arises from the annual publication of data—frankly, the media and politicians all collude in distorting the really good work that is being done in schools—I feel that we need to be extremely cautious about the impact of high-stakes testing and assessment and how we use those results nationally.

Education and Skills Committee 30 January 2019, Jackie Brock, contrib. 114, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150135

The University of Glasgow submission highlighted that international evidence reflects that perceptions can distort behaviours in the education system:

Even if data are not collected and published nationally, if there is a perception that data might be used to gauge performance in classrooms, schools, local authorities or nations, distortions are likely…

University of Glasgow School of Education. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181221Louise_Hayward_University_of_Glasgow.pdf [accessed 17 April 2019]

Professor Hayward added to this evidence when she appeared before the Committee, highlighting the long term impacts of divergence from the initial intentions of standardised assessments:

The history of every country that I have worked with internationally is that they start out with clear and coherent visions of what they want to achieve and, over time, divergence happens. As we do not go into the system to better understand why the gaps are beginning to emerge, it continues to develop until we get to a point when a new innovation has to come in.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 113, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147746

An example from a member of the IEAN, Professor Claire Wyatt-Smith, related to assessment in Australia (NAPLAN):

...the testing, now in place for more than a decade, was initially heralded as informing improvement efforts and supporting equity, the intent being to identify students at risk of not meeting the minimum standard. What is revealed in the paper is the undeniable, accelerating negative change (decline) in writing performance over the years 3, 5, 7 and 9, in all states and territories in Australia. Let me repeat: this phenomenon of decline in writing is across the country. The paper reveals the increasing numbers of young people falling below the benchmark or minimum acceptable level. I suggest caution in considering large scale testing of the type undertaken in Australia and the hazards ahead if the literacy tests from Australia are imported to Scotland and applied as they have been here. Importantly, the purposes of the test have become confused, with the case for measurement accountability and transparency effectively obscuring the original purpose of early diagnosis and evidence to inform intervention. The intelligent accountability argument has not been winning, sadly. [further details of the paper referred to is available in the full submission]

Wyatt-Smith, P.C. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125IEAN_Australia.pdf [accessed 17 April 2019]

Professor Hargreaves also reflected the impact of anxiety about a policy on the way it is carried out:

You need to build a culture within the teaching profession—among the headteachers and in the regional improvement collaboratives—in which all leaders clearly understand that the purpose is to learn and to find ways to keep moving forward, never to create a culture of fear or anxiety that will lead people to contrive the results.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 16, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150037

He later made clear, taking in proportion the potential for the stakes to increase for SNSAs, that:

The test is meant to be low stakes and is at risk of becoming medium stakes, but it is not at all high stakes.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 43, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150064

The Committee received a range of evidence on international approaches to standardised assessments, but has sought to focus on low and mid stakes examples as these are the useful comparators for SNSAs. James McEnaney also reflected on the experience in Australia:

In other countries that went down this road a few years before us, the tests themselves have had an impact well beyond what was intended. At this stage, that impact is relatively predictable. I direct you to Australia’s national assessment program: literacy and numeracy—NAPLAN—for example. Much of the comment that is coming from the teaching profession stems from that.

Education and Skills Committee 06 February 2019, James McEnaney, contrib. 27, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151518

Professor Hargreaves was integral to work in Ontario so his evidence included a focus on this example:

Ontario uses mid-stakes rather than high-stakes testing, so it provides one of the best examples that we might consider. A high-stakes approach means that assessments provide the power to intervene and to punish—for example, to remove a headteacher from a school or to close a school and reopen it as another kind of school. Ontario does not use such sanctions and provides a lot of support, but it has mid-stakes, which we probably have to pay attention to here. Knowledge of the results and their patterns can lead some school district directors to experience pressure from central Government to exert undue pressure on their schools to improve their results over relatively short periods. That creates all the negative impacts of large-scale assessments that we know of. Even in Ontario, the mid-stakes rather than the high-stakes approach produces some negative consequences.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 6, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150027

When assessing the SNSAs as being 'at risk of becoming mid-stakes', he highlighted the potential for unintended consequences of the assessments happening systemically:

The risk is that, if there is undue pressure from the Scottish Government or from local authorities to drive results up in a short period of time, in order to demonstrate success within a period of taking on leadership or before an election, that pressure will and does lead teachers to do strange but utterly predictable things.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 14, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150035

Two potential systemic impacts of the SNSAs, or forms of 'washback' as the University of Glasgow expressed it, was the narrowing of the curriculum and teaching to the test. A number of contributions have suggested that the focus on elements of the literacy and numeracy parts of the curriculum could have the effect of narrowing the focus of teaching away from the broader elements of the curriculum. Connect focused on this as an issue:

That narrows the focus and somewhat undoes the work that has been done on the assessment and the monitoring of progress in the curriculum for excellence.

Education and Skills Committee 06 February 2019, Lindsay Law, contrib. 9, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151500

My concern is that schools’ attention will be drawn away from some of the great work that they are doing on positive destinations by the pressure on them from local authorities, and by the pressure from official or unofficial league tables that might spring up off the back of the data from a narrow focus on literacy and numeracy.

Education and Skills Committee 06 February 2019, Lindsay Law, contrib. 15, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151506

Sue Palmer from Upstart Scotland looked at the narrowing of the curriculum specifically in relation to P1:

I adore curriculum for excellence because it tries—especially at early level—to nudge the Scottish system away from going in heavy on the three 3Rs as early as P1. That is a developmentally appropriate stage—much more like what you would see in northern Europe. Unfortunately, it has never really taken off, because we are still stuck in the cultural habit of starting the three Rs early. It horrifies me that, just as we are beginning to see some schools starting to move towards play-based pedagogy—developmentally appropriate pedagogy—in P1, the introduction of the SNSA will kill that in its tracks. The SNSA firmly puts the focus back on saying, “Get on with the literacy and numeracy skills. Crack on with it now.”

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 84, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150105

Previously, Singapore tested children at that age [six], but it has just abandoned that and it will not do any testing until after the age of eight, because it has realised that that changes the ethos of early years education in a way that is not productive for the children.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 107, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150128

Darren Northcott from NASUWT highlighted that the narrowing of the curriculum had reached a stage under the English system that the inspectorate has needed to 'recalibrate' its approach:

The problem comes if [literacy and numeracy] is all that they end up focusing on.

Take, for example, the experience from south of the border, where that has been a serious shortcoming in the education system. Even the Office for Standards in Education, which is the national inspectorate, now recognises that there has been a disproportionate emphasis on literacy and numeracy, which has been to the detriment of the rest of the curriculum. The inspection system in England is being recalibrated so that it is not just about assessing a narrow range of numeracy and literacy indicators, but about getting a broader balance.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 22, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151513

Professor Ellis set out in evidence that the key to managing or preventing the development of unintended consequences lay with the effective implementation of checks and balances alongside the implementation of SNSAs:

When talking about making low-stakes assessments, we need to look carefully at the checks and balances in the system. We should ask HMIE and Education Scotland, when they inspect schools, to ask parents about things such as teaching to the test and repetitive testing. That monitoring has to be built in. We also need to consider how the inspectorate thinks about, uses and talks about data and look at the language that we use in that regard.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 130, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143442

The Cabinet Secretary highlighted that the OECD's work had again been a notable influence in the Scottish Government's work, in this instance guiding the Government in designing assessments that seek to guard against unintended consequences, specifically:

We have learned from the Organisation for Economic Co-operation and Development’s 2011 report, “Student Standardised Testing: Current Practices in OECD Countries and a Literature Review”, which carefully maps out the benefits and dangers of standardised assessment and identifies themes and key lessons for countries to be mindful of when they go down that route.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 139, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153025

This work by the Scottish Government, and certain decisions taken as a result, received positive feedback in evidence:

I would argue that the shift to using the tests to lower the stakes and have them as part of the repertoire on which a teacher can draw is a positive move.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 107, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147740

To give the Scottish Government credit, its submission to the committee makes it clear that it recognises that there are dangers in high-stakes assessment and in narrowing the curriculum—we have seen the damage that that has done elsewhere.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 41, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151532

It should also be noted that the establishment of a broader measure of performance across more of the curriculum through ACEL, is intended to have the effect of limiting the development of an over-focus on certain elements of the curriculum that the SNSAs are based upon.

The perception of assessments is very important as, if assessments are perceived to be primarily an accountability measure, international experience shows that this can change the behaviours of teachers, school management and local authorities. These are the people that the successful implementation of SNSAs is reliant upon. The Committee has received evidence on the need to manage potential unintended consequences of introducing national standardised assessments. The Committee acknowledges that, by allowing the assessments to be taken at any point in the academic year, the Scottish Government has sought to limit the potential for the assessments to become high stakes.

As highlighted above the SNSAs cover elements of the literacy and numeracy part of the curriculum. The Cabinet Secretary highlighted that, as well as being bedrock competencies for much of the curriculum, literacy and numeracy have also been identified as priorities for improvement. Other witnesses explored the extent to which the SNSAs covered the curriculum and the value of focusing on these areas:

The test is not a test of everything but is a test of literacy—and not even all literacy, because it is primarily a test of comprehension and reading. If developing reading to a certain degree is important in your curriculum, the tests will have some value.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 41, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150062

In primary school, it is true that the tests assess only certain aspects of attainment. In some respects, however, those aspects are fundamental to any other progress being made. Unless a child can do the elementary operations of arithmetic, they will never make progress in any other aspect not only of maths but of science and many social sciences, too.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 130, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147763

Literacy and numeracy are foundational—that has to be recognised. One of the many professors who has given evidence to the committee said that although the areas of assessment for SNSAs are narrow, they are quite important. That has to be acknowledged.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 22, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151513

Professor Hayward highlighted that "SNSAs give you information about very limited areas"ⁱEducation and Skills Committee 23 January 2019, Professor Hayward col. 33 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11910&mode=pdf and also explained:

The idea of having information from tests that supports teachers’ professional judgment is an entirely appropriate approach. The issue is, however, that we have to decide what matters. If it is the curriculum for excellence, our assessment system should reflect all that matters in CFE. We have to find ways of gauging how much and how well children are learning in relation to all those processes.

Education and Skills Committee 23 January 2019, Professor Louise Hayward (University of Glasgow), contrib. 4, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147637

Dr Bloomer added:

...the assessment looks at a restricted area of the curriculum once every three years.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 15, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147648

...my overall conclusion is that the form of assessment does not yield a wide range of valuable information. It is not without value, but it is limited.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 26, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147659

Other witnesses suggested that there was sufficient information present in the Scottish education system already to inform teacher judgment in relation to literacy, numeracy and other elements of the curriculum. Susan Quinn, EIS, spoke of a 'broad assessment bank' that already exists. In relation to P1 she stated:

There are benchmarks that schools and early years establishments use for young people from the very earliest stages. The widest possible assessments go on...

Very few primary schools do not assess young people on their entry to primary 1...

At the end of primary 1, teachers make a professional judgment of the young person’s achievement at the early level based on three years of assessment of the young person. It is not done in, as we are generally told, a single half-hour session to assess their literacy and a single half-hour session to assess their numeracy, which would not give us information for closing the attainment gap. The data will always come from teachers’ professional judgment and informal assessments, and from the informal interactions that happen day and daily between teachers and pupils, between support workers and pupils, and between parents, teachers and everybody who is involved with the young person.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 14, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151505

As set out in the introduction, formative assessments are used to inform teaching practice. Diagnostic assessment, linked to formative assessment “often takes place at the beginning of a learning programme and can be used to identify pupils’ strengths and areas for improvement”ⁱNICCEA webpage, What is formative assessment?.

The Cabinet Secretary emphasised the importance of the diagnostic function of the assessments to teachers and by extension to improving the performance of the education system:

The fundamental point about SNSAs is that they are related to the performance of individual young people. They are for diagnostic purposes, to assist teachers in supporting young people to achieve curriculum for excellence levels. That improvement, in itself, helps to improve the performance of the Scottish education system.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 114, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153000

The Committee received some positive evidence on the quality of the information provided by the assessments to assist teachers in supporting learning. A number of perspectives, highlighting distinct qualities of the information, are set out below:

The Scottish National Standardised Assessments are of a high technical quality, with levels of statistical reliability that are satisfactory by the standards of good-quality testing. Notably, the reliability is high at every relevant school stage, including at P1. (Statistical reliability in this context may be thought of as a measure of the extent to which a test consistently measures what it is intended to measure, which in this case is attainment according to the criteria in Curriculum for Excellence).

Paterson, P.L. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181206Professor_Lindsay_Paterson.pdf [accessed 17 April 2019]

Standardised assessment can, under specific circumstances, raise pupil attainment. The evidence is from the USA where researchers investigated attainment rises across different states as they adopted standardised assessment policies at different times (e.g. Wong et al 2015). Polikoff, Korn and McFall (2018) indicate that a productive system requires assessments that generate data across a breadth of desired outcomes (which the SNSA does; in fact, the SNSA literacy outcome measures are far broader, and the results are given to the class teachers more quickly, than those of the other popular standardised assessments previously bought by local authorities.)

University of Strathclyde. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181218University_of_Strathclyde.pdf [accessed 17 April 2019]

The information available to teachers from the SNSAs is exceptional. It provides details of a child’s specific skills, knowledge and understanding, which enhances the teacher’s judgement when identifying strengths, as well as areas that might require more focus.

National Parent Forum of Scotland. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181220NPFS_letter.pdf [accessed 17 April 2019]

As cited elsewhere in this report, practical considerations that diminish the formative value of assessments include where SNSAs are not taken at a time of the teacher's choosing based on the pupil's needs, or of course where teachers have insufficient time to consider and act on the resultsⁱsee Alison Taylor submission in relation to P7 assessments in May and the transition to secondary school. There were also some criticisms of the assessments when judged on technical formative qualities. James McEnaney highlighted comments from Professor Dylan Wiliam that suggested the assessments had limited formative value, particularly at P1ⁱⁱEducation and Skills Committee 6 February 2019, James McEnaney col. 7 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11938&mode=pdf. GL Assessments, which provides assessments for use in schools in Scotland compared its tests with the Scottish Government assessments:

We also have a range of assessments to help uncover barriers to learning: whether that means engagement and confidence in learning or learning difficulties like dyslexia or working memory. The lack of formative and diagnostic elements in the Government’s tests highlight their weaknesses and without these elements, their usefulness when it comes to supporting pupils’ next stage of learning is limited. Our assessments are used at teacher level to assist them identify pupils’ areas of need and then to support implementation of support programmes as appropriate.

GL Assessments. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181220GL_Assessment.pdf [accessed 17 April 2019]

Moderation of teacher judgment is of central importance in ensuring accurate information on achievements is provided and collated to create the ACEL. SNSA data is one source of information to inform moderation amongst a lot of other moderation activity. One of the central messages from the International Council of Education Advisers to the Scottish Government is the importance of collective autonomy amongst teachers, and the role moderation plays in this:

In the teaching profession, we need collective autonomy, not individual autonomy—we have argued about that here. That means that we might have more autonomy from the bureaucracy but less autonomy from each other. By looking at the ways in which we make judgments together and moderate them, over time, we will create some consistency. The data can help teachers to do that. However, the data will always be imperfect,...

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 14, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150035

...the research on professional development in the UK and the US shows clearly that the best professional development is on-going, embedded in the profession, collaborative and seen as directly related to the learning. If the leaders of your schools and local authorities continuously bring together their teachers to look at what is happening as regards the judgments that are made, based on all the data that they receive, that will create consistency between the individual feedback and the national-level trends.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 28, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150049

He also focussed on the idea of unconscious bias, that effective moderation seeks to guard against:

We are all subject to unconscious bias, and we all tend to prefer people who remind us of ourselves. Getting consistency of judgment means that a student, at whatever stage they are at in any class, will get a reasonably equal and professional response from the teachers who deal with them.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 12, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2156312

The SNSAs allows pupils to be benchmarked against national averages. This feature, which was not previously possible in the absence of a national form of standardised assessments, assists moderation and received positive feedback in evidence:

I will make an initial point on teacher judgment. One effect of the tests is that it may assist teachers in relating their own judgment to national expectations and standards. That is quite helpful in itself.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 54, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147687

There was much emphasis in the Cabinet Secretary's evidence to the Committee on the importance of teacher moderation and the specific contribution the SNSAs can make in this regard:

The SNSAs provide the statistical consistency and rigour to inform moderation across the country.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 57, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152943

They provide the moderation of performance across the system...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 59, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152945

It is worth highlighting at this stage that, while the SNSAs are intended to be taken by all state school pupils, it is only teachers who teach certain year groups, and in secondary school it may only be teachers who teach particular subjects in S3, that will use the information to inform their judgement. Therefore, if moderation of teacher judgment is one of the main benefits of the SNSAs it should be acknowledged that large numbers of teachers will never directly use the information generated by the SNSAs. Alison Taylor, a teacher, stated:

Snap shots at P1, P4 and P7 do not help us in the other years. Teachers need to assess pupils all through their years at school. It is part of the teaching and learning cycle.

Taylor, A. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181206Alison_Taylor_Scottish_Borders_Council.pdf [accessed 17 April 2019]

Jackie Brock from Children in Scotland highlighted the work towards improving moderation that pre-dates the SNSAs, which she suggests should be valued and prioritised as opposed to undertaking SNSAs:

In 2005, the “Assessment is for Learning” guidelines stressed the importance of teacher judgment, supported by a range of assessment tools, which would be decided on locally...

A huge amount of development has taken place since 2005...All those principles were later reinforced under the 2011 “Building the Curriculum” guidance and there was a strong amount of reinforced pressure around moderation.

On the purpose of assessment and guidelines, and what we are actually doing with the information, it is interesting but frankly disappointing that we are not hearing about the thriving moderation that is going on in Scotland...

For some reason, we have decided that we do not believe in valuing and strengthening teacher judgment and moderation, strengthening assessment, and building on our learning strategies. We have decided that we do not believe in all that; instead, SNSAs are the way forward in removing inequity.

Education and Skills Committee 30 January 2019, Jackie Brock, contrib. 99, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2151989

Similar positions were set out by Professor Hayward and Susan Quinn from the EISⁱEducation and Skills Committee 6 February 2019, Susan Quinn col. 18 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11938&mode=pdf, who suggested a focus on SNSAs as the basis for moderation could detract from other valuable work:

We have a national self-evaluation system that is moderated by critical friends. We have a great deal of evidence in the system and, if we focus on only one tiny element, we risk ending up with a less-dependable judgment than we might have had if we had paid attention to the range of sources of evidence that are available to us.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 90, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147723

The Cabinet Secretary's perspective on teacher moderation and SNSAs was that the introduction of CfE benchmarks have provided greater clarity as to what the achievement of CfE levels looks like in practice.

The feedback that I get from the profession is that benchmarks have now made a significant impact in stabilising that understanding of what the levels are and, as a consequence, the profession is more confident about what the levels look like.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 195, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153081

The EIS suggested that this increased clarity in standards for teachers to moderate their judgment against diminishes the need for SNSAs:

The [ACEL] information is becoming more reliable as teachers and schools work together to moderate the information and get a better understanding of the benchmarks, which were introduced only in the past year or two—we were working with a curriculum, and then benchmarks were introduced after the fact, so a lot of work needed to be done.

That could have been resolved before the SNSAs were introduced, and I argue that it should have been resolved. Teachers should have been given the time, training and space that they needed to better understand the levels and the standards that were required to make the system more reliable. If they had, we might not have needed to have gone with the SNSA.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 7, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151498

A number of witnesses also placed importance on the role of inspection in assessing the standard of moderation work in Scottish schools:

The fact is that the way to get a standardised, equitable approach to assessing learning and teaching across the country is by looking at moderation practices and how they are inspected, to ensure that everyone is working to the same standards. The SNSA will not fix that. It will not do that. It deals with 10 per cent of the curriculum at a really narrow point in time—

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 83, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151574

It is not just about assessments; inspection has a really important role to play in giving policy makers at national and local levels an understanding of progress in the education system...

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 18, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151509

A final issue is that assessment is not the only part of the education system. It is the responsibility of education authorities to ensure that the quality and standard of education in schools are appropriate. We therefore have quality assurance officers and school inspectors—we have lots of sources of information that come together to give a picture of performance in a school.

Education and Skills Committee 23 January 2019, Professor Hayward, contrib. 88, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147721

Certain witnesses appeared to have different perspectives on the extent to which information from SNSAs should influence teacher judgment. Professor Paterson made the case for SNSAs being used to improve consistency by addressing teacher bias:

Collecting data that is neutral and reliable is always better than not having it. Whatever their faults may be, the new tests—or assessments, as we are supposed to call them—are more reliable, neutral, objective and independent of bias than anything that we have previously had in Scottish education in recent decades. I say that because all of us who teach—I include here university teachers as much as any other sector of schooling—are unavoidably subject to bias, which is sometimes unconscious. We know that when we do not allow students’ essays to be marked anonymously, there will be bias—for example, against women, whose assessment is more accurate when it is done anonymously. That is an illustration of the bias that all teachers inevitably have.

Education and Skills Committee 23 January 2019, Professor Lindsay Paterson (University of Edinburgh), contrib. 5, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147638

I referred earlier to the unavoidable bias that all teachers have. One way in which teachers can try to improve their judgments and correct for bias is by looking at objective data and comparing their judgments with its results. That is what other professionals do all the time, and teachers should do it. In that sense, although the SNSA’s measures look only at part of what a child can do, they are valuable.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 40, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147673

This would suggest a relationship with the data emerging from SNSAs whereby teacher judgment could be corrected by the results generated by a particular student, because the data would have identified the pupil's true ability free from the unconscious bias of the teacher. In this example, the teacher's judgment is being led by the information generated by SNSAs.

Professor Hargreaves described a situation where teachers are informed by a number of pieces of information, of which SNSAs are only one, and where ultimately the teacher's own judgment is the strongest factor informing their decisions:

The third approach is to consider the genuine importance of having an array of assessment measures and data, of which such tests are simply one part. Teachers’ judgment must have primacy all the time. If the situation does not get there or starts to deviate from there, there is the serious possibility that the great experiment will have failed.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 18, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150039

Professor Hargreaves' view would appear to align closely to the Cabinet Secretary's who stated in evidence:

The SNSA informs the process but does not dictate it. It is determined by the professional judgment of the teacher.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 220, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153106

A number of pieces of evidence have suggested that some teachers have not yet found that the information generated by the SNSAs has been of notable added value in informing their judgment, rather it contributes alongside other factors:

When asked if there was a value added by the SNSAs the responses suggested it was something else, amongst other measures, to back their professional judgment.

Education and Skills Committee 6 February 2019, teacher focus group note

It should be considered in the midst of all the other assessment information that a teacher will use daily about a child’s performance against the curriculum and the activities that have been set for that child to make progress with the curriculum.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 51, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143363

...in a large majority of cases, teachers found that the SNSA data confirmed their judgment of pupils' capacity. (Shetland Islands Council)

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

I do not feel that the results give any new information to teachers although they make interesting reading.

Anonymous teacher 3. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_3.pdf [accessed 17 April 2019]

...we are more than able to detail the progress of our pupils without these tests which fail to take into account the many factors affecting our children's performance on the day.

Anonymous teacher 1. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_1.pdf [accessed 17 April 2019]

The Cabinet Secretary responded to this thread of evidence in his opening statement to the Committee:

The information that the assessments generate is used by teachers expressly to direct to best effect the next steps in learning for individual learners...

On the comments that have been advanced about the SNSAs telling teachers nothing new, one perspective is that that is a strength, rather than a weakness, as it indicates that the assessments are correctly pitched and that teachers’ judgments of individual learners’ progress are predominantly sound. Equally, comments have been made about the value that SNSAs provide in ensuring moderation of performance across the system, resulting in greater confidence in the profession and vital diagnostic information when aspects of assessment outcomes suggest greater or lesser capacity than a teacher has expected.

Education and Skills Committee 20 February 2019, John Swinney (Deputy First Minister and Cabinet Secretary for Education and Skills), contrib. 38, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2152924

The Committee also heard evidence emphasising the need to ensure teachers felt there was sufficient trust in their abilities to form accurate judgments about their pupils. This builds on the idea of the importance of "autonomy from the bureaucracy" highlighted by Professor HargreavesⁱEducation and Skills Committee 30 January 2019, Professor Hargreaves col. 9 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11926&mode=pdf.

The class teacher is better placed to make judgments about the children they know and should be trusted to do so.

Anonymous teacher 3. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_3.pdf [accessed 17 April 2019]

A final theme in relation to the extent to which teachers attach value to the assessments and therefore would use them to inform their judgment is the accuracy of the link to the curriculum in the assessment content. For example the benchmarks that the assessments are based upon were questioned:

In my role as a [Support for Learning Teacher] I value and use assessments of pupils’ skills and knowledge to plan interventions and next steps in their teaching & learning. I am not in agreement with some of the CfE benchmarks and the Levels that some are allocated to, so this is reflected in my opinion of the content of the SNSAs. The SNSAs are based on the benchmarks which do not always tell me details about the core skills and knowledge of pupils that I need to know about with regard to learning to decode or encode. The science and evidence behind learning to read is not solely reflected in the benchmarks. In my opinion some benchmarks which are included are misguided as they are based on whole language/balanced literacy ideas.

Taylor, A. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20181206Alison_Taylor_Scottish_Borders_Council.pdf [accessed 17 April 2019]

Professor Paterson talked about SNSA—I am sorry, but I say “sensa”, because that is what teachers call it; I can never remember all the letters. He said that we have based SNSA on the curriculum. We have not; we have based it on the benchmarks, and the benchmarks for P1 are extrapolated from the experiences and outcomes. That extrapolation is quite distorting. There are 54 benchmarks for literacy, 22 of which relate to speaking and listening—that is nowhere near enough; speaking and listening are big things—and 32 of which relate to specific literacy skills.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 79, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2151988

The original experiences and outcomes use words such as “explore”, “play”, “discover”, “choose” and “develop”—they are major verbs. Once we drill down and turn those words into specific tasks, we move away from a holistic developmental approach to early education, which is what curriculum for excellence is about, to a drilled-down, skills-based approach. If teachers look, as I suspect they will, at the benchmarks rather than at the experiences and outcomes, that will affect whether they achieve curriculum for excellence levels assessments.

Education and Skills Committee 30 January 2019, Sue Palmer, contrib. 79, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2151988

Presented with evidence that some teachers do not consider SNSAs are providing additional useful information, Professor Hargreaves responded:

That should be a warning, and it should prompt the Scottish Government to work with the Australian Council for Educational Research on what the tests contain. If the breakdown of skills is not seen as valuable or useful, teachers must be able to say collectively what skills and competences should be represented in the tests. The evidence is not a reason to do away with the tests, but it is a reason to ask what kind of test is most valid for the skills that are important for CFE.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 30, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150051

A regular theme of evidence in previous Committee inquiries this Parliamentary session is the number of priorities putting pressure on teaching time. Scottish Government policy at a national level should reflect the existing pressure on the education system. The Cabinet Secretary made clear in evidence that "I constantly wrestle with the issues of workload".ⁱEducation and Skills Committee 20 February 2019, John Swinney col. 34 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11948&mode=pdf One submission highlighted the impact multiple policy initiatives can have:

It's also unfair to our young people. I find myself constantly apologising to pupils when yet another change to the system has to be catered for. Please refrain from further changes, promote and allow the professionals to carry out the work you have employed them to do.

Anonymous teacher 2. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_2.pdf [accessed 17 April 2019]

Context for considering the amount of time taken by teachers, other school staff and local authority staff includes that the SNSAs will replace local authority standardised assessments in many areas and this means additional work from SNSAs will be offset for many teachers by the removal of other assessments. Darren Northcott, NASUWT, stated:

An important part of the narrative is that one national approach replaces 28 different approaches to standardised assessments. There is no question but that that creates challenges, but we should not pretend that there were no imposed standardised assessments in schools before, because there was an awful lot of that.

Education and Skills Committee 06 February 2019, Darren Northcott, contrib. 90, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151581

As discussed earlier in the report, the number of teachers required to administer local authority or school level assessments is not known so the extent to which more teachers are required to administer assessments under SNSAs is unquantified:

An EIS representative suggested that assessments pre-dating SNSAs were not uniform, the application across the country was patchy with a lot of schools rejecting the idea in favour of formative assessment.

Education and Skills Committee 6 February 2019, teacher focus group note

Where SNSAs are taking additional time for certain teachers, the Committee heard evidence of the opportunity cost of undertaking assessments in relation to teachers' time. A converse argument from Professor Sue Ellis was that because of the pressures on teacher time performance issues can sometimes be overlooked and the quality of the SNSA data can guard against this:

In a class of 25 or 30, it can be very easy to miss poor oral story comprehension.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 52, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143364

A selection of evidence highlighting the opportunity cost of time spent on SNSAs is below:

• teaching time

The amount of time and resource diverted to administering these tests is also onerous, amounting to days and days of teaching and learning support time lost.

Anonymous teacher 1. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_1.pdf [accessed 17 April 2019]

We also got feedback [from consultation with parents and teachers] on resource time, by which I mean teachers’ time. It is a hugely costly approach. If there are three P1 classes, they will have three teachers, and there will be another three teachers taking children out one at a time alongside two support for learning assistants and another assistant. We are talking about a huge amount of resources and full-time equivalent teaching time being used for tests that are simply for gathering a baseline and, indeed, which are not comparable, given that teachers can pick when children between the ages of four and a half and six do them.

Education and Skills Committee 06 February 2019, Lindsay Law, contrib. 101, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151592

• supporting those with additional support needs

I am a teacher of Additional Support in a school of approx. 350 pupils... Since SNSAs have been introduced, at the instruction of the Head Teacher, I have had to stop my timetable of additional support to carry out P1, P4 and P7 SNSAs according to the 'window' of time we are given by [the] Council...The disruption to pupils in need of additional support is unfair.

Anonymous teacher 3. (2019). Submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_3.pdf [accessed 17 April 2019]

The Committee also heard evidence of other school staff administering assessments other than pupils' teachers in order to manage staff time in schools. This detracts somewhat from the concept that teachers interpreting the SNSA data are aware of the conditions students sat them in, including their frame of mind at the time:

Orkney [Islands Council] reported that staff were 'generally comfortable' with the administration of the SNSAs, which was done class by class and often carried out by Head Teachers or senior management.

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

We know that a significant number of schools directed their whole senior management teams to administer the tests for two or three weeks. We had reports of teachers giving up their non-contact time...

More worrying, a good number of people suggested that support workers who were being paid for out of PEF money were being redirected to administer the assessments or to support the teachers in some way in administering them.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 110, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151601

One teacher said teachers in their school had not been informed the assessments were taking place, an attainment adviser took the pupils to sit them.

Education and Skills Committee 6 February 2019, teacher focus group note

The Scottish Government's oral evidence set out the work undertaken in the development of the assessments to ensure that additional support needs considerations fed into the process. On request the Government then provided supplementary information to the Committee that stated:

In the spring of 2017 the Scottish Government’s Digital Transformation Service, in conjunction with ACER, conducted usability testing sessions with sixteen learners in three schools across the country with a range of disabilities including: hearing impairment; visual impairment; Attention Deficit Disorder (ADD); Asperger's Syndrome; language difficulties; reading difficulties; and anxiety.

Scottish Government. (2019, April 17). Supplementary submission. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/Scottish_Government_supplementary_information_addressing_needs_of_ASN_learners_paprticipating_in_SNSA.pdf [accessed 2019]

Some evidence received highlighted concern from teachers about the experiences of children with additional support needs taking the assessments, and the value of information that is gathered:

The headteacher had watched some students take the assessments including one with additional support needs, and saw a young girl being hesitant and taking a long time to take the assessment and then guessing answers quickly. On that basis the headteacher was able to understand why the results were not reflective of that particular pupil’s abilities.

Two primary teachers suggested that if they had the discretion they would not make pupils with English as a second language sit the assessments. The secondary teacher said they were from an inclusion school and found that the technology SNSAs used was a much bigger issue for those with additional support needs.

Education and Skills Committee 6 February 2019, teacher focus group note

The potential that results are not reflective of a child's capabilities because they become disengaged with the assessments, as mentioned by the headteacher, was also mentioned in a submission from a teacher, suggesting pupils become: "'click happy' just to move on and, as I have witnessed, if they randomly choose the right answer the assessment results will not be accurate".²Anonymous teacher 3. (2019). Submission. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_3.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20190125Anonymous_Teacher_3.pdf</a> [accessed 17 April 2019] This may be an issue where a particular condition impedes a pupil's ability to use technology.

ACER highlighted the work undertaken to ensure SNSAs take into account the needs of those with additional support needs:

We have implemented a lot of affordances in the programme to help children who have a visual impairment or motor skill needs to allow them to do the assessment. That was clearly a very high priority in the Scottish Government’s requests. We have had a lot of workshops and consultations with accessibility experts in and beyond Scotland on how to make the assessments accessible for children, and we have used the AA measure in the web content accessibility guidelines...

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 132, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143444

A line of questioning raised through a number of evidence sessions was the potential for the different formats that questions can be taken in to generate read-outs reflecting different levels of performance. A particular assessment question received focus during a number of evidence sessions. It related to the ability to identify a rhyming syllable in words by reading them, when an alternative format available was to identify the rhyming sound by listening to the words.

It was argued that the accommodations that allowed a child to answer questions in a different format, could in some circumstances have made it easier for them to get questions correct than if they had answered the questions in the standard format. The format each question is answered in does not appear to feature in the read outs available to teachers. In the absence of this information, it was suggested, the use of accommodations could actually inhibit a teacher's ability to accurately assess the ability of a child. This includes the possibility that the results could provide an overly positive reflection of the child's abilities in certain areas.

In response to the possibility the assessments could have this effect, ACER responded:

When we introduce affordances in the assessments to allow children with additional support needs to take the assessments, we are always conscious of the key intent of the question, and we do not adjust it in such a way as to obliterate what it tries to measure.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 134, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143446

This point was pursued with a Scottish Government official during the Cabinet Secretary's evidence session. In his response, David Leng gave a detailed explanation as to why hearing the word did not invalidate the particular SNSA question being discussed.ⁱEducation and Skills Committee 20 February 2019, David Leng Scottish Government col. 30-1 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11948&mode=pdf

In response to the point about the value of a teacher knowing what format a question is answered in Professor Paterson suggested that:

In principle, I agree that a teacher would want to know [which format of question was answered by a pupil]. Of course, whether it matters is an empirical question. The teacher would need to have information about whether the child had responded to the written form or the aural form, and they would then have to see whether one gave a better assessment of the child’s overall ability in language. With that information, the teacher could make the decision; it might turn out to make no difference or to make an enormous difference.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 60, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147693

The lack of detail in readouts for teachers on the exact format different questions were taken in, is a limitation in the current design of the SNSAs. The accommodations made in the design of the test when combined with an inability of the teacher to be able to assess which format a child answered in, is a consideration when assessing the usability and value of the assessments for those with certain additional support needs.

Responses from local authority Directors of Education suggested that SNSA data was being positively received and used in some areas, and that it was too early to assess how valuable the data would prove in others. Data can be viewed by a council down to individual level results, as well as school and regional level. It can also be interrogated by cohort, such as SIMD, ASN, gender etc. Councils cannot view data from other councils unless they seek and get permission to view it.

West Lothian used P7 and S3 results to 'inform next steps and curriculum decisions'. It set a window for assessments in P4 and P1, between February and May, to inform progress and next steps at the point of transition within the same establishment. West Lothian suggested "this has been very well received by school staff and promotes collegiate approaches to analysing data to inform improvements at class and school level".¹SPICe. (2019, February). Summary of local authority responses. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf</a> [accessed 17 April 2019]

South Lanarkshire explained how SNSA and other data combined to analyse data based on characteristics, using it to inform local authority improvement, set priorities and has "been very useful in helping us to further identify and analyse our poverty related attainment gap".¹SPICe. (2019, February). Summary of local authority responses. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf</a> [accessed 17 April 2019]

Glasgow City Council expressed caution about use of data commenting: "it is too early with just one year's data to be able to assess as to whether this information is useful and how effectively it would contribute to improvement. Orkney Islands Council stated that it is "anticipated in due course" that SNSA data will be used to support learning and teaching."

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

The SNSAs provide information in a format that was not previously available to teachers where national standards are linked to the curriculum. Although the Committee notes the evidence that questioned the interpretation of Curriciulum for Excellence levels when converted into the benchmarks that SNSA questions are based upon.

The Committee is encouraged by early indications from some local authorities that the data generated by SNSAs is proving useful when interrogated and used alongside other pre-existing datasets.

SNSAs cannot diagnose additional support needs. The Cabinet Secretary confirmed this in his evidence to the Committee. The Committee is concerned that the impression that they can has emerged including during the course of recent debates on P1 assessments. The Committee highlights this point to help to begin to counter this misinformation. There are other assessments in existence that can help to diagnose different conditions (for example dyslexia) but the SNSAs have not been designed for this purpose.

The OECD 2011 review noted that “without adequate training, teachers may not have the assessment literacy and ability to appropriately interpret results and to identify areas where curricular strategies may require adjustment”.¹OECD. (2011). Student Standardised Testing: Current Practices in OECD Countries and a Literature Review. Retrieved from <a href="https://www.oecd-ilibrary.org/education/student-standardised-testing_5kg3rp9qbnr6-en" target="_blank">https://www.oecd-ilibrary.org/education/student-standardised-testing_5kg3rp9qbnr6-en</a> [accessed 17 April 2019]

ACER set out in evidence to the Committee the teacher training that was initiated at the beginning of the assessment programme and how it is being developed over time:

The training programme that SCHOLAR at Heriot-Watt University is running, which has been implemented alongside the assessments from the beginning as part of our contract, is therefore a key element. It is unusual internationally that there was the foresight to bring forward a professional development and training programme from the inception of the national assessments to ensure that they were used in the way that was intended.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 15, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143327

SCHOLAR is developing the professional learning programme as the SNSA matures. In the first year, a lot of it was about the more technical dimensions, such as how teachers accessed the assessment, designed log-ins for the children and downloaded the reports. Increasingly, the emphasis will be on the interpretation of reports and what teachers do with the information that they have from those reports.

Those programmes are being developed and are available not just in the face-to-face meetings—which are extremely important and probably more fun than sitting and looking at a webinar—but also through webinars. There are also PowerPoint presentations and text guidance on the platform, to help teachers to become familiar with the assessment and how it might be used.

Education and Skills Committee 09 January 2019, Juliette Mendelovits, contrib. 183, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143495

The SCHOLAR programme has generated a 95% satisfactory rating for its online training. The Scottish Government also envisages more work on the consistency of data literacy to develop including through regonal improvement collaboratives. A number of local authorities in their submissions to the Committee also gave positive feedback on the rollout of training:

Glasgow City Council stated "...training in the analysis of SNSA data and diagnostic use of the assessments has been well received across the city."...

Highland Council highlighted that it has provided CPD for headteachers to support them in using the data to inform improvement...

South Lanarkshire reported that results of its survey found that 73% of respondents rated the training as good, very good, or excellent, going on to state:

"most respondents found that the implementation of the SNSAs had worked well and a number of minor challenges were raised, mostly around technicalities of implementation."

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

The Committee received feedback direct from teachers and unions that some teachers are yet to receive the training due to time pressures:

Two of the teachers suggested they and colleagues in their schools had not received training on the SNSAs...There was agreement more time was vital as other time pressures means that the training is not completed in there schools.

Education and Skills Committee 6 February 2019, teacher focus group note

The difficulty was the timing of the training. Generally, if the teacher training is, as we call it, a huv-tae—something that everyone or some individuals have to do within a school—it needs to be part of a working-time agreement or in-service days. The times at which the training was made available by SCHOLAR and the Australian Council for Educational Research meant that it could not be put into working-time agreements for the first year of the assessments, which meant that renegotiations had to take place around how people could get out for training and beyond. There were therefore challenges in certain local authorities and local areas with people being released. Given the shortage of supply teachers in parts of the country, it was difficult for people to get to the training.

Education and Skills Committee 06 February 2019, Susan Quinn, contrib. 132, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11938&c=2151623

The extent of the roll out of training was raised with the witness from ADES, Mhairi Shaw, who responded:

SCHOLAR is already providing training for headteachers and deputes. They can have a dialogue with a link worker from SCHOLAR and say, “This is where we’re at.” I take your point about such training not always being available to class teachers, but the expectation is that it will be cascaded down, particularly to the P1, P4 and P7 staff who will be using the data—although those staff will not always work with those age groups. Historically, secondary schools probably have more experience than primary schools of using attainment data.

Education and Skills Committee 09 January 2019, Mhairi Shaw, contrib. 181, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143493

Professor Ellis highlighted the importance of qualified teachers being afforded the time to train in assessment literacy, and also the importance of initial teacher education:

The assessments are a tool that teachers need to learn to use well, and they need time and space to learn to use them well.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 52, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143364

There is work to be done so that teachers learn about assessments not only in an abstract sense, but by navigating through what is in front of them. They need to know when and how they can take a deep dive and look at the granular information that is provided.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 179, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143491

The roll-out, in terms of both initial teacher education and continuing professional development, has not been as proactive as it could have been. However, it hit schools at a really busy time, with funding and a whole load of other things going on. Take two: now is our opportunity to improve it; that is a growth mindset.

Education and Skills Committee 09 January 2019, Professor Ellis, contrib. 182, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11879&c=2143494

Professor Paterson, Dr Bloomer and Professor Hayward were asked whether, in their experience, there was a sufficient level of data literacy in local authorities and schools. Professor Paterson responded:

No, there is not. It is demonstrable that local authorities do not have that statistical expertise. However, it must be said that the vast majority of Scottish teachers do not have that expertise either. Remember that one can do a primary teaching degree with a C in what is now called national 5 applications of mathematics—the equivalent of what those of us of a certain age would call arithmetic O grade or a standard grade pass. That is not enough to understand the complexities of statistical sampling and measures of reliability.

Education and Skills Committee 23 January 2019, Professor Paterson, contrib. 136, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147769

Dr Bloomer agreed with this statement and added:

Placing increasing importance on teachers interpreting evidence has implications for initial teacher education, which, so far, have largely not been considered.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 143, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147776

When asked whether a reduction in local authority staff with data literacy skills was a factor in capacity building of data literacy in the system, Dr Bloomer replied:

That is unquestionably the case. Local authorities have a declining capacity to offer support to schools. As long as local authorities remain an important tier of organisation within the system, that is decidedly unfortunate.

Education and Skills Committee 23 January 2019, Dr Bloomer, contrib. 147, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11910&c=2147780

Professor Hayward suggested that the ability to build capacity in the system "might vary from authority to authority, depending on their size"ⁱEducation and Skills Committee 23 January 2019, Professor Hayward col. 39 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11910&mode=pdf and that ITE varies too from institution to institution.

The Committee wrote to the Cabinet Secretary on the issue of ITE and whether data literacy specific to the SNSAs had been factored into course content.¹³Education and Skills Committee. (2019, January 25). Letter to Scottish Government. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/20190125Out_ltr_to_DFM_re_ccost_of_SNSA.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/20190125Out_ltr_to_DFM_re_ccost_of_SNSA.pdf</a> [accessed 17 April 2019] While SNSAs specific content was not mentioned in his response, the Cabinet Secretary made clear:

ITE programmes provide a comprehensive and developmental approach to ensuring student teachers are aware of both formative and summative assessment techniques.

Cabinet Secretary for Education and Skills 12 February 2019. Response to Committee correspondence

The Cabinet Secretary's response also detailed a number of pieces of work underway including by the Scottish Government and the GTCS in relation to course content including the Initial Teacher Education course analysis report. He added that:

The GTCS “Guidelines for ITE programmes in Scotland” makes specific reference to the need to develop the skill of data literacy. The GTCS expect this to be reflected in the content of programmes...

Finally, the GTCS are in the process of refreshing the Professional Standards and have made specific reference to data literacy being a requirement in the Standard for Full Registration. A revised version will be published in 2020.

Cabinet Secretary for Education and Skills 12 February 2019. Response to Committee correspondence

Data literacy and assessment amongst teachers is crucial for the formative element of the SNSAs to function effectively. The Committee notes the positive feedback in relation to the SCHOLAR training and appreciates that the programme is being developed to offer more sophisticated training to follow on from basic training. The Committee also appreciates that work is underway in relation to initial teacher education. The Committee intends to undertake further scrutiny towards implementation of its Teacher Workforce Planning inquiry recommendations later in the parliamentary session. This work may include a focus on ITE course content on the development of numeracy skills, which is linked to data literacy.

The Committee notes that the reduction in local authority support staff in some areas and the restricted ability of teachers to take training due to other workload priorities impacts on the ability of teachers and staff supporting them in local authorities to interpret SNSA data. The Committee considers that it would have been preferable for the rollout of training to have taken place before the rollout of the SNSAs across Scotland.

The SNSAs are taken by pupils on a tablet or a computer and are completed online meaning they require access to good quality hardware and internet access in schools. Teachers also require time after pupils take the assessments to generate the results, and to click through the assessment to view the different ways in which children responded to an adaptive range of questions. In addition the teacher training discussed above is largely reliant on the availability of technology in schools, as formats include powerpoint and webinars for the training.

In terms of usability, Professor Ellis was positive about the electronic delivery of the SNSAs for pupils and teachers, including the generation of immediate results. Professor Hargreaves suggested that the online nature of them combined with broadband limitations prevented the assessments from having more 'bells and whistles' to make them more in keeping with play based learning at P1.ⁱEducation and Skills Committee 30 January 2019, Professor Hargreaves col.15 http://www.parliament.scot/parliamentarybusiness/report.aspx?r=11926&mode=pdf Broadband issues were also cited by a headteacher in the focus group as impacting on a training session where SNSA data literacy techniques were to be rolled out to senior management.

The Cabinet Secretary and Professor Hargreaves, in discussing the ability of pupils to use technology to take SNSAs, suggested that the aim would be to allow pupils to acclimatise to the technology before undertaking the assessments.

If the committee members read the practitioner feedback, they will see that the practitioners wanted to deploy the assessments as part of the learning process of children in the classroom, so that one day a child might work with a piece of technology, with support or independently, and on another day, they might be invited to do the assessment using the self-same piece of technology, with—I would imagine—a degree of supervision...

Education and Skills Committee 20 February 2019, John Swinney, contrib. 171, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153057

Although I am broadly not in favour of a lot of technology in early childhood, a bit of familiarisation with technology in the classroom where possible, so that when children take the test it is not the first time that they have faced that technology, would make it seem less like an extraneous event and more like a continuous part of classroom learning.

Education and Skills Committee 30 January 2019, Professor Hargreaves, contrib. 41, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11926&c=2150062

As highlighted above, time pressure on some teachers or limited access to computers for others might mean there would be insufficient time to acclimatise pupils to technology as envisaged by the Cabinet Secretary and Professor Hargreaves.

A number of teachers and teacher representatives suggested some schools did not have tablets or computers in classrooms. This classroom setting is often what is envisaged as part of a low stakes environment in which to take the assessment. Evidence also highlighted the limited time available when some teachers could gain access to the computer suite in their school, including due to increased requirements from students studying technology to use these suites. This could sometimes result in assessments being undertaken as block bookings in computer suites, a practical constraint that detracts from the idea of teachers being able to decide when a pupil is ready to take an assessment.

A secondary teacher at the focus group said that their school had bought 30 ipads to overcome this issue. Two primary teachers also at the focus group suggested they had different experiences as their school did not have tablets that could be used in the classroom. One of the members asked whether the same issues with access to technology applied to sitting other assessments such as CEM [Centre for Evaluation and Monitoring, University of Durham] assessments. The primary teachers confirmed the same issues applied.

In response to a specific question on resources, a number of local authorities commented on ICT in their submissions to the Committee:

East Renfrewshire stated that: "general feedback from school leaders has been very positive with recognition of the benefit of SNSA". The Council noted that 'very few issues' had been recorded and that these tended to relate to IT problems and the differentiation of the questions.

SPICe. (2019, February). Summary of local authority responses. Retrieved from https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf [accessed 17 April 2019]

IT, and the availability of technology, was raised as an issue by four authorities [of 21 respondents]. For example, North Lanarkshire stated that feedback it had received suggested that practical issues such as the use of computers for pupils would need to be reviewed to ensure the SNSAs were carried out efficiently.³SPICe. (2019, February). Summary of local authority responses. Retrieved from <a href="https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf" target="_blank">https://www.parliament.scot/S5_Education/Inquiries/SPICe_summary__and_Local_Authority_submissions_pack.pdf</a> [accessed 17 April 2019]

As discussed earlier in this report, it is not clear how many schools used local authority standardised assessments to the same extent that SNSAs will be used in schools. Within that it is not clear that all of these local assessments required the same technology, including the same amount of teacher or pupil time using technology. On that basis, the rollout of the SNSAs will have changed the requirements on technology in schools, it is just not clear to what extent. However, the early signs of high participation levels in this year's assessments reflects that any issues with technology are not sufficient to impact on the number of SNSAs completed.

The Cabinet Secretary highlighted the high level of participation in the assessments in year 1 and the increase in the proportion of assessments completed at this stage in the second year:

We are just in year 2 of the process. In year 1, there was a theoretical maximum number of 613,000 assessments that could be undertaken; in fact, 578,000 were undertaken, which was a completion rate in excess of 94 per cent. Having taken the view that it is up to individual schools to judge whether it is appropriate and suitable for pupils to undertake the assessment, which means some pupils will not be given it, I feel that 94 per cent in the first year of implementation of a new system is a pretty high level of participation. As of yesterday, 144,941 assessments have been taken during this academic year. At the same date last year, the figure was 119,616.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 230, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153116

The Cabinet Secretary concluded that:

The significant increase in the proportion of assessments that were undertaken by 19 February this year is indicative of the system adjusting to the tests and taking them forward.

Education and Skills Committee 20 February 2019, John Swinney, contrib. 230, http://www.scottish.parliament.uk/parliamentarybusiness/report.aspx?r=11948&c=2153116

The Committee highlights to the Scottish Government the evidence received in relation to technology, including North Lanarkshire's evidence that "the use of computers for pupils would need to be reviewed to ensure the SNSAs were carried out efficiently".

The Committee also highlights to the Government that the understandable need, from a resources perspective, to rely on block bookings in a computer suite in many schools detracts from holding assessments in the ideal conditions, including at a time of the teacher's choosing, with one to one support for pupils and without time pressure on pupils.