Official Report

 

  • Public Audit and Post-legislative Scrutiny Committee 27 September 2018 [Draft]    
    • Attendance

      Convener

      Jenny Marra (North East Scotland) (Lab)

      Deputy convener

      *Liam Kerr (North East Scotland) (Con)

      Committee members

      *Colin Beattie (Midlothian North and Musselburgh) (SNP)
      *Bill Bowman (North East Scotland) (Con)
      Willie Coffey (Kilmarnock and Irvine Valley) (SNP)
      *Iain Gray (East Lothian) (Lab)
      *Alex Neil (Airdrie and Shotts) (SNP)

      *attended

      The following also participated:

      Caroline Gardner (Auditor General for Scotland)
      Graeme Greenhill (Audit Scotland)
      Leigh Johnston (Audit Scotland)
      Jillian Matthew (Audit Scotland)
      Claire Sweeney (Audit Scotland)

      Clerk to the committee

      Lucy Scharbert

      Location

      The Adam Smith Room (CR5)

       

    • Decision on Taking Business in Private
      • The Deputy Convener (Liam Kerr):

        Good morning and welcome to the 21st meeting of the Public Audit and Post-legislative Scrutiny Committee in 2018. We have received apologies from our convener, Jenny Marra, and from Willie Coffey.

        I ask everyone in the public gallery to either switch off their electronic devices or switch them to silent mode so they do not affect the committee’s work.

        Do members agree to take items 4 and 5 in private?

        Members indicated agreement.

    • “Children and young people’s mental health”
      • The Deputy Convener:

        The next item on our agenda is consideration of the Audit Scotland report, “Children and young people’s mental health”. I welcome our witnesses from Audit Scotland: Caroline Gardner, the Auditor General; Claire Sweeney, audit director, performance and best value; and Leigh Johnston, senior manager, performance and best value. I invite the Auditor General to make a brief opening statement.

      • Caroline Gardner (Auditor General for Scotland):

        Thank you, convener. This report looks at children and young people’s mental health services across Scotland.

        Services are under significant pressure. Demand is increasing. Over the past five years, the number of referrals to specialist services has increased by 22 per cent. Children and young people are also waiting longer for treatment, with over a quarter of those who started treatment in the last year waiting more than 18 weeks.

        The Scottish Government’s mental health strategy focuses on early intervention, but in practice this is limited. The current system is geared towards specialist care and responding to crisis, rather than identifying young people with issues and helping them at an early stage. Access to early intervention services such as school counselling varies across Scotland.

        The system is also complex and fragmented, making it difficult for children and young people to get the support they need when they need it. Accessing the right services needs to be easier for children and young people, their parents and carers and the professionals who work with them.

        We found examples of good practice and projects aimed at improving services. The challenge is how to sustain improvements in the longer term, especially when projects rely on short-term funding.

        Data on spending, performance and outcomes is limited. We do not know with any accuracy how much is spent on mental health services for children and young people, or what impact that spending has. The information that we have indicates that it is a small proportion of overall mental health spending.

        Without a clearer picture of what Is happening across all four tiers of the system, it will be hard to make the improvements that are needed. That will require the Scottish Government, national health service boards, councils, integration authorities and voluntary organisations to work together with children and young people to bring about a step change in how support is provided.

        As always, my colleagues and I will do our best to answer the committee’s questions.

      • The Deputy Convener:

        Thank you. We have many questions. Colin Beattie will start.

      • Colin Beattie (Midlothian North and Musselburgh) (SNP):

        Auditor General, one thing that jumps out in this report and which you touched on in your opening remarks is the issue of data. I cannot remember how many times you have raised that issue in reports. Without data, we do not know if we are getting the correct outcomes or if the money is being spent in the right place. Data is fundamental. Have those responsible for providing data services responded to the recommendations in your report about data?

      • Caroline Gardner:

        I am pleased to say that, on the publication of the report, the Government accepted the findings and our recommendations, which is an important step. The chair of the task force that was commissioned by the Government, which is reporting jointly to the Government and the Convention of Scottish Local Authorities, has made that one of her early priorities, as set out in her first report, which was published a couple of weeks ago. There is an acceptance of the need. The hard work that is required now is to collect that data and make good use of it.

      • Colin Beattie:

        We have seen many reports with comments on a lack of data. Looking across the public sector, is there an improvement overall?

      • Caroline Gardner:

        We see pockets of improvement. Overall, we still do not have the data that we need, particularly in developing policy areas, such as those relating to the provision of more early intervention and preventative services. In health and social care more generally, what is happening in community services and primary care is that we tend to be very good at collecting lots of data in the more traditional services, such as hospitals, compared with the more flexible services, which are often the preventative services. Lack of the data that would let people involved in those services plan and track progress over time is one of the blockages to making a reality of the Government’s outcomes approach.

      • Colin Beattie:

        Would it be correct to say that the majority of data disconnects come about when local councils are collecting data on one side, Government is collecting on the other side, and somehow they do not come together?

      • Caroline Gardner:

        That does not help, but I do not think that it is the whole story. There are certainly gaps in social care and gaps in the way health services and social care work together, but there are also important gaps in what we know about community health services and primary health services.

      • Colin Beattie:

        I vaguely remember that the Government put together some sort of task force on data several years ago. Am I wrong about that?

      • Caroline Gardner:

        That does not ring a bell. We have seen lots of initiatives around things such as the integration of health and social care and the wellbeing of young people more generally through the early years commitments, but I think that that is a question for Government rather than us.

      • Colin Beattie:

        One of the big questions is that of financial reporting, with regard to how much is being spent on this issue. It is hard to understand why that information is not available, at least on a local basis. Is it simply that the information is not being harvested nationally but is being reported locally?

      • Caroline Gardner:

        In exhibit 9, we try to pull together the available information on spending on these services across Scotland. The numbers are so variable as to not be credible. Claire Sweeney can talk you through some of the reasons we have found for that.

      • Claire Sweeney (Audit Scotland):

        We had a hard time getting a clear picture about how the resources were being used. We could see, to a certain degree, how much was being spent in health, but that data was very limited. In the report, we have presented as much information as we could gather nationally, but it is not good. There are key gaps in how the information is collected locally and in how it is reported nationally and publicly. We have highlighted some of the key areas in exhibit 9. There are inconsistencies in the ways in which organisations work out how much is spent, what is included and what is excluded. In some areas of Scotland, the information on how much is spent on community services does not feature. The information is by no means comprehensive but we have given you as much as we were able to collect, with those heavy caveats that there are some significant gaps.

      • Colin Beattie:

        In paragraph 50, you say:

        “the CAMHS workforce increased by 11 per cent between 31 March 2014 and 31 March 2018”.

        That is a fairly big increase. How does that feed into the results? It is not clear from your report that there is a direct correlation between the increase in the headcount and the outcomes.

      • Caroline Gardner:

        You are right; it is not clear. There are a couple of reasons for that. One reason involves our finding that the number of referrals increased by 22 per cent over a five-year period so, although there has been an increase in the workforce, the level of demand has also been increasing, and doing so more quickly. Just as important is that one of the data gaps is about outcomes and about what differences the services are making for children and young people, ideally helping to address their problems early and setting them back to thriving and being able to benefit from their education as they grow up as opposed to getting locked into a cycle of struggling with their mental health in way that limits their potential. We do not know enough about that. Dame Denise Coia, the chair of the task force, is very keen to fill that gap.

      • Colin Beattie:

        Surely the local councils, which are mainly concerned with CAMHS, must have some deep data on outcomes; they must have something to justify the increase in headcount and so on.

      • Caroline Gardner:

        It is not quite right to say that it is the councils that are mainly responsible for CAMHS. Again, Claire Sweeney can talk you through that.

      • Claire Sweeney:

        What we saw overall was a lack of clarity about how the whole system worked. We were looking for connections between different services but we saw a siloed approach in some areas. We would see particular specialities focused on certain needs of children, and also very broad services trying to support children in a range of different ways, through local authorities, charities and the private sector.

        All those things are good initiatives that try to support children, but we are clear that the matter can be addressed only by a range of different organisations working together more effectively. We saw gaps and problems throughout the system in terms of how the money is accounted for and, critically, in terms of what difference any of it makes to children. We have made a series of recommendations in the report that those things need to be sharpened. There needs to be a much clearer sense of what interventions work and where the money should be targeted, and there must be monitoring of what difference any of it makes to children.

        Throughout the report we have stories from the children and young people we spoke to when we were carrying out the work. They told us how frustrating it was to repeat their stories to different professionals and to be unclear about what services they could access when they needed help. The same points were raised by their families. We see that there are problems across the system rather than just in particular parts of it.

      • Colin Beattie:

        Who should be doing the assessment of the outcomes?

      • Claire Sweeney:

        Anybody who is providing services and support for children should be thinking about what difference their service is making and what impact it is having on children locally.

        As the Auditor General said, we can see data on waiting times. That data does not tell a great story but it gives us a picture of what is happening. There is a gap with regard to measuring outcomes, which is to say, the differences that services make.

        In the report, we highlight the fact that measuring outcomes is not seen as a priority in all areas of Scotland. The integration authorities have a key role to play in that area, improving the line of sight and the priority that is given to services across Scotland. From the evidence that we saw, measuring outcomes is not a priority everywhere.

      • Colin Beattie:

        Again, you highlight a concern, saying that each individual organisation involved should be assessing its own outcomes, but as you can appreciate, everybody might apply different criteria, so we could end up with data that is not of much use anyway, even if there was somebody who was bringing it all together. Who might bring it together?

      • Claire Sweeney:

        The issue links clearly to the national performance framework. It is difficult and challenging across the whole public sector to get good information on outcomes. The services need to be locally responsive to needs and differences, and there needs to be variety—we recognise that to be true—but there still needs to be something that brings the information together. We would ask some simple questions. How do you know that the millions that are being spent on these services are making a difference, and how do you tell that that is happening locally? At the moment, we do not see that thread throughout the way in which everybody is working. We see a system that is under significant pressure, we see a lot of effort going into harnessing the views of children and young people and we see a policy priority around the mental health of children and young people, but we do not see that being translated on the ground.

      • Leigh Johnston (Audit Scotland):

        During our fieldwork, we saw examples of measuring outcomes locally. What we are trying to say is that, at a national level, we have no idea of the outcomes that are being achieved, so we have no idea about where to direct funding and what to spend it on.

        The Scottish Government is working on developing a number of quality indicators for mental health across six different quality dimensions. However, the issue there is that we understand that boards will choose which ones they want to measure, and that will make benchmarking very difficult.

      • The Deputy Convener:

        Can I follow up on that point, in terms of the sharing of learning? It is a slightly different point from the data and the financial aspects. On page 21 of your report, you talk about NHS Ayrshire and Arran. It seems to have continued to improve performance to meet the 18-week standard consistently throughout the period. How did that happen? What is it doing that is significantly better than some of those that you reference on page 19, and how is that learning either being shared or going to be shared, either as a result of this report or in general?

        09:15  
      • Leigh Johnston:

        NHS Ayrshire and Arran is taking a whole-systems approach. It is working towards multiagency collaboration, for example seconding teaching staff to the CAMHS service, and vice versa—CAMHS staff sitting within schools. NHS Ayrshire and Arran is using its data to understand the challenges that it is facing and is piloting different initiatives to address the challenges.

        We have to be cautious. NHS Ayrshire and Arran is now struggling with how to maintain the pilot initiatives with short-term funding. Another thing to acknowledge is that different areas have different needs and challenges. Other areas could learn from NHS Ayrshire and Arran, but they would need to look at the situation in their own area and decide what pilot initiatives would suit them. Good practice will be shared in relation to various things that are going on. For example, I was at a conference yesterday and representatives of the youth commission on mental health services were also there. I know that the commission will make a number of recommendations and share ideas and good practice, and the task force will hopefully progress that work of sharing good practice.

      • The Deputy Convener:

        I am sure that we will come back to the nature of funding.

        Might I press you on a local issue? On page 19, you talk about NHS Grampian and NHS Tayside, which are of particular concern to me in my representative capacity. NHS Grampian seems to have a 21-week wait time for first appointments and NHS Tayside an 18-week wait time, which are significant waiting periods. Presumably this is happening when someone has an identified need and should get into the system as quickly as possible. What are NHS Grampian and NHS Tayside doing to address those wait times and/or learn from the likes of NHS Ayrshire and Arran how to improve that performance?

      • Leigh Johnston:

        Towards the end of our report we refer to some good work that is going on in Grampian.

        We have to be cautious about the waiting time figures. As Claire Sweeney has said, we have outlined the data that was available, but we know from having looked at the information in more depth that different areas measure waiting times in different ways in terms of what counts as treatment starting. We find that sometimes people go to an assessment and then possibly go onto another waiting list for the treatment that they need. The differences are also to do with workforce capacity and issues with the way that data is collected and monitored. We also found there were issues around the referral process—the referral criteria changing—and that, therefore, the number of young people that were seen fluctuated over a period.

      • Iain Gray (East Lothian) (Lab):

        Auditor General, you bring a lot of performance audit reports to the committee and they cover a wide range of services and projects. Some of them are good—and later in today’s agenda we will look at one that is good—some are bad, and some are downright damning. Where would you place the report in that spectrum?

      • Caroline Gardner:

        The report highlights a problem. We know that dealing with mental health problems early in their lives is difficult for children and young people, and distressing for them and their families.

        The evidence shows that it can make a difference to how well they thrive for the rest of their lives. If they get the help and support they need early, a relatively minor problem can be nipped in the bud and they can get back into full-time education, and continue to build relationships, their confidence and their ability to flourish as people.

        If they do not get that help and support early on, they can get into a cycle of depression and anxiety, doing less well at school and being less likely to fulfil their potential once they leave school. That is why the report is so important and why the failings that we have identified, and which the Government has accepted, matter.

        The team doing the work heard stories of young people and their difficulties in getting some of the help that they needed, often for quite small reasons, such as teachers not knowing what help was available or how to refer them, and referrals being made that did not meet the referral criteria for the service that they were being sent to. Those things should have been quite straightforward to fix and yet they had an impact on young people’s lives. That is why the problem matters to us.

      • Iain Gray:

        So the team identified exceptional problems that went beyond the usual difficulties that we find when public bodies work together, or problems with outcomes or data. The problems were greater than the team was used to dealing with.

      • Caroline Gardner:

        It was a combination of lots of the young people who we are talking about finding it difficult to access what ought to be relatively straightforward interventions in practice. Some of the pathways are described in the report. Leigh Johnston, do you want to talk about that?

      • Leigh Johnston:

        There were several barriers. One was the lack of early intervention and prevention services. Referrals to the CAMHS specialist service are increasing greatly. Because the data are not there, we do not have the evidence to understand what the demand for the lower-level services is, but we imagine that some young people could be helped with better services in place at the tier one and tier two levels, so they might not need the specialist services. We suggest in the report that the four-tier approach to the service delivery might no longer be fit for purpose and that we need to look at how we might provide a more person-centred service to save children and young people from bouncing between the different tiers. We need to make sure that the lower-level services are there to prevent people from being referred to CAMHS or their condition deteriorating further.

      • Iain Gray:

        Is it fair to say that the report could be summarised as saying that we cannot go on like this, and that something has to change?

      • Caroline Gardner:

        We say in the report that a step change is required. The Government and the chair of the task force recognise that. The challenge is to make a reality of the required changes.

      • Iain Gray:

        On that challenge, the report also says:

        “It is not clear how the Scottish Government’s mental health strategy will improve outcomes for children and young people.”

        Why do you feel that to be the case?

      • Leigh Johnston:

        There are 40 actions in the strategy, 15 of which relate to children and young people. We found that a lot of the actions within the strategy were focused on trying to understand how the system works and the challenges it faces rather than outlining action that was going to be taken and the outcomes that the Government wanted to achieve. The Government has said that it will develop a framework that will measure progress and outcomes, but there is no timescale for that work.

        The Government delivered the progress report against the strategy a couple of days ago and things are happening. However, the report focuses on things such as the youth commission and the task force, which will look at the challenges and figure out what is going on and what needs to change. That is why we feel it is not clear how the strategy will improve outcomes.

      • Iain Gray:

        Since the report that we are considering today was published, a couple of things have happened. One is the annual report on the strategy; the other is the programme for government, which made some announcements about counsellors in schools and additional funding. Even with those iterations, is the strategy still inadequate to meet the challenges identified in the report?

      • Leigh Johnston:

        We need to see the outcomes and recommendations of the task force and the youth commission. In our report, we say that we would like to see the task force consider the recommendations that we have made in our report.

        Funding for more school nurses and school counsellors will start to make a difference but, as we say in our report, it is also necessary to look at the way in which organisations work together and for them to work in a more joined-up and collaborative way.

      • Iain Gray:

        You say the plan to do those things still needs to be demonstrated.

      • Leigh Johnston:

        Yes, that is local and national bodies working together to take that forward.

      • Alex Neil (Airdrie and Shotts) (SNP):

        I want to focus on demand. The most worrying aspect of this is that nobody seems to have a handle on the level of demand and, more importantly, not just the numbers but what is behind the numbers—what kinds of services people need.

        Historically, as you have said, there has been a lack of sufficient data—qualitative and quantitative—on demand. How quickly do you think that gap can be satisfactorily rectified?

      • Claire Sweeney:

        In the report, we mention a case study in Grampian where some mapping was carried out to understand the levels of demand and need, and then start to think about what the services to respond to them should look like. The message is that it can be done.

        There is another issue that we have not touched on so far. We saw a lot of committed professionals who want to get this right. There is a lot of frustration within the system.

        The report also highlights that some groups of children and young people are more affected than others. That is well known and a lot of research around that is available. We would like to see that more clearly recognised and targeted.

        Exhibit 1 in the report sets out in much more detail information about those children in Scotland who are more likely to have a mental health problem, such as children who are looked after in the care system, and children who are living in the more deprived areas of Scotland. We did not see enough activity to target those children and to help give them the support they need. There is more to do, but it can be done. The case study in the report shows how it can happen.

      • Alex Neil:

        I presume that there is a close link between children with adverse childhood experiences and children with mental health problems.

      • Claire Sweeney:

        Absolutely.

      • Alex Neil:

        Filling the data gap on a permanent basis is a slightly longer-term exercise, by the time we set up IT systems and so on, and we know the problems that are associated with IT systems. Should the Grampian mapping exercise not be replicated throughout the country?

      • Caroline Gardner:

        It should. What happened in Grampian reminds me of what we have said on a number of occasions about genuinely transforming care for older people. If you are going to do that, you need to know what need and demand look like. A relatively small number of people require the most intensive support and, in this case, you start in schools, early learning centres and nurseries to identify the children who appear to have challenging behaviour or whose parents are struggling for a range of reasons, and you build that up from localities to the health board level, and gradually build up a national picture. Doing that helps you to not treat the data collection as a separate thing. As you are identifying those children, you are starting to understand what help can be provided in a nursery or a school and which children need to be referred to specialist services. Local intervention is always the best place to start.

      • Alex Neil:

        The data that is available is historical. It seems to me that it is a fast-changing world in terms of the requirements for children’s mental health. Take an issue such as autism: we are much better at identifying autism early on—I am not saying we are perfect, far from it, but identifying children who are showing signs of possible autism is a lot better than it was even 10 years ago. Is anyone looking at the changes in the nature of child and adolescent mental health?

      • Caroline Gardner:

        I will kick off. I am sure that colleagues will want to add to what I say.

        First, you are right that we do not know what is causing the increase in demand. There are two broad theories. One theory is that life is more stressful for children and young people with things like social media playing a part. Another theory is that the reduction in stigma and greater awareness of mental health problems is making it easier for young people to come forward. Nobody knows how far those the things are the case and what else might be happening.

        I was encouraged to see the proposal in Dame Denise Coia’s preliminary report. She suggested moving away from the current four-tier approach that aims to cover everybody to something that focuses first on children who have relatively mild levels of need and can be helped in school, secondly on people who need specialist services, then on children who have neurological problems such as autism, attention deficit hyperactivity disorder or Asperger’s syndrome, and finally, on children who are at risk because of deprivation and adverse childhood experiences. Without second-guessing Dame Denise Coia’s expertise, that approach is likely to give a way of understanding what is happening that is better than treating all mental health problems as though they were the same.

        09:30  
      • Alex Neil:

        It would also give us a better understanding of the resources and expertise that need to be put in place, and when and where.

      • Caroline Gardner:

        Absolutely.

      • Claire Sweeney:

        In the report, we mention the THRIVE—timely, helpful, respectful, innovative, values-based and efficient—model, which has been used in some areas in England. That model looks at different kinds of support that children and young people need and starts to map the resources that might be required.

        One of the reasons that the Grampian example is attractive is that it gets away from a siloed approach, where specialists in a condition treat children with that condition. There is sometimes a lack of connection between them and general practice, schools, education and so on. In Grampian, it was good to sense a whole system coming together to start to think about shared responsibility for children in the local area. We would like to see more of that.

      • Alex Neil:

        The Government has announced an average 6 per cent per annum increase in mental health resources over the next few years. I presume that that additional money should be focused first of all on developing something like THRIVE throughout Scotland so that we see better use of resources—more targeted and earlier intervention, and all the other good things that we have been discussing—and also that the resources go to where they would be most effective.

      • Claire Sweeney:

        We say in the report that it is really important to understand the levels of need and what makes a difference before you start to think about how you spend the resource. That is absolutely critical and it can only be done with the children and young people involved, as well as the folk who are providing the good services locally. There is a need to understand it all.

      • The Deputy Convener:

        Is that not one of the key issues? The section in the report on resourcing says lots of good stuff about money being put into various things. The committee is hearing, however, that no one has worked out where the most effective interventions are and at which of the four stages interventions should be made. Is there not also an issue about what I think you call non-recurrent funding and how third sector organisations are able to budget and say they will be able to deliver the service effectively in future? Could you tell us something about that?

      • Caroline Gardner:

        You have just summed up the end of the key messages of the report where we say that transformation will only happen if there is a clearer view of what works, a plan for how the system needs to change, and a move away from relying on short-term and isolated initiatives. It is easy to say that and much harder to do it. Claire Sweeney, would you like to pick up what that might mean in practice?

      • Claire Sweeney:

        We saw examples of voluntary sector initiatives and projects being introduced because there was a pot of money, but it was not clear whether the initiative had worked, whether it would be mainstreamed, and what had been the shared learning around it. If it had not worked, where was the decision making to say that it was not going to be continued and for good reason? Although we saw a lot of commitment to the overall idea of supporting children’s mental health services, and we saw some resource against it, we did not get the sense of a system learning together and working as one coherent whole. There are messages in the report from children and young people about the system feeling fragmented and we saw that through the way in which money is counted and performance is measured, and the current focus on short-term initiatives.

      • The Deputy Convener:

        Sticking with the demand issue that Alex Neil was talking about, can you tell us more about the benefits of early intervention and prevention, or the early stages of the four? What do you see as the positive outcomes that those measures can bring?

        Moving on specifically to the matter of the resources that Claire Sweeney was talking about, is the solution as obvious as saying that extra investment at tier one and tier two reduces demand on tier three and tier four, such that those who do require stages three and four get a better service and the money is better allocated?

      • Claire Sweeney:

        The idea that focus on prevention and early intervention is a good thing that will have a positive effect on reducing demand has long been held in the health and social care services; people will be less likely to get into a crisis situation.

        We were careful about our language around that issue. We say in the report that the service is focused on crisis and specialist need. We are not saying that that is a bad thing—of course there will always be children who need that kind of support—but we are saying that it is not a good thing if it operates to the exclusion of prevention and early intervention. A shift is needed. There needs to be a clearer picture of what works and a greater commitment to early intervention and prevention, which people know makes a difference.

      • Caroline Gardner:

        We also think that one of the factors underlying the increasing number of referrals to specialist mental health services and the increasing number of rejected referrals, is that people could be well supported by lower-level services closer to their homes and schools but because no specialist service is available locally, they are being referred up the chain to more intensive services that are not the best ones for them, and they are then rejected and left clogging up the system. The system is therefore under more pressure, the young people are not getting the help they need, and we are not breaking out of the cycle because we do not yet have the school counsellors and trained teachers who can spot a problem early and know who to refer it to, and a system that can respond in the best way for an individual child’s needs.

      • The Deputy Convener:

        I am going to press you, Auditor General. We are going to come back to the rejected referrals in two seconds. First, would you mind developing the point about access to early intervention? Having written this report, what do you see as the key barriers to the early intervention that Claire Sweeney is saying is vital?

      • Caroline Gardner:

        There are a number of things and the team knows more about them than I do. Briefly, however, there is something about having the services available in the first place, making sure that people such as teachers and GPs, who are in contact with young people every day, have some training in that general level of mental health and know what services are available, and making sure the system works smoothly so that it is easy to make a referral once, with the right information for that referral to be assessed and picked up by the right service. It comes back to looking at the system as a whole rather than having separate bits of it working in isolation.

      • Bill Bowman (North East Scotland) (Con):

        I echo what Claire Sweeney said. We should give due credit to the professionals who are working hard to deliver the services. We are asking questions about the system, and I will move on to ask some questions about rejected referrals.

        The key facts section of the report states that there has been a

        “24 per cent ... Increase in the number of referrals rejected ... since 2013/14”.

        Further on, in paragraph 25, you give some reasons why that has occurred, which include:

        “the ... young person does not meet the criteria for treatment”,

        “a lack of tier one and two services for children ... experiencing less severe mental health problems”

        and

        “the referral does not contain enough information.”

        First, you say that national data is not being collected on reasons for rejection. Are you aware of local data being collected? Secondly, is there any evidence to show that the NHS boards that are under the most pressure, perhaps because of the number of referrals that they receive or their level of service provision, apply the criteria for rejection more strictly than NHS boards that are under less pressure? Please do not feel that you have to be careful with your language.

      • Leigh Johnston:

        We found throughout our fieldwork that there are local examples of the collection of data. However, at the national level, we have no idea what the trend is for rejected referrals, or what the reasons are for them. It comes back to the lack of data. The Scottish Association for Mental Health recently published a report, which was commissioned by the Government, on rejected referrals and what happened to young people after their referrals were rejected, and it makes a number of recommendations. To understand the matter better, however, we need to come back to the matter of data.

      • Bill Bowman:

        Was that the answer to my first question or my second question?

      • Leigh Johnston:

        Sorry. Will you repeat your second question?

      • Bill Bowman:

        Are there hotspots in health boards that are under more pressure because they get more referrals or have fewer services available? Do they reject more people because they know that they cannot treat them, for whatever reason?

      • Leigh Johnston:

        Again, we simply do not know that because we do not understand the reasons behind the rejected referrals. The information is not collected at a national level.

        In our report, we outline why we think that some referrals do not meet the criteria. We heard that there are children and young people who would have benefited from lower-level early intervention and prevention services but, because they are not available locally, the young people are being referred up to CAMHS, and then they are rejected. Rejections can also be due to referrals not including enough information.

        Yesterday, I presented the findings from the report at a conference, and I met an academic from the University of the Highlands and Islands. About four years ago, she looked at the reasons for rejections, and she found that there were a range of reasons. Children and young people with behavioural issues were more likely to be rejected, and referrals from teachers were more likely to be rejected. However, that was a small-scale study. We need to understand what is going on at the national level, and that comes back to the collection of data.

      • Bill Bowman:

        Out of interest, did that academic give any reasons why teachers’ referrals might be rejected?

      • Leigh Johnston:

        She hypothesised that it was perhaps because of the language that they were using, because they were not clinicians. Knowing the language to use would perhaps make referrals more successful. However, she did not know the absolute reason.

      • Bill Bowman:

        Auditor General, do you have anything to add?

      • Caroline Gardner:

        No.

      • Iain Gray:

        I have a supplementary question. You talked about local strategies and what might be failing around data collection and the provision of services, which leads to inappropriate referrals. I do not want to lose sight of paragraph 70 of the report, which says that local mental health and wellbeing strategies focus on adults. Is the problem that, at the local level, there just is not a strategy for children and young people’s mental health services? The problem is not that the strategy does not gather data or that it is not working very well. There just is no strategy for children and young people. The strategy is for adults.

      • Claire Sweeney:

        We certainly got a sense of the level of priority that children and young people’s services have in certain areas. There is potentially a link to our message on waiting times. We say in the report that the waiting times have been a focus. That is not to say that we should not be concerned and know more about how long children are waiting to get services, but we have already highlighted the problems with the data that make it hard to answer some of the previous questions.

        Our sense is that children and young people’s services need to be a greater priority. There are lots of good and committed people working in the system and there is some clear evidence about where the problems are, but there is a lack of pulling together to make sure that those needs are responded to and that the money is there to make that happen. We definitely see this an issue on which a number of organisations need to work together very closely, including the Scottish Government and COSLA.

      • Iain Gray:

        That is surely a massive disconnect between stated national priorities and local priorities on the ground, is it not?

      • Claire Sweeney:

        We say in the report that we absolutely see a commitment in terms of policy on the issue. It is very clear that people recognise the policy as important in Scotland. What we did not see, however, was that translating into practice in all areas, and that is why some of the recommendations speak to that point.

      • Caroline Gardner:

        There is a related point that I do not want us to lose sight of. It is further up that page, in paragraph 68. A lot of the policy focus has been on the importance of integration authorities in getting oversight of what is happening for children and young people as a whole, but we found that only 11 of the 31 integration authorities across Scotland have responsibility for both children’s mental health services and social work mental health services. They are the ones who are best placed to do that. The other 20 will clearly find it more difficult.

      • The Deputy Convener:

        I will ask a few more questions. Going back to rejected referrals, which Bill Bowman asked about, I note that some reasons are given on page 18 as to why referrals are rejected. What I heard from your earlier answers is that there is no data. The report gives three reasons why referrals are rejected:

        “the child or young person does not meet the criteria for treatment”,

        “lack of tier one and two services”

        and

        “the referral does not contain enough information.”

        That is qualified later, at paragraph 28, which mentions

        “the level of detail provided by the referrer.”

        For clarity, is there any data on how many of the rejections fall into each of those categories?

        09:45  
      • Caroline Gardner:

        No. As we say at the beginning of paragraph 26,

        “National data on reasons for referral and rejection is not collected”,

        which makes it very difficult to be clear about that. As Leigh Johnston said, the Government commissioned some work from the Scottish Association for Mental Health to examine what has happened, and it published its report recently, but that has been a specific clinical audit rather than routine data collection, which we think is important.

      • The Deputy Convener:

        I agree with you.

        Going back to the point about the referrer, I note that a lack of services points us towards funding and supply, but the other two reasons for rejection seem to me to relate to the competence of the referrer. I do not use that term in a pejorative sense; it is just about their ability and whether they have the guidelines to do it. Somebody has identified a need and said, “I have a young person here who needs help”, but because of—again, I do not say this pejoratively—a failing on the part of the referrer, that young person is unable to access the help. That is hugely concerning. Is that a fair summary? What is being done?

      • Leigh Johnston:

        As we outline in our report, the referrals pathway is complicated. The criteria vary across the boards and they are often not easy to follow for young people, for parents and carers or for potential referrers who do not come from a clinical background.

        We outline some good practice in NHS Highland, where there is a primary mental health worker who undertakes a triage service. The young person goes to their GP and the primary mental health worker is there and can assess them and offer, almost, a step up or down. If they do not think that a referral to specialist services would be appropriate, they will step the young person down, and vice versa. If they think that the young person requires more specialist help, they will step them up.

        There are pockets of good practice to try to address some of the referral issues. However, as we also say later in our report, there is a need for more training for non-mental-health specialists such as teachers, school nurses and the like. That might help people to understand the referrals process and what is required in more depth.

      • The Deputy Convener:

        On that exact point, you mention in your report a revised role for school nurses. Are there sufficient school nurses such that, if there is that change in role, it will make a significant contribution to early intervention? Alternatively, is it the case that it is a good thing to do but, ultimately, early intervention will not be significantly impacted?

      • Leigh Johnston:

        As we say in our report, mental health and wellbeing is a priority for school nurses, but they have indicated that they require a bit more training and help in that area. It was announced in the programme for government that a significant number of school nurses will be brought on board. We will wait to see the impact of that.

      • The Deputy Convener:

        Finally, I want to press you on the data-sharing aspects. At page 23, you say that multi-agency working together is going to be crucial and that the

        “Young people ... found it very frustrating”

        having to repeat their histories and challenges

        “to multiple professionals.”

        What is the issue with data sharing? Is it about IT, for example? Is it the data-sharing regulations? What is going on and what can be done to fix it?

      • Claire Sweeney:

        We have been looking at the issue in the round as part of our work on the integration of health and social care services, and the Auditor General will bring a report on the issue to the committee later in the year. Some of the issues that we saw in our piece of work were about the quality of records. For example, some areas are using paper-based records, which by definition makes it difficult to share information. There are a range of factors.

        Earlier in the report, we talk about the need for trust in relationships, and that goes for the professionals who work in the system just as much as it does for the children and young people who need the support. We see information and data sharing as just one part of that. Use of paper records does not help, but we need to understand how the system is working together and what information everybody needs to know.

        Over the past few years, there have been developments in some parts of the health system to try to move that issue on. An example is the sharing of emergency care summaries for people who are going into accident and emergency services. It can be done. Work can be done to improve the way that information is shared across the system in order to make care better. However, we saw that there is a long way to go for the services that we have been discussing.

      • The Deputy Convener:

        As colleagues have no further questions, I thank our witnesses for their evidence, which has been very useful.

        I will suspend the meeting for five minutes to allow for a change of witnesses.

        09:51 Meeting suspended.  09:54 On resuming—  
    • “Forth Replacement Crossing”
      • The Deputy Convener:

        Welcome back. We will now examine the “Forth Replacement Crossing” report. I welcome back Caroline Gardner, the Auditor General for Scotland. She is joined by Graeme Greenhill, senior manager, and Jillian Matthew, audit manager, who are both from Audit Scotland. I invite the Auditor General to make an opening statement.

      • Caroline Gardner:

        Thank you, convener. The report looks at Transport Scotland’s management of the Forth replacement crossing project, including the construction of the Queensferry crossing. I will summarise the findings under three areas: the need for a new crossing; the management of the project; and demonstration of whether the intended benefits of the project have been achieved.

        First, the Government identified a clear need for a replacement crossing. Corrosion of the main cables of the Forth road bridge would have meant restricting traffic on it from 2017, and ministers made a timely decision to ensure that the road connection between Edinburgh and Fife was maintained. Transport Scotland’s decision to build a new cable-stayed bridge was cheaper than repairing the old one or building an alternative type of crossing or tunnel. Its design was intended to be easier to construct and more reliable and resilient. Exhibit 2 on page 10 of the report sets out the key features of the new bridge.

        Secondly, at £1.34 billion, the Forth replacement crossing project is one the biggest public sector infrastructure projects that Scotland has seen. On such a large and complex project, there are many opportunities for things to veer off track, and it is to Transport Scotland’s credit that they did not. In part 2 of the report, we highlight the good practice in the procurement process that helped to deliver value for money. The team had the right mix of skills and experience and it invested in the external expertise that it needed early in the project. From start to finish, the team demonstrated strong, consistent leadership and communicated well with contractors and stakeholder groups. It was strong on budgeting, governance, quality assurance and risk management. We think that there is a lot that the wider public sector can learn from how the project was managed, and we have recommended that the Scottish Government and Transport Scotland should share the lessons as widely as possible.

        The Queensferry crossing opened eight months later than first expected and 10 weeks later than the contract end date, but we concluded that the reasons for that were reasonable and that Transport Scotland managed the changes effectively to minimise the effect on time, cost and quality. There is still some work to complete on the new bridge, which is to be expected on a project of this kind. Our only criticism is that Transport Scotland could have communicated that better to manage the public’s expectations, and it should continue to keep the public updated on progress.

        Thirdly, it is too early for some of the project’s wider benefits, such as improving public transport across the Forth, cutting journey times and boosting economic growth, to be demonstrated. We set out progress against each of the eight project objectives in exhibit 9 on page 36, but more detail is needed on how success will be measured in future. Transport Scotland now needs to produce a clearer plan for how it will measure the success of the project’s wider benefits. It plans to carry out a full post-project evaluation later this year, and we will continue to review progress through our audit work.

      • Colin Beattie:

        The report seems to be unrelenting good news, so we ought to congratulate the Government. Throughout the report, reference is made to good practice and success story after success story. Given that other projects have been less successfully managed, how do we effectively transfer the good practice that has been learned in relation to this project to other projects in the public sector?

      • Caroline Gardner:

        We think that the Forth replacement crossing project was well managed, and it is to the credit of the Government and Transport Scotland that they achieved that. We have recommended that they should look at how they can spread those lessons more widely, not just to infrastructure projects but to big digital projects, to which many of the principles apply. On our website, we have produced a hub that pulls together all the materials that we have developed in this area, to which people can refer.

        I will ask Jillian Matthew what else we think can be done to make those lessons a reality and get some of the benefits in future.

      • Jillian Matthew (Audit Scotland):

        As the Auditor General said, it was a very well managed project, which is not something that, as auditors, we often see. You will know from audits of other projects that we have looked at that that is not always the case. The need to get the right building blocks in place right from the beginning is absolutely fundamental and cannot be underestimated. Time must be spent getting the scope right and understanding the costs, the risks and optimism bias, which we mention in the report. It is possible to underestimate what can go wrong and capacity has to be built in for dealing with things that are not anticipated. A lot of that is well known. We have looked at that in other reports and have talked about what good practice looks like and the need for projects to be well managed, but that does not always translate. There are various reasons for good practice not being adopted, such as time pressures, but I think that it is often the case that not enough time is spent getting things right at the start before proceeding.

        10:00  

        As the Auditor General said, there is a lot that can be learned from the Forth replacement crossing project, and not just for infrastructure projects. There are many generic project management lessons that can be applied to many other major projects. We have recommended that Transport Scotland should make sure that it applies those lessons in future work. Throughout the project, it made sure that lessons that were learned were shared across other projects, such as the dualling of the A9. The Scottish Government has a role to play in making sure that such learning is shared right across the public sector.

      • Colin Beattie:

        Has the Scottish Government given any indication of how it will do that?

      • Jillian Matthew:

        Not at the moment. Transport Scotland and the Scottish Government have accepted that the lessons learned need to be shared and have said that they will do that, but we do not know exactly how they intend to do that.

      • Graeme Greenhill (Audit Scotland):

        Can I chip in? One of the things that we do with every performance audit is prepare an impact report; we usually do that about 18 months after the publication of the original report. I am the auditor of Transport Scotland. At the appropriate time, as part of my audit of Transport Scotland, I will look at how the Government and Transport Scotland respond to the “Forth Replacement Crossing” report.

      • Colin Beattie:

        Having proven that a large project can be well managed and delivered in the way that the public would expect, it would certainly be a shame if we did not build on that success. There seems to be no reason why what has been done in building the Queensferry crossing cannot be done across the whole public sector if we can just take what we have learned and apply it.

        I want to raise what you say in paragraph 33 of your report. Is what is detailed there normal in the trade? I have not seen that practice in any previous project. Was it simply because of the scale of the project that it was agreed that such payments would be made? It was agreed that bidders would be paid reasonable costs up to £10 million, which is a lot of money, in the event that the contract did not go ahead. Is that normal?

      • Caroline Gardner:

        It is unusual and, as you can imagine, we looked at it very closely. I will ask Graeme Greenhill to talk you through what we concluded about it.

      • Graeme Greenhill:

        There are two separate elements here. First, there was the fact that the tendering was being undertaken at the same time that the Forth Crossing Bill was proceeding through Parliament. Therefore, there was always a risk that the bill might fail and the bridge would not proceed, which would have left bidders out of pocket. Transport Scotland took the view that if that happened, it would need to compensate the bidders for the costs that they had incurred.

        Secondly, there was the offer that Transport Scotland would pay up to £5 million towards the unsuccessful bidder’s costs. That was there to ensure that competitive tension still existed and that there would be more than one bidder in the process. As Caroline Gardner said, it is unusual, but it is not unknown for clients to go down such a route. I think that it was done in view of the size of the project and the likely costs that bidders would incur in developing their tender proposals.

      • Colin Beattie:

        From the point of view of Audit Scotland, does it seem a reasonable expense to incur?

      • Graeme Greenhill:

        We concluded that it was.

      • Caroline Gardner:

        The effect of the decisions that Transport Scotland took was to make sure that the procurement process remained competitive. It kept two bidders in the process to the point at which a decision was made, which helped to keep costs down and to generate a strong form of contract that was cost limited, thereby reducing the scope for cost overruns to come through. As always, there is a balance to be struck on where the risks are best managed, and we thought that it was a reasonable decision that paid off in the end.

      • Colin Beattie:

        Just to be absolutely clear, it is only the unsuccessful bidder who got £4.2 million. The successful bidder got nothing, because they made the profit on the contract.

      • Caroline Gardner:

        Yes. The successful bidder won the contract. The unsuccessful bidder was reimbursed £4.2 million for the cost of bidding, and that helped to make the procurement a competitive process.

      • The Deputy Convener:

        The procurement process and the bill that set up the project ran concurrently. I understand why that was the case. If I might use your words, you said that that paid off this time. I give credit where it is due—it worked this time—but it might not have done. Is that a practice that you would advocate in future or did Transport Scotland get a bit lucky this time?

      • Caroline Gardner:

        We concluded that, in this case, it was a reasonable decision. We were in the unusual circumstance in which all the engineering evidence suggested that restrictions were likely to be required on the traffic that the old Forth road bridge could carry by 2017. Given the timescale for a project of such a scale, waiting longer to start the procurement process would have run the risk of significant road closures, disruption and impact on the economy of Scotland, particularly in Fife and the Lothians.

        That is not to say that I think that such an approach should automatically be taken for future contracts. As always, what we are looking for is intelligent application of the principles of good procurement and good project management rather than a cookie-cutter approach, whereby something that worked for one project is automatically assumed to be the right answer for another one. That is certainly not what we are saying.

      • The Deputy Convener:

        On that point, you and the Accounts Commission recently produced a joint report summarising your findings on various major projects and procurement lessons. Do you have any plans to publicise that report or to disseminate it more widely?

      • Caroline Gardner:

        We are doing quite a lot on that. I ask Graeme Greenhill to pick up that one.

      • Graeme Greenhill:

        We have certainly made people aware of that report. As Caroline Gardner said, it appears on our hub that brings together all relevant reports on major capital projects, together with the summary of findings. We have made sure that Transport Scotland is aware of its existence, and we have been working closely with the Institution of Civil Engineers Scotland to make it aware of the hub. We are continuing to work on that.

      • Bill Bowman:

        I will take a slightly more personal view. I have been crossing the Forth since the original bridge opened in 1964 and I continue to do so on the new one. I do not want to rain on anybody’s parade, but Colin Beattie’s unrelenting good news is maybe as seen from afar. From a bit closer up, I see unrelenting delays. The bridge opened and then immediately closed. I have crossed in the evening and found two lanes closed and had to bump across on the hard shoulder, and I have had correspondence from constituents who have had the same experience. You say that some work is being done, but it seems to be more than some work. Was the bridge actually finished when it opened? You say that you are going to look at its operation in future. Will you be a little more critical of Transport Scotland? We have an empty bridge with virtually no traffic on it, but we still have traffic queued up on the new bridge.

      • Caroline Gardner:

        I will ask the team to come in in a moment. It is important for me to state for the record that the new bridge was not intended to increase capacity for traffic crossing the Forth. It was a replacement for the Forth road bridge, with any increase in demand to be met through increased public transport, which is one of the project objectives that is still not fully delivered. We make a recommendation that there should be a plan for delivering that objective, with clear measures of how that is affecting the situation.

        I ask Graeme Greenhill to pick up on the question of snaggings and our conclusions in that area.

      • Graeme Greenhill:

        It is important to recognise that snagging is by no means unknown for any major project. There undoubtedly was a list of snaggings that needed to be completed. The transport minister has informed Parliament of the list of works that needed to be done. None of the snagging work prevented the bridge from opening at the time, although Bill Bowman is correct that, subsequently, there was a need for a temporary closure to level off some tarmac joints in the bridge.

        The one main area of snagging that remains outstanding relates to painting the underside of the bridge. That was supposed to be completed before it will be completed. It will not now be completed until September 2019. That is largely because of the specialist equipment—the cradle—that is required to get underneath to provide access to the undersurface of the bridge. That work will not be completed until the end of 2019, but it is not having any effect on the operation of the bridge.

      • Bill Bowman:

        It is having an effect for those who try to cross the bridge in the evening and who find that it is down to one lane, with a long queue waiting to cross. We all know about snagging from buying a house. It is about there being a crack up in the corner; it is not about the roof not being finished or something like that. The term “snagging” implies minor repairs. If, as you say, the work needs to continue until the end of 2019—I presume that lane closures will continue until then—I cannot agree that there is not something wrong in the way that the bridge has been managed. However, your report is your report, and I hope that you will come back to the issue of how the bridge is being operated.

        I have a question on the costs. On page 29, you talk about inflation of 5.3 per cent being included in the estimates. Is that correct?

      • Jillian Matthew:

        Sorry, but which page did you say?

      • Bill Bowman:

        It is on page 29, in the bottom graphic. As I understand it, inflation probably came in at about zero, using the appropriate index. Does that mean that a lot of the cost saving comes from there being no inflation?

      • Jillian Matthew:

        There certainly were savings as a result of inflation, but there were savings from other areas. We set that out—

      • Bill Bowman:

        Can you give me a number for the inflation savings?

      • Graeme Greenhill:

        The third bullet point in paragraph 60, which is at the top of page 33, says that price fluctuation costs were £60 million to £205 million lower than first projected, which was essentially due to inflation.

      • Bill Bowman:

        So £200 million was saved by inflation. You say that the project was well managed but, if inflation came in at £200 million less than expected, that takes away quite a lot of the trumpeted saving.

      • Jillian Matthew:

        That was the range that was allowed for. The £200 million was the upper figure, but the saving was actually closer to the lower figure. When Transport Scotland was estimating the costs and trying to understand the various aspects of that, it was a very uncertain time. It was in 2009 and 2010, which was not long after the economic crash, and it was difficult to estimate what inflation would be for what was a 10-year project. Transport Scotland allowed for a range, but throughout the project—

      • Bill Bowman:

        When you say that there was a cost saving of whatever, how much of that was actually due to inflation not being as high as estimated?

      • Caroline Gardner:

        I am not sure that we are talking about cost savings; we are talking about the amount by which the total cost came in under the budget for the project.

      • Bill Bowman:

        Is that not savings?

      • Caroline Gardner:

        It is not savings in the way that you are describing. The project came in below budget, and international data suggests that nine out of 10 such projects do not; they overrun on cost or time. In paragraph 60, we set out the key changes to the cost between 2011, which is when the contract was let, and 2017, when it concluded. We break it down into things such as risk allowance, optimism bias, inflation, the costs relating to the principal contract and other elements of the project as a whole as opposed to simply the bridge construction.

      • Bill Bowman:

        I am trying to understand whether Transport Scotland did something good in the management of the project or whether it just benefited from inflation being less than was originally estimated.

      • Caroline Gardner:

        I would frame that a bit differently. I think that Transport Scotland let a good contract that placed the risk for those elements with the contractor rather than with Transport Scotland, and it was able to do that because of the work that had gone in beforehand, as Jillian Matthew suggested, to appraise the options and ensure that the form of the contract was competitive.

      • Bill Bowman:

        I do not want to labour the point, as we probably have other questions, but you have given an original number and said that the costs were less, so how much of that was due to inflation? That is all that I am trying to establish. It is about the estimated 5.3 per cent figure for inflation versus what it turned out to be.

        10:15  
      • Caroline Gardner:

        The price—

      • Bill Bowman:

        If you do not have the information, I am happy to get it later or to get an explanation later.

      • Caroline Gardner:

        I am not sure that we can add to that. Sorry—Jillian Matthew wants to come in.

      • Jillian Matthew:

        I have a breakdown of some of the costs and what they were at the beginning of the project and at the end. We are talking about ranges, and a lot of changes happened within the different figures for different reasons. At the start, the overall estimate was £1.4 billion to £1.6 billion. Within that, a lower figure of £91 million was allowed for price fluctuation or inflation, and the final figure was around £31 million. However, that was the lower range, and there was also a higher range. As we set out at the beginning of part 2, good project management involves putting in the time to understand the costs. Obviously, inflation is very difficult to predict. When the budget was being set, the Scottish Parliament information centre produced a report that examined the costs and it was satisfied with the way that the costings had been done, which included looking at economic reports. It was difficult to estimate what inflation would be after two years.

      • Bill Bowman:

        I understand that, but I think that an element in how well the project was managed is that there was good fortune because inflation was less than estimated.

      • Jillian Matthew:

        As we also say in the report, inflation is underestimated far too often, as is the element of optimism bias. We felt that those things were considered and built into the project very well. The situation was monitored all the way through and the budget adjusted accordingly.

      • Caroline Gardner:

        There was an element of good fortune but, as we say in case study 1 on page 21, a number of measures were built in to the contract approach that helped to deliver value for money. The fixed-price contract was one, as was the value-engineering approach, which let contractors suggest improvements to the project. All those things fed through so, although there clearly was an element of good luck with inflation, the project team did not just get lucky because inflation was low.

      • Alex Neil:

        I have two questions on the wider evaluation of the impact of the bridge. First, how broad and deep will the evaluation be? Will it be a wide-ranging economic and employment impact assessment or will it be narrower than that? Secondly, does the baseline data exist for making an objective evaluation at this stage? Is this not closing the door after the horse has bolted?

      • Caroline Gardner:

        That is a very good question. Exhibit 9, which is on page 36, sets out in graphic form the project’s eight objectives plus our assessment of whether they have been achieved or whether they are still to be assessed. The wider ones that you are touching on, such as supporting sustainable economic development and economic growth, are very definitely in the still-to-be-assessed category.

        Jillian Matthew might want to say a bit more about Transport Scotland’s plans for evaluation.

      • Jillian Matthew:

        Transport Scotland has plans to evaluate the impact at one year, three years and five years after opening the bridge. Some of the objectives, such as those on economic sustainability and development, are longer-term objectives. We say in the report that there was not enough detail in some of Transport Scotland’s plans for measuring some of the outcomes, although there was quite a lot of detail on issues such as traffic flow and other easier to measure things. Also, it was not clear at what point Transport Scotland was going to be able to say whether it had reached the outcomes. I do not think that Transport Scotland will have measured whether the outcomes have been reached by the time of the first evaluation, which is after one year, but we would not expect some of the outcomes to be achieved by that point anyway. However, it is not clear at what point we will be able to see what progress has been made on the different outcomes and whether that will be at three years or five years.

      • Alex Neil:

        To measure an outcome, you need to have figures for year zero, or the starting point before the bridge was built. Does Transport Scotland have that information?

      • Jillian Matthew:

        Again, that was less clear with some of the longer-term and more difficult to measure outcomes. Obviously, there is a baseline for things such as traffic flow. However, the methods that Transport Scotland described for how to measure some of the outcomes were set out at quite a high level. One of our recommendations is that there needs to be a lot more detail on how Transport Scotland is going to do that and what data it will use. It is talking about surveys and looking at decisions that businesses have made on setting up in areas around the Forth and how those have been affected by the project.

      • Alex Neil:

        Was there not some fairly well defined Treasury guidance—it used to be called the green book—about how to measure economic impact? Why is Transport Scotland not just following that?

      • Caroline Gardner:

        The green book focuses more on project appraisal than on project evaluation, but obviously you need to follow the line through. We say a bit in paragraph 66 about Transport Scotland’s plans for evaluation. The second bullet point describes its plans to compare pre-opening and post-opening employment patterns from a range of secondary sources. Transport Scotland has plans but, as Jillian Matthew says, we have not yet seen the detail of the particular data sources that it expects to use and the way in which it will pull that together at each of the three evaluation points. In broad terms, the plans are there. We are just raising the caveat that, at this point, the plans are not detailed enough for us to be clear that Transport Scotland will be able to demonstrate whether the benefits have been achieved.

      • Alex Neil:

        Do you think that Transport Scotland might not be able to do it, or has it just not got round to it and, if so, when will it get round to it?

      • Caroline Gardner:

        My experience over the past six years of doing this job suggests that people often do not pay as much attention to evaluation after a project has been delivered or a service has been developed. We feel that the plans that Transport Scotland has described to us are good in that context, but we cannot yet give you the assurance that it will be able to evaluate all eight of the benefits fully. Clearly, the ones on social inclusion and economic growth are the more difficult ones to evaluate.

        Graeme Greenhill might want to add to that.

      • Graeme Greenhill:

        I will be looking at that as part of the audit of Transport Scotland in the fullness of time.

      • Alex Neil:

        When is your next audit of Transport Scotland?

      • Graeme Greenhill:

        It is an annual audit of the financial statements. The first evaluation report is expected later this year, so I can have a look at that as part of the 2018-19 audit of Transport Scotland.

      • Alex Neil:

        We should ensure that that audit is circulated to the committee when it is published to see whether there is anything in it that we want to pursue. The whole justification for these huge expenditures is to a large extent based on the economic impact. Obviously, there was an urgent situation with the old bridge that in itself justified the investment, but we might as well look at the wider economic benefits for future reference, if nothing else.

      • Caroline Gardner:

        I absolutely agree.

      • Iain Gray:

        Some of those outcomes related to promoting public transport across the Forth. Paragraph 71 tells us:

        “Transport Scotland plans to publish an update on progress in late 2018.”

        We are almost in October now. Do you have any information about likely publication?

      • Graeme Greenhill:

        As far as I am aware, that is still the plan.

      • Iain Gray:

        But you do not have any indication of the likely publication date.

      • Graeme Greenhill:

        To some extent, the thinking around the public transport strategy will be influenced by the initial evaluation, which is due to be finalised around about now. I suspect that Transport Scotland will be feeding the result of that evaluation work into its wider thinking about public transport.

      • The Deputy Convener:

        I have a few wrap-up questions, the first of which is on cost. Key message 1 in the summary says that the final cost was £1.34 billion, but that total includes the costs from when the project was first scoped in 2007 up to the end of the five-year maintenance period in 2022. Given that that is projecting forward, is there a risk that the cost could increase?

      • Jillian Matthew:

        Not that we are aware of. Obviously, it is still to happen, but the on-going maintenance is being carried out by another contractor. The contractor that built the bridge is responsible for some of the snagging and initial repairs after the first year, then the responsibility for maintenance will be taken over by the other contractor that is looking after both the Queensferry crossing and the existing Forth road bridge. There are detailed costings around that and what is expected, but with a new bridge you would not expect there to be much maintenance cost involved.

      • The Deputy Convener:

        I want to bring it right back to Colin Beattie’s question about what went well. You talk about the co-location of the contractors and the Forth replacement crossing authorities, if you like. I think that you say that that allowed quick decision making and a fluid exchange of ideas. Could that be replicated fairly easily on other projects? Is it part of the best practice that should be replicated going forward?

      • Jillian Matthew:

        It will depend on the type of project. This one lent itself very well to that kind of set-up because the bridge was in one location. The site office was very close to the bridge, where all the construction was happening, and the benefit of that certainly came through strongly in our fieldwork when we spoke to all the people who were involved in the project, even without prompting. It helped that everyone was on the same site and was able to talk about issues as they arose. It meant that very good relationships could be built up within the FRC team as well as with the contractors.

        The success of that set-up is certainly in the lessons learned for future projects, but it will depend on how the project is set up. The nature of the dualling of the A9 project, for example, means that it is quite difficult to transfer lessons that have been learned. The A9 covers a vast area, so it is not easy to transfer things like that. However, that project is transferring other aspects of lessons learned around education programmes and the wider stuff around involving schoolchildren and universities and getting people involved in science, technology, engineering and mathematics. People would certainly look to replicate that set-up if a project lent itself to that, but you cannot apply everything to some projects.

      • Alex Neil:

        I have a wee supplementary, because that ties in with my question about the economic and employment impact.

        Is one of the lessons that the bigger the contract that is let—particularly when, for example, steel is procured from China—the smaller the economic impact will be on the Scottish economy? First of all, the big contractors take their profits out of Scotland and do not reinvest here, because they are not based here. Secondly, steel, which was a substantial part of the contract, is procured from overseas. The dualling of the A9 is being done in much smaller chunks for obvious pragmatic reasons, but for local Scottish contractors and indigenous employees, the potential benefits of that, such as training opportunities or apprenticeships, might be much greater than they would be if it was a huge contract like the Forth bridge contract.

        10:30  

        I realise that it would be very difficult to build the Forth bridge in anything other than one contract, but are we doing enough to maximise the economic impact? I worry at times that we are just taking an accountant’s point of view of these contracts and looking at the immediate savings to the public purse. When we look at the wider economic and employment benefits that we have potentially foregone, we see that the public purse might be losing more than it could have gained.

      • Caroline Gardner:

        I have to start by defending accountants and saying that we do not take a narrow view of just the immediate costs but instead look much more widely at what is being achieved. However, in broad terms you are right. Particularly at times when finance is tight, it is very easy to look at just the short-term cost and benefit of what one is trying to achieve, rather than the bigger picture.

        We know that best-value criteria let people take account of wider benefits and not just the cost in pounds of a particular project or initiative. There is room for best-value criteria to be applied right the way up through the economic strategy in thinking about some of the wider trade-offs that are involved not only in procurement, but other investments and services such as lifelong learning, retraining for adults—those sorts of things.

      • Graeme Greenhill:

        It is something that the public sector is growing more aware of. Increasingly, Transport Scotland is looking at the extent to which subcontracting to local companies is taking place in some of its big projects and at things such as the number of apprentices that are being taken on as part of major capital projects.

      • Alex Neil:

        I am thinking of Brexit. Obviously, there are restrictions on what we can do that are built into the Lisbon treaty and the like. Some European Union rules restrict how much preferential treatment we can give to local contractors, for example. Clearly, if those rules will no longer apply—we do not know yet whether they will after Brexit—there will be potential for enhancing the economic impact of procurement policies on the Scottish economy.

      • Caroline Gardner:

        I would feel more comfortable responding to that if we had more of an idea about what might happen when we leave the EU, so I will leave it there.

      • Alex Neil:

        Absolutely. I am an optimist.

      • The Deputy Convener:

        I have a final question on good practice and the successes. From your report, are you able to identify whether there was simply a confluence of serendipity-—all the right things coming together at the right time in a wonderful coincidence—or can you say whether the project’s success was a function of some key individuals and, if so, who those individuals were? Was it a function of some good planning at the start by certain agencies? Were there any key things that happened that can be replicated? If you can identify individuals, can they be used further?

      • Caroline Gardner:

        I will ask my colleagues to talk you through the handful of key things that people got right and which made a real difference. Although the project team got lucky on some aspects, such as inflation being low during the life of the contract, they also got unlucky on weather, which was worse than it had been in previous years. They had planned for the weather quite carefully. It certainly was not just serendipity.

      • Graeme Greenhill:

        I do not think that you could put your finger on a single point that made the difference. A number of factors were at play.

        Paragraph 21 on page 18 has a list of bullet points indicating factors that are critical to the success of major projects. The first three points are about planning. Good planning is essential—we are all aware of the five Ps when it comes to planning. The final three bullet points are about how you determine your likely costs, the extent to which you allow optimism bias, how you get independent advice on your costs and how you compare your expected costs with other similar projects.

        I would add leadership culture to that list. Leadership culture sets the tone for the entire project, and, as we have previously said, it is all about openness and transparency. It is all about willingness to discuss and negotiate problems, and all parties coming together. It is about people having clear responsibilities and there being governance arrangements in place to make sure that people are held to account for what they are doing.

        Success was due to an accumulation of factors, but the fact that the bridge was such an iconic structure played a part. There was a genuine source of pride in the project and everyone working together to deliver the bridge.

        If you were to push me, I would say that the project director came with a reputation that was highly thought of internationally, and he delivered.

      • The Deputy Convener:

        Can I press you on that? That is the point that I wanted to get to. We see an awful lot of bodies in here—agencies rather individuals. In most cases, there is no doubt that leaders have been as invested as you just described and have had the desire for things to succeed, but for some reason they have not been able to deliver. How much was the impact of the project director key to the project’s success? Is it just about leadership?

      • Graeme Greenhill:

        It is difficult to quantify the overall influence. All that I can say is that he was well thought of and pulled the whole thing together. Everyone was working in conjunction and co-operation.

      • Jillian Matthew:

        I think that you are trying to get to whether success is due to one person or one key thing. I do not think that it is. You could have all the good planning—the building blocks—in place and you could bring in an impressive person, but it could still go wrong if they do not have the right team under them, do not have the right costs or do not get the scope right. It is definitely a combination of all those things.

        The page that Graeme Greenhill referred to at the start of part 2 is where we tried to get at some of the key success factors. Paragraph 23 summarises them. If you have all the good aspects of project management in place, that is a good start, but it was the additional things—getting the right people with the right skills, getting people in early, and having good leadership, openness, transparency, team spirit and working relationships—that made the project work well.

      • Bill Bowman:

        I have a final comment as an accountant—although I am not giving an accountant’s perspective. If the money aspect was good, that is good. However, if users matter, I should say that I have been through the whole construction phase as a user and I am a continuing user, and my experience has not been good. I do not expect you to comment on that; it is just my view.

      • Caroline Gardner:

        We have to look at the project in the round. There are some real successes. We have made a criticism in the report in that we think that Transport Scotland could have kept people better informed about the need for further work after the bridge had opened, and I recognise that people’s expectations may have been for a bridge that removed congestion across the Forth, which was not one of the project’s objectives. I absolutely share your view that the user experience is an important part of this.

      • The Deputy Convener:

        As members have no further questions, I thank the Auditor General and her colleagues for their evidence this morning. I now close the public part of the meeting.

        10:38 Meeting continued in private until 11:02.