“Review of major capital projects in Scotland—How government works”
Item 3 on our agenda is a section 23 report on major capital projects. We are joined this morning by Sir John Elvidge, the permanent secretary, and by Alyson Stafford. Welcome to you both. We have not had the privilege, Sir John, of your attendance at the committee in this parliamentary session—although I do not know whether you have attended in previous sessions. We look forward to your contributions on two agenda items this morning. Thank you for coming along. Before we move to questions, do you wish to make some introductory remarks?
Only briefly—and thank you for the opportunity.
Thank you. Would Alyson Stafford like to say anything at this stage?
I have nothing to add. I am happy to respond to the committee's questions.
Before we move on to the committee's detailed questions, Sir John, can I ask you about your 200 trained reviewers? You said that they come from a broad range of backgrounds in the public sector, but how many of them have practical experience of major projects as distinct from academic knowledge?
It depends what you mean by practical experience. Almost all of them have practical experience of overseeing projects in a senior management capacity. Some of them also have practical experience in more direct project management roles. I do not have a feel for the breakdown of the skill set. I do not know whether Alyson Stafford does.
Have many of them actually managed projects and taken decisions about how they develop rather than just having oversight and an interest in projects? I do not mean any disrespect to academics, but they can pontificate on things without ever putting a hand to the particular task. I am trying to get a feel for where the 200 people are coming from.
Sure. I can certainly say that one would not describe many—if any—of them as academics. They all bring their working experience to bear on the process. For example, when I talk about the university sector, I am more likely to be talking about people who have been involved in the capital projects that the universities develop than about people with an academic interest in the issues. What we have is essentially a practitioners club—it is a peer-to-peer club.
Would it be possible to get some further information on that? It sounds interesting.
We are happy to arrange to provide that extra information. It is important to recognise the strength that we get from having a wide range of people from different sectors. Their experience is useful and they can draw upon it in providing peer reviews.
It is clear from Audit Scotland's report that, in many cases, the time and cost estimates for major capital projects are wildly optimistic and the final outturn costs are substantially higher. Will you say a little about the work that you are doing to try to improve the cost and time estimates for projects?
Certainly—I will say a little and then ask Alyson Stafford to continue, because that question relates to the work of the infrastructure investment group.
I will talk about the work around costs and then about the work around timing.
Thank you for that very full response. It sounds as though that is work in progress. Willie Coffey has more questions on that issue, but before I pass on to him, I have a question on a related topic. In relation to the departure of final costs and times from the estimates, has the Scottish Government done any comparisons between traditionally funded projects and those that are funded through public-private partnership or the private finance initiative?
For a comparison between the two, I would need to arrange a specific analysis to be shared with the committee. There is a distinction in the early stages of work before a contract is let. I am sure that colleagues will know that in PFI activity a longer period is required for project definition and specification. We want some of those disciplines to be supported in conventional projects, too, because the more we define up front, the more closely aligned are the early estimates, the prices at contract and the final delivery price.
You say that a standardised method of calculating the whole-life costs of projects has been published. Will that be applied to all projects henceforth, or will you apply it to projects that were commissioned in the past year and are still to be completed?
The expectation is that it will be applied to projects henceforth but, because it is part of a continuous review of existing projects, it is only sensible that we draw on it to determine whether any realignment is required.
So if we were to ask for the whole-life cost of any major public sector project that is still being developed, we would be able to get it?
You would for the new ones. We would need to test the standards at a particular point and it would depend on how developed the procurement project was. To be fair, we would have to consider it project by project.
Sir John Elvidge said that the level of good performance doubles as we move through the phases of a project. Audit Scotland's report showed that, between 2000 and 2007, 60 per cent of a broad range of 41 projects completed over budget and 66 per cent of them delivered late. I ask Sir John to comment on that in light of what he said about the level of performance doubling.
I was cross-referencing two statements in the Audit Scotland report, which I am not fast enough to find in the text. Broadly speaking, the first says that around two fifths of projects come in close to the original cost and time estimates and the second says that around four fifths come in close to the contract-stage cost and time estimates. I referred to the relationship between the two statements. Without taking up a lot of your time scrabbling through the report's individual paragraphs, I cannot give you the paragraph references, but I am confident that I am using the report's language in both cases.
If it helps, paragraph 26 on page 12 of the report and the exhibits above it demonstrate the movements in cost.
The green book guidelines apply from project inception. A lot of the projects that Audit Scotland examined started quite a bit before those guidelines were introduced, but the infrastructure investment group and the gateway process are part of ensuring that there are other means of ensuring that that work takes place and that the guidelines and policies that exist are followed. Gateway review was introduced in 2005 and it will be a matter of working the stages through. Gateway reviews are going through their various stages in parallel with projects as they develop.
Audit Scotland mentioned that estimates of construction costs varied wildly in comparison with those for general costs. That did not appear to be taken into account; it was as though there was a presumption that inflation would be the same for both, when in fact it was clear that construction costs were going over expected levels. That fact did not appear to fall back into project planning for future projects. Why?
That is a fair observation about the projects that were initiated in the timeframe covered in the Audit Scotland report. The infrastructure investment group focuses on that area, builds in such considerations and ensures that we keep in touch with the relative inflation rates at any particular time.
Stuart McMillan has some questions on skills and expertise.
The Audit Scotland report says:
Evidence suggests that smaller bodies often do not have sufficient in-house skills. I was struck by the distinction made in the report between the performance of organisations that undertake many capital projects and those that do so infrequently. As one would expect, we have seen a clear difference in that respect. One thing we know is that smaller organisations are less likely to contain the necessary expertise to do anything that they do infrequently. As a result, we are not seeking to build expertise in every organisation, because in some organisations such expertise would be redundant some of the time.
Is that akin to what is happening in the police service with the Scottish Police Services Authority? It provides back-office functions for all eight police forces in Scotland. Is that the idea with the construction procurement team?
It is different, in the sense that I would characterise work with the SPSA as the development of a shared service. It is bringing together in one place a capacity to do the job that everybody draws on. We offer individual bodies more of a source of advice, and we do not lift the management of an individual project away from a body to the central team. One is essentially a shared execution model; the other is essentially a shared advice and support model.
So a separate entity has not been created; the people are still in the current central Government unit.
Yes.
What systems exist to ensure that project teams employ recognised quality standards in the planning and delivery of projects?
That question takes us to the heart of one of the general issues: the distinction between giving people guidance and expertise on which to draw and what we might call enforcement—ensuring that people do what they are exhorted to do.
The infrastructure investment group is made up of the lead directors across the Scottish Government portfolio areas. Each of those directors has a relationship with a number of arm's-length bodies, either by being a Fraser figure for an agency, or by playing a leading sponsor role through a sponsor division in their own area.
Does your system act after the event or can it pick up on problems before they become major? You mentioned the arm's-length principle in your relationship with outside bodies, and you said that an alarm bell sounds if something goes wrong, but is that system reactive rather than proactive?
Yes and no. It is reactive in the sense that it is a review process so it always comes in after the event. However, it comes in not at the end of the project but at stages throughout the project. In that sense, the intervention can be made between the point when best practice is not being followed and the point when the consequences of that materialise in the way in which the project is being taken forward.
Does that mean that you, as the principal accountable officer, are made aware of problems that are developing in a project, rather than the end of the project being reached before someone reports that there is a problem?
I would not be made aware of that personally, except in the one area in which I am a budget accountable officer as well as the principal accountable officer. The accountable officer for the portfolio budget will be made aware if the gateway review shows that there is a problem with a project. That is one of the changes that we made in the past couple of years.
Are you suggesting that the accountable officers do not necessarily alert you, as the principal accountable officer, to emerging problems?
Yes. Broadly speaking, their first instinct will be to try to fix the problem rather than tell me about it. In honesty, it probably depends how severe the problem is. I regularly talk to all the accountable officers one to one—on average, no less than once every two weeks—so if something is particularly worrying them about a major project, they will tell me. However, they do not automatically pass on the warning of a red gateway review assessment.
Let us consider a specific example. Were you made aware of emerging problems in the Stirling-Alloa-Kincardine railway project?
Yes.
At what stage did that happen?
You are testing my memory there.
If you cannot remember, perhaps you could revert to us with that information, because the issue is important. Are problems being picked up, and if so, what action is being taken? When you reflect on that and revert to us, will you tell us how many projects were brought to your attention as having significant risks in the past two years?
I will do my best. As I said, the alerting normally takes place in conversation rather than on paper, so it will take a bit of digging into people's memories.
Can I tease that out? We are talking about major projects that involve millions of pounds, yet in terms of an audit trail and accountability, in general nothing is put on paper. Conversations take place. Is that not a strange way to manage a serious potential problem?
We are getting into the theology of levels of accountable officer responsibility. I designate a series of accountable officers to manage discrete budgets because, if everything flowed to me, individual issues would not receive the attention that they need. The model is not built around the idea that I intervene personally in everything. The purpose of having the designated accountable officers is so that they take responsibility for dealing with issues within their budgets and tell me only if they experience some particularly acute problem. There is no institutional complacency—some of the most senior people in the organisation are brought in through their accountable officer roles to bear down on the problems.
I presume, therefore, that there must be records of warnings and alerts at accountable officer level, rather than at your level. From an audit perspective, those should be available if anyone wants to find out when a problem was first identified and what action was or was not taken. I accept the fundamental point that you cannot be involved in the micromanagement of every project. However, you are responsible for managing the accountable officers. How can you manage them, and how do you know whether they are doing a good job, if they do not tell you that there is a problem and there is nothing on paper to indicate that there is a problem?
My safeguard system is sitting beside me. There is no realistic prospect of an accountable officer concealing from me a problem that has not been successfully addressed, not least because Alyson Stafford, as director of finance, would pick up the evidence of that, and she provides a second channel of information to me. I can say to you, hand on heart, that I do not suffer from accountable officers not telling me when they have problems.
That does not necessarily answer my question. I asked how you know whether they are doing a good job—whether they are competent—if you have not seen any written evidence of how they are managing.
I have a series of measures of whether they are doing a good job. Again, the infrastructure investment group's work is important in that regard because it gives me assurance across the piece about how individual projects are being managed, so I certainly find out about failures. I also find that people are not particularly reticent about telling me about successes, so there is not much danger of me not knowing about the good bits.
You say that you find out about failures, but does that happen after the event? We are more concerned that you should be in a position to take action to prevent emerging problems from becoming failures.
Let us try to be clear about what we are talking about: we are discussing the example of a project that had got into difficulty and in relation to which an accountable officer had not taken successful action. It must be right that the accountable officer is given the room to do their job, and I am confident that, if successful action was not taken, Alyson Stafford would tell me so through her overview. There is no absence of channels for telling me when the management of a major capital project might be in difficulty.
Okay. Does Alyson Stafford want to say something?
Yes. This would be a good point to say something about the gateway process.
Could you do so succinctly? We have a number of other questions to ask.
Of course. There are five stages in the gateway process: business justification; delivery strategy; investment decision; readiness for service; and gateway 5, which is the operations review and benefits realisation, after the delivery of the project. At each stage, a review is held by the review team that we described earlier. At times, more than one review will be held for each stage; there is scope for multiple reviews.
Before I bring in James Kelly, do Stuart McMillan or Andrew Welsh have anything further to ask on this issue?
You have set up a structure in Government, with a gateway team, a gateway review and a five-stage process, but we also have to consider realities on site—as we know only too well in this building. Reports come in, but how can you compare those reports with what is actually going on? How can you close the reality gap, to ensure that the information that you receive is accurate and states what is really happening on site? For example, we saw the flaws in the M74 completion project.
Gateway, with its reviews and peer reviews, is our mechanism for that. When I described the make-up of the review team, I said some external people are always on the team. When I say "external", I do not mean simply someone from another part of the public sector in Scotland. Although we have trained reviewers from the universities and from the NHS, for example, we also get external people on the review team. That offers another check and another degree of independence.
In life, we tend to hear of disasters rather than successes. Without breaching confidentiality, will you give an example of the system picking up and solving a problem?
I have an example in mind, but the confines of confidentiality are a problem. I will think about what I can say. The example that was prompted by your question is of the system being given what was in effect false information about what was happening on a project.
Did the system pick that up? That is what I am after.
The system picked up the situation and it worked. The project's management was changed and, at some cost, we got back on track. My difficulty is that, for reasons of confidentiality, I can tell you nothing about the identity of that project. That is not tremendously helpful for you, because you must trust me that it exists.
I do not know whether Sir John Elvidge has just encouraged people to think about freedom of information requests. He might have made a rod for his own back.
Post-project evaluation is critical. It looks back to decide whether a project has succeeded and allows lessons to be learned for future projects. Alyson Stafford spoke about the fifth gateway review, which was the benefits realisation process. What systems measure success in achieving the benefits of a project that are stated in its business case?
Audit Scotland's report brings out the fact that the need to improve is strongest at that stage. I return to the distinction between having guidance and enforcing it. For some time, the guidance has said that post-project evaluation should be mandatory. It draws a distinction between two kinds of post-project evaluation. A self-contained process, which we call the post-project evaluation, evaluates how the process went and goes into issues such as how effectively the project was scoped and managed, which the report mentioned. A separate process, which we call post-occupancy evaluation, is about whether a project delivered the benefits that it was intended to deliver, never mind whether the most fantastic procurement job in the world was done.
That links into the infrastructure investment group and the gateway process overall. At the helm of the work on post-project evaluation and post-occupancy evaluation is the Government's construction advice and policy division. The procurement manual for construction schemes sets out how performance is measured in specific areas. That means that not just a subjective assessment, but a substantive assessment takes place. It is important to do such assessment and to have comparability with other projects.
Where the business case states that there are particular cost benefits of going ahead with a project, will the benefits realisation process in the fifth gateway review look at the statement of those cost benefits and at whether they have been achieved, or are in the process of being achieved? Can you make that clear?
Yes. I will draw on the post-occupancy evaluation checklist that is used in reviews. As a minimum, it assesses the achievement of the business case objectives to date; the benefits to date against those that were forecast and the other benefits that were realised and expected; the extent to which there is continued alignment with the business strategy; the effectiveness of actual improved business operations, which may include function processes and staff numbers; the ways in which to maximise benefits and minimise whole-life costs; the risk assessment around that going forward; and business and user satisfaction. We also expect regular post-occupancy reviews to take place over the operational life of a facility. That chimes very well with gateway, which is not a single event process—we look to gateway 5 being achieved.
Learning the lessons from projects and feeding those into future projects is important.
The change is that it has been made clear that the gateway process is now mandatory for all projects that are mission critical and/or high risk, with a budget of more than £5 million. That is the clear standard by which the Scottish Government must live and work.
There is no consistent approach to the provision of information on major projects and no regular publication is made available to the Parliament or the public to show how major projects across Government are performing. What plans are there for the Scottish Government to report publicly on projects?
I will say something initially. We are on a journey. We now have two infrastructure investment plans, the first of which was published by the previous Administration in 2005, and the second by the present Government in 2008. The plans provide the first step towards a clear overall picture of investment activity. The second step is the database that the infrastructure investment group has constructed to provide a framework for tracking all projects.
Before we bring Alyson in, is the journey one on which you are doomed to wander for eternity, or is there a specific timetable? If so, what is it?
It is tempting to say that improvement is a journey that never ends. I am genuinely always hesitant to say, "Once we get there, we will have done everything that can possibly be done and that is it." This is not a "Maybe sometime" journey. We are looking to take regular steps and make progress. That is where the relevance of the work that Alyson Stafford is leading will help us to get a handle on the likely timescale.
Yes, but will you then report to Parliament and tell us what the target is for each step so that we can have some assurance that you are making progress?
I am sure that we can give you an account of our ambitions for the steps on the journey. We can say quite clearly that the next step is to develop our database, and Alyson Stafford might be able to give us a sense of the timescale for that.
Infrastructure projects have been going on for a long time. You have just said that you have mechanisms for monitoring them, so you must know exactly what is going on. Earlier, I mentioned the reality gap. What information is available as the project progresses? Is the project on schedule? Are things being delivered? Surely all that information should be going through the monitoring structure that you already have. What is the problem?
I will respond to both those points.
We are talking about informing the public and Parliament of progress on projects. We are talking about types of information. Everybody understands that some information will be confidential, but we want to know about the reality of a project as it progresses. We are also talking about clarity. It is important to give the public access to information and to state clearly what is happening. Projects have a start, a continuing process and an end. It would surely be appropriate from the start of such a process to let the public see simpler information and it could build up to greater complexity. The information that the committee would seek is probably different from what the general public would want. It becomes complex if you get all the information that is in the system.
I am happy to look at the read-across from what we are talking about today to Scotland performs and to see how that links in. Scotland performs is about charting our progress on outcomes, and infrastructure is part of achieving those outcomes. I am happy to take that suggestion away and work on it with colleagues.
We are all interested in outcomes: the end-product is what is important.
Yes.
Can I move on?
Nicol Stephen would like to comment.
My question relates to that point.
We will have to come back to you on the detail. I know enough from my conversations with the ministers to know that they are happy with the information flow that they get, but I do not know enough to describe either its form or frequency. We will come back to you with precise information.
Yes. The aggregate information is drawn on each month when I pull together all the information that comes from the feeds of different parts of the Scottish Government on the financial performance of the Scottish Government. Alongside that, there is a feed of information that breaks down the capital figure of our budget into programmes of activity with a specific attention on programmes within each area of £50 million or more. There is a degree of aggregation when looking at the whole picture. That information comes through and I use the infrastructure investment group gathering to scrutinise that and test the confidence of the individual directors who lead on those areas in the robustness of the information.
On maximising the use of budgets, what systems are there to ensure that priority is given to projects within the capital investment programme that will provide direct benefits?
That is an interesting question. The investment in individual capital projects is—as you would expect me, as finance director, to say—linked to budgetary decisions. It is fair to say that those budgetary decisions—the decisions about how much is allocated across individual portfolios and how those portfolios apply the funds—are very much ministerial decisions. Ultimately, it is for the Cabinet to decide on the mix across all the portfolios, and there are decisions and prioritisations within that. That is very much part of the whole area of ministerial decision making on prioritisation. The investment must be linked to the budgets, and that is where the conversation takes place.
Can you assure us that project proposals receive robust, independent challenge before they are accepted into the major capital programme?
That depends on what the word "independent" means in that context. I think that we can assure you that there is an increasingly robust challenge through the mechanisms that Alyson Stafford leads. That is a robust function not just of the finance directorate, but of a wider circle of colleagues with expertise in overseeing capital projects. In that sense, the answer to your question is yes. If you asked me whether there is a routine external challenge, the answer would be no.
I would like assurances about such challenges. At the beginning of the M74 completion project, there was no clear plan for managing and controlling the project's total cost. The only allowance for risk was implicit within the range of cost estimates for the project—there was no explicit allowance for risk or bias. There was no agreed basis for accounting, controlling and managing the significant risk element in the estimate, nor any explicit strategy for doing so. There were also various other flaws. Can we be assured that that will not be repeated?
Yes, I think that you can. Part of the answer is the development of Transport Scotland as a change that the previous Government made to the way in which transport projects are managed and the evolution of what that body of specialist expertise is able to bring to project management in the field of transport. The other part of the answer is the evolution of what we might call the challenge community, which Alyson Stafford leads through the infrastructure investment group.
So, there is a mechanism that has learned from past lessons.
Yes.
The Audit Scotland report recommends that the Scottish Government strengthen strategic direction and investment planning through stronger, Government-wide co-ordination and more constructive challenging of projects to help to deliver them better. Both Sir John Elvidge and Alyson Stafford have spoken about having a database that can provide a snapshot of what is happening, but that does not give any real-time update and it is not a check across the portfolios on major capital projects and performance outcomes.
We will do our best. I am disappointed that we have left you unconvinced about that. The work of the infrastructure investment group is, essentially, our response to that recommendation in the report, with which we agree strongly. My judgment—it is easier for me to say this than for Alyson—is that Alyson has driven forward our capacity in a marked way over the past year and a half using that structure.
It is fair to say that the infrastructure investment group has focused on ensuring that we get the maximum out of the budget. We can be judged by our outcomes, and in the past two financial years we used all the capital budget—which was in excess of £3 billion in each of the two years—to within a margin of less than £10 million. That evidence reinforces what we are saying about our work.
Shall I move on, convener?
I am watching the time. If you want to ask one further question, that is fine; but if there is anything else, we can follow it up in writing.
How does the Government decide on its priorities for projects? Also, we have heard from the Government about the Scottish Futures Trust. What improvements in the procurement and project management of programmes will come from that?
Those are two big questions. I will try to give some headline responses. We need to recognise that Audit Scotland's report describes project commissioning over a long period, in which the answer to your question about how prioritisation is done changed quite a lot. I will leap into the present. Now, prioritisation is achieved through the Government's structure of objectives—the single overarching priority of economic growth, the series of targets that relate to that and the five strategic objectives. Prioritisation is now handled in a way that tries to bring projects back to their contribution to those overarching objectives.
We will reflect on whether we have covered all the issues. If we have not, we will write to you. I thank you and Alyson Stafford for a full session. Some of the evidence was technical and specific. We might well wish to return to issues that relate to capital projects or other matters. We all realise the significance of good management and of using scarce resources efficiently. As Sir John Elvidge said, Audit Scotland's report was exceptionally good. It should drive better services for the public.
Meeting suspended.
On resuming—