“Performance management in Scottish Enterprise”
Item 2 is a briefing from the Auditor General on his report "Performance management in Scottish Enterprise".
Mr Robert Black (Auditor General for Scotland):
In December 2003, I published a report on my special audit examination of Scottish Enterprise. That examination was a response to concerns expressed about target setting and performance measurement in Scottish Enterprise, its delivery of major projects and its use of consultants and European Union funding. The Audit Committee took evidence on the matter early in 2004 from Robert Crawford, who was then the chief executive of Scottish Enterprise. The committee's subsequent report stated that it would welcome a more detailed review of Scottish Enterprise's performance measurement arrangements at some point in the future.
This latest report is my response to the committee's suggestion. It considers the range of information available to assess the performance of Scottish Enterprise, the adequacy of such information and how it is used in Scottish Enterprise to encourage continuous improvement. The main finding of the report is that there have been a number of positive changes in Scottish Enterprise. However, it remains difficult to judge the organisation's impact on the Scottish economy. To a large extent, it is fair to say that that reflects the complexities of measuring the impact of an economic development agency—those problems are by no means unique to Scottish Enterprise. Nonetheless, I suggest in my report that Scottish Enterprise needs to strive to provide better and more complete information in this area. I know that it is committed to that objective.
My report begins with an overview of the Scottish economy's performance because it is important that a context is set for the analysis of Scottish Enterprise's performance. The overview is followed by a review of the available performance information. Finally, the report looks at what systems are in place to support the development, monitoring and evaluation of Scottish Enterprise's activities. I will briefly comment on each of those areas.
First, on the Scottish economic context, the Scottish economy is of course subject to a wide range of external influences that are beyond Scottish Enterprise's control. They include, for example, interest rates, exchange rates and the performance of the rest of the United Kingdom's economy and of international economies. Scottish Enterprise has a role to play in influencing economic performance, but clearly it cannot be held accountable for any improvement or deterioration in performance as a whole.
Over the past decade, Scottish economic performance has generally been weaker than that of the UK as a whole. Gross value added—GVA—has grown at a slower rate every year, except in 2002. The gap between Scottish GVA per head and UK GVA per head has widened. However, that comparison is heavily influenced by the performance of London and the south-east. It is important to stress that Scotland's GVA per head is higher when compared with most other parts of the UK.
It should be noted that economic performance in Scotland varies widely. Wealth generation is concentrated in the three largest cities—Glasgow, Aberdeen and Edinburgh—and to a lesser extent in Dundee and other urban communities. Scottish Enterprise's thinking is evolving to reflect that. Its city regions approach seeks to identify how surrounding areas can contribute to and benefit from the growth of the major conurbations and cities.
Let me turn to performance information. The economic environment sets the challenges in context for Scottish Enterprise. However, as I outlined in my introduction, it is difficult to assess the extent to which Scottish Enterprise is succeeding in addressing the challenges that it faces. In "A Smart, Successful Scotland", the Scottish Executive set out the key indicators for assessing the performance of Scotland as a whole, but there is as yet only limited evidence to show the contribution that Scottish Enterprise is making towards achieving the goals.
The performance information that is published in Scottish Enterprise's annual report is limited and focuses on the activities and outputs of the organisation rather than on the impact of its activities. The annual report provides qualitative comment on the previous year's performance and quantitative analysis that focuses on 10 progress measures that are based around activities and outputs, such as the number of business starts assisted or the number of people achieving positive outcomes from training programmes. In the operating plan, target ranges are set for each of those 10 progress measures. Clearly, that type of performance information is important, but it tells us little about Scottish Enterprise's impact on the economy.
A much wider range of performance information is collected by Scottish Enterprise internally and is used to inform management and guide decision making. In addition, Scottish Enterprise uses a balanced scorecard approach to provide a rounded view of the organisation's performance in respect of four perspectives: processes, customers, people and stakeholders. However, the information remains predominantly activity and output based; it does not give a comprehensive picture of Scottish Enterprise's impact or of how the organisation is contributing to the goals of "A Smart, Successful Scotland".
In the field of economic development, measuring impact requires in-depth evaluation of projects and programmes. Scottish Enterprise undertakes an extensive programme of evaluation activity, but not in a way that allows overall impact to be regularly assessed. Attempts to measure overall impact have been infrequent and highly approximate. The previous such attempt resulted in an estimate that Scottish Enterprise's activities in 2001-02 generated additional gross domestic product in Scotland of some £1.6 billion over three years. I should emphasise that that is Scottish Enterprise's own estimate; it has not been subjected to audit and is probably very approximate. The exercise highlighted the considerable difficulties and costs involved in making such assessments. No further attempts have been made. The current focus is on improving the quality of the evaluation evidence that forms the building blocks for such overall estimates.
Scottish Enterprise is by no means alone in struggling with such concepts. We undertook an international review as part of our study and failed to identify any simple solution to the problem of measuring impact in the field of economic development. We concluded that Scottish Enterprise's approach appeared to be as well advanced as that in any other country that we considered.
There have been positive changes in Scottish Enterprise, a number of which relate to the way in which Scottish Enterprise develops, monitors and evaluates its activities. That work continues. A new system known as the gateway process was introduced in September 2003. It provides a clearly structured and rigorous approach to the appraisal, approval, monitoring and evaluation of projects. As part of our study, we reviewed those stages for a small sample of the projects being undertaken by Scottish Enterprise. We identified some weaknesses at all stages. We found a lack of factual evidence to support the rationale for projects, and that the consideration of other options was insufficient. We also found a limited monitoring of outcomes and a lack of consideration of value for money or additional impact. Scottish Enterprise expects that the gateway process will help both to introduce much greater rigour to the process of developing activities and to improve the strategic focus of its activities.
In view of recent suggestions in the media about overspends in Scottish Enterprise, I will at this point mention a section of our report that relates to how resources are allocated. Until this year, budgets were allocated to each broad theme at the start of the financial year. They were then broken down into individual local enterprise companies. That might have given LECs an incentive to spend up to their budget allocation, with a risk of some projects being selected to achieve a spend level rather than being chosen on the basis of the economic justification of strategic fit.
In 2005-06, Scottish Enterprise introduced a major change in the way in which it allocates funds to projects. Budgets have not been allocated in advance, either by theme or by geographical area. Instead, project staff apply for funds as project proposals are approved during the course of the year. Applications are reviewed throughout the year and approved or rejected following review. There is no set budget limit for specific areas of activity. Financial control relies on robust forecasting, and strategic alignment relies on quarterly management reviews. I say in the report that that rolling system should help to ensure that projects are driven by economic justification rather than by a desire to spend up to budget limits. As a result, the selected projects should more closely reflect the ultimate objectives and strategic intent of the organisation, provided that the appraisal and approval system works effectively.
However, last year was the first financial year in which that approach has been used and, as my report says, management reports to the Scottish Enterprise board suggest that demand exceeded available resources. When the report was produced, it was too early to comment on the effectiveness of management action to contain expenditure, but audited accounts, accompanied by the final auditor's report, will of course be available later this year.
Scottish Enterprise appreciates the need for better quality and consistency in the evaluation evidence that it gathers. The emphasis is shifting towards broader and more strategic evaluations that focus on the impact on GVA, which should help in assessing Scottish Enterprise's overall impact. If used effectively, better and more consistent evaluation evidence should help to ensure that resources are directed at achieving the greatest impact. However, it is too early to judge whether those changes will have the desired effects.
Several other changes, which are designed to improve performance management in the organisation as a whole, are also worth highlighting. I draw the committee's attention to exhibit 31 on page 39 of the report, which summarises the changes. I will describe some of the changes. First, a customer relationship management system has been introduced, which will help to track business customers over time and provide important information for assessing performance and impact. Secondly, Scottish Enterprise is rationalising the range of products that it offers, which should help to sharpen the strategic focus of activities and make it easier to assess impact. Finally, a new strategic focus has been placed on key industries and metropolitan areas, which should also help in identifying where the maximum impact can be achieved.
I suggest that those changes are to be welcomed. They should form a basis for improved performance management in Scottish Enterprise and provide a clearer picture of the impact that it is having. However, as the committee well knows, Scottish Enterprise is a large and dispersed organisation. There is some way to go before staff throughout the organisation are fully cognisant of the positive role of performance management in driving the business.
As ever, the team from Audit Scotland and I are happy to answer any questions.
I will pick up the last point about people outwith Atlantic Quay grasping performance management. Did you find evidence of buy-in from people who operate outwith the centre?
We did not examine that in great detail. As I said, the team formed the general view as it produced the report that there was still work to be done in the 12 local enterprise companies to achieve a full buy-in for the new arrangements. The new project appraisal system is significantly different from the previous system, so changes will be needed in the local enterprise companies to address management of that system.
What you say suggests that Scottish Enterprise is pursuing a top-down rather than bottom-up approach. That links into your point that local enterprise companies receive only core funding and all decisions are centralised at Scottish Enterprise. Given my stated opposition to focusing on metropolitan areas, I am obviously concerned about that centralisation. When you did your investigation, were you aware that most decisions were taken without local support or local knowledge and could therefore have produced the financial crisis that Scottish Enterprise is supposed to be in?
Unfortunately, I cannot help you with a robust answer, because that issue was not examined in detail.
It might be helpful if I summarised our understanding of what is involved in the approval process. The process differs depending on the nature of the project. Large national projects are reviewed by the Scottish Enterprise board, while larger projects that are specific to the LECs are considered by the LEC boards. Very small projects do not require board approval. There is a hierarchy of decision making. In the paper, we comment on the importance of the boards effectively reviewing projects and challenging them at an appropriate level.
My question follows on from Margaret Jamieson's questions. I recognise what you have said about how resource allocation is changing. Would you like to comment further on the present system of 12 LECs and the change that will make it more metropolitan focused? In your view, did the present system help or hinder delivery and the attempt to meet targets? Do you see a variation in economic performance between Scotland's regions? Did your investigations indicate whether such a variation was influenced by local business and economic factors or whether it was related to the management and delivery of particular LECs?
The move towards a new, more centralised set of arrangements is really a matter of policy, so it is difficult for me to comment on that. You asked about the relationship between local economic performance and the performance of LECs. I cannot recall the detail of that, although my team may be able to help me. However, the early report that I produced included evidence of significant variations in what one might call the application of resources by LECs to different projects. There was a diversity that was not fully explained by Scottish Enterprise's strategy. In part, that has led to the new arrangements for evaluating big projects at the centre, to ensure that there is consistency and a clear assessment of the added value that is expected to come from individual projects.
Exhibit 12 on page 15 of the report shows the estimated GVA per head at LEC level. Clearly, the major conurbations are showing higher levels of GVA per head than other regions. We would expect that, given the metropolitan nature of the Glasgow and Lothian economies. We could not make a direct link between the activities of LECs and GVA per head.
I was not sure whether the differences shown in exhibit 12 were due to local circumstances or the additional value provided by Scottish Enterprise. Was there any indication of interaction between LECs? You spoke about the centre being able to assess projects across the board, because the information is comparable. Was there any indication of LECs interacting to learn best practice or to share experiences?
Bob Leishman (Audit Scotland):
As part of the changes to Scottish Enterprise's performance management, over the past 12 or 18 months it has set up a number of groups that bring together individual members of LEC teams, with the purpose of sharing best practice in a way that had not been done before.
The overall nature of the situation is complex, as is the Scottish Enterprise solution. Mr Black described a hierarchy of decision making at Scottish Enterprise and said that it is a "large and dispersed organisation". What restructuring is envisaged? How far advanced is it, and is a timetable attached?
My understanding is that the restructuring is no more than a proposal. I do not think that Scottish Enterprise has finalised that proposal. There have been a number of suggestions over the past months about restructuring local enterprise companies, but nothing concrete has emerged yet.
So there will be continuing complexity and dispersal of decision making until that takes place.
Yes.
Is Scotland in an exceptional situation or are comparators available? In other words, can you point to successful models for such a complex situation elsewhere?
Our international review did not find any country that could clearly demonstrate the benefit of economic development and its impact on an economy. We have highlighted in the report a couple of examples of countries that we felt had better information. Canada, for example, made much more use of modelling to track the impact of economic development. Ireland uses a longitudinal survey of assisted companies to get a picture of the impact of its assistance over time. Those are the kind of areas in which Scottish Enterprise might learn a bit, but no one has the answer to that question.
Mr Black, you said that it is difficult to assess whether Scottish Enterprise is contributing to the goals of "A Smart, Successful Scotland". Are there in-house assessment and performance measures or is outside assistance available to solve those problems?
In answering that question, it might be helpful if I were to ask the committee to bear in mind a series of exhibits between pages 24 and 27 of the report, which attempt to capture the performance targets that have been put in place at different levels. It is a complex read. One of the main findings of the report is that it is quite difficult to relate some of the more detailed objectives back to the high-level objectives of "A Smart, Successful Scotland". We suggest that more probably needs to be done to try to relate the activity of Scottish Enterprise to the Scottish Executive's high-level goals. The exhibits highlight the fact that Scottish Enterprise has a number of measures of its own activities that are not reflected in its annual report. In particular, the organisation uses a balanced scorecard—captured at exhibit 23—that is essentially designed to capture information on four dimensions: what its stakeholders are looking for; how good its processes are; what its customers are looking for; and issues to do with its people. The latter is entirely appropriate. We have included some examples of the information that is collected.
However, a great deal of that information is not translated into the annual report, which tends to discuss activities that have taken place during the previous year. We wonder whether Scottish Enterprise could do more to join up some of the information that is gathered and the evaluation that takes place in the organisation in a more effective way that would mean more for stakeholders such as the Parliament.
Is the essential problem that we are trying to get focus out of what is obviously a complex situation? Is there a case for reconsidering those high-level objectives for the purposes of revision and clarification, which in turn may help Scottish Enterprise to focus?
There is good joint working between the Scottish Executive and Scottish Enterprise to monitor progress against Executive targets, but on the basis of our analysis we suggest that more needs to be done.
At the beginning of his report, the Auditor General set out the genesis of this work and the background to it. Years ago, through this committee, some of us were involved in that. It is, therefore, simply coincidental that this report has appeared at a time when parallel discussions are taking place about Scottish Enterprise. As a member of the Enterprise and Culture Committee, which is considering current issues that have arisen, I plead that we exercise a little joined-up thinking and ensure that this committee's deliberations and the work that Audit Scotland undertakes be fed into the discussions that the Enterprise and Culture Committee will have over the coming weeks.
At this point, I will exercise a little joined-up thinking myself and, remembering which committee I am in at the moment, move to my questions. For the avoidance of doubt, are you saying that, having conducted international comparisons, you believe that although there is scope for improvement Scottish Enterprise's system of performance management broadly compares favourably with that of equivalent agencies elsewhere?
That is a reasonable summary of the situation. Scottish Enterprise's thinking about how to evaluate impact is as advanced as that of any agency that our study examined.
That is helpful.
Would it be a fair summary of your report and your comments today to say that you believe that the new system for project appraisal is, as you say in your report, a "potentially helpful development"? In other words, regardless of the problems that have arisen this year with the operation of the system, is it your general view that the direction of travel in relation to project appraisal in the agency is good and that that approach should continue and be improved on?
That is a fair summary. The phrase "direction of travel" is appropriate in this context because the situation is dynamic. It was only in 2005-06—the financial year that has just ended—that Scottish Enterprise introduced its new system for allocating funds. It is therefore at an early stage. As I outlined, the new system is designed to ensure that any significant projects that come forward for approval are evaluated robustly in order to demonstrate the added value that they should bring. We believe, however, that Scottish Enterprise has some way to go before it beds that down, not least in relation to the quality of management information that we have available.
I note that you have talked about some areas that could be improved, which is precisely the sort of information that could feed in usefully to discussions elsewhere.
The overspend that has arisen this year—it has been suggested that it is about £30 million—has been widely talked about in recent weeks. I do not want to lead you into an inappropriate area, but words such as "financial crisis" are bandied about frequently in the world in which we live. You examine financial situations in every public sector body across Scotland. Can you benchmark SE's current financial situation for us and tell us how significant it is? How does the figure of £30 million compare with the organisation's overall budget?
I am not sure that I can do that directly and you will be aware that I am reluctant to say too much until I have the audited accounts and the report of the auditor.
However, I can say two things. First, for as long as I can remember, two features of public expenditure on major projects have been that big budgets tend to be underspent and there tends to be slippage in programmes. It is therefore entirely appropriate for bodies such as Scottish Enterprise to try to find ways to keep the programme up to the mark in terms of a spend profile that matches the resource allocation. That carries the risk that it might at the year end incur an overspend within one financial year. However, that is not the same as suggesting that there is a fundamental problem with financial control, taking one year with another. I hope that, when we get the auditor's report on the audited accounts, I will have more information to give the committee about the true extent of the underlying problem.
That is helpful. Thank you.
My final question is on the structure of the organisation. I think that I am correct in saying that you produced your report before the minister made a statement to Parliament on the issue. It was therefore written when the direction of structural travel in the organisation was towards a metropolitan region arrangement. Subsequently, ministers said that they will retain 12 local enterprise companies.
Again, without commenting on the policy decisions, will you say how that latest development will impact on the issues that you raise in your report, including the point that Andrew Welsh mentioned about the complexity of the organisation? You state that the organisation is "large and dispersed" and you link that to difficulties in effective performance management throughout the organisation. Given that the decision to retain 12 LECs was taken after your report was published, how might your observations change in the light of that subsequent decision?
I am not sure that I can answer that question terribly well because our understanding of Scottish Enterprise's intentions for its reorganisation is quite limited. The question should be addressed primarily to the management of Scottish Enterprise.
When might we expect the audited accounts to come before the committee?
The planning timeframe is that the audited accounts should be available in September. The final audit report normally follows a while after that, so I suggest that it will be October or November before I am in a position to prepare for the committee a report that will be based on the audited accounts. I will decide whether to prepare a section 22 report when the full facts are available from the auditor.
So we will not get that report until after September.
It will be a good while after the summer.
I refer to paper EC/S2/06/05/1, which is a summary of your report. It states:
"Key findings include that:
• The performance measurements used by Scottish Enterprise provide a limited picture of the agency's achievements and focus on activities and outputs, rather than impact.
• The Scottish Executive monitor progress towards the ambitions set out in A Smart Successful Scotland, but it is difficult to assess well the contribution that Scottish Enterprise is making towards these goals.
• The evaluation evidence available to assess the impact of Scottish Enterprise's activities is of variable quality and the methodologies used lack consistency."
Although Scottish Enterprise might compare favourably with other agencies internationally, is it fair to say that there is a general lack of information that allows a conclusion to be made about the agency's impact?
It is fair to say that there is a lack of publicly available information and that the evaluation evidence within Scottish Enterprise is variable in its quality and coverage. It is also fair to say that the management of Scottish Enterprise is aware of that and is working to improve the situation. There is evidence for that, first, in the balanced scorecard framework that it is developing, which is summarised in exhibit 23 on page 27, and secondly in the exhibit on page 39, which summarises the changes that are under way within Scottish Enterprise. In that exhibit, we list some of the changes that the organisation is introducing that will help to improve performance management. All are significant steps in the right direction.
We will have an opportunity to discuss the report under agenda item 4. Do members wish to raise any points now?
The report states:
"Changes to the systems for project appraisal, approval, monitoring and evaluation should help focus activities on the impact that can be achieved and deliver better performance information."
The changes are helping, but how effective are they? How and when will we know whether the changes have worked? What is the gestation period?
That question is probably best put to the management of Scottish Enterprise. Do the team have any information to add to that?
Not a lot. That would need to be followed up. We are talking about systems that have been introduced in the past 18 months to two years, which will take two or three years to bed in. You would need to evaluate their success as a specific exercise. We expect Scottish Enterprise's management to evaluate the changes on an on-going basis.