Audit Committee, 15 Nov 2005

Meeting date: Tuesday, November 15, 2005

Official Report 137KB pdf

---

“Scottish Executive: supporting new initiatives”

Item 2 is the Auditor General's report "Scottish Executive: supporting new initiatives", which is part of the "How government works" series. Caroline Gardner will brief us on it.

In November 2002, the Auditor General published the report "How government works in Scotland", which was essentially a reference document that explained the organisation of government in Scotland, the responsibilities of public servants and how they are held accountable, including the role of public audit in all that. We are following it up with a series of reports that comment in detail on aspects of the public sector's business that cut across individual organisations' responsibilities.

"Scottish Executive: supporting new initiatives" is the first report in the series and it examines how the Executive funds a series of initiatives throughout Scotland. It is worth mentioning that we are due to publish the second in the series later this week; it is on leadership development throughout the public sector.

The report on supporting new initiatives focuses on the growing trend for the Scottish Executive to work with partner organisations to deliver some of its policy goals. In 2004, the Executive announced 74 initiatives of that type; the overall funding will exceed £1.4 billion over time. Most of the funding will be distributed through a range of partners including local authorities and the voluntary sector. The funds available for the initiatives range from very small amounts—£0.3 million in one case—right up to substantial amounts such as that for the community regeneration fund with more than £300 million.

The report examines the Scottish Executive's arrangements for managing and monitoring the funding for a sample of 20 initiatives. In each case, the arrangements were assessed against good practice principles that Audit Scotland has developed. In a number of important areas, we also sought the views of delivery partners. That matters as much as it does because of the increasing emphasis on community planning as a means of planning and delivering public services locally, and because that can allow local authorities and their partners to develop a vision for their area and to decide who is best placed to deliver particular parts of that vision. Obviously, funding for initiatives can either contribute to work that has been well planned and co-ordinated, or get in its way.

The initiatives are relatively new and at this stage we are not examining whether they are efficient and effective. Instead, we are considering whether the Executive has arrangements in place to allow it to monitor what the initiatives are achieving and what is happening with the funding.

We have considered six key areas: the project strategy for each initiative; the way in which funds are allocated; the delivery arrangements; the contracting arrangements; monitoring and evaluation; and the views of delivery partners.

Overall, we have found that the management of initiatives is generally sound, but two important lessons have emerged. First, there is significant variation across the Executive, and there is some evidence that best practice in one part of the Executive has not been applied elsewhere. Secondly, there is scope for improvement in a number of areas, particularly in the setting of objectives, the agreeing of measures of success, the linking with partners daily, and the monitoring and evaluation of outcomes.

I hope that that has given committee members a sense of what the report is about. As always, I am happy to try to answer questions.

How did Audit Scotland decide which areas it would examine?

Do you mean the initiatives that we considered?

No. You said that you considered six areas. I was struck that those areas—and therefore the emphasis of the report—are geared towards the processes and internal arrangements of initiatives rather than towards the experiences of those on the receiving end and towards the impact of the investment. Why are you looking at one side of the coin?

You are right: we have focused on process. That is partly because, at this stage, it is not possible to consider the impact of each initiative. Most of them started in 2004. They will need to run for longer before we can consider the outcomes. The Executive will have to have these process elements in place at the outset, to work it through.

A significant part of our study was to seek the views of delivery partners—from local authorities right through to voluntary organisations. The report contains a lot of information on the issues on which those partners had generally positive views and the issues on which they felt there was room for improvement. We have considered the views of partners who made bids and then delivered services on the back of their funding.

I appreciate that answer, but if anything it reinforces my view that the report is just on one side of the coin. I am pleased that Audit Scotland is paying ever greater attention to wide-ranging aspects of how government works, but I have to be honest and say that I am very disappointed that the report does not address whether substantial areas of investment are joining up at the front end.

You said in response to my question that you sought the views of partner delivery organisations. That reinforces my concern not just to know what all the delivery bodies feel about this but, crucially, whether it is making a real impact on those who are supposed to benefit. Do the initiatives benefit the individuals and communities they are supposed to benefit? There is also a huge question about whether the stated objectives are being achieved.

Perhaps we could explore that later in the meeting, but I would be grateful for any comments Audit Scotland has. I was surprised and disappointed at the emphasis.

I recognise your point. We are looking at this piece of work from one end of the telescope, if you like. There is also a parallel piece of work—a joint study by which the Auditor General and the Accounts Commission will look at community planning. One of the issues that that study is looking at is how easy it is for local authorities and their partners, statutory and otherwise, to join the range of priorities and the funding available for them. They want to come up with a coherent set of strategies for what they are trying to achieve and to deliver that in ways that not only produce savings and efficiencies but outcomes for the communities they serve. This piece of work is one cut across that, but there is a separate piece of work that will look at the much bigger area of community planning. It will, I hope, join some of those pieces together.

I note that, too. However, community planning is just one part of the jigsaw. In previous meetings, the committee discussed the concern that community planning too is getting lost in internal process and organisation rather than actually achieving its stated aims.

I return to the question how and when outcomes will be assessed. A substantial part of the report talks about having in place yet more refined monitoring arrangements. I am not saying that such monitoring is not important, but if monitoring is to become ever more finessed, what we learn from it can be enhanced. When will we see in any objective shape or form whether the results are being achieved?

The point that we try to make in the report—although we are clearly not doing it very well at the moment—is that to demonstrate whether results are being achieved, the Executive needs to be clear at the outset what the intended outcomes are.

For example, local authorities and their partners spend a significant amount of money under the community regeneration fund, so a great deal of monitoring and evaluation is going on. The concern that we raised is that it is not entirely clear what outcomes the fund intends to achieve. Unless the process work is done properly, it is very difficult to establish what they are.

We keep coming round the loop to the question, "Were the intended objectives met?" Unless the process work is done in the first place, it is very difficult to be clear at the end of the process whether the objectives have been met.

Okay?

I am still a bit confused. There were two, different, issues there. One was whether objectives were clear. I absolutely agree that unless objectives are clear, we will not be able to establish whether they are being achieved.

There was also a separate point that implied that the objectives were clear and that what was unclear was whether they were being achieved.

I am still asking when we will get to the third point. Are the objectives clear? Are they being monitored effectively? And, crucially, are they being achieved? I will leave it there.

I think that that has now been registered.

Sorry.

I want to pursue the issue of clear and consistent monitoring standards. It sounds great in theory, but can all projects be judged by the same standards? What common criteria can be used to judge radically different projects and yet ensure that they have the flexibility they need to accomplish their aims?

A vast range of projects is involved in the study. Can solid monitoring standards be reached when the projects are so different?

You are quite right to identify the tremendous range in the size of the funding that is available and what it is intended to achieve. We strongly recommend a proportionate approach.

We would expect a significant fund such as the community regeneration fund to require much more rigorous and detailed monitoring than one of the small funds that are worth less than £0.5 million. However, the same principles apply—there must be clarity on the objectives and success criteria and on what information will be required from the people who spend the money to allow them to demonstrate whether and how the objectives are being met.

I understand why the very high-spending projects get more attention, but does that represent a problem? For example, exhibit 1 in the overview document demonstrates that most problems with oversight, monitoring and evaluation are experienced with projects that have lower levels of funding. We are not talking about a random sample. If one assumes that there are more medium-value projects than high-value projects and more low-value projects than medium-value projects, the total cost of the low-value programmes could be quite high, but there is comparatively little oversight of them. The two highest-value projects, which are worth £318 million and £127 million respectively, lie way ahead of the rest. There is a great bunching at the lower-value end. Surely that is where there should be much closer monitoring. From exhibit 1, oversight of such projects appears to be quite low.

There are two points to make on that. The first is that the size of the fund is not the only factor that affects the level of risk associated with it. Examples of other relevant factors are the number of partners involved, the size of the partner bodies and their ability to put good governance in place. Secondly, in doing our work, we were concerned to discover that good practice in one part of the Executive is not always picked up elsewhere in the organisation. The committee might want to follow that up with the Executive.

Oh yes—we certainly want to do that.

Can you give us an indication of how much money is bunched at the lower end of exhibit 1? Figures of £8.6 million and £600,000 are mentioned, but how is the money distributed at that end of the spectrum?

I can give you an indication of that, but it would be helpful if you could give me a moment to find the information.

No problem.

Thanks very much.

We can come back to that answer. In the meantime, Eleanor Scott has a question.

Can I follow up on Andrew Welsh's question before I ask the question that I wanted to ask?

Certainly—go ahead.

Is it possible to measure success against objectives, or are targets necessary? Is measuring achievement against an objective just too nebulous? Some of the projects that you mention in exhibit 8 might have been amenable to targets, because the measures of success were not clearly defined. You could have put a target on the business broadband incentive, for example, although that might have been harder to do with some of the other projects. Do we need more targets?

We do not necessarily need more targets. This goes to the heart of the question Susan Deacon asked, which was about how we can know whether we have achieved our objectives. First, we need to be clear about what we are trying to achieve. In some cases, that will translate directly into clear targets but, in others—with community regeneration, for example—the process is more complex and there is a range of ways of measuring achievement. We want a more structured approach to be adopted—we want the Executive to consider what indicators would tell it whether it was meeting an objective and whether they could be turned into targets that could be built into the agreements that are made with the delivery partners that receive the funds. We do not advocate a one-size-fits-all approach, but there is room for improvement in the management of the funds generally.

Can I ask my other question?

Certainly.

My understanding of an initiative is that someone has a big idea and a pot of money that will last for three years and they put the idea into practice. I have issues with that approach, but that is beside the point. Some of the projects that you mention do not seem to fit that model. An example of that is the rural stewardship scheme. I thought it slightly odd that you included as an initiative something that appears in the Scottish Executive Environment and Rural Affairs Department's budget. That scheme goes on all the time, not just for a particular period. It is funded by modulated money that comes from pillar 2 of the common agricultural policy. It seems not quite to sit with the other initiatives that are mentioned. There may be other examples of that that I do not know about, but I happen to know about that one because I was on the Environment and Rural Development Committee for a while.

We identified as initiatives projects that have pots of money that are outside organisations' mainstream funding and for which they bid. Some projects are short or time-limited and have particular issues associated with them. Some of them, such as the scheme you identified, continue for longer periods. We were interested in how the funds that are available are identified and allocated and how their outcomes are monitored and evaluated. In that sense, we considered a mixed bag.

So funding for some of the initiatives could come from different departments' budget lines?

That is right.

We are not talking about a separate initiative budget?

Funding for all the initiatives will have been included in the budget act, but will not have been earmarked for initiatives at the beginning of the financial allocation process.

One of my concerns about such initiatives is the way in which they move from being initiatives to becoming practice, or not. I suspect, as a result of hearing your answer to Susan Deacon's question, that this is too early to discuss how these projects will progress. I will therefore leave that matter hanging for now.

My other concern relates to your key recommendation that the Executive should

"consider the implications of individual project funding arrangements for the core work of prospective partners".

Can you say a bit more about the interface between the initiative and the work that is already going on? Can we see that it adds value, or is it a threat in some ways? I would see it as a threat when the initiative requires staff who currently deliver core services to do something special. Although that something special might be good, the impact that it has elsewhere might not be. Is there a way of measuring the impact of the initiative on what is already going on and judging, on that basis, whether it was the right thing to do?

The Executive has a range of reasons for funding initiatives in this way rather than through core funding, which vary from initiative to initiative. Two concerns that delivery partners raised were the amount of time and effort required of them when bidding for relatively small amounts of money and the impact on services when they are heading towards the end of the time-limited funding. They have to consider whether they can and should continue funding as the allocated funding winds down. The latter is relevant especially in the context of community planning, in which people are increasingly trying to look from the ground up at what they want to deliver, rather than doing the Humpty Dumpty thing of trying to put together funding from a range of different sources. In the case of initiatives, that funding is often ring fenced and difficult to use in the flexible ways that might make more sense across the piece.

That is one of the strong links between this piece of work and the piece of work on community planning. When you discuss the report, you might want to explore how the balance of greater control over how the money is used and greater flexibility for local delivery partners is being arrived at.

Initiatives can have a benefit. In developing new ideas and new ways of working across the piece, they can be good at highlighting issues that have lost out. However, how they fit into the plan and whether the decision is made to continue with them depends, in large part, on whether they were of benefit. Our judgment on that will come with time.

I am concerned that the allocation of funding to organisations is not always as open and transparent as it should be. Audit Scotland talks only about a clear and consistent approach; you make no comment about transparency. Is that deliberate, or is it something that you will revisit?

It is not deliberate; it is simply that we were looking at the variation in how funds are allocated—whether we start with an overall fund that is distributed or with several bids that add up to a total. The transparency of that is something we should keep in mind for future work with the Executive, especially in the context of community planning and the growing importance of this sort of funding.

Certainly for me, that transparency is long overdue. Elected politicians are unable to find out whether there is a structure to the allocation and whether it is consistent in addressing local needs. That is where the issue of the wide variation in the allocation of funding comes in.

In exhibit 8, on the CRF, you talk about it being

"difficult to measure the fund's overall contribution to national priorities."

As delivery partners, the 32 local authorities would have 32 different sets of needs. Does that not call into question the whole basis of the report being subjective?

I do not think that it does, but it takes us back to the importance of community planning.

Community planning partnerships always try to balance national and local priorities and to get the right mixture for their local communities. One of the things that initiative funding can do if it is not well planned is distort that balance so that people have to pay more attention to the things the initiative funding is there to support than to local priorities and circumstances. It is the balance that is important.

Do you think that it would be more helpful to delivery partners if there were a standardised bidding process and audit process?

The range and size of the initiatives that we are looking at could mean that in some cases that might be more of a burden, rather than less, but there is certainly scope for spreading good practice and ensuring that the guidance available to Scottish Executive staff is consistent and consistently used.

Have you had any indication as to when the Scottish Executive will respond to the report?

In the main, the Executive does not respond formally to us but responds instead to your inquiry, so there is not a date by which we would expect a response. It is part of the continuing dialogue that we have with it, unless there is an inquiry by the committee.

Is it an agreed report?

Yes.

I would just like to ask for clarification about something I may have missed.

The City of Edinburgh Council is down as being involved in the biodiversity action grants scheme. I would be surprised if that were the only initiative funding the City of Edinburgh Council got from Executive schemes. How have you chosen the things that you have looked at? Have the partner organisations themselves had any input into what you looked at? I would have expected the City of Edinburgh Council and other councils to have a number of such schemes.

Almost all of them do. We have come at it from two directions. We chose a sample of 20 of the 74 initiatives announced during 2004 and got a range from large ones through to small ones. We also surveyed a sample of more than 70 delivery partners. We asked them about their experiences on the basis of the good practice criteria that we have identified and about what has worked and what causes them difficulties.

The next thing I want to say is more of a comment than a question. You can probably tell from my colleagues' questions that our concerns are about the community impact of initiatives. You have talked about balance, distortion and identifying need, but I want to put on record the fact that I often get feedback from smaller and medium-sized organisations in my constituency to the effect that, for the amount of money involved, there is an awful lot of staff input and a lot of distortion of what they see as their core, mainstream business. I take on board your comments about the community planning report that is under way. It might be worth coming back to us to say whether there is anything beyond that that you think you could do to pick up on that side of the initiatives.

I have a general issue about consultation, and I see that your conclusions and recommendations raise some concerns about consultation. Could you give us a little bit more information about that?

The study looks first of all at the way consultation was happening in terms of what we identified as good practice principles. That was picked up again in the survey of delivery organisations and partners, which were asked how well they felt they had been consulted on what the scheme was intended to achieve and the way in which that was being done.

We found some good examples of consultation, but a consistent theme in the report is that there is a lot of variation. In some cases, the consultation was not particularly effective in reaching the right people or seeking their views, and it was often squeezed for time when it came to involving councils and their smaller local delivery partners. It was difficult for them to engage in shaping the initiative and how it was funded, and then to make an effective bid that tied in with their broader objectives and with the community impact to which you have referred.

Do you feel that, even with those limitations on the delivery partners, the consultations were geared towards those partners—even though the partners did not get the extra time to ask their own communities or to consult more widely?

It is difficult to generalise. There were some good examples, but there were also examples where, simply because of the limited time that was available, it would be very hard to enter into a wider discussion around how various things fitted with what we were trying to do locally. That variation is one of the key themes that we have identified. There is room for a better spreading of good practice within the Executive to ensure that, within the bounds of the need to keep government working, there is good local consultation and as much as possible is being done to tie the national priorities that the initiatives intend to achieve into the local circumstances and priorities of communities.

Before I invite a further question from Andrew Welsh, I wish to develop a theme that a number of members have touched on. In paragraph 49 of your report, you say:

"Six of the 20 projects examined had in place, or were developing, outcome-based agreements."

Have you identified any trend there that explains why that figure was six—rather than, say, 16—out of 20?

Are we aware of why not so many projects put agreements in place early enough to measure effectiveness? Your report mentions among its recommendations that it would be good to have

"monitoring and evaluation, including milestones".

Can we identify any trend that might explain why more projects do not have such agreements?

I do not think that it is possible to identify such a trend. I have said a couple of times that one of the striking things is the variation. It is not routinely the case that the larger initiatives have an agreement but the smaller ones do not. It is not the case that all the initiatives that come from one particular department or area in the Executive will tend to have outcome measures in place. There is simply a lot of variation.

We are not talking about significant risks in how public money is used in the sense of propriety, but there is scope to get a better measure of exactly the sorts of outcomes that Susan Deacon was discussing. That involves learning from what is already happening in some areas of the Executive and developing greater consistency.

I do not sense from colleagues that our concern is about a waste of money, in the sense of money being spent inappropriately; it seems to be more a question of value for money, what we are achieving, whether it is working and whether we are achieving our goals.

I want to follow through the idea of bunching. I realise that I have asked you a complicated question on that.

I have a note of it.

That is fine. I also realise that you are dealing with a complex situation and a range of organisations. If my arithmetic is right, a sample of 20 initiatives out of 74 is 27 per cent. How typical is that sample? Can we reasonably assume that those 20 projects represent the other 73 per cent and that the same findings will lie in the other projects?

We selected a sample of 20 not at random, but to reflect the types of initiative that are around and to get a distribution among different Executive departments. Because of that variation, it is probably not fair to say that if we multiply that 27 per cent by four, that is what the whole picture will look like. The most important finding is the degree of variation, which is explained neither by the size or complexity of the funds, nor by the outcomes. By spreading good practice from the outset, the Executive might have scope to be clearer about what it is achieving with a significant tranche of money.

Thank you for that explanation. Half of the initiatives complied with most aspects of good practice. Therefore, half did not. How badly out were that other half? Is that a major cause for concern? Why did non-compliance come about? Do you have any suggestions about that?

As the convener said, the shortcomings are to do with not achieving best practice and not always being able to demonstrate what outcomes are being achieved in areas of expenditure and policy that are always complex.

It seems to us that moving towards full compliance—simply by spreading what is already happening in some parts of the Executive—should be relatively straightforward. Some of these policy areas are extremely complex: it is not easy to set objectives and turn them into targets, but we think that more could be done to match what is already being achieved in the best projects in the Executive and that there would be benefits in terms of knowing what is being achieved with the money that is being spent.

So it is a matter of education and encouraging the spread of best practice.

Yes.

On the question that you asked earlier, we looked at 20 initiatives with a total value of £728 million. Nine accounted for less than £10 million each and came to a total of £30 million; six fell between £10 million and £40 million and accounted for a total value of £112 million.

Thank you.

Susan, I believe that you had a small question that you wanted to ask.

I wanted to clarify a first-principles point about the scope of the report. Forgive me if this has already been covered and I have not appreciated it.

The report considers initiatives for which funding was announced during 2004. I appreciate that they will be at a particular stage of development. Why was the decision made to study them rather than projects whose funding was announced in, say, 2001? If a different choice had been made, there would have been a longer period over which to track what has happened.

As always with these pieces of work, we are trying to come up with a specification that is tailored, does not take too long to do and is clearly bounded, but which has useful findings that can be applied more widely. This is the first follow-up "How government works" report that we have done and we thought that it would make sense to focus on live policy areas because the history of the way in which they were announced is still clear and readily available. We thought that we could use them as a baseline for examining the arrangements that were put in place and could determine what outcomes have been achieved at a future point.

One of the advantages of what we call baseline reports is that they give us an opportunity to identify best practice and encourage that best practice to be shared across the Executive.

The report contains an important finding for the Executive, which is that, on the basis of our sample, about a third of the 74 projects that were launched in 2004 are not as clear as they might be in terms of their objectives and targets. That must be an issue of concern to the Executive, I imagine. It is certainly an issue of concern to me, because if those projects do not have a clear set of targets it will not be possible for me to come back in a few years' time and do exactly what Susan Deacon is asking Audit Scotland to do, which is to evaluate the impact and value for money of programmes.

As members of the committee will recall from many of the sessions that we have had in the past, a prevailing conclusion in our reports is that the information is not present. We therefore think that it is important occasionally to produce reports that examine the current state of the procedures for implementing Executive priorities so that we can evaluate them later.

I thank Caroline Gardner for briefing us on that report, in which members showed a great deal of interest.

Under the next item on the agenda, we will discuss the approach that the committee will take to the report. That agenda item will be dealt with in private.

Meeting suspended until 11:17 and thereafter continued in private until 11:47.