The fourth item on our agenda is an evidence session with Scottish Government officials on the national performance framework. I welcome to the meeting Roger Halliday, the chief statistician, and Dette Cowden, a senior statistician. I ask Mr Halliday to make an opening statement.
Thank you very much for inviting me here to talk about the national performance framework. I have recently moved up here after working in Whitehall. At first, I could not believe that there was a single framework that sets out a clear vision of the kind of country that we want to live in and which is based on outcomes that improve people’s quality of life rather than on inputs and processes. We should be proud of that. More importantly, I have realised that this outcomes-focused approach is leading to a shift in the way that public services are planned and delivered.
Thank you for that very helpful opening statement, which has sadly kiboshed one of the first questions that I was going to ask you. Don’t you just hate it when that happens?
I do not want to pre-empt in great detail the Scottish Government’s response to the report, but I can say that we have included that area in the review. There are two new early years indicators around child deprivation and healthy birth weight, so you can see that we have responded.
Indeed. The committee also found it surprising that the spending review did not mention the national performance framework or the five strategic objectives that the Scottish Government had realigned to deliver. I wonder about the lack of linkage between the NPF, the draft budget and the Government’s economic strategy.
Again, that is a question for the wider Scottish Government to respond to. The programme for government was based around the five strategic objectives and they formed the chapters. Being a statistician, I counted the number of times that the framework or outcomes were mentioned in those three documents, and it was 142. This is the way in which the Scottish Government does business now, so I think that I will leave that question for the review to give more detail.
Okay, that is fine. I open up the discussion to members to ask questions.
From the suite of indicators, the one that leaps out at me is “Improve Scotland’s reputation”. How is that indicator measured?
That comes from a reputable study—an Anholt-GFK Roper national brands index—which is an international study of nations’ reputations.
Okay.
I have a couple of points that are based mainly on the question-and-answer paper that we were given, which is annex B in our papers. The first paragraph says:
In the conversations that I have had with colleagues in the Scottish Government—I should say that I am relatively new to the organisation—quite a lot of significant work has been done, starting with the national outcomes and working out how we will meet them. It is not just about the traditional outcomes that a particular part of the Scottish Government has been set up to do. In fact, the Scottish Government does not now have a departmental structure.
I accept that organisations report annually, or whatever, but are they actually doing anything different from what they would have been doing otherwise?
I cannot answer that on their behalf. I have certainly picked up on some examples in which the approach has made a difference to individuals, mainly through the integration of services, particularly those that work with people in transition at 16. We have stories about the choices that people have made that have led to smoother transitions. We can therefore certainly talk about the impact that the approach has had on people’s lives.
Secondly, the question-and-answer paper talks about simplification. The example that is given is dental health. The paper says that the previous indicator was that
You are quite right. When we did our lessons-learned exercise on the NPF, which started back in June 2010, we got the message that people did not quite understand the language and wanted it to be simpler at the top level. However, full detail on exactly what is being measured and the fuller description of the indicator title is given directly below that first line. The information is therefore still there, but the top line is simpler for people to understand.
Who did you consult about the lessons learned?
We consulted a lot of people inside the Scottish Government. External organisations that we consulted included VisitScotland, the Scottish Further and Higher Education Funding Council and Fife Council; we tried to build up a picture of what was going on with the framework, how it was being developed and what we needed to do to take it on.
I have a couple of questions, one of which is about an indicator that was removed:
I do not know, but I am sure that we can report back to the committee on that.
My other question is about another indicator:
The indicator is quite broad, but it will be based on data that will come from the Scottish household survey in which people are interviewed about different public services. They are asked whether they have been happy with those services, whether they have had the response that they wished to have, and what engagement they have had with different public services. It is a composite indicator that is based on certain information.
So local government is not expected to do its own surveys of what people feel in that regard.
The data comes from the Scottish household survey. Over the past two years, in anticipation of improving the indicators in the national performance framework, we have looked at all our surveys to see whether they meet the needs of performance measurement across the board. When the Scottish household survey was reviewed, we ensured that the questions it asked would be able to get the kind of information that was needed for the national performance framework.
Like Mr Halliday, I am pretty enthusiastic about the concept of the framework, but I find it quite difficult to use and not easy to follow. Picking up on the convener’s question, when the consultation took place—from the middle of 2010, I think—were any groups of citizens who are not tied or aligned to government or politics asked for their views of the framework?
We did not consult the public more generally, but we are thinking about a communications strategy for the general public going onwards. We feel that the framework is a tool for the outcomes approach and is for the use of us and our key delivery partners, which includes the third sector, in delivering the national outcomes. However, we know that we need to explain to the general public exactly what that means as far as they are concerned, so we will be working on that in the near future.
You are considering that and may take it on board going forward.
We know that we need to work more with the general public to see how we can explain the framework to them. On the Scotland performs website, which is our dissemination tool for the national performance framework, we have included a guide that explains the different terms and a glossary. For example, it explains outcomes as opposed to just inputs in a process. That is part of our response to how we get people to understand what the national performance framework is.
My second point is about having sufficient transparency to ensure that somebody visiting the Scotland performs website would have a good idea of how the country is doing.
That is an extreme example. On the first purpose target, which is to raise the GDP growth rate, the 2011 quarter 2 results were published in October, so there is a fairly short lag. We must balance a range of factors about what makes a good indicator. For example, it is important to get indicators that are related to the primary objective; that are practical to collect; that show changes over time or between areas; that are consistent; and that do not create perverse incentives. Timeliness is another factor. For each situation, whether we can get more up-to-date data depends on the data that is available at the time. However, throughout the statistics group in the Scottish Government, we always strive to produce more timely data.
There is no easy answer. I looked only at that one target. If I looked at all of them, would I find that most of them use 2011 data or 2008 data? Do you have a feel for how many of the 50 indicators and 16 outcomes are up to date?
I would like to be able to give an answer to that straight away, but the honest answer is that I do not know. My feeling is that the example that you gave is extreme. I hesitate to say that the majority of indicators use 2011 data, because there is always a time lag for data collection, but I expect that the majority of them are more likely to be information from 2011 or 2010 than from 2008 or 2007.
Obviously, I could go online and check them all, but can you provide that information to the committee in writing?
I would be happy to put that together for you.
I am grateful for that.
We will have a supplementary question from Paul Wheelhouse, to be followed by Michael McMahon.
I am sorry to interrupt, convener, but I just want to point out that, after receiving feedback from the general public on what they wished us to do, we placed a calendar on the website that shows when updates to each of the indicators are expected. People can see not only what is there, but when the next update for each of the indicators will be.
Thank you.
Not all the data on which you rely for Scotland performs is produced by the Scottish Government. Are there any bottlenecks in the data from the Office for National Statistics or other sources on which we should apply pressure, either through the Government or directly, to try to achieve an improvement in the timely production of data to assist your efforts? You mentioned GDP figures. There is a lag between the production of the UK GDP figures and the Scottish ones even though they are based on the same data set. That is because the ONS’s priority is to produce the UK figures and it does not prioritise production of what it regards as subnational figures. We therefore have to wait for an additional quarter to find out how we are doing. Are you aware of any other examples that we could highlight or apply pressure on to improve the production of the data?
Thank you for that offer. There is nothing that springs immediately to mind. We have strong relationships. Building relations with the Office for National Statistics and other UK Government organisations is an important part of what we do. However, that is an issue for us to take away and come back to you on.
Will you give us some information on one of the national indicators that were removed, and the reasons for that? According to the information that we have, the indicator
There are a few things to say about that indicator. First, we continue to report on all the indicators that are no longer part of the performance set and we publish that information online in a transparent way. That indicator was not an outcome-focused indicator that reported on the quality of life for people in Scotland, and that is an important reason why it has been removed. Those are the two main things that I want to get across.
I do not understand your answer that it is not an indicator. It is a target—a 25 per cent reduction by 2011. Either it was met or it was not. Can we have an indication of whether—
To answer the first question, the data is not yet available for us to fully report on that. It will be reported on Scotland performs and you and everybody else will be able to make a judgment on whether it has been met.
I find that a bit strange. Surely you know whether the public bodies still exist, so we should not need a report on that. Either they are there or they are not. Surely the indicator could have been changed to “reduce the number of Scottish public bodies to”—and then you would state the number that you would like to see it reduced to, whether that is 100, 120 or whatever. Your answer seems a wee bit strange to me, and I think that other committee members feel the same.
If you are saying that you took the indicator away because it did not indicate what it was supposed to indicate, that is a much better explanation than to say that, as we are now in 2012, a target that had to be met by 2011 no longer stands. That offers no indication of the success or otherwise of the objective.
We have put on the website what we call legacy targets. Any indicator that has been dropped that has a target associated with it will still be reported on Scotland performs. We hope that that information will be available in the spring, so we will be able to answer the question whether the target was met. You will be able to see whether the target in the indicator from the previous national framework, which has now been removed, was met.
As some of my colleagues said in asking their questions, it is difficult, when speaking to people about the link between the objectives and the indicators, to see exactly how those things match up. I have been looking for a good example, and this is the best one that I can find. The objective
In the technical notes behind each indicator, there is a section that explains which national outcome we believe the indicator is helping us to achieve. We hope to be able to put up a table for each of the national outcomes that explains which of the indicators apply, to look at it the other way round. We will show, for each national outcome, which indicators in the national performance framework we believe will help with it. There is that cross-link. It is probably information on what are we doing to achieve the indicators that will help us to answer your question on what will help us to achieve the national outcome.
Was the fact that people were expressing concern about the lack of a clear link between the objective and the indicator something that came out during the review?
The technical notes have always said which national outcome each national indicator is linked to. The concern was more the other way round: in other words, which national indicator we think is the most important in helping us to achieve each national outcome.
We are talking about the national performance framework, but below it are the local authorities and community planning partnerships, which have responsibility for single outcome agreements and have their own performance framework and performance indicators. Are you working with them to ensure that what they are measuring matches what you are now measuring? As a councillor, I know that some of the performance data that we were looking at in committee seemed to have no relevance to anything that we were discussing. Can we ensure that relevant performance data is produced locally as well as nationally?
Part of my team works alongside the project led by the Society of Local Authority Chief Executives and Senior Managers on improving the local outcome indicators. A paper has already gone to the project board, which I believe is meeting on 26 January, to make recommendations on the refresh of the menu of suggestion for things that CPPs can include in their single outcome agreements if they wish to do so.
That ties in quite well with what my colleague Mark McDonald said. In our inquiry on preventative spending, we received quite detailed evidence on indicators to do with child poverty. I welcome the new focus on preventative spend in the refreshed set of indicators.
I am happy to take that suggestion on board.
One of the indicators that were removed was:
There is nothing that specifically mentions the social economy, but third sector organisations will be important planning and delivery partners in virtually all, if not all, the indicators. Our approach to that indicator was to say, “The third sector is important across the board.” The same applies to equalities. A couple of indicators mentioned doing things in deprived areas, but we are saying that those things are important not just for those areas but across the board. In developing and presenting the information, we are looking at protected equality groups, areas of deprivation and urban and rural areas to ensure that the framework is delivering for all people of Scotland.
How can the national performance framework be used to better organise the Scottish Government and deliver better co-ordinated policies?
The Scottish Government’s response to the committee’s report will go into a bit more detail on that. I reiterate the point I made earlier that the Scottish Government has removed its departmental structure and is now working as a single unit to enhance cross-cutting work. Talking to a range of people across the organisation, I have picked up that work and the development of strategy has started on the national performance framework and the national outcomes indicators and, working back from that, on how those are delivered.
Earlier, you mentioned your experience at Whitehall and talked about how the approach is proving to be an effective tool in improving departmental working. Could you say a bit more about that?
Yes. I should say that, where this approach has been tried internationally, the evidence suggests that the period of time before it will have a significant and meaningful effect is something like four to five years, if not slightly longer. It is important that, in the refresh, we maintain the framework and pursue it in order to reap the benefits down the line.
You obviously believe that the recent refresh will make the national performance framework even more effective. Are you looking to tweak the NPF every year? Is it an on-going process or is the framework now fairly well set and not needing to be refreshed for another two or three years?
We have decided that the NPF should probably remain as stable as it can be but, if there is a refresh of the spending review, that will be the appropriate time for the NPF to be looked at again to ensure that it is meeting the current priorities of the Government.
It might be from the technical point of view, but it will certainly excite Professor Bell.
We are asked whether we should be doing more to express the longer-term trends rather than just showing the arrow between the previous two data points. We are going to look at how we can better express longer-term change, such as the change since baseline.
The model that you use is based on one that was developed in North America—I cannot remember where; perhaps it was Pennsylvania. Presumably, that model has been operating for longer. Is there any independent international assessment of whether this type of model leads to improved performance?
We have some research evidence to suggest that it does exactly that. It is important, as the convener said, that we have a framework that stays stable over a longer period of time, as the benefits are not immediately evident—it may take four or five years, or even longer.
Do we know whether this way of monitoring performance stimulates improved performance? Is there international evidence to say that using this method helps people to lift their game?
That is the evidence to which I was referring; I am happy to share it.
One thing that came up in our scrutiny of the budget was that measuring outcomes for different organisations—the NHS, local authorities and so on—can involve comparing apples with oranges. Does the national performance framework help to reduce that type of differential, and help those organisations to focus on the same direction so that outside bodies can more easily examine their accountability?
I certainly think that it would help in that joint working to understand what a national outcome is and what we are trying to achieve in a joint partnership. I would hope that the national performance framework helps people to work jointly and thereby achieve the outcomes.
I have an afterthought based on the discussion that you have just had. In my previous life, I used Her Majesty’s Treasury’s green book a lot for economic appraisal, which was developed with a different policy framework.
That is an interesting suggestion—I cannot necessarily comment on it today, but I thank Paul Wheelhouse for that.
I do not think that colleagues have any further questions, so I thank you for your evidence today and for answering so many of our questions; I know that they were not always easy. I wish you well in continuing to develop the framework.