Item 2 is oral evidence from members of the David Hume Institute, in a round-table format, as part of the committee’s scrutiny of the draft budget 2014-15. I welcome to the meeting Professor Jo Armstrong, Robert Black, Stephen Boyle, Donald MacRae, Jim McCormick and Jeremy Peat.
The session will address four key questions posed in the call for evidence on the draft budget, as set out in the paper from the clerk. I intend to allow 20 to 25 minutes for each topic. If the witnesses wish to speak, please indicate that to me or the clerk. I am sure that you all know the round-table format. People are not restricted to speaking once or twice; you can speak as often as you wish.
I invite Jim McCormick to address the first question, which is on the progress that the Scottish Government is making in meeting its targets as set out in the national performance framework.
Thank you, convener. Good morning, everyone. Thank you for the invitation to join you today.
In the papers that have been submitted by panel members, you will see some common themes in our answers to that first question. A genuine answer to the question whether the Government is making progress would depend on at least two things that we do not have yet. The first is much clearer links between the priority setting and resource allocation decisions that are made by Government and its partners in contributing to the targets and outcomes. The second is a demonstration of the strength of the evidence base. To what extent do our policy choices in Scotland build on what we know about what works? Are we investing where the evidence is strong? We are not really able to answer those questions to the level of accuracy that we might like.
Within the limits of what the national performance framework tells us, progress is being made. However, we may have some questions about the how and the why of what it tells us.
The targets are for the whole of Scotland, not just the Scottish Government. We would frame the question in terms of the sum contribution being made by all public agencies as well as by business and the third sector. There is some reference in the paperwork to the way in which the national health service, through its health improvement, efficiency and governance, access and treatment—HEAT—targets, has been trying to get closer to the links between decision making, spending and outcomes. Some interesting consequentials may flow from what the NHS is doing in relation to what other public agencies might do.
It is hard to tell how much progress we are making despite the many strengths of the framework partly because the purpose targets measure progress—that is what they are trying to get at—in different ways. I can give you an example of that. We are benchmarking Scotland’s performance against that of the rest of the United Kingdom and the Organisation for Economic Co-operation and Development in terms of participation in labour markets. We are measuring only the employment rate; nonetheless, that is an outward-looking, ambitious measure of how we are doing. When we come to population and healthy life expectancy, however, we are looking at the improvement in healthy life expectancy over a decade. We are not looking at Scotland in a benchmarking sense—it is not a comparison with the rest of the UK or the rest of the OECD.
Therefore, we have different types of measures going on. We can take a view on whether they are ambitious enough, but they do not really do the same thing. The population measure really matters. Although Scotland’s health performance has improved since devolution, we still lag behind comparable parts of the UK, such as the north of England. It therefore matters that we are outward-looking on that measure as well.
What kind of snapshot might be taken by a layperson who looks at the framework and asks, “Are we making progress?” We can discern more information by scratching the surface than the framework tells us at a glance. If we take a step back and look at the story of the 16 national outcomes, it is possible to drill down and look at the indicators that are mapped against them, which tell quite an interesting story. There is now improvement in 21 of the indicators, 19 of them are stable or flatlining and 10 of them are going in the wrong direction. That is net progress on 11 indicators. At a time of recession and austerity, that could be considered a pretty solid performance, but we cannot easily tell from the framework whether that net progress is better than it was a year ago or five years ago. Therefore, to be able to answer the question “Are we making progress?” we have to do quite a lot of digging and detective work. Much of what we need is there—many of the building blocks are in the performance framework—but it could be made easier to draw conclusions.
The national performance framework is a genuine asset for policy making in Scotland and for accountability. However, the committee and others have asked how it can be made more transparent. I think that we need to be able to draw clearer links between the contributions that are being made by different parts of the public realm in Scotland and the outcomes that we are seeing. For example, which of our policies in Scotland make a great difference to positive outcomes and which of them do not make much difference at all? Frankly, at a time of pressure on our budgets, should we stop doing certain things or do less of them? We need to be able to answer such questions if we are to get closer to knowing how we can make progress and get our limited public resources to make more of a dent in the issues that we are all interested in.
My final point is specifically on some of the indicators. You will have seen from the written submissions that there is a view that some measures probably matter more than others at this point in our development. We can take a view on which those are, but there are a couple of the indicators that we might regard as flashing red because of their wider significance. The first of those is the number of emergency hospital admissions, which continues to rise—it has gone up 3 per cent from six or seven years ago. Why does that matter? It matters because of the costs involved and because such admissions are at the far end of the spectrum from the work that we are doing on prevention. We might ask whether the work that we are doing on reshaping care for older people is strong enough, fast enough and being delivered with the right degree of urgency for that indicator to start to go in the right direction.
The second of those indicators is the supply of new housing, which has decreased significantly. We know that the Scottish population is rising—that is good news—and has reached its highest level on record. Therefore, it is concerning, to say the least, that our record on the supply of new housing is going steeply in the wrong direction. That will have big knock-on effects for the economy, employment, investment and so on.
Therefore, we might ask ourselves harder questions about particular indicators, as we have strong evidence about their multiplier effects and should be more concerned about them than about other indicators if performance against them is going in the wrong direction.
Thank you. I will quote your written submission—committee members can come in as they wish. You state:
“Local discretion rather than delivery of a national strategy is a more dominant theme in Scotland’s annual reporting than in the other countries. There is a risk that local priority-setting and resource allocation won’t be based consistently on the best available evidence of what works in reducing child poverty”.
My submission takes one policy challenge, which is the challenge of reducing child poverty—we could have taken any other. It tries to track through what we know about the evidence, about measurement frameworks and about national strategies. There is a decent story on performance in recent years. However, single outcome agreements do not make many references at all to the national strategy, to the evidence base and to what we know works. We can characterise where we are with devolution by saying that we are strong on analysis of such problems but rather weak on investing in what works.
When it comes to the challenge of reducing child poverty, we know that we are on the cusp of another big rise in child poverty driven by what we might call externalities, in this case the downside of UK welfare reform. We must satisfy ourselves that we maximise, within the limits of current powers and resources, all that we can do at both the national and local levels to drive down that impact. I am not sure that we can satisfy ourselves that we are doing that yet.
I am interested in that example and in your general point about evidence. Is the key point that if the approach is localised in the way that you describe, it is less likely to be based on evidence? I do not suppose that that is necessarily the case in principle, as that would assume that the national strategy is evidenced and the local strategy is not. Is there a fundamental problem? If so much delivery is localised, is that consistent with having such strong national targets and objectives, given that there is no reliable way of ensuring that the local priorities match up with the national ones?
That is a really good question. There are different types of accountability across the public services in Scotland. For example, what can be achieved in the NHS through HEAT targets—the national dementia strategy is a great example of what can be achieved by aligning national and local action—is a bit different from what can be achieved through local government and other community planning partnership partners, because of discretion and autonomy. To some degree, that is right, but there is probably demand at a local level for more support to build know-how and to develop the ability and capacity locally to invest in what works. Local action does not mean that there will be a weaker link to evidence, but we could do better at providing support and guidance to local partnerships to ensure that they put their limited amount of money into actions that we know already work better than others.
Stephen Boyle’s submission states:
“One of the Committee’s questions is whether the national indicators and purpose targets are an effective means of measuring the performance of government. My short answer is, ‘yes’”.
Can you expand on that?
I am happy to do so, convener.
I will be fairly brief and will make four points. First, a virtue of the framework is that it recognises the complexity of government and the world in which the Government and its agencies operate. Secondly, it allows us to measure progress in the round as well as in individual areas of policy. Thirdly, I regard it—and I think that it is properly regarded—as being a tool for measurement. It is not a substitute for making judgments about matters of principle or policy priorities; it measures progress towards what has been decided through the democratic process.
Finally, to echo a point that has been made extensively by others, I agree that if we are looking for enhancements in the evolution of the framework, it should make clearer the links between aspects of the performance framework and decisions about resource allocation. I would not say that the framework should drive such decisions but, when the Government makes decisions about resource allocation, how those relate to aspects of the framework should be clearer.
09:45
You talk in your submission about the cost of measuring things. I sometimes struggle to get my head round all this. We have heard the example of a 3 per cent increase in hospital admissions. That could be seen as good: the increase could have been 10 per cent had we done nothing. The figure might be only 3 per cent because the Government has done wonderful things and we might be pleased about it, but it shows up as a red light and looks bad on the surface.
We could dig for ever into such figures and never reach a conclusion. Are we sometimes too optimistic or confident that we can tie in the pound that we spend with a little impact miles down the line, or is that a cynical question?
I will make three points. First, it is often difficult to trace the link between a decision that is made to spend money and the ultimate impact, because life is pretty complicated. In my submission, I asked what would happen to the supply of mathematics teachers in Scotland if we trained more maths teachers. Would the number go up or down or stay the same? We do not know, because the position would depend on the wages that were paid for people with maths degrees in Standard Life, actuarial firms and so on.
Secondly, to extend a point that Jim McCormick made, much of the best evidence that we should seek concerns the evaluation of what is going on, to try to understand a bit better the links between cause and effect and to find out what works. I would instinctively prefer to spend my last pound on evaluation than on measurement.
That takes me to my third point, which is that measuring things can cost a lot of money. I am not familiar with the health service measurement system that tells us how many people have been admitted as emergencies; I am more familiar with labour market measurement, which involves pretty costly surveys. Sometimes, it is important to spend such money, but at some point, a judgment must be made about whether having a bit more accuracy is worth spending a bit more money that could be spent elsewhere.
In his submission, Jim McCormick says:
“Governments are well-versed in producing targets, milestones, indicators and monitoring frameworks.”
Our session is mainly about the Scottish Government’s national performance framework, but you also assess the approaches of the Welsh Government and the Northern Ireland Executive. You say:
“Each country”—
I presume that that means each country of the UK—
“would benefit from developing the evidence base underpinning its approach and reporting on evaluation findings in annual progress reports.”
That begets the question of how we can put the national performance framework in a wider context. Stephen Boyle just said that he feels that the general approach is pretty good, but how does it compare with the approach in other jurisdictions in the UK? I am also quite interested in the approach further afield.
Donald MacRae made the point in his submission that it is almost the case that a performance framework is never a prophet in its own land. Scotland performs may be getting more recognition and admiration from other parts of the world, including the rest of the UK—certainly Wales and Northern Ireland—than it is getting at home.
In a sense, we are setting the bar pretty high by saying that we would need to know those things to assess progress. We would need the inner workings of Government to be shown more effectively and to see the links between spend decisions and the national outcomes. I do not think that we will ever have all the data or evidence to allow us to do that across the board, although we might be able to do it in, let us say, half a dozen areas that happen to be the six priority policy areas that have been agreed with community planning partners in this year’s SOA framework. It might be possible to marshal good enough evidence and good enough linkages between spend decisions and outcomes for those six areas. If we did that, a year or two from now, we would be in a much stronger position than we are today to answer some of the committee’s questions.
To answer Jamie Hepburn’s point, that would be a move from our having a good framework for measuring and monitoring, as Stephen Boyle said, to one that is genuinely better at informing local action and practice in priority setting, as well as allowing us to take a view nationally about how we are doing and, crucially, what is contributing to better outcomes. It is possible that, in some areas in which we are doing fairly well, such as crime reduction, we just do not know enough about the links and contributing factors. However, in other areas, such as employment, education and older people’s independent living, we can tell a stronger story. There are areas in which we could be on quite strong ground if we chose to do more to evaluate the impact of what works and then put more investment into that, especially at a time of austerity and tight public spending.
But, by comparison with other places, what we have here is actually pretty good. I am interested in others’ perspectives on that.
The framework that we have for measuring and monitoring is probably as good as any of the examples that we have seen elsewhere. I have spoken previously about Virginia and Oregon, and British Columbia also takes an interesting approach. I am less well versed in European examples but, in so far as it goes, the national performance framework is as good as any of the examples that we have seen.
Yesterday, at the Welfare Reform Committee, the expert working group on welfare and constitutional reform made the observation that it would be useful to have national information from organisations such as the Office for Budget Responsibility broken down to the level of Scotland as a region. The Royal Society of Edinburgh has said that
“independent assurance of data quality and the matching of data to the measures used in the NPF”
would be welcome. What would be the downside, if any, of that? Everyone seems to agree that we need that sort of data analysis, but nobody seems to make any arguments about the downside.
Jeremy Peat was going to come in next, so that is perfect timing—he can respond to Michael McMahon’s question and then make his point.
Thank you. I should admit to having chaired the group at the Royal Society of Edinburgh that prepared the response that Michael McMahon has referred to, so I can pick up his question, which actually links to the point that I was going to make. Having looked at the issue a bit more than I did in the past, I think that Scotland should be proud of the NPF. It is a remarkable achievement to have got something so detailed, so regularly presented and so transparent in terms of the data, targets and information that are set out. It is a hugely positive base from which to proceed.
The RSE makes the valid point that it would be appropriate to have the data validated to ensure that everything that is in the NPF has a stamp of approval. We referred to Audit Scotland or some other body being engaged in that; it is just good governance to have an external look to ensure that the data are fine.
To pick up on Stephen Boyle’s comment, a more substantive point is that, in the end, the data are never a replacement for judgment. One cannot hope to achieve a mechanistic approach whereby we insert all the data into a sausage-machine model, which then comes out with answers about whether outcomes are meeting Scotland’s objectives and what they imply for policies.
What is required is a sufficiently strong foundation for judgment to be applied, which must often be related to individual policies and individual elements of the objective function of Government. The data must be as clean, crisp, well presented and transparent as possible, which includes ensuring that we have the right comparisons over the right timescale, with the right comparator countries and so on. Judgment must then be applied through more detailed analysis, taking the NPF elements as a starting point but looking more specifically at how outcomes have been determined, how they relate to the set objectives, how policies come in and what that means for future resource allocation.
I therefore think that this is the starting point rather than the end point, but it is a hugely strong starting point for Government, Parliament and committees such as this.
Jeremy Peat talked about mechanisms for independent assurance of data quality and the matching of data. A number of written submissions referred to Audit Scotland’s role in that. What do you feel about independent monitoring? How could the NPF best be monitored? Is it okay as it is or should it be done independently?
I am suffering from sinusitis a bit this morning, so—
I apologise for not speaking as loudly as I often do.
It is my fault.
I am asking for your view on assessment of the data independent of the Scottish Government.
Right. I very much endorse what Jeremy Peat and other speakers have just said. I am a fan of the Scotland performs framework, which is a great start and foundation, and is a work in progress.
However, the point about avoiding a data-driven exercise is really important. I will flesh that out. To give meaning and significance to our discussion, it is important to populate it with practical examples. We talked a minute ago about the problem with emergency admissions and so on. As, I think, everyone in the committee would recognise, the difficulty is in seeing emergency admissions as some kind of stovepipe in the NHS. Over the years in my former role as Auditor General of Scotland, we did a number of pieces of work that looked at whole systems. Back in about 2005-06, we did a really interesting piece of work with NHS Tayside and the councils in its area on bedblocking and emergency admissions, in terms of discharge procedures and blockage in systems. We analysed the situation and—it will be no surprise to anyone in this room—we found that starting planning for discharge as soon as the patient was admitted is really important, as are community support services and the right kind of housing environment, particularly for frail and elderly people. We could therefore not look at the NHS issue in isolation, just as we cannot look at a HEAT target in isolation.
To come back to Jeremy Peat’s point, we can analyse the data to death, but what really needs to happen is that this and other parliamentary committees are helped to get a good understanding of how the systems out there are operating. That probably involves further work—which could be done by Audit Scotland or by another entity—to present committees with an analysis of how the system is operating in the round, rather than its being a data-driven exercise. That will be the next stage in the development of the process.
I am sorry to speak at such length, but the issue is really important and raises another important issue that is very closely related to it. Take the HEAT targets, which are owned and run by the Scottish NHS, which has developed them well over the past few years. As this committee knows very well, health service spend is roughly one third of the total spend, so it is really important.
10:00The other big chunk of spend is local government. The Scottish Government—quite rightly, in my view—is putting a lot of emphasis on partnership working. That is right because it is through partnerships that you can start to analyse the complexity of systems of delivery at local level and get all the joining up done. There is an issue there about evolving Scotland performs alongside the single outcome agreements, and with what is happening in the partnerships, so that Parliament can get a good understanding of what is happening, for example, in the social care environment and community health service in Edinburgh, in Highland and so on.
That way, there will be a narrative that truly explains what is going on, which is difficult for the committee to do, because you are busy and have many other commitments. There is a really big issue about finding a resource that can help you to analyse and interpret the information and to present the whole-systems picture of what is going on at local level.
Stephen Boyle’s paper says:
“a framework should recognise uncertainty and avoid creating perverse incentives for government. The NPF does this well by avoiding quantified targets and setting specific dates ... by which they have to be met.”
In a way, you are saying that we should not have absolute, detailed targets. Can you expand on your thinking in that area?
For a framework of that nature, which tries to look across the whole of Government and the whole of the country, it is better to set a direction of travel and a broad set of objectives. It is then for lower levels of government—departments and agencies—to set, or have set for them, more specific targets. One of the reasons why I like the framework is, as I say in the paper, that it recognises that there is uncertainty. To reflect Jeremy Peat’s point, it recognises that Government is not about sitting in Edinburgh and pulling a lever to make action happen in a predictable way on the ground.
Let me give an example from my own world. In my day job, I work as an economist in a bank. I did not know in 2007 that there was going to be the worst recession on record—I do not think that any of us did. If we have frameworks or systems for planning that are firmly wedded to an expectation about how the world is going to be, we will be disappointed and let down, which is why we need flexibility and recognition that there is uncertainty.
I ask Jo Armstrong to talk a wee bit about the link between performance information and spending priorities.
Some of the stuff that has come up clearly fits into the discussion around whether, when we say that we are going to spend £30 billion, we can know that there will be a positive outcome on key targets. The budget document talks about the key targets, and it is certainly important for the committee to talk about the higher-level longer-term targets. It is not the shorter-term input, or even output, measures that are important; it is the outcome measures around gross domestic product, productivity and employment. It is also important to think about the longer term, but it is always difficult for a one-year budget document to readily identify the impact that it will have on longer-term, broader macroeconomic measures.
It is difficult at the moment to say with a high degree of certainty that the budget allocations that are made, particularly in times of austerity when the outlook is that funding will not be as plentiful as it has been in the past 10 years, are being allocated as efficiently and as effectively as they could be in order to achieve those targets. The targets have not fundamentally changed and we are still looking at increasing GDP per capita and at increasing employment and reducing inequality. We need to think, as Jim McCormick said, about whether we are allocating the budget as efficiently and effectively as we can, and I do not think that we can say that at the moment.
The issue comes back to whether we have effective evaluation measures that allow us to say—having spent what have, arguably, been record levels of funding over the past 10 years—which spending measures have worked and which have not worked. Probably more important is to know, at the margins, where each marginal pound has had the best effect. I agree that we need not short-term measures and indicators but longer-term runs of data and longer-term evaluation techniques.
Some of the quangos are asked to carry out evaluations, but it is not clear to me that all departments use the same level of rigour in evaluating their spending. For example, if a department’s expectation is that putting money into an initiative will improve productivity, has that actually been delivered four or five years down the track, and would it know that? If it just keeps feeding the beast, there may be a diminishing marginal value of return.
On the issue of local versus national that was raised at the beginning, the evidence on what works might be better at national level because the Scotland-wide data on many issues are, although not perfect, better than what is available at local level.
However, there are areas where local evidence is clearly much more valuable. For example, there is not a Scottish housing market, but there are local housing markets. In order to identify how to allocate scarce public funding for social housing, one needs to look at local housing market analyses rather than at a Scottish housing market analysis. I think that that has been a driving force, so there is not necessarily a conflict between the national and the local, because it depends on the particular market that you are talking about.
For me, the most compelling statistic in the national performance framework is the one on productivity. If our aim is to grow GDP and to reduce inequality, productivity is the critical factor. Unfortunately, over the past 10 years there has not been a significant change in our relative position on productivity. That begs the question: in allocating funds, how do we make best use of an increasingly scarce resource in order to affect productivity? We have not moved from the top of the third quartile in the past 10 years.
Thank you for that. I will probe just a wee bit further. What additional evaluation methods could we introduce to make evaluation more effective and rigorous?
For me, there are two elements. First, we do not have a model for Scotland that enables us to say that we can monitor the effects over time that various interventions will have on Scottish GDP. As we do not have an effective model for that, we need to ask why not, given that we have one for the UK. We really need something like that to allow us to track the effect over the long term, rather than over the short term.
We also need to ensure that, for large interventions—I am talking about budget lines not at level 4, but at a higher level—the responsible officer in the department is asked what evidence justifies asking for the funding to be allocated in the first place and provides a baseline against which the project should be measured, and at what point the department will step in to evaluate against that baseline. The baseline must reflect current expectations of the economy based on what we know.
As Stephen Boyle pointed out, in 2007 virtually nobody predicted the downturn that we faced, so it can be legitimate to recalibrate. However, at least the responsible officer will have been asked what evidence justified the spending, what timescale the spending should have delivered within, and at what point an evaluation report will say whether the spending delivered, which might be due to the interventions, or did not deliver, which might be due to external factors. Therefore, there will be a justification either for providing further funding or, if the funding has delivered on its targets, using the funding elsewhere.
Ms Armstrong, one of the lines in your paper says:
“What is less clear is the relative importance of each of the chosen targets.”
You have also said that we need a long-term view. However, if the budget tightens up from one year to another, how do we keep that long-term view? For example, I totally agree that we should put money into housing, which has been mentioned a couple of times. Therefore, should culture suffer in the short term? Culture is probably a good thing in the long term.
As an economist, I would pass that back to the politicians, because that is clearly a political decision. There is a legitimate political decision to be made; one needs to be balanced against the other. However, if you are asking me as an economist to say where to get the biggest bang for your buck, I would say that you need to think through what the longer-term effect of that spending will be.
Some statements have been made that short-term interventions have helped. They might have helped with employment in the short term, but they might not have helped with economic growth in the longer term. It is legitimate to have an impact on employment in the short term and to make short-term interventions, but it is also legitimate to say that we might have done that at the risk of affecting economic growth in the longer term, because culture might have been affected, as a consequence.
Would you go as far as to put numbers against the chosen targets? I think that someone suggested a 5, 4, 3, 2, 1 system, or something like that. Is there a danger that that might become too fixed and the judgment side might be lost?
I fear that the judgment side would be lost. At the end of the day, where the budget goes is a political decision. In asking whether the best use was made of that funding, given the priorities, it is legitimate to get an economist’s answer, but if a fundamentally different view is taken because the budget is different and the world is different, it is necessary to reset the priorities. Once the priorities have been reset, it is legitimate to ask the economists where the money should go.
I note that whenever John Mason talks about money being taken from somewhere, it is always culture that he mentions.
I am sure that that is just accidental coincidence.
It is.
To pick up on that last point, I totally agree with Jo Armstrong that, in any given context, it is for Government to set the priorities. That context will change, but I think that the more clarity there is on the Government’s priorities, the more economists and other analysts can help to determine whether those priorities are being met and how they could better be met. Clarity is always important, but it is for the politicians to form the priority list, based on their view of the will of the country. I agree that one should not rank the priorities in absolute terms or put numbers against them, but clarity is important.
I want to return to how we determine what success has been achieved and what lessons can be learned. I would like to link that to the proposal for an independent fiscal institution, which the committee is considering. There is a great advantage in having an arm’s-length body that is appointed by and responsible to Parliament, and which can provide it with informed and independent advice and develop a close relationship with it and its committees.
Such a body would not have to do just what the OBR does at UK level. Scotland can choose what an independent fiscal institution might do and what its priorities might be. I suggest that consideration be given to whether such a body might produce periodic reports on particular aspects of policy that could feed into the work of this committee and other subject committees. That might be a good way of evaluating the policies of Government, based on the NPF, and of providing advice and analysis to Parliament and its committees that could help it to perform its function of challenging the Government at budget time and at other times.
I am a great believer in the importance of Parliament having the strength to perform that challenge function and I think that, although it receives great advice from the Scottish Parliament information centre and elsewhere, there is a lack of the type of well-resourced, arm’s-length and objective evidence-based input that we get at UK level from the OBR and, to an extent, the Institute for Fiscal Studies. A newly established independent fiscal institution that undertook that role among others, and which was backed by Parliament to make it better placed to carry out rigorous evidence-based examination, could really help in bridging the gap between the data that are provided by the NPF and the firm evaluation and judgment of policies and resource allocation that come at the end of the day.
I found the submission from the Centre for Public Policy for Regions very interesting. The issue of trade-offs could perhaps be looked at in the context of analysing the whole performance framework.
I want to ask Jo Armstrong about what we measure. She emphasised the importance of productivity but said at the end of her paper that there is not a “robust” measure of that. Will she say a bit more about the deficiencies of our current measure of productivity?
An even more interesting issue, of course, is the GDP-GNP one. My understanding is that there is not a measurement of gross national product. I do not know how easy it would be to achieve that. Can you speculate on whether it would make a significant difference? It might even make a significant difference in the next few months, because obviously GDP is always quoted in our referendum debates.
10:15
I will choose my words carefully. There are questions about the appropriate quality of data for Scotland for productivity and the economic growth target, whatever that might be. The CPPR has certainly argued for improved quality of data around those two measures for quite some time. That is not new.
To be honest, I am loth to go into the space of GDP and GNP per capita in this forum. We are not looking at short-term measures, and we have to be careful about using the right measure for the right reason. If we are talking about increasing returns to individuals in Scotland, a GNP measure is probably more appropriate than a GDP measure, but if we are using GDP as the measure over time, it will be just as good for showing improvement, although perhaps not the best. We have a measure that we can use and, as long as we use it and measure consistently, we will show the change.
For me, the issue relating to where we are with the data is that we have not necessarily put a floor on the measures. That is a fairly techie issue, but perhaps it is worth thinking through when we take information off the national performance framework database. What is there is fantastic: it is extremely transparent and totally up to date. The finance secretary is quite right—what more could he do? He could not do any more to make things clear and transparent.
The matter is not debated very much; indeed, this is about the first time that I have heard this level of debate about what measures exist. Such debates do not necessarily feed through to discussions in the chamber, which is unfortunate. What is shown is that certain measures are moving up and certain measures are moving down, but it is not said whether the movement up is because Scotland is not performing as badly as comparator countries. I do not think that we would necessarily think that it is a good thing that we are not performing as badly as others. Perhaps we also want to put floors on some measures that say that we do not want to fall below a certain level and that we want to improve relative to our comparator nations, regions or whatever.
There is quite a bit that we could do with what is already a very good basic tool to improve the discussion on priorities and the budget allocation.
I have asked in meetings with the cabinet secretary and people from other parties whether, if things are improving, that is by 0.1 per cent or 20 per cent.
Yes, that is a big difference.
The scale of improvement or not is important.
The question about our GDP and GNP, particularly the per capita issue, is really good. The UK, of course, tends to talk just about the GDP. If the population of England has gone up by 1.5 per cent a year and the economy has grown by 1.2 per cent, it could be said that there has been a real-terms shrinkage of 0.3 per cent. We have to use a common measure in the argument.
I accept that, but a GNP measure is more appropriate for the local population of Ireland or Luxembourg, for example. We have an atypical structure, and we have to think through both measures. They tell different things; they have different purposes.
Indeed.
On the theme of whether we use GDP or GNP, as Robert Black said, we can use HEAT targets, but the question is who uses them and how they use them.
I was interested in a comment in Donald MacRae’s submission. He said that we need to identify “missing indicators”. I wonder whether we are into the territory of known unknowns or unknown unknowns. Do we know what indicators are out there that we need to use or which ones we use that we do not need to use? I think that he said that we need that for scrutiny. Is that right?
I will bring in Donald MacRae in a minute or two, if that is okay, as the next section of questions is for him. Jean Urquhart has asked to make a comment; we will then go to Donald MacRae.
Two or three written submissions refer to Scotland performs and suggest that, although it is recognised internationally as a good thing, it needs to be promoted more and understood in Scotland. Who needs to understand it better to make it more efficient, and to what end?
Funnily enough, after bringing in Donald MacRae I was going to raise the issue of public engagement and exactly what you have mentioned. You have raised a pertinent point that Jeremy Peat touches on in his written submission.
Donald MacRae mentions the issue of performance budgeting. Can you talk to us a wee bit about how NPF can better inform policy development and spending decisions? I would be keen to hear from Robert Black, as well, on that issue.
My written submission is targeted towards making better use of the national performance framework.
The NPF is partly on the right track because it is based on outcomes as opposed to inputs or outputs, which is greatly welcomed. It also purports to measure Government performance overall—that is an important point—and to do that it has no fewer than around 50 indicators. Those indicators use numbers, which, I warn the committee, are only one way of expressing or describing reality—members should bear that in mind. I suggest that the numbers and the indicators should be validated annually by Audit Scotland.
I want to move the NPF from just measurement to what I have called performance budgeting, which links the spending decisions to outcomes. There is little evidence that that is happening at the moment, and it would be hard to find evidence that an increase in expenditure in one area leads to an increase or decrease in a particular indicator. Such evidence would be almost impossible to produce. Nevertheless, there are indirect linkages and the question is how we can encourage them. I believe that one way of doing that would be to have reference to indicators in all budget statements on expenditure.
In all budget statements, there should be reference to the indicators that are relevant to whatever expenditure is being announced. To do that, we must evolve some degree of prioritisation—that has been discussed already. I would advocate the attachment of a ratings scale to each indicator, which would be challenging but that is what Governments must do in deciding their priorities. Where possible, we should also provide as many international comparisons as we can.
On scrutiny, I suggest that the Finance Committee should undertake an annual assessment of the national performance framework. Why the Finance Committee? Government expenditure is the ultimate expression of policy, which is why the Finance Committee should assess the national performance framework.
That would involve taking a view on the thresholds for each indicator in the NPF. It would also involve taking a view on whether there were any missing indicators—I would certainly question whether there are some missing and whether there are too many. The committee should seriously question whether this or that indicator is the right one, whether it should be in the NPF and whether there should be more or fewer indicators. Finally, the committee should come up with an assessment of how well the Government is doing across all the indicators.
The most important part of my submission is about the need to identify the links between the spending decisions or budget allocations and the indicators. The only way that I can think of doing that is to ask for reference to the indicators in all budget statements.
There are 50 national indicators, each of which refers to a really important aspect of public life and welfare and each of which involves a range of systems and services for its delivery. Even something as simple as reducing the number of road deaths contributes in all sorts of different ways.
I would go part of the way in agreeing with Donald MacRae’s comment about the need for regular scrutiny, but I cannot get my mind around how, in practical terms, the Finance Committee could find the time to do that given its very busy agenda.
In addition, there is an issue with advice and expertise. For some time now, as some of the committee members may dimly recollect, I have suggested that the Parliament could usefully decide on a programme of five or six major reviews of policy areas—such as services for the elderly—over the lifetime of a session. It could commission decent pieces of work that would take the national indicators as one input while also looking at other sources of data. Reports could be produced for the committee on a managed timescale.
To some extent, that would represent a development beyond the Audit Scotland reports, as the reports that I am suggesting would look at some of the policy issues, too. They would be brought to you as an offering, and they could then be used for scrutiny. Over the lifetime of a session, you might have one look at how the criminal justice system is working, one look at how services for the elderly are working, one look at child services and child poverty, and so on.
Otherwise, the meal is just too big—it is indigestible. You have to divide it into bite-sized chunks. That is rather a big topic for today, but I strongly recommend that the Parliament thinks seriously about doing it; otherwise, there is a risk that we will be constantly stuck having a conversation every year about wanting more scrutiny when it is perhaps not practical to do it in that way.
Since I am speaking, convener, can I come back to the important issue of productivity?
Of course.
Thank you. It is often useful to give examples, and I will go back to my theme of looking outside the stovepipes and looking at all the systems. Audit Scotland did a really interesting piece of work a few years ago on orthopaedic services in Scotland, which contained good news and challenging news. The good news was that levels of measured activity increased over the best part of several years, and the less good news was that inputs—the number of orthopaedic surgeons and the spend—increased much more.
If one did the crude math, one would say that productivity is falling, but the health professionals said to us, “Ah yes, but the quality of what we do is changing. We are using surgeons in a different way.”
There were more difficult cases, for example.
It is very difficult to get below that level, so the real question to ask is: “What are we trying to do here?” We are trying to prolong the healthy life expectancy and quality of life of the elderly, so we need to ask which services contribute to that and where orthopaedic surgery fits in. We need to consider how it should be appropriately resourced, given that, if we improve falls clinics, we might have less need for orthopaedic surgeons and, if we improve the aftercare, we might get better outcomes.
It is important that we move away from looking at issues too narrowly. It is difficult for any committee in the Parliament to do that with the time and resources that are available, and the question is, therefore, whether you should go to a body and say, “Would you look at that system for us and come back in a year’s time? We will programme into our work a serious examination of that policy area.” That would take the national performance framework—which is already a leading international exemplar—to a whole new level.
Thank you. Jamie Hepburn will go next, followed by John Mason.
I certainly share Robert Black’s perspective. It might be difficult for this committee to look at the national performance framework in its entirety on a regular basis, given the volume of work that we have.
I want to return to the issue that Michael McMahon raised in his question on Donald MacRae’s paper on the need for further indicators. Donald answered the question to some extent, and he went on to suggest that we might need to consider whether some of the indicators that have already been set out are necessary.
That brings me back to Stephen Boyle’s paper, in which he makes the interesting point that
“changes to indicators should be on a one-in, one-out basis.”
What do the other witnesses feel about that suggestion?
10:30
I will let you in on that one, Donald.
Thank you, convener.
The framework should be considered to be dynamic. In other words, it is always possible to envisage new indicators coming in. Crucially, taking Stephen’s point, if one comes in, one has to go out. What is magic about 50? Could it be 40?
That is exactly the question that went through my head. Why are there 50 indicators?
There will be a huge temptation always to have more. I have succumbed to that temptation and suggested two more. I have not done the difficult part, which would be to say which two I would leave out. The question should be addressed seriously on an annual basis. It will always be a dynamic situation.
I noticed one indicator, “increased physical activity”, which I would replace with one of my suggested indicators, if I was forced to do so. There will always be competing views about which indicators should be in and which out.
To pick up on that point, we had a witness last week who said that the single most important thing that could happen in health would be an increase in physical activity. She would have dropped practically all the other health targets apart from that one.
She said it more than once—she almost had us doing 20 press-ups on the floor in front of her.
Mr MacRae, you talked about reference being made to national indicators when, for example, the finance secretary presents his budget. How does that tie in with Mr Black’s comment that things should not be too narrow? If it is just a reference, that is very general. The finance secretary could say we will spend more money on maths teachers and refer to the indicator, but that leaves open the question of measurement and how to hold it to account. You said that sometimes it is impossible to say that a pound spent here has a particular effect so far down the line. How can we tie all of that together?
You are absolutely right to refer to the difficulty of the linkage. Let us take the indicator of increased physical activity: can anybody tell me what the connection is between particular Government expenditure and that indicator? I find that particularly challenging. We have to accept that.
Nevertheless, the reference could be required in budget statements. For example, if the finance secretary is giving the expenditure on economic development in the Scottish Enterprise budget, I would fully expect there to be a reference to some indicators, saying that they are too low or need to be improved, for example. That would force the recognition and acknowledgment that movement of expenditure will affect the indicator. It might also flush out situations in which there has been an increase in spending but no increase in performance—we have to flush out those areas as well. That is the only way that I can think of to improve the linkage between budgeting and indicators or outcomes.
Would you go so far as to make a prediction? If we go back to the 3 per cent increase in emergency hospital admissions, do you think that we should say that expenditure will reduce it to only a 1 per cent increase or to a minus value?
I would not go that far. I would not recommend that—it asks far too much.
You would just say that the expenditure would have an impact on that indicator.
Yes.
I found the suggestion about references to indicators extremely interesting. It could be very useful when we think about the performance framework in general and the budget. I am not clear about the relationship between the indicators, the purpose targets and the strategic objectives. Why did you pick the indicators? One critique of the strategic objectives is that any activity could meet any one of them because they are so general. That may be a weakness of the strategic objectives.
There may be a great deal of sense in going for the indicators but you have already opened up an interesting discussion. The other thing is that, if you are going to do that, you have to get into the area of relative importance. Like Jim McCormick, I am interested in poverty. I think that there are only two indicators related to that, but my sense is that they are probably more important than some of the others. I believe that physical activity is a particularly important health indicator, so I disagree with you there.
You seem to be saying that every spending decision should be related to an indicator, but I am sure that we could all think of major spending commitments that we cannot relate to any of the indicators. If I mention any of those, we will immediately get into areas of controversy but, without giving a view on free tuition—in fact, I have always supported that, but we know that there is a debate about it—I note that none of the indicators, as far as I can see, has any relevance to that. I am sure that we could all think of other examples. People may draw a certain conclusion from that, but I am not drawing it. However, the suggestion is really interesting and I think that it would bear fruit in relation to our consideration of the budget.
The subject of prioritisation has been raised. Some have questioned my idea of attaching ratings, but if we do not do that, what do we end up with? We end up with 50 indicators and we have to see all of them going up at the same time. I find it challenging to accept that as a concept that would actually work. We have to recognise that beneficial upwards movement is harder to achieve for some indicators than for others, and that there will be some competition between some of the indicators.
When we do a bit of digging and look under the surface, there is an interesting variation between the 16 national outcomes in terms of which indicators are related to them. One outcome has only three indicators related to it, whereas another has 18. My sense is that the relationship between the outcomes and the indicators is a bit random, to be frank, and a case could be made for doing it quite differently. I would like to see a clearer narrative about the linkages and which ones are considered to be headline indicators for the delivery of particular outcomes versus others that might be important but are not headlines.
I want to pick up on a couple of other points that committee members made. First, Jamie Hepburn asked whether we add to the indicators and change them. When a policy agenda changes, we have to ask how the framework adapts. For example, in this year’s guidance to community planning partners around the priorities for the single outcome agreements, the first new priority is on economic recovery and growth. It is interesting that the word “recovery” is used, because that might suggest that the purpose target on labour markets is about getting back to the level of labour market participation that we had just before the recession. To pick up on Jo Armstrong’s point, there might be something about absolute performance—not just performance relative to that of England and Wales and Northern Ireland, but absolute performance. That might be a worthy addition to the suite of measures so that we know whether we are getting back on track following the massive impact of the recession.
Secondly, John Mason made a really important point about externalities when he asked whether things would have been worse if we had not made some interventions, even though it looks like we are going in the wrong direction. The example that I would give from the Scottish Government’s programme is around fuel poverty. That programme has a high quality evaluation and we know that it makes a positive net difference but, if we look at the figures on fuel poverty, they went substantially in the wrong direction. The answer to that conundrum is that things would have been much worse without the Scottish Government’s programme. Should we look at doing programmes that work on a much bigger scale to mitigate the impact of externalities, which in this case are fuel prices, UK reforms around tax and so on?
It is right to say that the framework tells us something in terms of numbers, but we should be careful about how we interpret the trends. The story on what Government is doing might be a good one, but the outcome might still be negative.
What Jim McCormick has just alluded to is really important. As Donald MacRae said, with 50 targets it is extraordinarily hard to say, “They’ve all got to move up, and they’ve all got to move up at the same time.” That just will not happen.
Dare I say it that the level 4 spend has driven what the targets are? Having allocated funding, we have found targets to ensure that that funding is justified. That might be a bit unfair, but there is probably an element of that in it.
We might be saying that the world has changed because we now have a harder economic environment in which we will have to work with less money, but overall the headline purpose is still the same. The overarching outcomes that we seek to achieve are probably roughly the same. We must be careful about being so prescriptive about what we do underneath that that we end up perversely allocating funds in the wrong direction.
I am minded to think about the water commission—this is only partially relevant, but it might be a useful example to think about. When Ofwat started out, it set a massive number of targets for this beast to become more efficient. As it got more comfortable with what it was trying to achieve, its headline target remained the same—effective water at a quality that is set by European Union standards. How it delivered that was much more in the hands of those who were responsible for allocating the funding.
As we get more comfortable with our headline purpose being what we are actually about, and that those relatively high-level outcomes are what we seek to achieve, how we allocate the budget is probably a departmental or quango decision. We allow them to get on with it, in the expectation that they will deliver and influence the delivery of those high-level purpose and outcome targets. However, they are not short-term measures. Short-term interventions will be for short-term, expedient purposes. The longer-term budget allocations must be to deliver the longer-term, higher-level purposes and outcomes. Therefore, it seems to me to be extraordinarily hard to justify, monitor and use effectively 50 indicators for allocating large chunks of public funding.
What number would you suggest?
Four to seven high-level objectives are quite hard to manage and monitor effectively and realistically.
In the work that we did back in 2008, we liked the idea of setting high-level targets but we were not convinced that they were necessarily the right ones. There is definitely justification for asking whether they are the right targets; if they are, we are comfortable that we can use the data that we have to monitor them, which we do over long-term not short-term movements. It is then up to the departments to decide how they will allocate their funding, in the expectation that at some point within a budget cycle or spending cycle, they would be monitored against the delivery of those targets.
I move on to the issue that Jean Urquhart raised. Paragraph 13 of Jeremy Peat’s paper says:
“While Scotland Performs is an excellent online resource which brings together the most up-to-date information available on how Scotland is performing, unless people are aware of its existence, it will not fulfil its potential.”
When I started looking at the NPF, I was surprised about how much very valuable work there is there and how much very valuable information is available. I was surprised that I was not more aware of it—that is, in part, my fault and responsibility—but that implied that it was not being used as much as it should have been in aiding decision making in the Parliament and elsewhere.
There is a case for getting the NPF better known, but that will be extremely difficult because it is not exactly the most sexy topic to try to get across. Asking the BBC, Radio Forth or whoever to have a series of programmes on the NPF may not be a winner.
There is a need to bring to policy and resource allocation discussions the fact of the NPF’s existence, the results that are emerging from it and the analysis that is in there, so that those who are interested in decision making are better aware that the NPF is a part of the process and so that it does not just sit there being ignored, but is used to inform and assist decision making.
10:45I like Donald MacRae’s idea, which is the last point in his submission, about linking the NPF indicators to the annual budgeting process and the budget statements. That is important, and it should be much more up front that the NPF exists and is being used. If that continues to be the case, the NPF might be reported more and more people might become aware of it. That, in turn, would lead to better decision making, as people become aware of what is there and the implications of what is there, as they start to consider how it could be improved, as they draw conclusions from the data in relation to their priorities and as they make comparisons with how decisions are being taken.
Getting the NPF more in the public eye is desirable. Donald MacRae’s suggestion about how to start that process is the best that I have seen, but it will not be easy or straightforward. A public relations exercise for the NPF is not one that I would particularly like to take a commission for.
The group that is considering the NPF with the cabinet secretary is mulling over how we raise its profile so that people know more about it and are more happy to engage with it.
I do not take quite such a pessimistic view about the potential public interest in it. Considering our brief discussion about the national indicators, I think that we could have quite a good discussion with that focus group about the relative importance of the national indicators, and the group might come up with other ones that they think are more important. Through the indicators, the framework could actually be quite accessible and interesting for people.
No one else has indicated that they wish to speak, so I ask each of our guests in turn to make any final points that they wish to make.
Who is my first volunteer? I thought that it would be you, Bob.
I have a relatively specific, important point to make. Jo Armstrong mentioned this a while ago during the discussion about extra indicators. I will not go into the bear pit of suggesting which indicators should be omitted, but it has struck me for a number of years as rather surprising that there is so little about housing in the framework. I will not say much more than that but, as I am sure we would all recognise, there are major issues regarding the future direction of housing policy and the housing stock in Scotland. That is such a fundamental issue with regard to public health, welfare and wellbeing that it might be worth revisiting it.
The housing scene has been transformed during my professional lifetime in terms of tenure mix and so on, and in terms of housing delivery; public housing is now much more focused on the housing association movement than on local government. That has pushed the subject slightly to the side of what I would call the main departmental interests. It is such an important issue for the wellbeing of Scotland that it may well merit a rethink.
Does anyone else wish to contribute on any aspect of what we have discussed today, or to comment on anything that we have missed?
I would like to view this as an opportunity to refresh the high-level indicators, principally because, in periods of severe budget constraint, the decision-making processes around budget allocations are quite different from what they were over the past 10 years. Now is probably as good a time as any to start that process such that some of the targets clearly identify where the priorities might lie.
If there is one thing that would help in the next year, it would be to tie together the national performance framework and the guidance document for community planning partnerships, in particular the six priority policy areas, which are supposed to be being delivered on in every part of Scotland. If we better marshalled the linkages, the evaluation evidence and the other evidence across those six priority themes, we would be doing the whole of the public realm in Scotland a service. To pick up Jean Urquhart’s point, that would improve ownership at the local level.
On a more specific point about indicators, to me the current approach underplays the importance of the education attainment gap in Scotland. In Scotland, the top 20 per cent of schoolchildren perform at the level of schoolchildren in Hong Kong or Singapore, but the bottom 20 per cent perform at the level of those in Turkey. Closing that attainment gap would have consequences for many other indicators, but the framework is too high level and too generic to capture the importance of doing that.
If I may, convener, I want to try to draw together three or four threads. Is that all right?
Of course.
First, I thank the committee for permitting us to have this session, which I have found to be extremely constructive. Members will all have realised that we came to praise the NPF, and that any comments have been made in that general context. We also accept how difficult the process is.
To me, three or four things have come out in the discussion. One is that any improvements will be at the margin, but there can be improvements. There are issues about having better comparators and how we take account of what would have happened without action—the counterfactual point that John Mason raised. If we add a few indicators, we should take away a few. I agree with Donald MacRae that we should have an indicator on business investment, and preferably on business investment in research and development, as that is sadly missing. However, we should not simply add on—we should take away at the same time. The point about vetting data merits the committee’s attention. Also, some form of re-examination of the link between the indicators and the objectives might be desirable. However, I think that all that is at the margin.
We have had a good discussion on the prioritisation and how far to go on that. Again, I have a lot of sympathy with Donald MacRae’s point that one cannot just leave everything as equal and that one has to make an attempt to give a degree of ranking. That would not have to be done to the extent of quantifying and giving 2 per cent here and 50 per cent there, but an indication should be given of the order of priority.
We have also discussed how to use the outcome of the performance framework to further inform decision making. Bob Black gave the idea of taking a number of topics in the course of a session of Parliament and investigating them in depth. I suggested potentially linking the approach to the IFI. Really, that is the same story, which is about using an informed and objective arm’s-length group to look in detail at a select group of issues. We cannot do it all but, if we take the issues in turn, over time we would have a much better grasp, which must aid decision making.
That brings me to my last point, which is on the link between resource allocation and policy decision making. In virtually every submission, we have said that that is the area in which development is most needed. We all think that the NPF is valuable and has great potential, but it is not easy and it is not a mechanistic process. However, it is important that, in a selective and periodic way, resource allocation between policy areas is examined on the basis of the information in the NPF and further evidence-based analysis.
So, there are pointers for where to go. We are heading in the right direction and it is all very valuable. The process is not easy but, for me, it has been constructive to be involved in it because what had been a glimmer at the back of my consciousness has come much further up front, where it should be. I have realised the value that has been extracted and the further value that could be extracted in future.
I have a very quick point. The committee should remember that the NPF is the measurement of government and that the Scottish Government does not have the full ability to influence all the indicators in the NPF. It might be right to have indicators on issues such as GDP growth, but the Scottish Government cannot prevent an international recession and clearly it has limited powers under the current political settlement. Just remember that the NPF is a measurement of government, but that the Government has limited powers to influence the outcomes.
Some of us are acutely aware of that.
Stephen Boyle will have the last word.
To go back to where Jim McCormick started, the NPF is a real asset for Scotland and we should be pleased and proud of it without being complacent about it. In that context, we should not ask the framework to do too much. It is a measurement framework and is not a substitute for judgment on matters of principle. I would resist strongly the temptation to tinker with it too much. It is easy to measure more things, but I would resist that. I would resist the temptation to go for ever more amounts of data. As I said, my personal preference would be to go for more evaluation rather than just more measurement.
I will abuse my position as the last person to speak and say that, if I was to make one change to the NPF, I would take out any one of the economic indicators and replace it with a measure of median household disposable income. The productivity point is important, but we know that, across the western world, economies can grow and average incomes can rise while most people do not get better off. I would include that indicator.
I thank everyone for their positive contributions, which the committee very much appreciates. That ends the public part of the committee’s deliberations today.
10:56 Meeting continued in private until 11:19.