Data Collection
Item 2 is a discussion on data collection. The issue has been raised in discussion with the committee a number of times over the past year. I welcome to the meeting Antony Clark of Audit Scotland; Roger Halliday, from the Scottish Government; Jim and Margaret Cuthbert; and Jenny Stewart of KPMG.
There are no opening statements, so we will proceed straight to questions. In time-honoured fashion, I will get the ball rolling. I have a question for Ms Stewart first. In your written submission, which we received within the past 24 hours, you state:
“The little cost data that is readily available in the public domain is provided by the Scottish Government but ... the most relevant output data are produced by Audit Scotland. The categorisation of each set of data is slightly different. Thus, it is difficult to compare the cost data with the output information from these sources.”
Can you advise us about what difficulties that actually presents?
I had not realised that the paper that I sent to the clerk by way of background had been circulated around the team, but I am happy to take questions on it. The paper was produced jointly by KPMG and the David Hume Institute to assess not so much the data collection area but how performance management in the public sector, particularly in local authorities, could improve. The particular focus was cost information, because that is a difficult area for us.
On your question about the issues that might arise, we find that there are various sources for publicly available information, but it is difficult to match the three important things that I imagine the committee is interested in, which are the outcomes that are being achieved, the cost of the provided service and how we benchmark those against each other.
If we take one of the easy areas in the paper, we can see how much it costs each local authority to collect the council tax, which varies hugely; we can also see how much income is generated from that, so we get an output measure. However, that is more of an exception, because it is quite hard in other areas to work out where cost information is available and why there are such discrepancies.
In areas such as social care, which takes up a large part of the public sector budget, large differences can be seen—for example, cost per care can be something like two and a half times as much in different areas. Clearly, there will be good reasons why something costs a lot more in the Western Isles than in Falkirk, for example.
11:30
However, the question is why the discrepancy should be so great. If more benchmarking data were available, we would be able to cross-compare performance in local authorities and the NHS much more readily in order to understand those differences. The basic premise of our paper and behind the areas that we chose to examine quite closely is that even a shift to average performance—not even best performance—up to the top quartile can save substantial amounts of money. I realise that this is dull performance management stuff, but it is really important and could release significant resources for reallocation to other priorities.
I am sorry, convener—I hope that I did not go on too long.
Not at all. You should not feel constrained in any way.
I have a number of questions for Mr Halliday. In his excellent submission, Professor David Bell says that it is sometimes
“difficult to compare performance across local authorities and across NHS boards”.
Indeed, it is an apples and oranges issue, even in Scotland. Furthermore, he suggests:
“There are strong arguments for producing data that are comparable both to other parts of the UK and to wider international groupings.”
What is the Scottish Government doing to improve the situation?
With regard to Professor Bell’s comment that
“the key issue is establishing the linkage between the spending of public money and the outcomes that such spending is expected to produce”,
what progress are we making in that respect?
Finally, Professor Bell also says:
“There is generally no point in collecting data which government, nor any third party, can usefully use to aid decision making. Scotland has a historical legacy of collecting vast amounts of data on agriculture, which may now lack significant usefulness.”
He has mentioned that issue before. I realise that you have made a very detailed submission, Mr Halliday, but I have to ask you not only what the Scottish Government is doing to improve data collection—and, as people have suggested, to collect more data—but whether it intends to put into abeyance data that are collected for no purpose and, if so, the types of data collection that would be affected.
Roger Halliday (Scottish Government)
I am very happy to talk about those matters. First, I must point out that the approach to collecting data on agriculture is different from data collection in other areas because the vast majority of that data is collected to meet European requirements and we face particular penalties if that does not happen. Nevertheless, we have made clear and significant progress with regard to data on farm accounts. By using the payments data from the farm accounts systems to collect data that would otherwise have been collected as part of what was called the farm census survey in order to meet EU requirements, we have been able to stop parts of the census itself, saving about £500,000.
Coming back to the point about comparability, I note that as far as Scottish Government statistics are concerned we have a very strong tradition of engagement with users of statistics and have put in place the ScotStat process, which allows us to immediately engage with 2,000 users of statistics. People register with their interests, and those interest groups are used to set priorities and consult on changes that we might make to statistics. For example, they drive reductions in data collections that are deemed lowest priority and the tailoring of statistical products to the needs of those who use them in the first place.
On the point about producing data in Scotland that are comparable with the UK and internationally, users have told us that, in certain situations, such comparisons are indeed at the heart of what they need in statistics. However, the majority of users of statistics in Scotland want to make within-Scotland comparisons or to look at what is happening in Scotland as a whole, rather than to make comparisons with the UK or with what is happening internationally; I would say that that is the case with 19 out of 20 uses of statistics in Scotland.
As far as prioritisation is concerned, Professor Bell might have made his comment because we are doing some great work to improve the coverage, depth and reporting of statistics about Scotland and to make possible comparisons within Scotland, which is what users are telling us that they want.
On comparisons between Scotland and the UK, we are doing an exercise across the Government statistical service as a whole to signpost and to explain to people the various sources of data and the differences that might exist in certain areas to make more informed comparisons possible.
The final question was about within-Scotland comparisons, on which I have two things to say. Scottish neighbourhood statistics is a fantastic resource, which provides a range of statistics. It consists of thousands of data series that cover all different aspects of society and the economy. The neighbourhood statistics programme is about getting comparable data across Scotland, down to very small areas. In addition, in conjunction with the Society of Local Authority Chief Executives and Senior Managers, we have been supporting a benchmarking project, which not only involves the presentation of data, but enables sensible comparisons to be made between different parts of Scotland.
Thank you very much. That was very helpful.
In your submission, you rightly say that
“To collect and quality assure statistics takes time”.
I think that we all understand that, because when people ask questions the answers to which are based on data collection, they expect information right away, but it is hard to provide such information quickly without compromising accuracy. You talk about ensuring
“that population surveys meet key information needs while maximising the analytical potential of the data they generate, the precision of estimates and value for money”.
As far as your methodology is concerned, what do you believe could be improved? Are you happy with the way in which the methodology is working? Are we at optimum levels in that regard?
Do you mean methodology in terms of collection of data?
Aye. I am talking about maintaining accuracy while making data available expeditiously. Sometimes there can be a lag. In his submission, Professor Bell commented on the fact that the 2011 census data will not become available until 2013. How do we improve matters and ensure that information is given more timeously, without compromising its accuracy, to allow policy makers to act on it?
A number of things spring to mind. I was involved in a discussion last week in which we looked at potential ways of getting hold of information more quickly. There have been some great examples of that recently, such as the Scottish exchange of education data programme. There is now a platform for collecting education and other data. We are looking at what other possibilities there might be for using that within the Scottish Government.
We have a programme that is looking at what we will do on the census in the future. It will be possible to improve on how things have traditionally been done over time. That programme will run between now and 2014, because making a change as significant as the one that we might need to make in the census gives rise to quite a lot of significant methodological problems. There is definitely a strong programme of work on that. Making results more timely would be one dimension of assessing options.
I hope that I have described a couple of things that we are doing or planning to investigate further.
We are to undertake an extensive inquiry into demographics, because of the demographic challenges that Scotland faces in the next few years. Which minister will be in charge of addressing demographic change and who in your department will be responsible for collecting data on that issue? If you do not have the answers to hand, you can always get back to us.
It would be best if I got back to you, because a number of interests are probably involved.
You will want to tie them in.
My next point is to Mr Clark and to Mr and Mrs Cuthbert. As we do not have submissions from you, I wonder what you feel the gaps in data collection are and how it could be improved. Margaret Cuthbert seems to be the most excited about the question, so perhaps she can answer first.
Thank you for inviting us to give evidence. Jim and I could make quite a number of points about existing data, but the most important issue that faces us just now is the dynamic situation in Scotland and ensuring that we have statistics that are fit for purpose. Are the statistics that we collect suitable for what we hope to achieve in this century?
One big point is that, although we are happy with many aspects of statistics that are collected by us and dealt with by the Scottish Government, there are various big holes. One big hole is that, while Scotland deals with devolved subjects and devolved finances, what happens in reserved matters is also extremely important. The presumption is that one hopes to get a good relationship going in which reserved moneys help out devolved moneys, so that money is not wasted.
We have found a dearth of information on what is happening with reserved moneys, other than what is seen in “Government Expenditure and Revenue in Scotland”. As members know from the history of GERS, we have found a lot of problems with it. We are still finding problems—for example, as has been found in other areas, the Treasury funding statement does not connect up with public expenditure statistical analysis or Government expenditure and revenue statistics. We have found that, despite GERS, it is almost impossible to find out in large areas of reserved expenditure what is being spent on Scotland and that moneys that are allocated to Scotland in the official statistics are not in fact being spent in Scotland. Those are big problems.
As for problems in relation to tax, we need to put an awful lot more effort into statistics on income tax. We do not need just the headline information about income tax. As members know, most income tax is collected from high-income earners, sections of whom have special relationships with HM Revenue and Customs. They do not come into the statistics at all, so we must deal with that.
We need to have much more information about what is happening with regard to income tax in trusts in Scotland—especially trusts that involve large land ownership arrangements. It could appear that some of our noblest peers do not earn much money, whereas tax is being dealt with as if a place was a trust. Such matters need to be dealt with.
I have described what we think needs to be done in the future. There are also big issues in extending our statistics way beyond GERS. We have specific problems with ordinary statistics that are calculated in Scotland—particularly those that affect policy—but I have talked long enough, so I will give somebody else a chance.
11:45
Okay. I will bring in Mr Clark, and then your hubby can come in.
Antony Clark (Audit Scotland)
Audit Scotland has repeatedly made the point at the Public Audit Committee that there are often significant gaps in performance, cost and activity data across the public sector. An obvious example is the NHS, in relation to which we have often highlighted significant gaps in financial information in the context of our ability to track the impact of policy implementation and form judgments on value for money. We have often highlighted gaps in management information at various levels of the health service and gaps in relation to post-policy implementation and review. When it comes to evaluating complex, cross-cutting policies there are often significant gaps.
Our interest is not just in the health service, of course. We have an interest in local government, and the overview reports that we present to the Accounts Commission for Scotland every year have a common theme, which is the need for local government to continue to improve the quality of local performance management information. Our best-value audits have identified improvements in performance management over a number of years, but there is still some way to go to strengthen the availability and usefulness of cost and performance information, to demonstrate that best value is being achieved.
I will be more positive. The context of this discussion needs to be considered, and there is clear evidence that the financial pressures that public bodies face are driving bodies, individually and collectively, to look more closely at the information that is available to them on how they spend and use their resources. Many of the points that Mr Halliday made about engaging effectively with users of statistics across the public sector give us cautious optimism that there will be improvement in the area, which will be driven by external factors as well as the internal driver of people recognising that it is a good thing to do.
We jotted down a number of themes before we arrived, and I will be happy to give you a note of them. I will restrict my comments to four themes.
The first theme is administrative data, and is underlined by what Roger Halliday was saying. The question is where the boundaries of Roger’s empire are. An awful lot of important data is produced by administrative systems. We came across an example recently when we were looking at procurement. There is a database service and a portal has been set up, for administrative purposes. The portal does its job excellently, but it is also being used to produce statistics, although it is not fit for purpose in that regard.
An administrative system can be perfectly successful but it does not necessarily collect data to the common and scrutinised standards that are necessary for statistical purposes. If we want to use data from administrative systems for policy purposes, we need professional input—that is a plug for Roger Halliday’s people to get out there and get their hands on administrative systems before misleading inferences are made from the data that come out of such systems.
The second theme is openness. It is incredibly important that the statistical system is open, so that insiders and outsiders can dig down and assess the quality of data. A classic example is the old PESA database, which had a big input into GERS. The Treasury ran the PESA database and did not let the public or even other departments, such as the old Scotland Office and the Scottish Executive, have access to it.
On the advent of freedom of information we requested access to PESA; we were the first people outside the Treasury to get access to the database. We found horrendous things. For example, we found that £4.4 billion of expenditure in England was being classified as non-identifiable expenditure, when in fact it was expenditure on prisons and so on. The spend was non-identifiable only within the regions of England, but it had been classed as non-identifiable overall, which completely distorted comparisons with identifiable spend in Scotland. When GERS was done, we got a share of the English non-identifiable expenditure on prisons—the whole thing was ridiculous. That emerged as soon as the database was opened up to public scrutiny. Only after we got hold of it did the Scottish Executive get hold of it.
Openness is an essential part of quality of data. If a system is not open, we really cannot believe in the high-level figures that we are given. It is unfortunate that the tide is against openness. The move towards greater privatisation—I am thinking of the private finance initiative and utility privatisation, for example—meant that much of the stuff that previously took place in the public sector moved into the private sector and disappeared under the shield of commercial confidence.
That has been the most retrograde step, and it need not have happened. If the public sector is the client, it is perfectly open for it to insist that it is written into the contracts that information should be made available, perhaps after a decent six-month delay when the issue is no longer red hot and sensitive in terms of commercial confidence. The public sector should be able to insist that decent information is available so that things such as PFI can be adequately analysed and scrutinised.
The third theme, which is dreadfully important, is the careful specification of output measures. One example of that is PFI. An awful lot of the party-political heat that surrounded the issue of PFI—and much of the big public policy mistake that was made—was related to the fact that the public sector was using completely inadequate output measures to scrutinise the process.
That was the fault of the Treasury, which effectively said, “We are happy if the equity funders in PFI are earning a 15 to 20 per cent return.” The Treasury was taking the rate of return on equity as a basic performance measure. One side in the PFI debate was saying that the funders were earning only 15 to 20 per cent, while the Treasury was saying, “That’s fine—everything is okay.” Actually, that measure—the rate of return on equity—fails to encapsulate the degree of profit that the private sector equity holders can take out. It is not the measure that those funders use themselves: they use a measure that is based on net present value at a lower discount rate.
We wrote that up in a paper that has just been published in Critical Perspectives on Accounting, if you are interested in the detail. The fact is that the measure that the public sector was using was understating the potential profit to the equity holders by 30 or 40 per cent. That fact contributed to a vast public policy mistake, and meant that the debate that was being conducted divided on party-political lines in a way in which, if the facts had been known, it should not have done.
The fourth theme is a gap, as we see it, in the available statistics at present. A lot of the debate about Scotland concentrates on GERS, which is an analysis of the Government account in Scotland. The fact that we concentrate only on the Government account is a nonsense. For the UK as a whole, one would look not only at the Government account, but at the pink book and all the flows in and out of the UK economy. Similarly, we would argue that GERS should be broadened into a proper overview of all the inflows and outflows from the Scottish economy.
That is difficult to do, because it means getting a proper grip on exports and imports, but an awful lot of the effort of the statistics and economic section is put into the input-output tables. Those tables must arrive at an estimate for the export-import balance. If they are not arriving at an implicit estimate that is reasonably accurate, we should not be producing them. If they are arriving at such an estimate, we should be using that to build at least a broad-brush approximation of the overall flows in and out of the Scottish economy. That would transform the political debate. That is one suggestion of a gap; I will leave it at that.
I want to add to the general themes that others have picked up with regard to where the gaps are.
Statistics in Scotland are pretty good, and the Scottish Government is very well resourced—I am sure that Mr Halliday would not necessarily agree, but it is comparatively better off than elsewhere in terms of staffing levels, statistics and analytical services. There are some really good long-term series, which David Bell picked up, that make a real difference to Scotland. They include things such as the health morbidity data, which are a real competitive advantage for us because they allow so much more research to be done into health.
While we are talking about the gaps, we should stress some of the advantages that make a real difference. As Audit Scotland has highlighted, the gaps are often around the financial areas and the cost, rather than the long-term series that help to inform long-term policy.
I will pick up on a couple of issues. I absolutely endorse the point about transparency. If local authorities, health boards and others all had to publish performance data, we would soon find that questions would be asked in any event that would help to manage performance. That is quite important.
To pick up on the point about users being predominantly Scotland based, I suggest that there is perhaps a slight chicken-and-egg situation in that regard. Because the data relates predominantly to Scotland, it makes it difficult to make that comparison. For example, we sponsored a piece of research on health spending that was carried out by the Centre for Public Policy for Regions and which found that it was quite difficult to make a direct comparison between what is happening in Scotland and what is happening in England. I welcome the fact that work to address that issue is on-going but, given how much the NHS in England and Scotland is diverging in the way that it operates, it would be very helpful if it were made a priority area as far as comparability is concerned.
My final point is more of a plea and relates to something that is not so much a gap but which brings us back to the issue of timing. We have to wait quite a long time for Scottish GDP figures, which come out later in Scotland than in the rest of the UK. I absolutely understand the point about the trade-off between timing and accuracy but it would be lovely if the publication of the Scottish GDP figures could be expedited. I also acknowledge the point about concentrating on GERS, which will become even more important as the debate moves forward.
Mr Halliday seems quite keen to come back into this lively discussion. I will bring in my patiently waiting colleagues very soon.
I must point out that when I referred to comparisons with local government and health I was talking about comparisons within Scotland, not across the border.
Some very helpful points have been made. One of my biggest problems—it is quite big barrier to carrying out some of the work that people have alluded to—is in getting hold of the necessary data, especially on the underpinnings for economic statistics, from other parts of the UK Government, particularly HMRC, the Department for Work and Pensions, the Office for National Statistics and the Bank of England.
Jenny Stewart helpfully suggested that we have loads of resources, compared with countries of similar-sized population. I must point out, however, that while we have 150 statisticians, those other countries have at least 1,000 people who cover such things. Given that, I think that we do pretty well. Moreover, resource constraints are one of the reasons for engaging with those who use statistics.
As for using administrative data, we have been looking not only at how the resources in the Scottish Government are used. We now have 13 public bodies who are also producers of official and national statistics. In recent discussions, we have said that we want, as a strategy, to work with other potential producers of statistics and to train those organisations to improve their capacity and capability in collecting data, in order to improve the quality of administrative data and the uses that are made of it. I am sure that members will want to ask about the uses of data—in particular data linkage.
I thank the witnesses for their evidence so far and want to drill down into a number of issues that have been raised. With regard to the GDP figures, I suggest to Roger Halliday that there is no reason why the ONS cannot provide similar GDP estimates for Scotland. I might be wrong, but is it fair to say that its priority is publication of UK-level data and that data for Scotland are a second-order priority?
12:00
That might be true for the ONS, but it is not the barrier to our producing more timely GDP figures. In fact, there are two barriers, the first of which is that Scotland is a smaller country than the UK. The ONS has relatively definitive numbers for some bits of GDP but has to forecast other bits, which is why revisions are made in the UK figures; the last time, for example, there was a 0.1 per cent revision. There is no point in our putting out a number if it is going to be revised by percentage point, so we try to balance timing and availability of data. Our first-cut GDP series is based on more data but, even so, there were revisions of 0.4 per cent to the previous quarter. Without that level of data, it is difficult to produce a credible series.
Moreover, a number of the data sources that we use—and that need to be used, irrespective of whether we or the ONS put the series together—are businesses that report on a UK level, which means that all we might have would be UK-level changes in value added from those organisations so we would have still to estimate what is actually going on in Scotland. To change such a system would be quite a big undertaking. Those are probably the primary reasons why we do not do something different.
Irrespective of the timing issue and the other issues that you have outlined, I wonder whether, given that the Government’s primary overarching national objective or purpose is to raise Scotland’s long-term sustainable economic growth rate—I will ask about the characteristics of growth in my next question—the more timely production of GDP figures is important, particularly in the light of the fact that we are, as Mrs Cuthbert has pointed out, taking on additional tax-raising responsibilities.
It would be good to be in an ideal world in which we had perfect data and we could get figures out faster. However, there is no point in being faster at putting out figures that are not useful indicators of where the economy is going. GDP is an important indicator, but there is a range of other indicators and forecasts that the committee and others should bear in mind.
I accept that and do not mean to be difficult. All I am suggesting is that there is—in principle at least—no reason why the processes for producing a UK GDP estimate for a certain timeline cannot be deployed to provide a GDP figure for Scotland on the same timeline. It is just that the current processes do not facilitate that. As you have made clear, UK-level data that are difficult to disaggregate are collected. However, that does not need to be a permanent state of affairs; we need to look at ways of securing more timely GDP data to inform policy in a more timely way. However, I do not want to get bogged down in that matter.
Jim Cuthbert referred to the wider picture and talked about flows into and out of Scotland. I suppose that we lack an estimate of gross national income and other measures that would help us to understand Scotland’s macroeconomy. However, we have just taken evidence on the welfare reform agenda and I am aware that the Welsh have formulated a macroeconomic model that allows them to model the impact of welfare reform on Wales. As I understand it, the Scottish Government does not have the same ability. Do Mr or Mrs Cuthbert or Mr Halliday have views on how good the Welsh model is and whether we are yearning for something that is actually not very good? Is there any reason why we cannot develop a model in Scotland? Would it be prohibitively expensive?
First of all, I will make a point about GDP that is related to those comments. Over the next 10 years, GDP might well lose its place as the number 1 indicator. After all, the current crisis has made us very aware of the fact that GDP values can suddenly fall, largely because of the element of GDP known as imputed rent. It is a fictitious amount that reflects the fact that lots of countries have different mixes of owned and rented homes. In order to be able to compare things across Europe, for example, in the case of a country such as Britain, where the level of home ownership is high, we impute what rents would be. If house prices soar, so does the imputed rent value and if house prices sink, imputed rents go down, so GDP goes up and down depending on that.
We need to move away from such indicators if we are interested in growth in Scotland and in the real mechanisms that affect welfare. I suggest that, instead of the endless flight towards the goal of getting GDP figures out faster, we devote a bit of Scottish intellect to creating an indicator that would be part of the GDP indicator but would be more help.
Do any of you know what is done in Wales? Is there any value in considering that example?
I sat on the Howat review from 2005 to 2007. I think that, around 2005, the Scottish Government considered whether it could have a similar macroeconomic model, but did not progress the idea. It might be worth investigating why that was not taken forward at that stage.
That is an oddity to me. We have some excellent universities and academics: why do we not use that intellectual power to develop a model for Scotland?
We have heard today and previously, from other witnesses, that the cost of living in rural economies can be 10 to 20 per cent higher than it is in urban situations because of the costs of fuel, transport and so on. The Scottish index of multiple deprivation essentially assumes that living costs are uniform across Scotland when examining income deprivation, for example. Do we need to reflect on that and to develop a more nuanced version of the SIMD in order to take account not only of income levels in an area, but of the cost of living, which might show us that there is a different pattern of deprivation across Scotland from what we are familiar with?
That is an interesting suggestion, and I thank you for it. We will produce a new index of multiple deprivation in a few months, but it will be difficult to consider those issues within the timescale. My only obvious concerns are about whether there are data to support what you suggest, and what people’s priorities are in development of the index. I know that there is going to be significant development, mainly due to the welfare reform agenda, because data from the DWP are a significant input into the index.
My question is directed to Mr Halliday, but if anyone else wants to comment, given their expertise and knowledge, that will be fine.
In my experience—and the experience of colleagues to whom I have spoken—there is nothing more frustrating than putting a question to the Scottish Government and getting back a single line answer that says, “We do not hold that information centrally.” It occurs to me that we do an awful lot of things with money—we spend money on the health service, local government and so on—but we do not know what is happening to it because we do not chase up the information.
My question has two parts. First, why do we not have that information centrally? The Scottish Government is giving money to all those public services, so why does it not collect the information that would tell it the outcome of that spending?
I will introduce the second part of my question with a short anecdote. The other day, I wanted some information about a primary school in my constituency and I wondered whether the same information was available across all 22 of the primary schools in my constituency. I got a member of my staff to phone around all the primary schools, and it took her less than an hour to get the information. We have 11 health boards, 32 local authorities and eight police boards. If an MSP asks, “What are we doing with this?” or “What are the statistics on that?”, why cannot the Scottish Government produce an answer?
First, as you know, we are now in the world of outcome-based Government; our priority is the information that the Scottish Government has on overall outcomes. When it comes to individual processes, resources or outputs, the report indicates that there is no centrally collected information in a number of areas. It is partly about proportionality; all such things come at a cost and we must ensure that we do not spend more on data collection than is reasonable in the light of the level of spend.
It also comes back to my previous point about listening to users. You are flagging up that we maybe need to do more on that score.
It strikes me that you have outlined a situation in which you do not know why you cannot get information. Currently, a dozen or more MSPs might want information or statistical data on what health boards are doing. Rather than those 12 MSPs lodging a question to the Scottish Government and it getting back to them, they have to write to all the individual health boards and collate the information themselves. That hardly sounds very efficient.
Transparency would help to mitigate that. If health boards and local authorities were to publish more management information, as I would call it, which is probably what you are looking for, and made it readily available, it would not be necessary to ask the Scottish Government for it. Transparency is therefore very important.
To hark back to the Howat review, one of our key recommendations was about the need for a very active challenge function in the centre in the Scottish Government. Every three years there is the comprehensive spending review and the work is done very thoroughly at that stage, but there should be a standing challenge function throughout. I reiterate that, if something like that were in place, more information would be available.
I know that it is hard for Parliament, but I am pretty sure that somewhere in the report we also touched on the resources that are available to Parliament and to the Scottish Parliament information centre. If those resources were increased, more research capability would be available to the Finance Committee and other parliamentary committees to exercise their proper function in terms of what is happening to vast amounts of public money.
I am not aware that there is a lack of resources in SPICe to do anything that I have ever requested.
In that case, it might be able to help with some of these queries.
I will make a brief comment. Relatively few questions can be answered on the basis of a simple phone-around. On most topics, if you want data that are consistent and meaningful, we are talking about a properly structured information-collection system and about having systems in a local authority or whatever that can provide answers to such questions. If you are dealing with a serious topic, the collection of meaningful information involves specifying your requirement, going to the local authority and coming to an agreement with it that it will put in place its own internal systems that are capable of producing that information, then it will collect the information. To meet a serious requirement, it will take a minimum of several years to set the process up. That is not to defend what goes on now. If you want a good data collection system, that is the sort of inherent time lag that you would be dealing with. Obviously, you want to be sure that your systems are up to date and that your user requirements have been identified as far as possible in advance so that you have the process in place to meet them. However, this does not involve a simple phone-around.
12:15
I want to build on Jenny Stewart’s and Jim Cuthbert’s points relating to benchmarking. Your frustration about having easily available comparative information about the cost of performance or outcomes delivered by range of public bodies in many ways reflects the fact that we have a relatively underdeveloped set of approaches for benchmarking in the public sector. Although it is entirely reasonable to expect the Scottish Government to have information, there is an onus on public bodies themselves to gather and publish that information and to be publicly accountable for how they use their resources. It is well worth the committee’s while to think about there being an equal interest for public bodies.
I am slightly in opposition to that, in that I think that the Scottish Government should have more control. I can give two examples, the first of which is the flagship policy on free personal care. In a series of papers of free personal care that we have produced over the years we have pointed out an inconsistency in the numbers in the tables; the Scottish Executive added up figures from across local authorities to find out the total spend on free personal care when at least one of the areas put in a zero amount. In other words, those areas had not produced the data for the Scottish Executive so the Executive just put in zero and added the rest up to get a total. That is the type of thing that we are interested in—areas on which there should be data but there are none.
The other example is in the final business cases of PFI, which should be of interest to every MSP. There was a unit in the Scottish Executive—predating the Scottish Government—specifically for PFI. We had to request final business cases, except for a handful, by way of freedom of information requests because the information was not available for all local authorities centrally. When we did get them, most of them were conformed copies; in other words, any financial data of any worth were blanked out. That information should have been available.
I hope that Mr Cuthbert did not misunderstand what I was saying. I was trying to simplify the question—the point being that when we look for information, the Scottish Government’s response is not that the information does not exist, but that the Government does not hold it centrally. That leads to a paper chase, via SPICe or some other resource, to collect information that I think the Scottish Government should have available to it. That is the point that I am trying to make.
Is there a lack of harmony between management information systems, which might be partly responsible for the inability to answer the kind of questions that Michael McMahon has asked? Do we collect information consistently across local authorities and health boards?
Professor Bell is shaking his head.
There are different management information systems, but some of the ways in which we collect data, such as through the ScotXed system that I mentioned earlier, are designed specifically so that we work with the MIS companies to ensure that the data are extracted.
The support that we are able to provide in those areas has significantly improved the quality of the data. As Jim Cuthbert mentioned, the area that we need to progress is the quality of the data that are held locally. For example, on the education side, we are aware, after collecting data, checking it and publishing it, that before that happened, 2.5 per cent of children’s gender was recorded incorrectly on school systems. That is just an indication of the levels of quality that maybe exist out there.
Professor David Bell (Adviser)
I will make just a couple of remarks. In his first contribution, Mr Clark mentioned evaluation, about which we do not talk enough. We put in place policies, but we do little thereafter—or, indeed, beforehand—to evaluate them properly. An example—it is not necessarily a bad one—comes from the alcohol pricing stuff. Some pretty good work was done in Sheffield for the Scottish Government, which used international evidence on responses to changes in alcohol prices.
Much more extensive pilots than we undertake are often used in England—we can understand why such an approach is used there, where budgets are larger. For the English equivalent of self-directed support—individual budgets—a huge piece of work was done to compare what happened in some areas where the policy had been introduced with what happened in areas where it had not been introduced. I do not want to go down to specifics; we are doing evaluation reasonably in some respects, but we do not approach it systematically over the piece, and statistics are key to that.
My other point is optimistic. Most of the stuff that I deal with is about how policies impact on individuals and is not the macro stuff that Jim and Margaret Cuthbert talked about. Mr Halliday mentioned data linkage. The notion is that we have administrative data sets on health—and maybe on social care and education—and that Scotland is better placed than almost anywhere else in the world to link together those data sets.
If I am thinking about preventative spend and if I know from a survey that somebody now has cirrhosis, for example, I can link that person’s data to their health record, so I might go back 10 years and find out that they visited their doctor regularly because of alcohol overconsumption. That capability is available in Scotland to a greater extent than almost anywhere else in the world.
My evidence for what I say is that I am part of a consortium of five universities that have just made an application to the National Institute on Aging in Washington, which part-sponsors surveys on ageing around the world. There are such surveys in England, Ireland and Europe, but not in Scotland. The institute is thinking of coming to Scotland because the potential for linking data here is possibly better than that anywhere else in the world. We have spoken to the NHS about its data and to National Records of Scotland about education data.
I do not want to be too negative about what we do. I have described an interesting possibility that is available only in Scotland.
There you go—some brownie points for Mr Halliday and his colleagues.
It is clear that some things are easier to measure than others. Does that skew our decision making? In recent weeks, the committee has spent a bit of time on employability, and youth employability in particular. How is employability measured? Can it be measured?
We can measure positive destinations, so we know whether people are in jobs or at college. However, somebody could be in employment but not have become more employable than they were a year before, or somebody could have become more employable in the past year but not be in employment—the easy thing to measure—because no jobs are available. I am a bit concerned—I am an accountant, so the issue relates partly to my background—that we measure the things that are easy to measure but we find it more difficult to measure other important things.
Of whom are you asking your questions?
I am throwing the points out—
You are throwing a hand grenade.
Employability is quite an interesting issue and brings us back to David Bell’s point about data linkage; indeed, the paper that Fergus Cochrane produced for the committee touches on the same matter. I believe that data linkage might be able to make a significant input here. For example, there are a number of interventions—further and higher education, school education, training and so on—and if data on the things that people did could be linked with, say, benefits data or income and employment data, and if those people could be tracked over time, researchers might be able to understand better the interventions that actually work in moving people towards or into work and the kind of work that they move into.
Secondly, someone mentioned a survey that was carried out on school leavers and their destinations. That might not be needed if we were able to build much richer data around the issue. I point out that that is not happening at the moment—I am simply illustrating the potential of linking different sorts of data. However, as I said, there are certain difficulties, including around the ability to access really rich and informative sources of data on benefits, tax and work from HMRC.
As has already been pointed out, it might be a long time before we see results from some of the things that we might do now. Professor Bell suggested that we might be able to go back 10 years but my guess is that that would be unusual and that for many people who have cirrhosis of the liver, for example, or some other health issue we simply do not have data that goes that far back. I wonder whether, in 60 years’ time, we will be able to measure the impact of the family nurse partnerships that we introduce now. Are we even going to try to do that?
This is exactly the point of data linkage. We are developing a data linkage framework, the consultation on which closed just last week and a significant objective of which is the ability to track people over a period of time for research purposes. A significant use for that framework could be the study of health-related situations; in other words, instead of having to follow up people over time, with all the costs that that would entail, we could, as Professor Bell suggested, look at people’s health records to identify specific causes and effects. Indeed, that is already going on to some extent. In the health service, for example, a statin that was introduced to deal with heart conditions was taken off the market after an examination of people’s health records down the line found that it was associated with a lot of mental health problems. As a result, the approach had a distinct advantage for patient safety.
As John Mason was right to point out, sometimes we do not even try to measure the things that, although important, are difficult to measure. That is why the shift to outcomes has been really helpful, and the next stage on from that will be outcome-based budgeting and the ability to track budgets to outcomes. It is a difficult issue; although I have done some international research and my colleagues have carried out some research in Canada and New Zealand, there is not a huge amount of other research out there. That said, the matter is worth pursuing.
As for Mr Mason’s reference to family nurse partnerships, I think that two distinctive issues emerge, the first of which relates to the statistical theories that we have over the long run. David Bell made the important point that we need to keep those things going, irrespective of changing policy priorities. Secondly, post-project evaluation must be carried out on particular policy initiatives, requiring access to a variety of data. I echo the points that David Bell made about the need to do more of that. When the Howat review looked at how much of Scottish Government spend was subject to post-project evaluation, we found that it was quite limited, mostly to projects where there were EU requirements. EU funding was always subject to post-project evaluation because that was a condition of the funding. There was then a bit on top of that. The situation has improved and there has been more post-project evaluation since then, but a lot more could still be done.
12:30
One has to be clear about what one is looking for. For a lot of subjects, one may never be able to develop a simple indicator, which may be an inherent problem. The important thing is to have a rich enough database that one can research. Often, one can get a handle on something in a research study if one has appropriate data to work with, even if one cannot end up with a simple indicator.
In a sense, it is the other side of what Margaret was saying about GDP. With GDP, we have latched on to a simple summary indicator that may be a bit misleading. It would be better if we could dig below that and find out what is going on. The important thing is to set up the databases and vet them thoroughly so that there is good-quality data at the individual case level. One can then do research studies to find out what is going on. One may never end up with a simple indicator, but one may nevertheless learn an awful lot. The databases are essential to the process.
I have two questions, the first of which is about ensuring a consistent approach. There will sometimes be data that is collected locally that does not translate up to a national level. There will always be statutory performance indicators that authorities are obliged to collect and key performance indicators that authorities collect, but there will sometimes be local performance indicators that authorities choose to collect themselves. The question is how we can ensure a consistent approach to the gathering of information. For example, I recently wrote to local authorities requesting information about tree preservation orders in connection with the bill on high hedges that I am going to introduce, and I received 32 very different responses, ranging from one sheet of paper to hundreds of sheets of paper, depending on the amount of data that each local authority held—I recognise the irony of a response on tree preservation orders stretching to 100 pages. How can we ensure that there is a consistent approach to the collection of data at a local level? Perhaps Mr Halliday has a view on that.
I apologise, but I have to leave to get a flight to London.
Just as the conversation has turned to tree preservation orders.
I was advised of that in advance, but the clerk told me that you were going to Majorca. [Laughter.]
I wish that I was. Thank you for the invitation to give evidence today. If the committee has any other questions, I will be happy to provide responses in writing.
Thank you.
We have found, especially in chasing up the spend on free personal care, that even if a well-defined series of questions goes out, what comes back to the Scottish Government can be very different from what it had hoped for. When we pursued that with the Scottish Government, the answer came back—after maybe four years of complaint—that it was very good for people such as us to phone in with our complaints or suggestions for improvement because that was the only way in which the department would get some muscle in trying to get local authorities to do something about the situation.
Even after all that, the most recent edition of the “Free Personal and Nursing Care Scotland” bulletin, which is for 2009-10, states, under the heading “Limitations of the data”:
“For some Local Authorities, it appears that gross expenditure on personal care services at home is higher than overall net expenditure on home care services.”
I could go on. In essence, all the points that we have made have still not been addressed nine or 10 years after the policy came into being. That means that like with like is not being added up across local authorities, so there are massive differences in the spend per hour that have no meaning. In some places, it appears that less than the minimum wage is being spent on personal care—it is just nonsense. I do not know how we can get Scottish Government clout over local authority data collection processes so that the data that is finally produced does not make us look stupid.
That shows that the issue is not straightforward. Some people might think that statisticians are a bit dull and boring, but their skills are important in ensuring consistency of definition, and they work with others towards that end.
We have done two things in particular. As I said, we have engaged with public bodies as producers of official statistics to train their staff and improve their capability in that regard. In addition, we have undertaken projects over the past two years to address improving local indicators—that issue was just referred to. I am working with Antony Clark on that. The priority is to get consistency in community planning partnerships’ single outcome agreements. We are now thinking about how to use that approach for other issues.
I have a more general point. There is no straightforward answer to the question of how one gets better comparability across organisations. Our experience is that there tends to be better comparability where there is ownership of the information to be used for the purpose of comparison. The process of using information inherently helps data cleansing and comparability.
That is a good point.
Obviously, as we refresh how we collect data, there will undoubtedly be data that we choose to collect no longer because it is not relevant. However, because there is a natural human tendency towards cynicism, there will be some out there who will imply that certain information is no longer being collated because it is convenient not to do so or because information is being hidden. How do we get to a stage at which we refresh the data collection process while making it clear that that process is providing data that are more relevant and that we are not simply brushing things under the carpet because we would rather that people did not know about them?
The answer is that we have user committees and groups of people who decide whether something is important and what the lowest priority is, for example. What has worked for me in the past is signalling ahead what the overall plan is for a particular theme, topic or group of users, to ensure that people are not caught out by problems.
That appears to have exhausted questions from members. However, I have a couple of questions to finish off the session.
Robert Chote from the Office of Budget Responsibility said in evidence to the committee:
“The obstacle to producing a Scottish macro forecast as an underpinning for a Scottish fiscal forecast is not only a lack of resources to put together the model to do that—we would need to do it in a way that was consistent with the UK forecast as a whole—but the fact that some of the data that underpin the UK-wide forecast are not available at a Scottish level. The raw material that you would need to input into the model is not there ... you are some way away from having the data infrastructure available to be able to do something more dramatic on that score.”—[Official Report, Finance Committee, 28 March 2012; c 910.]
Mr Halliday, what kind of work is being undertaken to prepare for the powers that will come via the Scotland Act 2012?
The analysis that you quoted is absolutely right about there being resource and consistency difficulties. The most important one from the committee’s point of view is probably our attempt to improve the way in which data from other parts of the UK Government is shared—particularly data from HMRC. There are strong legal barriers to doing that at the moment, but we are exploring how we might overcome some of them. Any help from the committee would be useful.
How useful is the national performance framework with regard to data provision?
I have a slightly negative view of that, but that might just be a reflection of the interests that have been steering us over the past 15 or so years. However, I am slightly sceptical about macro-level indicators, because I believe that the devil and the value always lie in the detail. The thrust towards overall performance indicators has the potential to be a bit of a diversion of resources away from getting the quality and the analysis right at the micro level.
The answer to your question, convener, depends on what you think the NPF is there to do.
What do you feel it is there to do in terms of data provision?
I was going to say that the NPF seems to be a useful summary of the direction of travel in a number of quite important areas. However, there is probably quite a bit more work to do to understand the interrelationships—this is Jim Cuthbert’s point—between what drives performance and the contributory factors that might lead to good performance or poor performance.
The position is the same in relation to almost all outcome measures. For example, if I am doing a bit of research into higher education in Scotland, I might want to look at graduate first destinations as an output measure. Obviously, universities are keen to promote the message that they are doing extremely well in relation to first destinations, but I believe that the only figures that they give are those that give a good impression of them. When we dig further, we can find that some graduates’ first destination may be the job market but that they are working in Tesco or Pizza Hut. Parents look at first destinations data and say to their children “Look, you are guaranteed a good job when you come out of university.” However, the first destination figures hide a mass of outputs under one heading that says that graduates have got jobs. We fool ourselves a lot about outcome measures, because they show only what the person who collects the data wants us to find.
On that interesting point, I bring the evidence session to a close. I thank the witnesses for their evidence.
That was our final agenda item, but I remind committee members that our meeting next week will be on Tuesday, not our usual Wednesday, and that it will start at 2 pm, not 5 past or 10 past 2.
Meeting closed at 12:42.