Public Audit Committee

Meeting date: Wednesday, May 29, 2013

Official Report 467KB pdf

---

Scottish Government Progress Report

The Scottish Government progress report has been circulated to colleagues. In the report, the Government provides details of progress, as requested, in relation to two of our previous reports—“An overview of Scotland’s criminal justice system” from February last year and “Cardiology Services” from September last year. The progress report is one of a series of reports that we asked for. The next one will be on progress in relation to our “Report on the management of patients on NHS waiting lists”.

I open the session up for anyone who wants to comment.

I am generally happy with the responses that we received. I was drawn to the comments on cardiology services and particularly those relating to the keep well programme. I have picked out some paragraphs on page 11 of NHS Health Scotland’s “Keep Well impact evaluation 2012-14—End of year progress report 2012-13” about the availability of data and how we estimate value for money, outcomes and so on. They are in an interesting section that describes how difficult it is to assess the success of the programme as a whole by a single means or otherwise. It mentions the difficulty in gathering data and the lack of technical capacity in health boards to extract data.

We have raised the issue a number of times, so I was hoping for something a wee bit stronger than the response, to allow us and future committees to assess the performance and the success or otherwise of things such as the keep well programme. I suppose that there is a common message. I would like us to ask for this to be looked at and strengthened so that data is gathered consistently across the national health service. That would allow us to properly scrutinise matters.

I admit that I was struck by the fact that a whole section in the keep well report is entitled:

“Why is it so hard to tell if Keep Well is ‘working’?”

That is a little worrying, is it not? The paragraphs on the availability of data are a significant part of that section.

Fundamentally, we need to be able to rely on consistent data in any system if we are to assess its effectiveness. This is another example of that.

Yes.

I picked up on exactly that point. Page 3 of the Government progress report mentions

“the fact that Keep Well was not originally designed as a research study.”

That is not an excuse for not having data on where the money goes and what outcomes are likely. It also states:

“a single overall measure of the effectiveness of Keep Well is not possible.”

I seem to remember that in the evidence—I think that it was Willie Coffey and the convener who met the general practitioners—the GPs in particular were quite sceptical about the amount of money that goes into the keep well programme, and they implied that it could be better spent elsewhere. The situation is unacceptable.

Having said that, the bottom paragraph on page 3 states:

“The Outcomes analysis study is not yet complete.”

I suppose that I took some comfort from the fact that, having spent a few million pounds, the Government is now beginning to look at some sort of outcomes analysis in auditing the trail of money to see whether there has been effective spend in reducing inequalities and addressing cardiology issues.

Like Willie Coffey, I thought that that was an open admission. There does not have to be a research study to have a decent trail of money and to see whether we are getting value for money and whether it is being spent effectively.

At least the outcomes analysis is being worked on and the results will be reported in 2013-14. That seems a long time, but I suppose that something is happening, albeit late.

I do not disagree with what Willie Coffey said. It is clear that there is again an issue with the availability of data. People are saying that that problem has come up again and again. We keep looking at a slice of it. We are looking at the keep well programme and saying, “Well, they should keep better data on that. Maybe we should be asking them to improve that.”

Is there not a bigger problem, in that the legacy systems in the public sector do not seem to be fit for the purpose for which we now need them—for information gathering and analysis? Increasingly, we are trying to look at more sophisticated analyses and better ways of doing things, but we are not getting the data extracted. That is a historical matter—the systems were never really set up to cope with that.

I am not sure how effective or how cost or time effective we are being in looking at a slice at a time and saying, “This needs improved and that needs improved.” The reality is that the whole public sector has an issue with data gathering. Perhaps we should consider whether it is possible to suggest that a long-term programme be put in place comprehensively across the public sector.

I apologise for being slightly late this morning.

I will make a brief comment, which impinges slightly on the Health and Sport Committee’s remit. I know that we are the Public Audit Committee and I agree that we have to look at the best spend of money and quality outcomes but, when we talk about tackling health inequalities, it is reasonable to say that short-term outcomes analysis is notoriously difficult, although important, in healthcare. I commend our committee for continuing to scrutinise the matter, but we have to give the caveat that a long-term approach to tackling health inequalities is vital. Just because we do not have very focused outcomes in a relatively short period, that is not a reason not to stick to the long haul in tackling health inequalities.

The Health and Sport Committee is doing a significant inquiry into health inequalities. I think that that committee would be keen to look at how we can get the best outcomes from spend. However, I put it on the record that we have to be in this for the long term, and we should not rush to judge based on short-term outcomes.

Is the keep well programme one of the programmes that the Health and Sport Committee is likely to look at?

The short answer is yes, but I do not think that that committee will look at it specifically; it will look at the programme under more general themes. I would be keen for the Health and Sport Committee’s clerks to update members on where we are going with the keep well programme for their information, if that would be helpful.

I am sure that it would be.

It is not a case of the short term or the long term; there is no single overall measure of effectiveness—that is not possible to have. I appreciate that inequalities are generational, but there is still nothing to measure for the next generation.

It would be useful for the committee to have some understanding from the accountable officer of how they see their evaluation of keep well. The programme is doing a really effective job. It was established in 2006, and self-evaluation tools are available to a range of organisations. I would expect a wee bit more self-evaluation at least to give us an indication of how bodies think that they are performing. I am pretty confident that they are doing quite well in the areas that we have covered.

We need to decide how to deal with the progress report. We can simply note what the report says or we can write to the accountable officer for further clarification. Mr Coffey has suggested that we write to the accountable officer to pursue the point, which Mrs Scanlon also raised, of how the keep well programme’s success and value for money, which we are interested in, will be judged, given that the interim report indicates problems. I will come on to Mr Beattie’s point in a second. First, do we agree to do what has been suggested?

Members indicated agreement.

Mr Beattie makes a fair point—Mr Coffey referred to this as well—which is that, whatever we look at, we continually seem to come up against issues of data handling and of information and communication technology. However, to look at all that across the public sector would be a very big piece of work. We must also bear it in mind that we could not at our own hand launch such an inquiry.

I suggest that we consider writing to the Auditor General for Scotland. She has heard the exchange, as she is in the public gallery, but I suggest that we write to ask her formally whether there is any way in which Audit Scotland could look at that issue across the public sector rather than constantly doing so in slices, as Mr Beattie described it. Would that be helpful? It seems quite a difficult undertaking, but I think that Audit Scotland might have a view about how it could be done.

It might be worth while getting the Information Services Division’s view on the matter as well, because it has a broad picture of everything that happens across the public sector and in health boards in particular. It must have a view about the clarity and consistency of the data that it gets from different areas in the public services. It would be useful to get its take on the issue.

I think that we could do that as well.

Yes.

Do we agree to write in the suggested terms about keep well to the accountable officer, who I guess will be Derek Feeley of the national health service; to write to the Auditor General about the possibility of undertaking a thematic investigation; and to ISD to ask the question that Willie Coffey suggested?

Members indicated agreement.

Is there anything else on the progress report?

How regularly do we get reports on the criminal justice system?

I will ask for the clerk’s advice on that, but I think that we would have to ask for a further progress report. The progress report in front of us is the one that we asked for.

So we do not get such reports as a regular item.

No.

Okay, that is fine.