Audit Committee, 26 Nov 2002

Meeting date: Tuesday, November 26, 2002

Official Report 237KB pdf

---

“Measuring Up? A follow-up report on performance measurement in the Scottish Environment Protection Agency”

I call the meeting to order. Agenda item 3 is evidence taking for the committee's inquiry into the report by the Auditor General for Scotland entitled "Measuring Up? A follow-up report on performance measurement in the Scottish Environment Protection Agency".

I welcome Mr John Graham, who is head of the Scottish Executive environment and rural affairs department, Mr John Ford, who is the acting accountable officer for SEPA, and their colleagues. We will examine issues that are raised in the follow-up report and ask questions on two main areas: SEPA's impact on environmental improvement and the management of SEPA's regulatory role.

As there are a number of witnesses, I propose that we should first direct questions to Mr Graham and Mr Ford, who can indicate if they would prefer one of their colleagues to answer. I hope that that is acceptable.

Do any of the witnesses wish to say anything before we begin? I should also welcome Mr Ken Collins and apologise for not mentioning him before.

We are pleased that we have been asked to come to the committee today. The Auditor General for Scotland's first report was helpful to us. Its findings focused on important areas, which we then tackled through structural organisational change, the adoption of an outcome-based performance measurement approach and by beginning a complete overhaul of the guidance and support provided for operational staff. Those steps have produced a fair amount of change in the organisation. The second report, and recent feedback from stakeholders, indicates that we have made quite significant progress in achieving the objectives, although we acknowledge that a great deal still has to be done. We are working to improve our performance even further and some of the impending legislation in the Scottish Parliament, notably in relation to water, will help us. We particularly welcome the committee's and Audit Scotland's continuing interest. Thank you again for inviting us today.

The committee thanks you for your remarks. I would be grateful if you would introduce your other colleagues for the record.

On my left is Mr John Ford, who is director of finance and support services. On his left is Mr Willie Halcrow, who is director of operations. On my far left, which is a strange place for him to be, is Dr Campbell Gemmell, who is director of strategic planning.

You are all welcome; we look forward to your evidence. My question is on whether results supported by SEPA provide an accurate record of performance. The Auditor General notes on page 5 of his report that SEPA's performance results are based on "unaudited information". In those circumstances, how do you gain assurances that SEPA is producing the results that you expect from it?

We rely on SEPA's data. We have had no reason in the past to find fault with the performance data that SEPA has supplied us with. We are also aware that many of the stakeholders who are watching SEPA's performance would alert us if anything was systematically wrong with its data.

Mr Graham gave us the view from the department. Paragraph 2.5 of the report tells us that the Scottish Executive has recently produced new guidance, providing

"a clearer policy framework for SEPA".

In what way does that additional information allow SEPA to focus its work to meet the Executive's expectations?

The structure that SEPA has followed for performance management and performance reporting is based on outcomes. We are in the process of bedding that in; we went through the full process once, during our corporate planning round. We found it extremely helpful that the Executive set out systematically new policy priorities, because that allowed us to begin the process of proper alignment between the activity of SEPA and the Executive's priorities. We have made good progress in that direction already, given that we were working closely in any case, but it is helpful to have the priorities spelled out explicitly.

So you see improved focus and co-ordination in what you do.

Absolutely, and that will improve over time as we go through further corporate planning rounds and are able to align everything more tightly.

You have tried to show your anticipated results for 2001 against a range of indicators of progress towards expected improvements in the overall state of our environment. Going through the tables in exhibit 2, it is quite difficult to measure some of that progress. I pick the example of biodegradable waste. Why are you not able to assess whether we are on course to meet the European Community directives on the reduction of biodegradable waste that goes to landfill?

The target that is described in exhibit 2 is, by 2002, to reduce by 25 per cent the amount of biodegradable waste that goes to unengineered landfill. That target reflects previous practice and was dropped in 2000. It has now been superseded by targets that have been drawn from the national waste strategy and the landfill directive. The new targets are to reduce biodegradable waste to 1.5 million tonnes by 2006, which sweeps up part of the previous target, and to eliminate the disposal of any active waste to an unlined landfill. Although the directive has only just been implemented, it is interesting to note that 43 landfills in Scotland have already closed, following the production of site conditioning plans this summer.

Progress is being made on both fronts. However, I should point out that there are inherent difficulties in accurately measuring the waste streams that are disposed of or that go to landfill. In the past two years, we have established with the Executive's assistance a specialist unit and are gathering much better, more accurate and more timeous information.

That is helpful. It might also be helpful for people to understand the significance of the new targets when we track progress in future.

We move on to ask how, as far as the Executive is concerned, the revised performance measurement framework will highlight SEPA's contribution to environmental improvement.

Paragraph 2.8 of the report says:

"SEPA's 2002/2003 Corporate Plan represents a significant change in its approach to its strategic planning and performance and monitoring processes."

What are the benefits to the Executive of having a clear picture of SEPA's work in relation to regulatory targets and measurable outcomes?

Our key response to Audit Scotland's initial report on SEPA was the introduction of the policy priorities document, which is designed to bridge the gap between the Executive's published environmental aims and objectives and the measures that SEPA can take to contribute towards them. The document is in the public domain—in fact, it is on the website—and articulates SEPA's role in contributing to the Executive's overall objectives. We have made it clear to SEPA that, in future corporate planning rounds, we expect the agency's performance measures to be set within the policy priorities framework to ensure that they are clearly tied in to the Executive's overall objectives.

Exhibit 10 in the report shows how SEPA is developing performance information to support its outcome measures and to improve the focus of its performance results. What benefits do you expect to gain from those improvements?

That forms part of the process of moving from a concentration on inputs and activities, as it were, to an examination of the impact of those activities in the real world. The most important benefit is that we are now concentrating on real delivery, not on what lies behind that. I should stress that moving towards such an approach is a major challenge; after all, outcome delivery is not solely in SEPA's hands. For example, one of our outcomes involves engendering respect for the environment. I am sure that we all agree that that is very important, but it is a task not for SEPA alone, but for a range of bodies. The same is true of water outcomes.

Such outcomes are difficult to measure.

Yes. However, we can use public perception surveys and other, more objective, measures wherever we can. Nonetheless, it is important that we measure outcomes that are hard to measure. I should point out that we are capable of dealing with more obviously robust items, such as water and air quality, although it has to be said that air quality is not solely SEPA's responsibility. We must also take into account the transport agenda and the role of local authorities; it is not a simple matter.

We are all going to have to work with the process over time. We cannot make radical changes to the situation quickly; it will take a lot of effort. We have already taken important steps, and clear guidance helps in that respect. However, we also need to develop relationships with other bodies and to ensure that, in its dealings with Scottish Natural Heritage, Scottish Water, Scottish Enterprise and Highlands and Islands Enterprise, Government recognises the connections that exist between SEPA's performance and the delivery of overall outcomes in which those other players are involved.

The crucial phrase that you used was "over time". In paragraph 2.10 of the report, SEPA has suggested that it will be 2007 before

"better performance information becomes available".

Why will that take such a long time?

The situation varies from subject to subject; perhaps some of my colleagues will amplify that point. Some of the data processes that legislation requires us to put in place will not be introduced until 2006-07, which gives us an obvious target for having everything ready. That said, we are already making progress towards that target. We must ensure that we do not wait until the last minute to comply with important pieces of legislation, but that we comply in a planned and structured manner. As the year-on-year targets in our corporate plan make clear, we will make progress; however, we must do so on as even a front as possible to ensure that we have data that prove that we are performing to the standards that we say we are performing to.

What assurance can SEPA give the committee that the improved information will not be delayed and that it will meet the target of 2007?

I can say with confidence that the systems are being developed and will be in place by 2007. With the help of the Executive on, for example, the water framework directive and other issues that are being addressed through Scottish legislation, SEPA is ensuring that the data and processes are in place to deliver on time. SEPA has a great deal of work to do in some areas, and it is reasonable to plan for a robust structure by 2006.

Will you give the committee a quick outline of those areas?

That question would be easier for my colleagues to handle.

Three critical areas are the main focus of regulation: the protection of air quality, the protection of water quality and safe, managed disposal of waste. It has taken many years to develop the reasonably accurate measurement systems that we have today, and further development of those systems will take considerably more time as the regimes are widened. That is why SEPA wants to proceed carefully in a structured manner.

By 2005, SEPA hopes that, for the first time, Scotland will proportionately regulate abstractions from watercourses and groundwaters. It also hopes that Scotland will regulate river works and other aspects of the water environment that have not been regulated previously. It will take considerable time before we are able to establish control systems in harmony with the rest of the United Kingdom and Europe, and SEPA does not anticipate that work being completed for some two to three years. In short, water is the biggest area in which regulation is required, as it involves a regulatory regime that is new to Scotland. It will take two to three years to develop quality measurement systems, because Scotland's systems must mirror those of other European countries.

As someone remarked earlier, the control of air quality is a divided responsibility. SEPA regulates prescribed processes, whereas local authorities have a more general responsibility—for traffic pollution, for example. The general measurement of air quality is focused on cities and is covered by a UK-wide network. Scotland does not have its own comprehensive network. SEPA is investigating that and is developing a classification system for air quality. Again, that will take time.

As we shift the focus from municipal waste to all sorts of important priority waste streams, we have to develop the means of capturing information and ensuring that it is correct and publicly available. There is a great deal to do but, as Dr Gemmell said, we have plans in place to do all that. Those are the vital areas.

How sure are you of 2006? Are you not aiming at a moving target?

That is our experience. Inevitably, the target will move. We expect the work to expand and change because of the forthcoming landfill directive, the national waste strategy for Scotland and the water framework directive, but we have a core target.

What we have heard so far is very much a comment on outcomes. The Scottish Executive—I presume—is helping ministers to set the outcomes that will be considered. SEPA has responsibility for delivering those outcomes, measuring progress, giving guidance and so on. In general, the Parliament is moving away from number crunching—asking how much money is spent, what one gets for the money and why it has to be spent. Will the Executive and SEPA comment quickly on how they are developing outcome measurement systems? There seems to be a slight difference in focus in what we have heard from them today.

I would not characterise it in that way. SEPA and the Executive work together closely. The difficulty is that the targets that we have in place year on year are different in type and in detail from the outcomes. The outcomes are at a much higher level. We probably need the experience of a number of iterations with Government to be sure that the full suite of outcomes—not the SEPA outcomes, as it were, but the Government outcomes—fit better together. Where our contribution fits in will then be much clearer to us.

We are only one year into the outcomes approach and we have some methodological difficulties. Some of us have been exploring how outcome-based budgeting is done worldwide. We have learned a great deal from talking to the United States General Accounting Office, which in some ways has made more progress than many others on that front. The one note that we get from all the experts—people in Australia, in the health service and in the US—is that outcome-based budgeting is difficult. It is challenging and demanding. Apart from anything else, we all focus day to day on what is done and what it costs—people always tend to focus on those details. It can be difficult to connect that high-level vision and policy to the nuts and bolts. We are already talking and working together to try to focus at a higher level. The Government's strategy on that front and the views of the Scottish Executive and the Parliament are evolving. We have to work with that and see how we, too, can make progress.

Are you suggesting that, in the interim, outcome-based budgeting is more of an art form than a science, but you are trying to develop the science?

That is not a bad characterisation of the situation. What we have learned about outcome budgeting is that it is a slightly mysterious art form; it is not terribly precise. However, we know that we probably have to go there if we are to be able to give the Parliament the confidence that it knows what bang it gets for the bucks.

Will the Executive comment?

I am not sure that I recognise the discrepancy that Mr Davidson sees. In my judgment, the Executive has moved progressively towards making it clear to the Parliament in the budget documentation what it intends to buy with the resources that are at its disposal. In "Building a Better Scotland", which is the most recent example, a clear set of high-level objectives for my department is associated with the resources allocated to the department, and an explicit target—or pair of targets—is attached to each of the objectives. Those targets demonstrate specifically where we intend to get to. On waste, for example, which is a matter on which we are working closely with SEPA, "Building a Better Scotland" makes it clear what the target is and where we want to get to by when with the considerable increase in resources that we have.

The document details what we are looking for across the range of areas for which SEPA is responsible and translates the high-level intentions of the Executive into what we want from SEPA. Exhibit 9 in the Auditor General's report shows that set of relationships for the range of activities for which SEPA is responsible.

On page 21 of the report, under the heading "Conclusion", the top section seems to indicate that a measurement framework is already in place to go with the objectives that are set for SEPA. It strikes me that you are not both in harmony, but I am happy to leave the matter there.

Margaret Jamieson will now consider how changes in SEPA's operations will improve consistency and efficiency in its regulatory role.

Paragraph 2.13 sets out new guidance and a new code of standards of service that will make the quality that SEPA expects to deliver more transparent. How will the improved information lead to improved consistency in the way in which you deliver your services?

You refer to the final bullet point in paragraph 2.13. The code of standards for service for regulated persons—which is a somewhat cumbersome title, but the best working title that we could adopt—sets out what a person who seeks to apply for a licence from SEPA can expect. They can expect to have a named officer assigned to look after them, to be told the amount of time that the process will take and to be informed of the transparency of the process. Those standards are all aimed squarely at the applicant—"the customer" is perhaps a useful phrase in this case. Although the code should ensure consistency throughout Scotland in those customer-related standards, it is not aimed at achieving the consistency of regulatory and environmental decision making that was behind the formation of an operations directorate. It is not unrelated, but it is not the heart of it.

The rest of paragraph 2.13 describes fairly fully the other aspects of what we have done. The act of forming a single management unit supports consistency of decision making, although it does not bring it about without action, of course. We have harmonised all our procedures so that decisions are made in the same way and all the same factors are taken into consideration. We have introduced quality controls, which ensure that any important document—a licence or notice—is subject to appropriate scrutiny beyond the issuing officer. Above all, we have allocated resources to the generation of simple, clear guidance so that the people who conduct SEPA's business—the front-line staff—have an easy source of reference that is uniform across SEPA, legally correct and reasonable. All the actions are guided by simple guidance.

We also have a support network of officers and a helpdesk, where consistent, single answers are given to the many diverse questions that SEPA is asked every day through its field staff. We have brought together all SEPA's operational functions and all the other functions that are necessary for consistent operational decision making in Scotland-level, cross-functional teams that have responsibility for approving guidance, supporting the networks and generally giving the direction that is necessary to achieve consistency of decision making.

That leads into my next question. Exhibit 12 lists a further nine quality improvement initiatives that are currently in train. Why are those additional initiatives important and when do you expect to see the benefits from them?

Exhibit 12 sets out fully and in detail what I have just touched upon, and all the initiatives are important to that. Officers of SEPA can be remote because they are located across Scotland. It is vital that each officer has a common standard of training and an identical guiding philosophy on how SEPA regulates so that it can protect the environment effectively but at the same time be fair, reasonable and have regard to local conditions. As I said, it is also vital that an officer has immediate access to support by telephone or through guidance. Each of the initiatives is part of that network of guidance, advice and direction.

I am pleased to say that we are already receiving feedback—through surveys of regulated persons and directly—that people are finding SEPA more consistent. Mr Collins remarked on that at the start. Eighty per cent of respondents found us professional in our dealings with them. More than 50 per cent—I cannot recall the exact figure—believed that SEPA was consistent. We know that ourselves from what we see internally.

The restructuring was put into effect only just over a year ago. Although it is early days, the first signs are that we have achieved a great deal. However, the road is endless, and we aim to improve even further.

I want to emphasise Mr Halcrow's point. Rather than saying when we expect to see results, we can see the comparison by considering the experience of the past two or three years. The entire point of restructuring SEPA a year and a half ago was to reduce and—if we are being really optimistic—perhaps even eliminate the disparities that existed in the three regions. The point was to ensure that we had consistent delivery across the community—I mean country. That is my old role coming back. All the same, let us not lack ambition.

The stakeholder feedback during the policy and financial management review—PFMR—process shows that there is quite a high degree of satisfaction with the restructuring. The stakeholders are already seeing the benefit and are anticipating the likely benefit in the next period. I wanted to give a little emphasis to what Mr Halcrow said. I hope that it was helpful.

Ken Collins, the chairman, has made the point that I was going to make, which is that we are in the middle of the periodic policy and financial management review process for SEPA. The quality and consistency of regulation are issues on which the team has been focusing and on which it has taken much evidence from stakeholders, the Confederation of British Industry and other business interests. As Ken Collins said, the feedback has been quite positive, and there is a clear perception that there have been improvements in the recent past, stemming from the reorganisation and from the other measures that are mentioned in the report. Of those measures, ISO 9000 accreditation is especially significant. That is the independent external accreditation of the quality of the process.

We will move on. Rhona Brankin will pursue the issue of delivery.

I am interested in whether the accreditation of the process includes ensuring that the benefits of the additional initiatives are delivered on time.

More than one of us may wish to contribute, but I will start. I have a feeling that the question is in two parts, and I regret to say that I am struggling.

Perhaps I can help you. The second question that I was going to ask was about the improvements in how you monitor your efficiency, which seem to depend on the development of a time-recording system. Is that system fully operational and how is it being used to supply information on relative efficiencies? I suppose that the two questions are linked.

That is helpful, and Mr Ford will start us off.

The system is fully operational and is used by approximately 95 per cent of SEPA staff, so it is representative of the activities that go on in SEPA. It provides SEPA with information for planning its workload and, post-event, allows us to monitor actual activity against the plan. That in turn allows us to compare by office or team the length of time that it takes to deal with an application under the Control of Pollution Act 1974, for example. We can now see whether there are significant disparities compared with the average and examine to see whether best practice is being used. The system provides key information to help us to target the offices or teams that are dealing with applications in the shortest time and disseminate information from them.

The system is 95 per cent operational at the moment.

Yes, 95 per cent of all SEPA staff are on the time-recording system and utilise it.

Presumably, that will rise to 100 per cent.

We exempted some staff from using it, such as our boat crews and cleaners, so I think that the figure will sit at about the 95 per cent mark. That is probably the best that we can hope for.

Would you regard 95 per cent as best practice?

Yes.

Our final questions are about how changes in performance measures will provide a more complete picture of the extent to which SEPA is addressing its regulatory function.

Both exhibit 12 and paragraph 2.15 of the Auditor General's report note that SEPA intends to introduce guidance on the frequency of inspections. That appears to be basic information for a regulatory authority. How did you manage without such guidance until now?

We introduced a risk-based inspection and sampling frequency some two years ago. Before that, there was no guidance in the United Kingdom, except when the Department of the Environment, as it was then, published "Waste Management Paper No. 4" in the early 1990s. It set out the frequency of inspections for sites governed by waste management licences depending on their size and nothing else. It was an attempt by the Department of the Environment to bring regularity to the activities of the local authorities, which were both regulator and disposal authority at that time. With the formation of SEPA and the Environment Agency down south, the dual role disappeared, and both agencies have been working to improve the inspection regime.

The only other matter on which guidance is given at a national level concerns sampling of sewage works covered by the urban waste water treatment directive, for which the frequency of sampling is set by statute. Otherwise, before we introduced a structured risk-assessment procedure, the frequency of site visits was based largely on custom, practice and the individual officer's perception of risk. Although that has served us well over the years, it does not always make best use of resources or ensure beyond doubt that zero-risk sites do not receive excessive numbers of visits or that sites that are at risk of failing to comply receive an adequate frequency of visits.

The Auditor General's report says:

"Managers will be expected to record decisions taken where the frequency of inspection and sampling differs from those recommended in the manual."

You just made reference to the previous structure, wherein individual officers' perceptions were the basis on which decisions were made. As an element of that approach will be left, how will you ensure that your inspectors comply with the new guidance and what management information will you use to ensure that the necessary level of inspection is undertaken? Will there still be the freedom for individual officers' perceptions to form the basis of decision making?

It would be wrong to remove all discretion from field officers. We rely very much on their judgment. We can guide, measure and change the way in which things are done, but, at the end of the day, the field officer must make a judgment.

We are keeping the guidance that we have issued under review, working closely with the Environment Agency. By taking account of the features of a site—whether it stores petrol, its compliance history, how it has been managed and so on—you arrive, through a formula, at a number that will guide you to the number of inspections. We have found that system to be satisfactory. Given that no formulaic system is perfect, there will always have to be an element of judgment.

Field officers' perceptions can be positive and they can be dangerous if they do not move with the times. Will the key to the structure be monitoring and the development of training for field officers?

Absolutely.

Is that in-built?

It is.

It has to be appreciated that, previously, instead of one authority being responsible for the process, a great many local authorities whose practices varied were responsible. I was once a member of a town council that became part of a district council. When the boundaries changed, we discovered that the practices in the old county area were not acceptable to the people who had been in the borough because the standard was considerably lower. That is anecdotal, but the fact is that differences in practice account for some of the points that Mr Quinan is making.

On the developments that are under way, in the past year, the board of SEPA has acquired a human resources committee that has established an HR strategy with a commitment to organisational development and so to training. We will be taking a much more systematic approach to training in the future than we have in the past. That will help considerably.

The third element concerns the need for managers to talk to their staff and listen to them. That can account for a great deal of the otherwise fairly mechanistic measurements that can be acquired in this area.

Will further funds be made available for HR to put in place a proper structure for professional development?

We do not ring fence our funding to SEPA in that way.

Will the HR requirements be taken into account, though?

Yes. Obviously, we consider all the new obligations and duties that the organisation is taking on when we assess the funding. We have significantly increased SEPA's funding for the forthcoming spending review period in recognition of a number of new responsibilities and burdens that it will have to take on.

Although you refer to burdens, it is clear that SEPA is looking not at burdens but at moving forward and developing its staff in a positive way, to allow them to deliver a better service. In its budgetary provision for SEPA, does the Executive fully recognise SEPA's desire to achieve its aims in that regard?

We certainly think that we have taken that into account. You might wish to ask the SEPA representatives whether they think that we have given them enough.

We should do that.

Mr Collins seems tempted beyond belief. I suspect that he would like to contribute.

One always hesitates when one is sitting in front of Executive representatives. They would say that they are fully behind me. We are confident of our ability to convince the Executive of the worth of what we do. We work together. Although our relationship is not without occasional marital tiffs, they add to the general colour of the life that we lead.

I want to follow up on the inspection business. You have described the different mechanisms and procedures that trigger inspections. In recent times, there has been additional pressure on SEPA, in particular, I suspect, from individuals who contact SEPA. In my constituency, there was recently an application for an extension to a landfill site, which has been withdrawn. That triggered a significant amount of individual contact with the organisation. How do you build that into the inspection system?

Individual contact does not necessarily influence the frequency of inspections, which is fixed. Any complaints that we receive are investigated. We visit all sites that are the subject of a complaint, unless they are so familiar to us that they are a class action. We take complaints very seriously—we seek to respond to them within 24 hours and we always seek to find an answer.

Public interest in licence applications is growing fast. SEPA can take account only of those things that statute allows it to take account of. Nevertheless, we attend public meetings and we feed back to people who make representations to us. Although it is true that individual contact is a burden that is increasing rapidly, it is good that the public are interested in such matters. The consultation process does not affect the regulatory process.

I do not want to delve too much into the marital tiffs, but does SEPA think that it has too many targets? Has it been put in a straitjacket of targets that inhibits its ability to do what it thinks is necessary to fulfil its objectives and functions?

That will emerge from the PFMR process, the outcome of which I would not want to anticipate. It is true that any non-departmental public body has a constant dialogue with the Executive about the difference between the Executive's expectations and the NDPB's struggle to achieve its aims. I would be quite worried if there were no tensions between the Executive and SEPA. If there were no tensions, the system would not be working. If the Executive ever found that we were totally compliant and that we agreed with it in every respect, it would realise, in its quieter moments, that life was not nearly as interesting as it ought to be. The Executive would also think that it had appointed to SEPA people who were not worth appointing. I am relaxed about occasional differences. A divorce is not pending.

It is not the case that we sit at Victoria Quay dreaming up targets, which we produce in a puff of white smoke and then hand over to SEPA. SEPA usually provides the initial proposals for the targets, after which we hold a series of discussions about what a sensible set of targets would be. The targets should reflect the Executive's expectations of the organisation. For the first time, the policy priorities paper has set down the key expectations on a single piece of paper. The intention is to give a clear signal to SEPA about the range of targets that we expect to appear in the next round of corporate planning. As Mr Collins said, we are considering the issue as part of the PFMR process, as it is vital to our relationship.

As the witnesses have no final comments, I thank them for their attendance and their contributions. Their work is important for all of us, because it affects the whole country. We wish them well in that work. We will pause for a few minutes to allow for a changeover of witnesses.

Meeting suspended.

On resuming—