Homecoming 2009 and the Gathering (Evaluation)
I am pleased to welcome Dr Geoff Riddington, who is the director of Grid Economics (Scotland) Ltd. Members will remember that we commissioned Dr Riddington to carry out an independent evaluation of the evaluation of the homecoming 2009. Dr Riddington will give a presentation on the outcome of that, which will last about 15 minutes, after which we will ask questions.
Dr Geoff Riddington (Grid Economics (Scotland) Ltd)
I ought to start by saying who I am and what right I have to criticise others. I taught at Glasgow Caledonian University for a long time and, like many others, I took a little bit of early retirement but kept on working. I am now a teacher at Glasgow Caledonian University and on the postgraduate courses on economic appraisal at the University of Glasgow. I am also one of the sinners, in that I am a consultant who does economic impact analyses. I have done them for the Government, the Department for Environment, Food and Rural Affairs and a whole host of people. I hope that I know what I am talking about, but I might be found out—you never know.
The remit that I was given was to consider the research on homecoming Scotland and the gathering and to report on the quality and appropriateness of the evidence, the appropriateness of the methods, the rigour with which they were applied and the validity of the conclusions that were drawn. Those of you who have waded through what was actually an edited version of my report will know that some of those conclusions were not terribly valid.
In my 15 minutes, for those who have not managed to get through the paper, I will quickly explain the concepts and methods that were used and then consider how those were applied in the case of HS09 and the gathering and the limitations that I perceive in that. Probably more important, I will then open the discussion on how we should use economic impact analysis.
It is important to recognise that impact analysis is a forecast and forecasts can be wrong. It provides information, but it is not an evaluation methodology. It can be used in two ways. One is to say what we expect as a result of an action; the other is to say what would have happened if we had not taken an action and to try to evaluate what has happened on that basis.
It is important to recognise that impact analysis relates to a specific time period. For example, if we were considering the Trump development in Aberdeen, we would look at a particular figure that related to a specific time in the future—not tomorrow, but in the future. Often, the figures are never reached. For example, the economic impact analysis for the Loch Lomond Shores development relates to some mythical time in the future when there are two hotels and 5,000 shops and so on. The analysis relates to a point in time. Normally, the proposer chooses the maximum, so it is not particularly useful information.
However, it is extremely important to recognise that impact analysis does provide useful information—on jobs, value added and regional effects. The method is part of the evaluation process, if it is post-evaluation, or part of the decision-making process, but it is not the one thing on which decisions should be based.
The procedure is fairly straightforward. One tries to work out the direct change—ex ante or ex post—then sits down and tries to say what would have happened if we had not done the activity. That is very important in the case of something such as homecoming Scotland. We take the total change that we think was the result of the activity and see what effect it had in the wider economy. We normally use input output analysis or multipliers to do that. That estimates indirect effects, which are effects on companies that supply directly, and induced effects, which relate to the wages that go through the system.
A full model can forecast employment. It can also forecast gender distribution and full-time or part-time structures, but that is not normally done, because most models are not so sophisticated. The value added in each region can also be considered. If an event takes place in Edinburgh, the model can say, “As a result of that thing in Edinburgh, we expect so much more in Falkirk.” The ripple effects from an event such as the gathering can be traced. That is an important feature, but it is not normally used, which is disturbing.
In the presentation of such analyses, there is tremendous confusion about what we call output. Output is the aggregation of all the sales so, if we are not careful, it double-counts. It depends on the length of the supply chain. Outsourcing increases the output, without anything being done.
Output is always rotten to use, but everybody uses it. In impact analysis, everybody tries to show that output is wonderfully big—it has become a hype thing. I worry that output has become so associated with hype and with selling projects or institutions that it has become degraded, although it need not be. Output is a starting point. We are not talking about output as we understand it in terms of the national economy. If we consider the economy’s output, we want to examine the gross value added or the income, but the output that we are discussing is not that.
I met people at EKOS, which conducted the analysis of homecoming Scotland. I must admit that I was impressed—I had confidence in the guy who was responsible. EKOS did two tasks. It looked at the events, the people who went to them and the people who said that they would not have been there except for homecoming, and it allowed for multiple attendances. Those figures are very reliable. The problem is that homecoming Scotland was a series of events and a marketing campaign. EKOS found that a number of people who said that they were attracted by homecoming Scotland did not go to events, so EKOS had to try to estimate those figures, which is extremely difficult. EKOS’s way of doing that was to hook itself to an internet survey that VisitScotland undertook.
VisitScotland has a huge database of inquiries. Every year, it finds out through an internet survey how many inquiries were converted into people coming to Scotland, which gives it an idea of how many people from the database came here. Of course, that does not tell VisitScotland how many from the database did not come to Scotland, which is quite important. The survey is also probably biased—the people who are likely to reply to the internet questionnaire are those who came here. I guess that the survey is biased towards a high conversion rate, so I would not trust the rate. Those are the two reasons why I find the figures a bit problematic.
EKOS looked at the proportion from the survey who did not go to an event, which gave it some idea of how many people who came as a result of homecoming Scotland did not go to an event. The number who were not on the database and who did not attend events is not covered and might be quite large—we do not really know. On the other hand, I think that the survey was biased towards people who came here for homecoming Scotland. The area is problematic.
10:45
I tried to check that using the tourist statistics, but anyone who knows tourist statistics knows that they are terribly difficult to work with because they are not particularly accurate. I had a bit of a debate with Scherie Nicol about how reliable I thought they were, but I am fairly sure that the report’s figure of 72,000 extra tourists is about right. That is just a feeling; I would not swear to it, but I think that there is a good probability that the number is of that order. It is certainly not more than 100,000, and I do not think that it is less than 50,000. A figure of 72,000 is about right.
Estimates of tourist expenditure per head are fairly standard, and the figures that EKOS obtained are fairly standard. The additional gross expenditure figure is pretty good, but then we encounter a problem, which arises all the way through. If someone came to Scotland and said that their main reason for coming here was to go to the gathering, should we allocate everything that they spent on their trip to Scotland to the homecoming Scotland factor or should we just do an events-based allocation?
The report allocates all such people’s spending to homecoming Scotland—in other words, it adopts the whole-trip approach. That is fine—it makes a lot of sense—but the problem is that the economic impact studies that other people have done, such as the wildlife trips study that was done last year, have been done on an activity-day basis, whereby if people came and took a wildlife trip, the economic impact is said to be all those people. However, some of those people will have said that they were here because of homecoming Scotland, so we have a conflict. I tried to figure out how to resolve it, and in the end I decided that we should combine both approaches—we should take the whole trip and break it up into the various activity days, which is quite possible to do.
The other advantage of the activity-day approach is that if someone came to Edinburgh for an event but also spent 10 days outside Edinburgh, we can say that they spent such-and-such an amount of time on the Clyde coast, for the sake of argument, whereas with the whole-trip approach everything would be attributed to Edinburgh. To me, that information is valuable. We want to know what the impact was in the regions, because that is extremely important as far as employment is concerned, which is primarily what we are concerned with.
Incidentally, the local figure that is given for Edinburgh in the gathering report is, as far as I can see, completely bogus. The fact that whole trips have been allocated to Edinburgh, when a lot of that time may well have been spent elsewhere, is totally misleading.
I have no argument with EKOS on any of those issues. EKOS has an agreed methodology, whereby it does X, Y and Z and then uses the Scottish tourism multipliers. It takes the total expenditure and multiplies it by the multipliers to get “the output”, then it says that the output to cost ratio is X. The problem, basically, is that the figures are wrong. I know that EKOS did not realise that they were wrong. That is a consistent feature; the Scottish tourism multipliers have been used for years—they were first published 18 years ago, since when things have changed quite dramatically.
Moreover, the initial work on the Scottish tourism multipliers was wrong, so we have a problem. That the multipliers are wrong is about as much as one can say. They are out of date, they do not treat retail correctly and they deal with VAT inappropriately. Basically, they include VAT, so if VAT increases, the output of the economy apparently increases, too, which is clearly not true. We have a big problem there. The effect of that is to overestimate output and employment by a significant amount, so the figures that we have got are not right.
There was a bit of a dispute about the cost. VisitScotland and EventScotland said, “Okay, we were given £5.5 million. That was our budget and this is how much we spent,” but another £3 million was spent on homecoming Scotland in recurrent budgets. The question is, should that amount also have been included? After a bit of a debate, I decided that I was sure that it should have been. The £3 million was diverted from promoting walking or sailing, for example, to promoting homecoming Scotland. If walking or sailing is not promoted, the expected effect would be that fewer people go walking or sailing. If that was not the effect, there would be no point in doing the promotions. One problem is that the tourism statistics give a net result, whereas EKOS looked at additional numbers. In the end, as I have said, the figure should have been £8.5 million.
The assessment of the gathering went the other way. It included various grants in the costs for developments that related to people who were not additional visitors. The costs for the gathering were overestimated. EKOS used a ratio of output to cost, which it calls return on investment. I see that done all over the place, and I hate it, because it tries to associate some sort of measure of output and cost with the rate of return on investment, which is the investment analyst’s way of looking at how well an investment does. It is a cost, not an investment; it is an output—accumulated sales—not profit or return in any sense. I hate it; it should not be used. If you are going to have fixed multipliers, you might as well use some sort of gross expenditure to cost criterion, although I do not think that you should do that. If you are interested in measuring employment, the cost per job is a better measure to use. The development agencies have been using it for a long time. Doing that allows comparisons to be made. The return on investment measure is totally inappropriate.
The location and quality of jobs are critical. That measure should have been used, but it was not.
The other point is that if we are interested in how many visitors we generated and how much that cost, we should calculate that. Doing that, we see that we spent £118 per visitor and on average we got back £438 per additional visitor. That may be good or bad—I do not know—but that is what the measure says, whereas the return on investment says nothing at all. I could say, “It is four,” but that means nothing.
I also do not like the single period aspect. If we are talking about investment, we are looking at what occurs over time. Impacts occur over multiple time periods. For example, one wonders to what extent visits were brought forward to 2009 from 2010. On the other hand, if a lot of advertising was done in the States, it might have enhanced visits in 2010. We do not know. However it is done, it is important to look at the time dimension.
One obvious question is whether homecoming Scotland should be repeated in 2014. The economic impact analysis says that it was quite a cheap way of generating jobs, if that is what you want. However, it is critical to say that the jobs were temporary, were not of high quality and might have been located in areas of full employment. There is no point in the Government putting money into tourism in Edinburgh. That is not a very sensible way of spending money. That said, there are lots of reasons why we should promote Scotland’s heritage. There are a lot of internal reasons why the event should occur. We should not be saying that jobs are the only thing that we are interested in. That is the only thing that an economic impact analysis is interested in, but it is not the only thing that we or any decision maker should be interested in. We need an extended framework.
As an old quantitative economist, I always return to cost benefit analysis. A lot of work is going on in areas such as the arts and culture to try to get an idea of benefit in monetary terms. That forces decision makers to value their objectives, which is a very good thing anyway, but I can see the difficulties and the way that people react when you try to put a value on energising the diaspora, which was one of the objectives of homecoming Scotland. I can see why there are problems there.
I feel that as a staging post, we should be moving to a situation in which we have some sort of standard appraisal guidelines. Those should include economic impact and job creation, but those should firmly be seen as only part of any appraisal.
The other issue that worries me is who does the evaluation. I was worried when I talked to EventScotland, because it was fairly obvious that it was being very defensive. It had commissioned the reviews, and it was being defensive because it was its responsibility to ensure that HS09 worked. In principle, I do not think that the paymasters should commission the evaluation. I know that the English Government is going completely the other way—it wants to privatise the Audit Commission and get organisations to do their own evaluation—but I do not like that. Evaluation should be done by a third body. Obviously, Audit Scotland would be the ideal body. Perhaps there are too many evaluations to do, but something like it should be the third body.
The message—which Jim Dewar said I should tell you—is that EKOS was competent and professional in its assessment, and I do not want to criticise it. The output model that was used to gross up was basically flawed. In any case, whatever we say about these things, the appropriate measure was not necessarily used. We should have a better evaluation methodology that incorporates economic impact assessment.
I hope that that was not more than 15 minutes—my apologies if it was.
Thank you very much for that summary. I have read your report, and your summary has helped to clarify some of the issues that I was slightly unclear about, given my rather distant half a degree in economics.
To summarise your summary, would it be fair to say that EKOS did a good job in its analysis of the overall impact of HS09 using what is currently the standard methodology, and that its assessment is therefore useful for comparing with similar events, but that the current standard methodology is not very useful in giving us meaningful information?
Yes, that is fair comment. EKOS used the same multipliers. Given that, it does not really matter that much if they are wrong, because what we are really interested in is output versus cost.
Is it your view that one outcome from the exercise is that the Scottish Government should consider a new model for evaluating tourism-related projects and perhaps other matters, such as the development agencies, which you mentioned? We are looking at the enterprise network as part of our work, and one issue is how we judge the development agencies’ effectiveness in using public money. Is it time to consider how we evaluate those agencies and the economic benefits that they provide?
The museum and art gallery sector has just started on exactly that exercise. EKOS used the agreed standard methodology; it just happens that the figures that it used were not right. However, the approach is too narrow anyway. Museums are in the same situation. A museum or art gallery does not exist solely to generate employment; it exists for lots of things. Knowing that it generates employment is useful and important information, but that is not its purpose. One could say exactly the same for homecoming Scotland or the gathering. They did not exist to generate employment; they existed for lots of reasons, and our evaluation mechanisms should look at all of them.
The experience is most interesting because, as we investigate the tourism industry, we know from the inquiry that we carried out previously that there are concerns about how to measure the success or otherwise of projects, and that that is not established. If the 1993 model is to be brought up to date, your suggestion about activity days and so on seems an interesting development. Would you say that, as has been suggested to us, with bodies in the tourism area, large-scale events tend to be improved upon for the future, after the event, rather than during the time when they are being run?
11:00
I am not sure that I understand the question.
The gathering and the homecoming were assessed by EventScotland and others as they went along. Were lessons learned during that process or, as is more likely, after it?
I think that it is both. Obviously, one hopes to learn after the process, but one problem is that, because of the way in which the process is set up, the evaluation is almost a judgment of what has been done. Naturally, agencies want to say, “We’ve done a marvellous job,” so it is not as if they learn anything from the evaluation; too often, all that the evaluation does is defend what they have done.
For example, with homecoming Scotland, and particularly the gathering, an awful lot of the big events were focused here in Edinburgh. Was that appropriate? Is the reason why a regional analysis was not done that it would have shown that the impact in areas where one would have hoped for a big impact—the sparsely populated areas, for example—was absolutely minimal? As the event was concerned with clan gatherings and what have you, one would expect a big impact in Skye or the north-west Highlands. However, I guess that it had very little impact there, and if a regional analysis had been done it would have shown that, which, I suppose, you would have been a little upset and annoyed about. However, that work should be done, because it is important information.
The gathering of that information is interesting. As a member who represents the Highlands and Islands, I must say that the perception was that the homecoming did improve business in that year. If we want to find out how, we must in future find a method to do that.
The method exists. It would not be difficult to do that next time the event happens, assuming that it happens again. It is a question of trying to find out what visitors did when they were not in Edinburgh looking at clans walking down the street.
I move on to the investment of £5.5 million that the Scottish Government made through VisitScotland. You want to include in the analysis the £3.5 million from VisitBritain and VisitScotland. However, they would have spent that anyway, whereas the £5.5 million was new money. Any calculation of the success of homecoming must be measured against the new money and not against recurring funds. No matter how we explain the spending of VisitBritain and VisitScotland to encourage people to come to Scotland—with the homecoming as a theme—we are not clear, from what you previously said, what activities they were involved in. Therefore, it is surely not correct, in criticising the success and the methodology that was used to interpret that, to aggregate the £5.5 million with the £3.5 million to get an overall picture. In fact, the evaluation should have been, and was, made on the basis of the new money.
We asked what effect the action had. If we take £3 million from one source and say, “We’re not going to do that; we’re going to do this,” we have to say what happened as a result of not including that. Let us assume that that £3 million would have been spent on promoting outdoor activities in Scotland in Europe. There is undoubtedly a loss if such promotion does not happen, or we would not bother to do it—I do not know the size of the loss. If we do not include the £3 million with the £5.5 million, we have to exclude its effects. What we see in the tourism statistics is the net effect, which is why it is lower than the effect that EKOS looked at. EKOS looked at just one thing; if it was going to look only at new money, it should have taken off the effects of the diversion. Does that make sense?
I understand why measuring the effects is complex. At this stage, when people are raking over the coals, it is helpful to have your view that the issue is complex and to have it recognised that from before devolution until now, one Government after another has been happy to accept the approach to measurement that we have been discussing. Should we say to the Government that it is time for a change in how we measure the success of events such as homecoming Scotland?
Yes, absolutely, if you are talking about return on investment, which is a horrible phrase and does not mean anything—it is not investment, although I suppose that it is cost.
When we were following up our tourism inquiry it was suggested to us that the evaluation of homecoming did not start quickly enough. We were told that to have effective evaluation we really need to evaluate 2008, 2009 and 2010.
That is a fair comment. We certainly need to go back and consider what happened in 2010. Of course, the figures are not out yet, but a significant fall in the figures from North America and Australasia would tell us something about what has gone on. We need that information.
In many respects your critique is devastating, although some of your comments on your criticisms have been very polite. I want to understand the balance between areas in relation to which you think that information has been “bogus” or “misleading”, as you said in the report, and areas in relation to which you are satisfied with the methodology that was used.
When I read the report, two things jumped out at me. First, at the end of your report you expressed concern that the organisations that had spent the money sponsored the EKOS assessment. Secondly, if I recall this correctly, you said that the decision to commission EKOS to evaluate homecoming followed the submission of EKOS’s evaluation of the gathering, which was extremely positive. I think that you described the assignation of benefit to Edinburgh from the spend on the gathering as “completely bogus”.
I did say that.
You did. I am keen to understand that. It is clear that the gathering generated some benefit for somebody. The businesses that were left out of pocket were principally Edinburgh businesses. One of the defences for its approach that the Scottish Government mounted was to say, “Ah, but look at all the benefits that came to Edinburgh.” On the basis of your understanding of the EKOS assessment and your critique of its methods, can you offer an alternative estimate of the benefit in the local area that might have come from spending on the gathering?
I would not want to do so. It all depends on how many days the people who went to the gathering spent outside Edinburgh. The key people who generated the money were the people who were on two-week packages, who were spending an awful lot of money. It is unlikely that all those people spent the whole of their two weeks in Edinburgh. We know that an awful lot of people do that, but we do not know how many. That is the problem. People tell me that VisitScotland has done surveys that tell it how much goes out, but I do not have that information. Could I offer an alternative estimate? I do not know. No—I cannot really. The truth is that I have not done that, but it could be done relatively easily.
I think that you said that, when EKOS used multipliers that were ill founded, it was not aware that it was doing so. Did EKOS understand that ascribing the whole benefit of someone’s trip to Scotland to the Edinburgh economy might be, as you say in your report,
“potentially extremely misleading and should not be undertaken”?
No. EKOS applied a standard routine with a local multiplier for a city such as Edinburgh, which produced a figure. So, the answer is no—it probably did not understand that. I think that the analysis was undertaken as a matter of routine without the consequences being thought through. EKOS used a standard methodology that has been developed, in part, for things such as bringing the rugby league cup final to Murrayfield, whereby people come to Edinburgh, spend two days here and then go away again. Its use is perfectly legitimate in that context; however, the situation is very different when somebody comes from Australia for two weeks for an event such as the gathering. They say that that is the reason why they are here, but we know that they spend a lot of time in the Highlands and Islands, tracing their roots or what have you.
Also in relation to the assessment of the gathering, you were concerned about the way in which the Scottish Government loans that were written off were not ascribed at all as a cost of the gathering. Why did EKOS decide that those loans were “irrelevant”, as it described them, to the investment in that event?
I go back to my exchange with Rob Gibson. EKOS was looking at the impact of the investment that VisitScotland and EventScotland made; it was not interested in the impact of public funding as a whole. The simple answer is that EKOS was not concerned with the cost; it was concerned with what it termed the investment and the additional money. Personally, I do not think that that is appropriate. We should try to get information about how much those extra jobs cost.
So, in your interpretation, the fault lies not with EKOS but with the remit that it was given. From your description, it sounds as though EKOS was given a very narrow remit that was concerned only with one or two organisations’ budget outgoings, rather than with the total cost to the public purse.
In part, it was about applying a standard method to a non-standard activity. There are great advantages in having a standardised approach to everything, but sometimes that does not work at all. This example is at the edge of where it works. I have seen the same thing in other situations. For example, I have looked at the results of a cost benefit analysis of railway signalling and wondered how the people doing the analysis could conceivably ignore the potential for a massive crash—a disaster. They told me that that did not fit in with their standard routine and that they made no provision for such things. It was absurd and there is an element of that in this case. Applying a standard routine for good, comparative purposes was not adequate in this particular case.
Let us move on to the homecoming evaluation. EKOS’s project of evaluating the homecoming followed the submission of its report on the gathering, but you found what seem to be some enormous discrepancies. The one that jumped out at me is your conclusion, on page 21 of your report, that
“EKOS overestimates the number of jobs created or supported through HS09 by around 75%.”
That is an enormous gap, not a marginal methodological quibble. You are saying that, fundamentally, EKOS’s conclusion on jobs was way off the mark. Can you explain why you came to that view?
11:15
There were two reasons. First, the multiplier was far too big. The multiplier that was used would have been okay if VAT had been stripped out right at the beginning. I looked at the original documentation for the Scottish tourism multiplier study, which said that VAT should be included; basically, the original manual was wrong. The multiplier should have been something like 1.35, instead of 1.7, so there was a very substantial overestimate.
Secondly, the figure for the output that is required to generate a job—effectively, the cost of each job—was really wrong. That is predominantly down to the changes that have occurred over the years. The figure was updated for inflation but not for changes in structure—such as the fact that tourism jobs have become much more sophisticated and people are paid much more in 2010 than they were in 1992. The situation is fundamentally different, which produces an enormous difference in the figures.
One of your less critical conclusions was that the EKOS study was broadly accurate on additional visitor numbers and spending per visitor. Did you assess its methodology and decide that there was plenty of evidence that it was right, or did you simply conclude that there was no particular evidence that it was wrong?
I did some work on the national statistics. Initially, I applied standard econometric models and reached the conclusion that there was no statistical improvement; I could not prove one. However, if you use a 5 per cent confidence interval, you come up with one answer, whereas if you use a 30 per cent confidence interval, you come up with different one. I found weak statistical evidence that there had been an increase. I could find evidence in the changes from year to year, but those are horrible figures to use. As Scherie Nicol pointed out, we can say that growth in Scotland was 10 per cent greater than growth in England, but if we compare Scotland with Wales, we find that Wales is doing even better. Perhaps England was doing particularly badly, rather than Scotland doing particularly well. The evidence is not convincing, but I was fairly convinced that the 10 per cent figure was real. The numbers for Wales are so small that you have to be a bit hesitant about using them. The independent statistical evidence that I had pointed one way—in the same direction as EKOS’s findings, if that makes sense.
Right. However, you drew that conclusion on the basis that it was a reasonable assumption, rather than on the basis that you were able to demonstrate that it was accurate.
That is fair. There is a statistical basis, in the national statistics, for saying that growth occurred, but that evidence is weak.
I return to the question of jobs created and employment. Lewis Macdonald mentioned the overestimate of about 75 per cent for homecoming in general. EKOS said that the figure for the gathering was 288 full-time equivalent jobs; you suggest that it was 115. My maths is not great, but that is a disparity of substantially more than 75 per cent. The reasons that you have given for the disparity in the figures for homecoming relate to the treatment of VAT and the output that is required to generate a job. Why is there an even bigger disparity in the figures for the gathering?
Goodness, that is a horrid question. I would have to look at that. To be honest, I do not know the answer. One takes the numbers and multiplies them by various things. I do not know. The EKOS study deals with different groups of people with different total expenditures per head and so on, so that might well be the explanation. However, I would need to trace it through. I cannot answer the question now.
Will you submit a brief explanation to us in writing?
Of course.
It would be an explanation of why there is such a big difference. You explained the 75 per cent figure, but it would be interesting to me as a local member for Edinburgh to know why there is a bigger difference for the gathering.
You explained that the difference in employment or jobs created results relates to the multiplier analysis and an out-of-date approach. Do experts and economists simply disagree with one another on the approach that ought to be taken or do you think that if we had a panel of experts from the same field before us today they would all agree with your analysis? Would EKOS hold up its hands and say, “No, actually, we applied the agreed methodology, but the approach Dr Riddington is suggesting is the one that we probably ought to apply going forward”? Or do you think that it would dispute how the approach should be applied? I know that you cannot answer for EKOS, of course.
EKOS will say that the Scottish multiplier approach that it used was out of date and should be changed and that it should adopt a new approach. Arguments about whether HS09 should have cost £5.5 million or £8.5 million will rumble on. I hope that EKOS will be convinced by my suggested approach, but I am not necessarily sure that it will.
I am fairly confident that an independent analyst would agree with what I did—I can see no reason why they would not. I did not set out to rubbish anything; it just came. That is it, really. The problem is that there is a tiny world of people who understand the techniques of national accounting and the way that it applies to local input-output tables and so on. I happen to be closely associated with one of those people—I would not know any of this stuff if I had not made all the mistakes that EKOS has made. Eight or 10 years ago I would have done exactly the same sort of thing. It is still being done.
We are looking at some museums and galleries work at the moment and exactly the same sort of thing occurs. People are saying, “Here’s a multiplier. Let’s multiply by 1.7 or 1.63,” without understanding where the multiplier came from. Even if they understand the basic matrix methods that lead to a multiplier, which I think not many do, they do not understand the accounting system that underpins that. It is a very specialist area and people do not want to know about it. They want to give an estimate of how many jobs have resulted, so they start off with a certain number of jobs, multiply it by a certain number and then get X jobs. That is what they want. It is rare that you get a chance to come forward and say, “Hang on a bit, you’re doing it all wrong.” I suppose that I am in that lucky—or unlucky—position. I am sure that I am not particularly popular with some people, anyway. The analysis needs redoing: that is all there is to it and I think that EKOS would accept that.
I have a final question. You also talked about the economic impact on Edinburgh of the gathering and suggested that the figures were not correct because the whole-trip cost had been applied to the Edinburgh economy as opposed to the wider Scottish economy. Do you have any sense of how big the margin of error is in that case?
I think that Lewis Macdonald asked me exactly the same question. I do not know, but it is large. We know that the people who generated the impact were those who were on the long trips, which are unlikely to have been wholly in Edinburgh. Perhaps it is an overestimate of a third, or a double estimate. I do not know. I would think that the impact was something like a half of what EKOS said, but that is a guess. In the write-up, EKOS mentions the general Edinburgh area, but I think that it is a little bit uncomfortable about doing that, for good reason.
On page 11 of your report, you highlight that the homecoming might have generated income for England as well as for Scotland because people might have visited England as well. On page 12, you go on to highlight what I suggest is a positive point for the tourism sector as a whole in Scotland:
“On this evidence it is difficult to argue that the Scottish tourist industry was dependent upon the success of Homecoming Scotland.”
With those two points in mind, would you say that homecoming Scotland was a positive thing for Scotland in 2009, or would you say that it was a negative thing?
I am confident that we had an increase in tourism as a result of homecoming Scotland. As I said, there was always the problem that there might be a significant fall in 2010 as a result of things being brought forward into 2009, but it might be too early to say. The question is whether we got value for money, is it not, rather than whether we got any value? We certainly got extra people, but whether that was value for money, I do not know.
It could be argued that there are major challenges in trying to promote a year-long event. In fact, I do not think that anyone could suggest otherwise, but the organisers were doing something that was really quite unique. It has never happened before. For those reasons, we might not get it perfect first time. Any lessons from 2009 should certainly be learned for 2014.
If we take the return on investment thing, I do not know why they were saying, “If we go ahead and say the return is 4.2 and not 8.5, that will be terrible.” I am not sure why you would say that, because there are so many other good reasons for doing something like homecoming Scotland, not least that it helps us to learn how to do these things. The point that I was trying to make is that economic impact per se is limited in determining what we should do. We should not say that, because we will not get a return of 8:1 or whatever, we should not do something. That would be silly. That is the message that I am trying to get across. That is not the way to evaluate things. It is only a partial measure. One always worries that we judge based only on things that we can measure, and that is what we have done in this case, in effect. I do not think that that is sensible.
That point illustrates what you state on page 24 of the report:
“there were significant other objectives to the campaign most of which seem to have been substantially met.”
It is not just about the money.
Absolutely.
The bigger picture is the whole concept.
Yes—it is more than that. The EKOS report relates particularly to overseas visitors, but benefits sprung from the event for ordinary citizens, such as people in Edinburgh who got something out of it. It is not just about jobs. People got the chance to see the clans walking up the street, the Highland games and what have you. There are other significant benefits, and we are silly if we do not take those into account.
That being the case, would you recommend a second homecoming in 2014? Would you go to any of the events?
Thank goodness I do not have to recommend it. I think that I would, to be honest, but I would predicate that on going back and thinking about whether and how we could improve regional dispersion.
There is no point in repeating things. I am not sure whether I would repeat the gathering, for example, but I certainly would not repeat it in Edinburgh, because that is not a sensible way of spending money. I realise the accommodation difficulties that would arise if it went to anywhere except Glasgow and I recognise Edinburgh’s significance as a location, but I would want to rethink the whole event because it is not particularly clever to put yet another event in Edinburgh in the middle of summer.
I recommend a linked theme, but I am not sure that I would include golf. That seems to me to be pretty spurious to the objectives. The event would be about history, genealogy and culture; golf does not fit in any of those categories.
It would be well worth considering another homecoming. Castles and kilts are parts of tourism.
I suggest that, if there is going to be a new gathering event, it could be held somewhere such as Greenock. There, we have a historic custom house, from which hundreds of thousands of Scots left to go the new world.
That is the sort of idea that we should pursue. I like the idea of focusing on the expatriates.
Thank you very much for your presentation and for answering our questions, Dr Riddington. It is worth emphasising that you were not asked to redo the research but simply to evaluate how it had been conducted. Your report highlighted some interesting issues with how we evaluate the value of investment in tourism.
Does the committee agree that it would be useful for me to write to the Cabinet Secretary for Finance and Sustainable Growth and the Auditor General to ask whether they have any comments on the points that Dr Riddington’s report raised?
I certainly agree with that. It would also be helpful to draw ministers’ attention to the weaknesses in the EKOS assessments that were clarified in today’s evidence. There are lessons for future assessments, but the assessments have been fairly comprehensively discredited in some respects by the things that Dr Riddington had to say.
Christopher Harvie (Mid Scotland and Fife) (SNP)
Has any assessment been made of the impact that world financial collapses have had on Scottish tourism? Mervyn King said to us that 2008 was the worst setback that the economy had experienced since 1914. I do not expect Dr Riddington to go back to 1914, but I expect some—
Chris, you will have an opportunity to ask that of the tourism officials who are coming for the next item.
On that note, we will suspend briefly while we change witnesses.
11:32
Meeting suspended.
11:35
On resuming—