Case Study: Impact Evaluation

Challenge: How do we present a fully rounded argument about the value of the Festivals beyond the economic?

Solution: Commission a 360° impact study covering cultural, social, economic and environmental impacts.

Project Outline

There was a shared recognition among the Festivals and their stakeholders that it was important to present a fully rounded argument about the value of the Festivals beyond the economic, following an economic impact study by SQW in 2004/05. It was agreed therefore to commission a study with two main objectives:

• produce an updated impact study for 2010 that looks at economic, social, cultural, media and environmental impacts, and that allows comparison with an earlier impact study produced in 2004/05
• develop a method of ongoing assessment that is owned by the Festivals themselves in the form of a new impact assessment framework and data collection process

After an open tendering process BOP Consulting were commissioned to undertake such an impact assessment.

Project Outline

An evaluation framework for the project was developed by:

• Reviewing existing data sets – to identify existing data collection currently undertaken by the Festivals and partner agencies and assess their suitability to provide the information required for the impact assessment

• Reviewing best practice in evaluation and performance management frameworks – to identify and review approaches and indicators from other existing frameworks

• Undertaking one-to-one consultation with Festivals and stakeholders

Having established the core activities of the Festivals, the scope of the impact assessment study was defined in terms of three aspects:

• organisational remit – in a number of cases the Festival is not the only activity run by the organisation: for the purpose of this study, only activities that directly relate to the Festival operations were included

• timing – most of the Festivals (whether they are run as part of a larger organisational structure or not), run some activity outside the immediate Festival period: for the purpose of this study, only activities that took place during the Festival period were included

• geography – a few of the Festivals programme events and activities that take place during the Festival period but outside of Edinburgh: for the purpose of this study, only activities that took place in Edinburgh itself were included

The next stage in understanding the impact of the Festivals was to think about ‘who the Festivals have an effect on’. This is important in order to know who to include in the research activities. These stakeholders can gain a direct benefit from the Festivals, or also act as ‘proxies’ for assessing the Festivals’ value.

The Evaluation Framework was then developed using a logic model [see above]. This helped to illustrate the ‘pathway of change’ as it maps out how what the funders and organisers input into the Festivals (e.g. financial resources, organisational capacity, expertise) can be logically linked to what is produced (the Festival events and activities), how this determines the actual Festival experience of the various stakeholder groups, and how this is further connected to a series of wider cultural, social, media, environmental and economic changes. Once this overall logic chain was established, the detailed outcome and impact areas were developed - and in order for the evaluation framework to become a usable tool for data collection, a suite of indicators needed to be developed under each of the sub-outcomes.

Following the development and workshop testing of the evaluation framework, the data collection phase began. There were two main data collection sources and processes: Festival-specific primary research and management information. More than 50 separate surveys were conducted across the twelve Festivals, involving in total more than 15,000 respondents. This did not only include audiences, but also sought to understand the impact that the Festivals have on a range of other groups and stakeholders (performers and delegates, attending journalists, volunteers, temporary staff, teachers and Festival sponsors).

Results

According to the independent consultants appointed to undertake the research, BOP, the response rates achieved were extraordinarily high for this kind of research. This was particularly true for audience responses, but response rates were very good across all research cohorts. This meant that, in terms of the response rates, the research findings were highly robust across all the Festivals and strong at the individual Festival level. Amongst the key findings were

Cultural Impacts

• 88% of performers agreed that attending the Festivals offered them the opportunity to see international work that they wouldn’t have seen otherwise
• 79% of performers agreed that the Festivals offered them the opportunity to meet other practitioners
• 90% of audiences agreed that the festivals were ‘must see’ events
• 92% of audiences agreed that the Festivals had given them the opportunity to access work that they are not otherwise able to
• 80% of audiences agreed that the festivals gave them an opportunity to discover new talent, styles or genres
• 64% of audiences agreed that they are more likely to take greater risks in their cultural choices as a result of their Festival-going

Economic Impacts

• Attract audiences of four million, over 25,000 artists and 2000 accredited media
• 82% of visitors from outside Edinburgh and 77% of visitors from outside Scotland said the Festivals made them more likely to visit Edinburgh again in the future
• Generate £261 million of economic impact for Scotland and 5242 full time jobs in Edinburgh
• 93% of rooms across the city during the month of August are occupied, with £41 million spent on accommodation and £37 million in cafés and bars

Social Impacts

• 89% of local audiences agreed that the Festivals increase their pride in Edinburgh as a city
• 89% of audiences agreed that the Festivals promoted an outward looking, positive Scottish national identity
• 75% of audiences agreed that the Festivals had given them the opportunity to spend more time together as a family
• 69% of teachers felt that the Festivals made a difference to the children’s creativity
• 65% of parents agreed that the Festivals had improved their children’s well being

Understanding such impacts, and the role of the festivals across a number of different sectors, was not only of momentary media interest but allowed the festivals to make the case for future investment by partners in specific project areas of mutual interest and demonstrable value.