A European report has just been released which reviews the available evidence and methodologies for cultural and creative ‘spillovers’ into other areas of society. The researchers, who included the creative industry veteran Tom Fleming, looked for evidence in:
- knowledge spillover (new ideas, innovations etc from the arts and cultural sector which might spillover into the wider economy)
- industry spillover (e.g. productivity and innovation from having dynamic cultural presence)
- network spillover (e.g. impacts on society and economy from the presence of cultural clusters)
This might not surprise you, but Fleming and his team found that there is not enough evidence in all the many research articles out there to demonstrate a causal link between the arts/culture and many claimed spillovers. The research for these areas were persuasive, but fell short of proving causality.
He did find that there were several areas where causality had been established to a scientific standard:
- communications can be boosted within organisations
- culture-led regeneration has a positive impact
- cross-fertilisation occurs between commercial and no=commercial sectors
- investment in design has an impact
- spillovers play a role in boosting uptake of new technology
- networks are important in spreading innovation
We were excited to see Fleming recommend the following to really look at if and how arts/culture can cause social change:
- experimental studies which include control groups
- action research, testing hypotheses through interventions over time
- proxy research approaches
We have our fingers crossed extremely tightly that some funding goes towards this in the academic and cultural space. As we all know, the kind of research to find out if the arts ‘changes the world’ and if so, exactly how and where and when and who, costs the kind of money most arts organisations don’t have.
At the moment what we mostly do in our evaluations is look at the existing research into the determinants of particular social outcomes e.g. wellbeing, inclusion, and see if we can hypothesise a sound theoretical chain of contribution to these determinants by the arts activity. Or not.
This is a perfectly acceptable way of evaluating arts projects (and is used in other areas with similar complexity around causation, such as health promotion). But you do need the big research on which to rely upon when looking for whether your arts project is contributing to the determinants of wellbeing, inclusion, increased productivity, or whatever it might happen to be. So let’s hope academics and institutions can support Fleming’s recommendations and put the question to rest (or perhaps, to work) – how do the arts really change the world?
Yesterday I gave a presentation at the Creative Victoria Expert Arts Panel session on evaluating arts impact. Along with me, Deakin Uni’s Hilary Glow and Anne Kershaw presented about a recent evaluation they did for Vichealth on arts and wellbeing, and Mark Hogan from Regional Development talked about the Clunes Booktown regional transformation story.
Once Creative Victoria upload the full session to the web I will upload a link. In the meantime, I have uploaded my powerpoint presentation here.
This is just a little something I thought I would include here. I have been working on it for a client and thought it might be useful to others too. Nothing top secret, so completely fine to share.
Summary of key types of evaluation
||This is a tool to establish the needs and baseline of the community before a project is designed and commenced. It can include:
- A needs assessment
- Community consultations about needs and priorities
- Benchmarking research, e.g. surveys of the community about characteristics which the project will aim to change (e.g. wellbeing, social inclusion, arts access)
Mission Australia Youth Survey
ABS Community Profiles
The Warwick-Edinburgh Mental Wellbeing Scale
||This looks at how the project was implemented. It can include:
- program fidelity – the extent to which the process was implemented as intended
- feasibility – problems and solutions that were implemented along the way
- reach, equity, intensity of involvement – the level of uptake and participation in the project
UNSW Centre for Social Impact guides
||This looks at the impact the project has had over the short, medium and long-term. In this approach, parties agree on a ‘theory of change’ – what they think will happen as a result of the program. Then they look at whether this occurred, and what unintended outcomes may have arisen. They look at impacts for each major stakeholder group e.g. participants, partners, community groups, donors, and the arts organisation. Some evaluators use a ‘Program Logic’ approach, or a ‘Most Significant Change’ approach, where they try to identify the key change resulting from the project for each stakeholder group.
Methods of measuring impacts will depend on access to participants and what can be feasibly and reliably measured. Methods regularly include a review of participation numbers and data, surveys, interviews, ethnographic observations and focus groups.
UNSW Centre for Social Impact guides
The event is free but I think places might be limited, so for more information and to book, click here.
I will be talking about evaluation methods in the field of arts, culture and social inclusion projects. Hilary Glow and Anne kershaw from Deakin Uni will talk about evaluating the impact of arts and culture on wellbeing, and Mark Hogan from Regional Development Victoria will talk about evaluation methods in social impact for regional centres and hubs.
Please feel free to let your peers and friends know if you think this might be useful for them.
BYP Group would like to congratulate the Creative Industries Innovation Centre, especially former director Lisa Colley, for the release of the e-book, Creative Business in Australia: Learnings from the Creative Industries Innovation Centre 2009-2015 by the University of Technology Sydney.
The book pulls together key lessons from the Centre’s six years of operation, during which time the Centre delivered hundreds of business reviews to creative SMEs and collected a rich trove of data about Australia’s creative industries.
Amongst other useful things, the book includes a summary of BYP Group Principal of Creative Industries, Hung-Yen Yang’s findings about the opportunities, strengths and issues facing Australian creative businesses (see in particular pages 30-31). Yen distilled these issues via a comprehensive analysis of the Centre’s business reviews in nine creative industry sectors.
2-page summary ‘Forensic Reports’ based on Yen’s findings can still be accessed via the UTS website and via the links below. For more information contact Yen: yen at bypgroup dot com or +61 414 462 189.
We will be presenting at a number of events over the coming months. We will attempt to upload any powerpoint slides to this website, but just in case you think can come to any of the below, here are the details:
Jackie will be a panellist on an Expert Arts Panel on evaluation and the arts, Creative Victoria, Melbourne, 28 October 2015
Jackie will be delivering a Keynote Address – ‘Meaningful Measurements – How do we articulate the value of public libraries in a contemporary context?’ SWITCH 2015 – NSW Public Libraries Association, Sydney, 17-20 November 2015
Sarah will be delivering a free regional workshop for Creative Victoria – ‘Evaluating your Work – Measuring Success in Arts and Culture,’ in Kilmore, Victoria, Friday 27 November.
Jackie will be hosting the panel – ‘Evaluation and Research – 6 Minute Learnings’ at the Australia Council for the Arts Arts Learning Forum, Footscray, Melbourne 25-27 November 2015
Key findings from our evaluation of the Red Room Company’s schools poetry education program have now been published as part of the Red Room’s annual report. We interviewed students and teachers and conducted a survey of participants from the 2014 program. The company is using the findings in strategic planning and philanthropic work.
You can read the key findings and more about the Red Room Company here.
If you would like to talk to us about the evaluation or similar work, please contact Jackie jackie at bypgroup dot com or +61 428 576 372.
The Australia Council for the Arts has published fact sheets about research into artists residencies in 2014-15.
We did qualitative interviews and focus group discussions whilst the Australia Council implemented a survey of artists who had participated in residencies.
Fact sheets are available about the motivations and benefits to artists, the ingredients of a successful residency, the location of residencies and the support needed by artists on a residency.
The research is most useful for residency providers, grant makers and artists deciding about whether to take up a residency opportunity.
If you’d like to talk to us about the research, call Jackie on +61 428 576 372 or email jackie at bypgroup dot com
We are happy to share the recently released Australia Council for the Arts report, International Arts Activity – Australian Arts Sector.
The report summarises the research which BYP Group did on international arts activity by Australian artists and arts organisations. This was a pretty massive piece of work, consuming much of our time and brain space for the better part of 8 months in 2014-15.
We interviewed something like 100 artists and arts organisation representatives, international arts brokers, producers and residency providers. We also surveyed the national artist population, receiving close to 500 complete responses (which you stats nerds will be green with envy about ;-).
The Australia Council has released a free summary report, detailed report on survey findings and an accessible version of the summary report.
If you would like to talk to us about the research, contact Jackie: jackie at bypgroup dot com or +61 428 576 372.