Validated instruments to use when assessing the impact of the arts

evaluate-imageHi guys,

There was a lot of interest in my blogette about resources for measuring cultural value, so I thought I might share some other tools which I find quite helpful in BYP Group’s work around assessing and measuring the impact of the arts.

I have compiled a table of useful and even better, VALIDATED instruments which might be used when assessing the impact of the arts on personal capacity development e.g. self-esteem, self-efficacy, self-concept, emotional regulation, theory of mind, creativity etc.

Validated-Arts-Research-Instruments-Table

Enjoy! And of course email or comment on this blog if you have any more to add to the list or have any questions.

Jackie

BYP Group does not take responsibility for the accuracy of the information contained in external links.

Useful resources about quantifying cultural value

pmiy

Street Pianos (c) Luke Jarman

A quick scratch pad of useful reports to do with quantifying the value of arts and culture (and also a bit about alternative financing for a bit of light reading).

 

Understanding the alternative finance market (NESTA 2016)

Quantifying the social impact of culture and sports (UK 2014)

Quantifying and valuing the wellbeing impacts of culture and sport (UK 2015)

The 2015 report of the Warwick Commission on the Future of Cultural Value (UK)

Validating the links between arts and liveability (US)

 

The Matrix for Making Decisions

question-markA friend of mine contacted me today, saying she wanted to do ‘the Bailey matrix’ to help her decide whether to start her own business.

The matrix is something that, back when we were graduates in Canberra almost 20 years ago, I would draw up for friends in the hope of it helping them decide what they wanted to do next with their lives. Using a spreadsheet.

Another friend, Andrew Dempster, used a similar matrix which we all called the Dempster matrix (Hi Andrew!). I can’t remember now if it was Andrew or I who innovated the weighting approach – where you weight different options according to their fulfilment of various important goals. It doesn’t really matter who did it – but it was a turning point for the utility of the matrix, allowing you to give a higher score to things which mattered the most to you. Effectively, it introduced the variable of subjective value into a spreadsheet.

All jokes aside, the matrix is actually a very useful tool for making decisions. Not so much that you use the scores religiously to determine what job you might take, but that the process itself helps you to clarify whats important to you, and how various options will help you to achieve those things.

I actually use a variant of the matrix these days with clients when trying to establish if their program budgets are going towards the right areas, based on their overall goals or mission. It is slightly more complex – it includes things like duration of engagement, level of subsidy, and stakeholder target groups – but the principle is the same. It is a way of measuring your actions (real or proposed) against your values. Yes, using a spreadsheet.

So here it is in all of its glory – the Matrix of Making Decisions. Use it as a guide only :-).

I am happy to share the Matrix under a Creative Commons License – Share Alike/ Non-Commercial. This just means, please don’t go out and use it to make your millions (or if you do, please tell me how you managed it. I’ve only ever gotten the occasional beer or block of Dairy Milk from it ;-).

Jackie Bailey

Principal, BYP Group

 

 

Validated instruments for use in arts impact research

surveyI was putting together a table of various validated instruments which have been or could be used in arts impact research, and thought it might be useful for others working in this area.

The table is a list of instruments which can be used to measure empathy, self-esteem and self-efficacy, wellbeing and so on. I have also included links to the instruments if they are available for free or purchase online.

You can access the table in html or pdf.

Please feel free to comment – the list is not exhaustive and doubtless there are some in there which have more or less efficacy than others!

Happy measuring :-)

Jackie Bailey, Principal, BYP Group

Can we measure the value of enduring artworks?

Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched

Mona Lisa by Leonardo da Vinci – how much has this enduring work contributed to global culture over the years?

(This post is a continuation of my ruminations on Ann Markusen’s work)

There is something else I saw in Ann Markusen’s report on California’s arts and cultural ecology which I found really interesting.

I saw these words:

‘California’s arts and cultural nonprofits generate new and enduring artworks—they commission an estimated 41,000 theater, dance, musical compositions, and artworks annually (p.33)’

and had one of those moments of nerdy, impact evaluation realisation (you know the ones ;-).

I have never tried to capture the value of the enduring nature of works of art. Sure, we all talk about heritage value, bequest value etc (thanks to John Holden). But normally when I do impact or value assessments for arts clients, I am very focused on the intrinsic experience of the art and the social, educational and personal outcomes for the participants.

I have always included ‘contribution to society and culture’ as an element of Artistic Vibrancy (a framework for measuring the health and impact of an arts organisation). But I never thought of measuring the contribution to culture through the creation and continued public experience and enjoyment of enduring works.

How can you value the contribution of Shakespeare, for example? Well, it’s invaluable – as you can can always say with the arts. But you can try to translate the value of enduring works into a language that funders can work with (‘Invaluable’ does not cut it in the Cabinet room).

For example, could you start to count the number of works likely to endure over say, 2, 3, 5 years and longer? You could probably come up with a statistical formula based on big data (PhD project anyone?!) And then how many people are likely to see it, engage with it, perform it, derive some meaning from it, and be transformed by it (a kind of pyramid effect)?

I think this would be completely awesome or terribly hard to believe ,depending on how carefully it is done. There is nothing worse for arguments about arts impact than seemingly ambit claims of creating $X in value. But when it’s done well, it’s – well, invaluable.

I’d love to hear other evaluators’ thoughts on this. Do you think it could be done and done well? Or perhaps it has been done?

Jackie Bailey – Principal, BYP Group

Appreciative Inquiry – change and evaluation rolled into one

We are often asked about the different types of evaluation approaches.  We will try to summarise them from time to time here, on our site, and hope this helps people sort through the jargon of evaluation and get to the bottom of what suits you and your needs best.

We have started with an explanation of Appreciative Inquiry, because I was asked about it at a presentation in Melbourne to arts organisations.

What is an Appreciative Inquiry Evaluation?[1]

An Appreciative Inquiry evaluation is based on the principles of Appreciate Inquiry (AI), an organizational development method which is focused on building on what an organization does well.  The principles of AI are summarized below.

In an AI evaluation, evaluators and participants work together to share their views of the present, and “co-create” the future. An AI evaluation does not ignore problems, but approaches them as opportunities for change.

In an AI evaluation, the evaluator and participants:

  • become fully engaged in the learning journey
  • work to “co-create” the future
  • acknowledge that there are multiple, equally valid interpretations of reality
  • share their individual interpretations of reality, with an aim to gain a shared understanding of experiences
  • envision possible positive futures which build on present strengths
  • use language and foster relationships which create that positive future

In a pure AI evaluation, traditional evaluative methods – eg qualitative and quantitative research – are used only as the need arises, and are driven by participants.

Principles of Appreciative Inquiry

“Appreciative inquiry” is an approach to evaluation based on the assumption that an organization wants to improve.  Accordingly, the evaluation has a fundamentally positive focus on what the organization does well, and how it can build on this.

The core principles of Appreciative Inquiry are:

  1. Constructionist principle: people’s realities are “constructed” through their social interactions.
  2. Simultaneity principle: change and inquiry are simultaneous.  Inquiry can itself effect change.
  3. Poetic principle: the “story” of an organization is a product of the ongoing narrative of its members and others.
  4. Anticipatory principle: envisioning a positive future can help to guide people towards one.
  5. Positive principle: focusing on the positive can help create a positive energy for the future.
  6. Wholeness principle: wholeness brings out the best in people, so supporting people to share the whole story from a position of individual wholeness can build a “collective capacity for change.”
  7. Enactment principle: positive change occurs when people create the future through their words, images and relationships.
  8. Free choice principle: free choice stimulates positive change and liberates personal and organizational power.

[1] This explanation of AI is drawn from Howieson, Jill, “A Constructive Inquiry approach: blending Appreciative Inquiry with traditional research and evaluation methods,” Evaluation Journal of Australasia, 11(2) 2011.

Libraries conference – presentation

Screen Shot 2015-11-20 at 9.48.25 PMI gave a presentation yesterday at the NSW Public Libraries Association annual conference, ‘SWITCH2015.’ I talked about how to conduct ‘meaningful measurement’ in the libraries context.

I have uploaded the presentation here (see the link, below) for those keen on seeing my meandering thought processes up close and personal.

Download Meaningful measurement – library conference presentation November 2015

Social Return on Investment and other stuff

I was just bookmarking a few useful guides and tools for SROI and the like, and then thought, if this is useful for me, why not share it? So here are a few of my favourite links (is that a song…?) and apologies if I am telling you stuff you already know.

Other awesome websites:

  • Better Evaluation (international collaboration with loads of resources about evaluation)
  • New Economics Foundation (I am a fan-girl of this UK think tank about economics ‘as if people and the environment matter’)

 

How do the arts really change the world? European report recommends experimental research to answer this question.

CCS_coverA European report has just been released which reviews the available evidence and methodologies for cultural and creative ‘spillovers’ into other areas of society. The researchers, who included the creative industry veteran Tom Fleming, looked for evidence in:

  • knowledge spillover (new ideas, innovations etc from the arts and cultural sector which might spillover into the wider economy)
  • industry spillover (e.g. productivity and innovation from having dynamic cultural presence)
  • network spillover (e.g. impacts on society and economy from the presence of cultural clusters)

This might not surprise you, but Fleming and his team found that there is not enough evidence in all the many research articles out there to demonstrate a causal link between the arts/culture and many claimed spillovers. The research for these areas were persuasive, but fell short of proving causality.

He did find that there were several areas where causality had been established to a scientific standard:

  • communications can be boosted within organisations
  • culture-led regeneration has a positive impact
  • cross-fertilisation occurs between commercial and no=commercial sectors
  • investment in design has an impact
  • spillovers play a role in boosting uptake of new technology
  • networks are important in spreading innovation

We were excited to see Fleming recommend the following to really look at if and how arts/culture can cause social change:

  • experimental studies which include control groups
  • action research, testing hypotheses through interventions over time
  • proxy research approaches

We have our fingers crossed extremely tightly that some funding goes towards this in the academic and cultural space. As we all know, the kind of research to find out if the arts ‘changes the world’ and if so, exactly how and where and when and who, costs the kind of money most arts organisations don’t have.

At the moment what we mostly do in our evaluations is look at the existing research into the determinants of particular social outcomes e.g. wellbeing, inclusion, and see if we can hypothesise a sound theoretical chain of contribution to these determinants by the arts activity. Or not.

This is a perfectly acceptable way of evaluating arts projects (and is used in other areas with similar complexity around causation, such as health promotion). But you do need the big research on which to rely upon when looking for whether your arts project is contributing to the determinants of wellbeing, inclusion, increased productivity, or whatever it might happen to be. So let’s hope academics and institutions can support Fleming’s recommendations and put the question to rest (or perhaps, to work) – how do the arts really change the world?

 

 

 

Evaluating arts and social inclusion projects

Screen Shot 2015-10-29 at 11.57.56 AMYesterday I gave a presentation at the Creative Victoria Expert Arts Panel session on evaluating arts impact. Along with me, Deakin Uni’s Hilary Glow and Anne Kershaw presented about a recent evaluation they did for Vichealth on arts and wellbeing, and Mark Hogan from Regional Development talked about the Clunes Booktown regional transformation story.

Once Creative Victoria upload the full session to the web I will upload a link. In the meantime, I have uploaded my powerpoint presentation here.