5.1 Develop an evaluation design

The 5th step of the narrative change planning process – Evaluate reach & uptake - consists of two elements and on this page, we focus on the first: Develop an evaluation design. 
 

1. Develop an evaluation design
2. Collect the data, share and reflect

 
In putting together your evaluation design/plan, you need to start by working back from your campaign objectives. If the campaign has a clear short-term, policy outcome as the goal, as in the case of planning permission for a Muslim prayer centre in Shrewsbury, this process is relatively simple. However, for most campaigners working on narrative change, their objectives are focused on shifting the narrative or “changing the weather” as it is sometimes referred to. In this case, such a narrative shift approach means efforts to put diversity and inclusion back on the public agenda around migration. Designing the evaluation of these narrative shift focused campaigns is our focus in the section and is significantly more challenging than a focus on specific policy change.
 

Work on metrics to measure the reach, response and uptake of your campaign.

In putting together an evaluation plan for a narrative change campaign, we propose focusing on three levels as realistic measures that inform your understanding of the effectiveness of the campaign:
 
 

Specifically, the focus of leach level is as follows:
 
A.     Reach -  How many people in the target segment(s) did you reach? How many people in general paid attention? 
 
In this first evaluation level, you are simply looking at how many people were exposed to the campaign messages, whether they were from your target segment(s) and also how many times they were exposed to your message. For example, knowing basic estimations and segmentation of readership for your various target publications, viewership for a target TV programme, or attendance figures for a public event you’ve organised or the campaign is part of would help you to estimate the exposure. This was the approach taken in some of the campaigns we have showcased in these guidelines.
 
The third dimension of reach looks at how many times you reached the same audience. According to marketing and political campaign practice, the so-called “Rule of 7” principle significantly increases the chances that the audience will notice and also eventually buy your product or message1 . This goes back to the basics of our frames definition and the so-called salience of frames. Put more simply, people tend to notice and eventually believe the things that are repeated a lot. This is harder to measure accurately, but given that it is a fair assumption that people tend to get their news from the same sources, looking at how many times the campaign got covered in each source would be a good start in this regard.
 
B.     Response - Did you get the coverage and responses you wanted? 
 
Moving to the second evaluation level, the focus moves past exposure to how people responded to the campaign. The target for many campaigns is to have key opinion leaders and ultimately representatives of the public from the target segments responding positively to the campaign. Finding out if key opinion leaders have responded well often involves a media and social media analysis and you can get a quick feel for the public response by looking at comments and likes on websites and social media. However, this is also risky as these comments sections are home for trolls of all kinds, who tend to be very active in migration debates. If your campaign entails offline activities, such as an event of some sort or conducting face-to-face campaigning, you will quickly get an idea of how well the message is playing out and what type of responses the campaign messages induce and trigger. 
 
Building an evidence base for this evaluation level will combine collecting more structured data from a media analysis process, but also anecdotal evidence from meetings, events and coverage. So, it is worthwhile to have campaign team member(s) record such data on an ongoing basis in a campaign evaluation log that is a shared and easily accessible document. 
 
C.     Uptake - Are the right people adopting and using the campaign messages or proposing the policy position you put forward? 
 
This is where the ultimate goal of your campaign resides – meaning that you are looking at whether the public discourse has shifted in the way that you wanted it to. Simple metrics are focused on whether key opinion leaders have started to use your language or talk about the problem and/or solution in the way you describe it. More challenging metrics involve looking at the language, values and policy options the target segments of the public are talking about in the discussion – often done using focus groups or polling. 
 
One of the key challenges in this debate is how much credit you can take for any shift in opinion in the public discussion. Obviously, people’s opinions shift based on multiple inputs and in a free and open democratic culture, this is a good thing. So, the advocacy principle we propose you follow is to look at contribution rather than attribution, i.e. to what extent did your campaign add to a narrative shift, rather than trying to take full credit. In order to track a campaign’s influence in the evolution of a public discussion, many people use tracer phrases or hashtags that easily flag that some contribution has been made. For example, in the campaign case by British Future we researched, ‘poppy hijab’ as a phrase or even the artefact itself can serve this tracer function. The timeline for measuring uptake also poses a challenge, as if often takes time beyond the natural timeline of the campaign to see a shift in the public discourse. So, considering some ex-post measurements in the evaluation design is also a good idea.
 
The following is an example from one of our cases of how response and uptake were measured:

 
Clearly define the data sources/methods at each level to generate a focused and comprehensive plan.

The table below builds on a short overview of the each of the levels of evaluation defined above and adds the dimension of common methods of measuring the associated levels.
 

 
 
To put together your plan, you need to dig deep into the specifics and find credible and reliable sources that allow you good data for each level. Based on the specifics of your context, you will undoubtedly need to adapt the measurement and approaches to suit your purpose, budget and the data available.
 

<< 5.0 - 5.2 >>

• 1Tutorialspoint (2018) The Rule of Seven.