Evaluation Partners
To determine the value of your programme, and to attract funding for its continuation or scale-up, you will need an evaluation.
Ideally this would be conducted by an external, independent evaluation partner. Your evaluation partner could help with some or all of the following elements:
Developing a theory of change for your programme, showing how your actions lead to your desired impact.
Developing an evaluation plan, including study design, sampling strategy, measurement strategy and analysis strategy.
Data collection and analysis.
Presenting the results so that core learnings and insights are summarised and disseminated (including potentially through conferences and/or peer reviewed journals).
Evaluation Brief
To recruit your evaluation partner, you will need a clear evaluation brief that sets out:
Details of your programme, its channels, approach, implementation period and objectives, as well as details of any previous evaluations that have been conducted.
Scope of work (e.g. Theory of Change development, evaluation plan development, etc)
The key questions you want the evaluation to answer
Timings and budget for evaluation
Additional Resources
SKY Girls Grant 4 Evaluation Partner Brief
Evaluation Criteria
The OECD DAC Network on Development Evaluation (EvalNet) has defined six evaluation criteria that allow evaluative judgements about programmes to be made.
You may want to assess your programme against some or all of these criteria.
Types of Evaluation
A range of evaluation designs have been used to measure social marketing programmes such as SKY Girls.
Quasi-Experimental Matched Design
SKY Girls Ghana was evaluated from 2017-2018. Experimenters selected three cities in Ghana where exposure to SKY was minimal, and compared these to Accra, where the programme was being implemented. A representative sample of girls in all four cities were given questionnaires at the start and end of the implementation period to test changes . The observed effects were compared across the different cities.
Dose-Response Analysis
The SKY Girls Ghana evaluation also measured individual girls’ level of exposure to the SKY Girls programme, and compared this to the observed changes in girls over the evaluation period. Implementers hypothesised that exposure to more than one SKY channel (e.g. magazine and radio show) would result in greater changes over the evaluation period.
Randomised Controlled Trial
MTV Shuga, a drama featuring educational storylines about HIV/AIDS, was evaluated in Nigeria from 2014-2017. Evaluators randomly selected young movie fans to watch either MTV Shuga or another drama with no educational messages. Participants were surveyed before and after the viewing to assess its effects on them.
Stepped Wedge
The SKY programme in Ghana is currently undergoing independent evaluation. This employs a stepped wedge design where neighbourhoods are randomized into different wedges which receive SKY programming sequentially – the wedges which have not received SKY act as controls until activity begins. A representative sample of girls from each wedge are surveyed at regular intervals throughout the implementation period to assess the impact of SKY’s launch in their community relative to other wedges.
Pre/Post Questionnaires
SKY Botswana was evaluated using a schools-based survey with a random sample of teenage girls, conducted once a year from 2014-2018. Researchers then analysed the differences at a population level between the period before SKY’s implementation and the period afterwards. They also assessed the differences between those exposed to SKY and those not exposed.
Data Collection Methods and Tools
There are different ways of collecting data to feed into the impact study. Methods fit broadly into two categories: quantitative and qualitative.
Quantitative Methods
Medical tests to show physical changes in participants from which you can infer behavioural shifts (for example, testing a participant for common STIs to make inferences about condom usage).
Surveys with your target audience to measure shifts in attitudes or self-reported behaviours among your TA. These could be done face-to-face in schools or households, or via the phone or social media.
Surveys with a representative sample of the wider population to measure shifts in attitudes or self-reported behaviours at a population level.
Stakeholder surveys reporting observed changes in a target group or of contextual factors that could have impacted your chosen outcomes.
Epidemiological / population / market data that provides objective insight into behaviour change (e.g. number of new HIV diagnoses, annual cigarette sales).
Qualitative Methods
Interviews or focus groups with your target audience to provide fuller information on how they experience your programme and what effect it has had on them.
Interviews or focus groups with stakeholders on their experience of the programme or the context surrounding it.
However, using any of these tools to collect data on a social marketing programme poses challenges.
Complexity of issues
Social marketing programmes tend to focus on complex, socially-embedded issues. Distilling the desired outcomes down to standardised quantitative questions is difficult.
Spillover effects
Randomised controlled trials allow you to infer causality. But with a mass-media campaign across multiple channels, it’s hard to prevent a control group becoming accidentally exposed to the intervention.
Desirability Bias
Many of the issues targeted by social marketing programmes are likely to be highly sensitive (e.g. alcohol usage, smoking behaviour). Surveys with the target audience rely on self-reporting of behaviour change and attitudes. There is always a risk that respondents will be influenced by desirability bias: trying to give ‘the right answer’. They may also find it hard to accurately remember their behaviour. This effect is compounded when working with younger age groups, especially those answering the survey in a school or home environment where they could be overheard.
Need for Robust Data
Qualitative methodologies are often better suited to the complexity of the issues and the need to build a deep level of trust and rapport with the audience before they will be honest. However, sample sizes for qualitative data tend to be small, and robust data based on a large sample is often required to give confidence in observed impact.
Survey Questions
Survey questions often rely on self-reported attitudes and outcomes. However, these can be subject to desirability bias. Different ways of asking questions can help to mitigate this effect.
Implicit Association Tests can help to measure subconscious attitudes towards a substance or behaviour (e.g. alcohol) at a group level. For example, participants could be shown images of people wearing different outfits, some of whom were drinking alcohol, and asked to rate them as cool or uncool. The average difference in the ratings between images with and without alcohol could be used to assess participants’ attitudes towards alcohol.
Knowledge testing can help to measure the educational value of a programme. For example, participants in an HIV treatment programme could be asked to identify common HIV treatment methods from a list.
Demand for services can test participants’ real-life behaviour in seeking out a service. For example, participants in a programme that seeks to increase demand for mobile money could be offered either a store voucher or a mobile money voucher of higher value. Their choices would indicate their likelihood to use mobile money.
Triangulation
Because of the reasons described above, in our experience it has been challenging to draw conclusions about the value of social marketing programmes from quantitative data alone. At SKY, we aim to use a multi-method approach, incorporating both qualitative and quantitative analysis, to enable us to build a body of evidence that stakeholders can use to make a considered judgment about the impact and efficacy of SKY.
Ongoing monitoring of your programme is also essential to keep your programme on track and drive continuous improvements.
You can use a range of tools to provide ‘feedback loops’ on your programme, and this data will also feed into the body of evidence over impact.
Feedback is important to provide indications of how effectively you are creating impact on key measures or attitudes, as well as to help optimise and evolve specific elements of programme implementation such as channel use or messaging.
Digital Metrics e.g. Social media following, engagement rates or number of video views.
Traditional Media Metrics e.g. TV station viewership figures, event attendence or circulation of print resources e.g. magazines.
Ongoing Phone, Digital or In-Person Surveys to measure awareness of and attitudes towards your programme, as well as indicative progress towards your impact indicators. These can be done with a smaller sample of your target audience than for a formal evaluation.
Focus Groups with your target audience or other influential figures to dive more deeply into relevant issues.
Social Listening tracking social media platforms for conversations related to your brand or focus issue, and analysing them for insights.
Additional Resources
SKY Girls Kenya Phone Survey Discussion Guide
Responsive Feedback in SKY Girls Ghana
Testing our Messages
We use feedback loops from our target audience to provide guidance on our campaign strategy and on specific messages and assets.
Pre-testing our initial messages through paid surveys with our target audience.
Running informal focus groups to gather ongoing feedback on content and shape future direction.
Setting up short surveys on our social media channels for followers to vote on multiple options for an asset (e.g. their favourite magazine cover of three).
Creating WhatsApp groups of influential teenagers to provide a second opinion on content and design, and advise on popular influencers and content.
Setting up paid advisory panels made up of members of our target audience.
Monitoring and emulating popular content among teenagers in our target market.
Ensuring our target demographic is represented on the SKY Girls implementation team to guide visuals and tone of voice.
Additional Resources
SKY Girls Message Testing Discussion Guide