• Measuring our impact in a social context can pose many challenges due to the lack of universal and objective truths 
• In combatting this, using a mixed-method approach allows us to better understand and articulate the impact achieved through our programme
• Here, we explain the steps we've taken to demonstrate this both internally and externally and what we have learned since our pilots

Throughout our Investigating our Impact blog series we have highlighted the various ways in which we think about impact at WLZ. First, highlighting how one size does not always fit all, and most recently how we are working with external partners, the Centre for Education and Youth to understand our provision in the early years. This time we explore how our internal teams use the perspectives of stakeholders around the programme – rather than data – to understand delivery and impact.

When we think of the terms evaluation, impact, or data, the first thought that often comes to mind is numbers. We use numerical data such as academic English and maths assessment scores and school attendance rates to determine which children and young people would most benefit from the WLZ programme. Furthermore, we attempt to measure the emotional wellbeing of a young person using the Strength and Difficulties Questionnaire (SDQ), a verified behavioural screening survey. In brief, these quantitative measures help us identify the most pressing risks affecting young people in our schools and how we can best support their development.

Numbers can never tell the whole story

But of course, numbers can never tell the whole story. Unlike in the physical and biological worlds, quantitatively measuring impact in a social context is extremely difficult due to the lack of universal and objective truths. For example, the SDQ is a verified tool that attempts to measure emotional wellbeing, but how can we objectively define ‘emotional wellbeing’? In addition, the questionnaire cannot account for home conditions, relationships, or any of the myriad of unobserved contextual factors.

Alongside the consideration of context is the question of impact. One of our core values at West London Zone is ‘evidence-led’ and this has helped to embed a culture of data and quantitative analysis throughout the organisation. However, this focus has sometimes been at the expense of understanding and articulating the impact achieved through our programme that is hard to measure – but often at least as meaningful to the children and families themselves. For example, a child talking to adults confidently by the programme end, whereas at the start even making eye contact was a challenge.  Stories of this kind are often relayed when speaking to school leaders and teachers, but have not always formed a part of the ways we try to understand our programme.

Complementing quantitative learning with qualitative learning

Of course, we have always used case studies to communicate to our funders and stakeholders about our contextual and hard-to-measure impacts. These have helped to build a connection with our work in a way that numbers are not able to on their own. These are important to balance with aggregated statistics to avoid cherry-picking and give a balanced view. We also incorporate context and stakeholder testimony into how we learn about our programme - as well as communicate about it. We complement quantitative learning with qualitative learning. Methods such as interviews, case studies, and focus groups can help fill in the gaps and reveal insights that cannot be represented numerically.

Using qualitative methods at WLZ 

Our first blog outlined the value of a mixed methods approach and we have started to incorporate qualitative methods into our internal evaluation processes. Following the completion of our fifth two-year programme in Summer 2022, we embarked upon a deep dive analysis to better understand what aspects of the programme worked well and what could be improved for future cohorts.

One of the most significant and insightful components of the analysis involved semi-structured case study interviews with five Link Workers. We asked each Link Worker to choose two young people who they worked with - one who made significant progress on the programme and another who didn’t – we asked them to reflect on how each part of our Theory of Change (Specialist Support, Trusted Relationship, and Joining up the System) contributed to the child’s progress. A thematic analysis revealed four key learnings:

• All three components of the Theory of Change are important, but the relationship with the Link Worker seems to be the keystone because they are a trusted person in school to who young people can talk to. In addition, they provide a safe space for young people to learn how to express their emotions. This trust helps to enable engagement with the other support provided, as well as having a therapeutic quality itself.

• Link Workers really recognised the benefits of one-to-one academic specialist support to improve children’s academic outcomes, as it can boost their confidence and gives them time to work on English and maths at their own pace outside of the classroom environment. This is something we already recognised and so over the past year, we have started to deliver a range of one-to-one intensive academic support at higher rates and to more children. 

• Link Workers recognised that sometimes it can be hard to support children and young people with behavioural challenges. Through our ongoing work with young people who have experienced fixed-term exclusions, we are aiming to further understand how our programme currently supports young people with more complex behavioural challenges, and what we should be doing in future to better support them. 

• Children and young people really enjoy having the opportunity to learn new things through our partner programmes, such as circus skills or woodworking. These play a role in driving school engagement, peer relationships and confidence.

We are really excited that we have taken this first step into qualitative evaluation methods. Whilst we still have work to do to incorporate qualitative insights into our everyday work, we recognise the benefits of this, as qualitative learning gives a voice to our Link Workers and the children and young people who they support every day. Without this voice, we reduce the programme to numbers and lose what makes WLZ so enriching to so many children and young people across West London.