The benefits of civil servants volunteering, to both themselves and charities, were laid out in a recent CSQ article. In this spirit we’ve recently completed a volunteer research project for the British Science Association (BSA). Given the amount we learned from the project we thought it worth sharing our experiences, some of the techniques we used, and what we found. Before reading on please note that the work was undertaken independently, we were not acting on behalf of Government.
Getting involved in pro bono work with the BSA
Pro-bono economics (PBE) is a charity that matches volunteer economists, like us, with charities, like BSA, who could benefit from an economist’s skill-set. They also provide economic advice, help to manage the work and ensure published work is anonymously peer-reviewed. BSA were interested in building the evidence base behind whether or not participating in their CREST awards programme has an effect on students’ science attainment and STEM subject selection.
The CREST programme is an inquiry-based learning intervention which BSA describes as ‘hands-on science’ which ‘builds transferable skills for further education and future employment’. There are four levels of Award, but we focused on evaluating the Silver Award, which consists of around 30 hours project work typically undertaken by 14-to-16 year olds. Given BSA research interests, we sought to answer:
(i) What are the characteristics of students taking Silver CREST Awards?
(ii) Does participation in the Silver CREST Award programme have an impact on attainment in science subjects at GCSE level?
(iii) Does participation in the Silver CREST Award impact on the likelihood of taking a STEM AS level?
What did we do?
We linked the data we had on Silver CREST pupils, held by BSA, to the National Pupil Database (NPD), held by the Department for Education (DfE). This enabled us to compare the characteristics and outcomes of CREST and non-CREST pupils. When trying to ascertain the effect taking the Award has had on outcomes (GCSE science performance and STEM subject selection at AS level) it’s important to make a fair comparison, which isn’t biased by confounding factors. To do this we used a matching method called ‘Propensity Score Matching’ (PSM) which, out of the millions of pupils in the NPD who didn’t do CREST, helped us to select a much smaller group with similar characteristics to the CREST cohort. This comparison group had similar prior attainment (at Key Stage 2), similar gender and ethnicity profiles, similar proportions of pupils eligible for free school meals (FSM), and so on. There were, however, a number of unobservable factors, such as teacher and pupil enthusiasm, which we could not match on.
What did we find?
We found that students who participated in Silver CREST were broadly representative of the wider pupil population in terms of gender and ethnicity. However, the CREST students were also different in many respects. For example, they were substantially less likely to have been eligible for FSM and had achieved stronger results at KS2 across all subjects.
Even after controlling for these differences we found that CREST pupils achieved half a grade higher on their best science GCSE result, compared to a statistically matched control group. The effect was slightly larger when looking only at FSM pupils. When looking at the impact of CREST on STEM subject selection at AS level we found CREST students were 21% (or 14 percentage points) more likely to take a STEM AS level than students in the comparison group. Again, the difference was larger for FSM pupils.
Because these results come with the significant caveat that we weren’t able to control for influences like pupil enthusiasm, these cannot be interpreted as the ‘causal’ effects of taking Silver CREST. However, our research suggests that CREST awards appear to be having a positive impact. To test this, our main recommendation is to design a randomised control trial (RCT) to take the evidence behind CREST to the next level.
What did we learn?
We learned a lot from the project, too much to detail here, but these are our main takeaways:
- Pro bono work is rewarding, we’re extremely glad we did it. It culminated in us being able to present our analysis at the House of Lords (courtesy of David Willetts, BSA Chair) which was a great development opportunity.
- It’s also challenging, the day job comes first, and there were a number of unforeseen obstacles which meant the project took significantly longer than we thought it would. PBE played an important role in helping to resolve issues.
- The NPD is a valuable national data source, and DfE were especially helpful in helping us to use it as we needed. They linked it with our CREST data for us, which was both challenging and time consuming for them as well as us.
- Linking two datasets is tricky, but valuable. Due to the need to ensure pupil anonymity, we weren’t able to look at the CREST data we wanted to link to the NPD. This made advising DfE how we wanted the data linked pretty difficult. However, once this was done the data we were left with became of great value to BSA. Other charities interested in the educational impacts of their schemes should consider following suit.
- PSM is a good alternative to standard regression methods when assessing the effect of a ‘treatment’ like CREST, but it also has relative disadvantages.
You can find the full report here. If you’re interested in finding more out about CREST awards, look here. If you’re either a charity or an economist and you’re interested in the work PBE do, look here. If you want to know more about our experiences with pro bono work, or if you’ve done some yourself, please share your thoughts below.
Recent Comments