Back Back

BLOG: The uncomfortable truth of access evaluation

14/12/2018
dan-howells

Lee Elliott Major, the Chief Executive of the Sutton Trust, recently noted that with an estimated £800 million a year spent by UK universities on outreach activities, the low level of evaluation that takes place is “criminal”.

The heightened need to focus on outcomes rather than just inputs in the new Office for Students access and participation regime underlines this view, writes Dan Howells, Head of Access and Outreach at the University of Wolverhampton.

The pursuit of truth

The Cambridge-Somerville Youth Study (1939-45) provides a vivid example of why the pursuit of truth through robust evaluation is so important. Indeed, I would recommend that every university outreach practitioner listens to the Freakonomics podcast produced by Stephen Dubner entitled: “When Helping Hurts: Good intentions are nice, but with so many resources poured into social programs, wouldn’t it be even nicer to know what actually works?”.  The study itself was pioneering in its use of a control group to see the effect of mentoring on 250 young boys deemed at risk of delinquency.

As Dubner notes, this is important “because whenever you’re trying to establish cause and effect, in anything — whether it’s a chemical reaction or a mentorship program — you need to be able to isolate the inputs and reliably measure the outputs. That’s a lot easier to do in chemistry than it is with something as complicated as people’s lives.

Through use of a control group and a steadfast commitment to the pursuit of truth over a significant period of time it was revealed that the intervention produced negative outcomes against all seven of its key measures: these included life expectancy, mental and physical health, alcoholism and likelihood of a criminal record.

What appeared to be an incredibly positive source of support for the boys involved (and this was echoed by the views of the participants themselves) was revealed to have had a detrimental effect.

Shifting the focus

What does this mean for universities and access and outreach practitioners? I would suggest that like those involved with the Cambridge-Somerville Youth Study, we must maintain a determined focus on establishing the true impact of our work. Major is right – not enough time or resource is dedicated to evaluation of outreach and widening participation activity led by universities.

On more than one occasion I have heard outreach practitioners say that they do not believe in randomised control trials because it means some young people will miss out. I would suggest that this fundamentally misses the point. This position assumes from the beginning that the outreach work in question will make a positive impact; we simply should never assume this. Programmes cannot be continually delivered based on the great intentions of their designers and deliverers, but on the proven outcomes they achieve.

What should focus the mind further is the widening gap in HE access between poorer students  and their more affluent peers which has continued since the Second World War. An increase in poorer students progressing to HE has taken place, particularly since the expansion of university places in 1992. And more children eligible for free school meals are progressing to HE than ever before. But the gap in HE progression between free school meal and non-free school meal pupils has remained stubbornly fixed for the past decade at around 17-18%. This includes during a period of significant investment in this area in the form of Aim Higher.

The jury is out

In his article, “Proud to Be Wrong”, Ben Thomas refers to how determined the great biologist, Charles Darwin, was in the pursuit of truth. In one vivid illustration, Thomas shares a conversation between Darwin, his wife Emma and a close friend on their happiest memories, when Darwin recalled memories of visiting a scenic park. He later returned to them after going to bed, bursting into the room in his nightshirt to correct himself, stating “I was wrong! That wasn’t my happiest memory! The happiest day of my life was one summer in Cambridge”.

This was an anecdote that members of Darwin’s family would retell with humour and pride. Thomas notes: “It’s easy to hear this story and laugh, and picture Darwin as an ageing bookworm, too flustered by details to follow a normal conversation. But even that picture hints at something deeper: what mattered most to him wasn’t keeping the conversation flowing, or even being right, but providing only the most accurate information possible to his loved ones”.

A jury assembles

We should aspire to something similar in our access work. In my view the jury is still firmly out on the broader impact of university access and outreach provision in how effective it has been over a sustained period in narrowing the HE progression gap. Indeed, in the many cases where evaluation is severely lacking, we could say that the jury has not yet even been assembled. This is not acceptable.

The Sutton Trust’s 2015 Report, Evaluating Access found no UK-based studies evaluating access strategies and approaches using robust designs to establish effectiveness. It cannot remain this way.

My personal commitment is that through the National Collaborative Outreach Programme consortium, Aspire to HE, and the wider outreach provision delivered by the University of Wolverhampton, we will be determine the full true impact of our work. This involves being bold in adjusting our provision when required to ensure we can always answer two key questions; are we making the positive difference we set out to make, and how do we know? Young people and local communities that our work seeks to support deserve this from us.  

It’s too early to say if our approach is working. But we are committed to finding out.

For more information please contact the Corporate Communications Team.

Share this release