John Jerrim and Robert de Vries argue a radical overhaul is needed of how social science is published and produced for it to provide a helpful basis for public policy. More progress is needed in particular over the lack of transparency of the research process, publication bias for positive findings and improved quality assurance mechanisms for peer review.
Governments have started to wake up to the fact that public policy should be informed by scientific evidence. Here in the UK, our government says it is committed to the idea that policy should be based on evidence of ‘what works’. But what if that evidence is not as reliable as it should be? What if the rock on which we want to build policy is actually made of sand? In our new paper, we describe the major problems with social science research, which severely limit its use in developing public policy.
The first problem is transparency. Just as schoolchildren are told they must show their workings in their maths homework, one might presume that academics are required to show how they arrive at the figures they report in their studies. But this is typically not the case. The precise analytic steps taken are rarely seen by others – even by peer reviewers. This makes it very difficult to check the quality of academic work.
This is a problem because academic social scientists are just people – they don’t always get things right. A host of studies now illustrate how, when attempts are made to reproduce academic evidence, it is not uncommon to find very different results. Consequently, there are likely to be hundreds of studies out there, quietly influencing policy, that could turn out to be incorrect.
The next problem is that of perverse incentives. The idealized view of science is that papers are published based solely on scientific merit. You formulate your hypothesis, you test it, then, regardless of the outcome, you publish the results in an academic journal. The problem is that the journals are less keen to publish boring negative results than eye-catching positive ones. And it’s not just the journals – scientists themselves are more likely to submit their exciting positive findings for publication in the first place.
This is called ‘publication bias’, and it’s fundamentally bad science. If ‘boring’ negative findings are less likely to be published, the evidence base becomes dominated by positive results. This creates a serious problem for a policymaker trying to assess the scientific evidence on any given topic. They will end up getting a highly distorted picture of the real world; one in which the extent and severity of social problems is probably exaggerated, as is the extent to which policy interventions can change things. This might leave policymakers too active in public policy investing in potentially ineffective solutions to social problems that might not even exist.
The final problem is the lack of a thorough quality assurance process. Currently, ‘peer-review’ is meant to be science’s stamp of quality. When a politician invokes ‘peer-reviewed research’, they mean ‘this is proper, solid science’. But in reality, this process is less thorough than one might think. It simply means two or three other researchers think the work is worthy of being in the public domain.
One would hope that, as experts in the relevant field, reviewers would tend to agree on what should be published. In fact, levels of agreement are scarily low. Reviewers are human beings with their own biases and prejudices, and they often have no set guidelines or standards upon which to review a study. In the words of the former editor of the British Medical Journal, this can make peer review ‘little better than tossing a coin’.
Together, these issues are devastating for the integrity of social science as a foundation for public policy. A lot of published studies are probably unreliable, but a lack of transparency leaves us no easy way to check. Problems with the peer-review system mean that the ‘peer-reviewed’ label is of little use as an indicator of whether a study is any good. And publication bias means that even an unprejudiced review of the evidence on a topic will give an answer that is unlikely to reflect reality.
This definitely does not mean we should abandon evidence-based policy altogether. A distorted view of reality is better than proceeding with our eyes closed. Nevertheless, a radical overhaul is needed of how social science is published and produced:
- Organisations which fund social science should, where possible, require researchers to share the computer code which generated their results
- Open publishing models should be extended across the social sciences. These models (as exemplified by the journal PLoS One) emphasise publication of all research that is technically sound, regardless of the results.
- As is (or should be) the case in medicine, social science research projects should be pre-registered before any analysis takes place to prevent them going ‘missing in action’ if they find negative results.
We are optimistic that progress is being made in some areas. But while we wait, policy will continue to be based on unreliable science. The longer we drag our feet, the more damage will be done.
About the Authors
Dr John Jerrim is a Lecturer in Economics and Social Statistics at the UCL Institute of Education and is a leading thinker in the evidence and methodology underpinning social policy research. He was the first recipient of the ESRC ‘Early career impact award’ for his research on the attainment of UK pupils compared to their peers in other countries (2). His research on education, and particularly East Asian teaching methods, has featured extensively in the mainstream international media since 2013.
Robert de Vries is a lecturer in quantitative sociology at the University of Kent. His research is primarily in the area of inequality and social hierarchies. He is a research fellow in sociology at the University of Oxford and co-editor of the collaborative social science and policy blog Inequalities.