Substances, adolescence (meta-analysis)
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

Substances, adolescence (meta-analysis).

Roona M.R., Streke A., Marshall D.
In: Gullotta T.P., Bloom M., eds. Encyclopedia of Primary Prevention and Health Promotion. New York: Kluwer Academic/Plenum Publishers, 2003, p. 1073–1078.
Unable to obtain a copy by clicking title? Try this alternative source.

The most influential finding in drug education research – that interactive teaching methods have the greatest prevention impact – was confirmed by the featured report but later questioned by unpublished analyses using better statistical methods, an episode which has left concern and uncertainty in its wake.

Summary

Note that this document has recently been acquired and processed by the Drug and Alcohol Findings Effectiveness Bank but dates from 2003. We feature it because of its importance in itself and the importance of the questions raised over this and similar analyses outlined in the comments from Drug and Alcohol Findings.

This book chapter synthesised research on the impacts of school-based programmes intended to prevent use of alcohol, tobacco or other drugs by young people. Meta-analysis A study which uses recognised procedures to combine quantitative results from several studies of the same or similar interventions to arrive at composite outcome scores. Usually undertaken to allow the intervention's effectiveness to be assessed with greater confidence than on the basis of the studies taken individually. was used to compute an average effect size A standard way of expressing the magnitude of a difference (eg, between outcomes in control and intervention groups) applicable to most quantitative data. Enables different measures taken in different studies to be compared or (in meta-analyses) combined. Based on expressing the difference in the average outcomes between control and experimental groups as a proportion of how much the outcome varies across both groups. The most common statistic used to quantify this difference is called Cohen's d. Conventionally this is considered to indicate a small effect when no greater than 0.2, a medium effect when around 0.5, and a large effect when at least 0.8. across studies of similar interventions in order to compare their relative effectiveness at delaying the onset of substance use over the following 12 months. The analysts found relevant evaluations of 207 programmes, which could be classified in to two broad types:
• 'non-interactive' programmes which rely on didactic teaching methods to lecture or otherwise convey information to pupils;
• 'interactive' programmes which give pupils the chance to exchange ideas with their teachers and classmates, try out new roles, or explore alternative ways to address potential future drug-related predicaments.

Interactive programmes

Overall, interactive programmes caused a small but statistically significant delay in the onset of substance use – an effect size of 0.15. This impact was particularly pronounced (effect size 0.21) among pupils identified as specially prone to (or 'at risk of') substance use. Of the interactive approaches, the highest effect size (0.27) was computed for studies of programmes which sought to create system-wide changes to enhance the school's atmosphere, to engage pupils more fully in school life and work, and to involve parents and local communities. However, such studies were few, methodologically weak, and the advantage of these programmes was reduced somewhat when the analysis was restricted to the better studies.

Next in their degree of impact were 'social influence' programmes intended to help pupils identify and resist pressures to use substances from peers or the media, or because the pupils overestimate how 'normal' substance use is among their peers. Such programmes had an effect size of 0.12, or 0.17 when allied with comprehensive life skills teaching to (for example) improve assertiveness, coping skills, communication and the setting of goals. Across all substances combined, only with these additional elements did programmes for primary school or older secondary school pupils improve on whatever comparators were used in the studies. But in the in-between younger secondary school ages (US middle school range), both types of programmes were effective to roughly the same degree. This also applied to the prevention of smoking and cannabis use in particular. In respect of drinking, the additional life skills elements remained essential for older secondary school pupils, but such programmes were ineffective among younger secondary school pupils.

With or without the additional life skills elements, social influence programmes were considerably more effective at preventing heavy drinking than drinking as such. This was exemplified across five studies which measured both. If preventing drinking was the criterion, the programmes tested in these studies would have to be judged ineffective, but if heavy drinking was the measure, they were unusually effective, delivering a statistically significant effect size of 0.25, one of the largest among universal prevention programmes.

Non-interactive programmes

In contrast to interactive variants, more didactic 'top-down' (or 'teacher-down') approaches were on average ineffective whether applied to all pupils or those identified as at-risk of early substance use. Among the latter they may even have been slightly counter-productive when set against whatever comparators were used in the studies – another contrast with the interactive programmes, which were maximally effective among at-risk pupils.

The authors' conclusions

School-based prevention programmes which use scare tactics or non-interactive lecture formats do not work. Others of a more interactive nature seem to work quite well for some substances at some school grade levels, but less well or not at all for other substances or grade levels. Perhaps most effective of all are the comprehensive approaches which seek to create system-wide changes in the school and involve parents and the community. But these are costly; perhaps a more justifiable approach is to adopt similar kinds of system-wide programmes, but with the aim also of advancing the school's core education mission.

It is also clear from this analysis that preventing 'use' of substances is not the same as preventing 'abuse'. Some approaches which very effectively prevent heavy use do not prevent use. This finding mandates clarity in objectives, including the choice between focusing on legal and moral imperatives to prevent illegal and disapproved activities, or the public health imperative to focus on damaging patterns of use.

On the basis of research to date, it seems advisable to consider prosocial schooling and other approaches which create protective schools and provide adolescents with sufficient developmental assets to ensure resilience, supplemented for particularly 'at-risk' pupils by individualised counselling, assistance and harm reduction programmes to develop the skills required to avoid dangerous behaviours like drinking and driving.


Findings logo commentary The same authors drawing on essentially the same body of research have focused on the issue of the objectives of school-based prevention, concluding that "promoting abstinence may not be a viable objective when substance use is normative in the culture, but preventing abuse and its attendant harms may be viable". The authors' advocacy of non-drug focused school programmes designed to improve pupil engagement with school and improve the school's climate has support from a limited body of research, which indicates that such programmes may not only be more feasible than drug-focused approaches, but at least as effective.

The featured report does not specify the precise analytic methods used to reach its conclusions. In the light of the re-analyses described below, these could have been a crucial. The re-analyses were brought to light by the British researcher Jim McCambridge, who drew largely on information received from the first author of the featured report. This new work questioned that report's finding that interactive teaching was most effective – a finding among the most influential ever reached about drug education. The series of analyses which culminated in the featured report was identified with the lead analyst Nancy Tobler, whose work was commemorated by Drug and Alcohol Findings in a reproduction of one of her last publications before she died in February 2000. Partly due to her death, it seems a decision was taken not to publish re-analyses of her work which undermined her major achievement – the discovery of the importance of interactivity. These new analyses used more rigorous and appropriate statistical methods Namely maximum likelihood estimation instead of least squares regression to find which estimate of the true impacts of the programmes best fit the available data (for which see: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74.671&rep=rep1&type=pdf) and a random effects rather than a fixed effects model, more appropriate for estimating the true impact of a type of programme from the limited sample of impacts available from the implementations of that programme which happen to have been researched (see for example the explanations at http://mindhive.mit.edu/node/92 or http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.5902&rep=rep1&type=pdf). to estimate the true impact of prevention programmes from the 'snaphsots' available from research studies. Not until 2007 were these findings published, and then only briefly on the web site of the funder of the work, not in a journal or book.

As highlighted by Jim McCambridge, assessed according to these more rigorous methods, most of the differences between drug education variants were no longer statistically significant. In particular, and contrary to previous analyses in the series, the web report concluded that across all grade levels and substances, interactive programmes could not be shown to be more effective than lecture-oriented, non-interactive programmes. The exceptions were smoking and the younger secondary school ages (in US terms, the middle school years).

Moreover, across all substances, interactive programmes which sought comprehensively to foster life skills were no more effective than those which focused more narrowly on countering social influences to use substances. However, as the featured analysis found, such programmes were sometimes more effective among primary school and older secondary school pupils – in the reanalyses, in respect only of cigarette smoking and 'hard' drugs, not cannabis or alcohol. This finding is in line with conclusions that the best-known and most widely researched of these life skills programmes (Life Skills Training) is most thoroughly proven in its impact on smoking. Again echoing the featured analysis, social influence programmes were (at least in the younger secondary school range) better at preventing heavy drinking, than drinking as such. However, these findings appear to have been based on the earlier statistical techniques, and it is unclear whether they would have survived the more rigorous and appropriate techniques.

Though not a blanket repudiation either of drug education in general or of interactive or life skills teaching, these findings have shaken confidence both in drug education and in research in to its effectiveness. The far-reaching implications are explored further below.

For the editor of the journal which published the featured analysis, "These findings raise huge doubts about the value of school drug education in the prevention of drug use and related harm". In some ways this slightly overstates the case. The doubts raised by the featured analysis are about whether sophisticated interactive and lifeskills programmes are better in preventive terms than routine drug education, not about drug education as such. Also the typical outcome in the featured analysis was any substance use, not harmful patterns of use or related harm. Had these outcomes more often been targeted and measured, they may well have been affected more by interactive teaching. Lastly, there is (as opposed to any narrowly defined preventive impact) the possible extra educational and personal development value of interactive teaching. Social and emotional teaching which incorporates active learning has been found to confer a wide range of benefits on pupils including pro-social and healthy emotional development and academic advancement, while teaching the same topics using less well developed or well implemented teaching methods has narrower impacts. Such findings raise the possibility that this form of teaching better equips pupils for taking autonomous responsibility for themselves, even if they then go on to make decisions many adults would rather they didn't.

For the director of the project which produced the featured analysis, a key lesson was that "different ways of analyzing the same data may yield very different, even contradictory, results". Commenting on Jim McCambridge's paper, the lead author of the featured analysis described the pressures on researchers to downplay uncertainties and negatives in order to reassure policy makers that they really do have reliable tools to prevent drug misuse. Such researchers gain in influence and credibility, while more sceptical colleagues are sidelined. Commentaries on Jim McCambridge's paper highlight the co-option of research to the policy priority of showing that something does work, rather than rigorously testing the claims of programme makers as far as possible to destruction – a reversal of scientific method made more likely by the fact that the researchers often are the programme makers. On other occasions it seems that conscientious accounts from researchers have been bowdlerised by publicity outlets with an interest in presenting attractive findings.

More broadly, there is a suspicion that drug education's patchy preventive record is (except in respect of smoking, where any use really is harmful and use really does often lead to addiction) due to inappropriately targeting and measuring use rather than harmful use, and to teaching methods which, even in 'interactive' programmes, are not interactive enough. In particular, they usually mandate set prevention goals rather than allowing these to emerge during multi-directional pupil and teacher interactions (1 2 3). Whatever the subject, in Britain teachers have rarely been able to implement truly interactive relationship and communication styles as opposed to surface innovations which retain the essence of the didactic, closed-questions and one-right-answer teaching antithetical to deep pupil participation.

Thanks for their comments on this entry in draft to Michael Roona of the University of California, Jim McCambridge of the Centre for Research on Drugs and Health Behaviour at the London School of Hygiene and Tropical Medicine, and Andrew Brown of the Drug Education Forum based in London. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 13 May 2011

Comment/query
Open Effectiveness Bank home page


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2010 Does successful school-based prevention of bullying influence substance use among 13- to 16-year-olds?

STUDY 2013 PROSPER community-university partnership delivery system effects on substance misuse through 6½ years past baseline from a cluster randomized controlled intervention trial

STUDY 2011 Effects of a school-based prevention program on European adolescents' patterns of alcohol use

STUDY 2000 Education's uncertain saviour

STUDY 2010 The effectiveness of a school-based substance abuse prevention program: 18-month follow-up of the EU-Dap cluster randomized controlled trial

STUDY 2010 One-year follow-up evaluation of the Project Towards No Drug Abuse (TND) dissemination trial

REVIEW 2015 Prevention of addictive behaviours

STUDY 2001 Prevention is a two-way process

REVIEW 2009 School-based programmes that seem to work: Useful research on substance use prevention or suspicious stories of success?

STUDY 2010 The effects of Project ALERT one year past curriculum completion