Send email for updates
Faggiano F., Vigna-Taglianti F., Burkhart G. et al.
Drug and Alcohol Dependence: 2010, 108(1–2), p. 56–64
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Faggiano at firstname.lastname@example.org. You could also try this alternative source.
The largest European drug education trial ever conducted tested whether US-style social influence programmes would prove effective in Europe. There were probably some real successes, but these were limited and may have been artefacts of the implementation and analysis of the study.
Summary Funded at European level by the European Commission, the European Drug Addiction Prevention trial (EU-Dap) aimed to test whether 'social influence' school-based drug prevention programmes of the kind developed in the USA will prove effective in Europe. Across seven countries Belgium, Germany, Spain, Greece, Italy, Austria and Sweden. and 170 schools it recruited 7079 12–14-year-old pupils, the largest sample ever in a European drug education trial.
Developed by the EU-Dap project team, the 12-lesson curriculum they tested is known in English as Unplugged. Materials are available on the EU-Dap web site and the programme's development and approach has been extensively documented. As well as informing pupils about substances and their use, such curricula aim to affect substance use by training pupils how to resist pressure to use, reinforcing attitudes which sustain commitment to continued non-use, and enhancing decision-making, social and life skills. Unplugged particularly emphasised correcting pupils' beliefs about the pervasiveness of substance use ('normative beliefs') by contrasting these with data from surveys of pupils of the same age which typically reveal that average use levels are lower. To make the programme more feasible for schools, it was limited to 12 lessons which can be completed within a school year. The schools' own teachers taught the lessons after two and a half days' training in the lessons and materials, and in how to teach them using methods which encourage interaction between pupils and between pupils and teachers, such as role-play and giving and receiving feedback in small groups.
This basic curriculum was supplemented either by meetings led by pupils selected by their classmates, or by workshops for the pupils' parents. While the curriculum was moderately well About half the classes were taught all of the curriculum units, and each unit was taught to 78% of enrolled pupils. However, some units were typically implemented only partially. implemented, peer-led activities were rarely conducted, Just 8% of classes conducted all seven meetings, 71% did not conduct any, and on average there were just under two meetings. Poor implementation was generally due to peer leaders withdrawing and lack of interest among their classmates. few parents attended On average no more than 28% of pupils had a parent participating in any given workshop. the workshops, and an important element – role-play – was generally omitted.
Schools were randomly allocated to one of these three variants of the Unplugged intervention or to act as 'control' A group of people, households, organisations, communities or other units who do not participate in the intervention(s) being evaluated. Instead, they receive no intervention or none relevant to the outcomes being assessed, carry on as usual, or receive an alternative intervention (for the latter the term comparison group may be preferable). Outcome measures taken from the controls form the benchmark against which changes in the intervention group(s) are compared to determine whether the intervention had an impact and whether this is statistically significant. Comparability between control and intervention groups is essential. Normally this is best achieved by randomly allocating research participants to the different groups. Alternatives include sequentially selecting participants for one then the other group(s), or deliberately selecting similar set of participants for each group. schools which simply carried on with their normal lessons. Taken singly, none of the three variants significantly improved substance outcomes compared to the controls, so reports to date have concentrated on comparing outcomes for all 3547 pupils in the 78 Unplugged schools, to the 3532 pupils in 65 control schools. Excluded from this total were the 27 schools which dropped out of the study after being randomised to the interventions but before their students could be surveyed. Among these were nearly a quarter In contrast, fewer than 5% of the control schools faced only with the research elements of the project refused to carry on. of the schools allocated to Unplugged. Another five did not conduct the latest follow-up surveys. Of the 7079 pupils surveyed before the lessons, 18 months later (15 months after the Unplugged lessons had ended) 5541 provided usable data at the latest follow-up. In between a further survey assessed pupils' reactions three months after the lessons.
The surveys indicated After adjustments for various influences on the outcomes including the starting level of substance use among the pupils. that 15 months after the lessons, over the past month pupils in Unplugged schools were significantly less likely to have been drunk (14% v. 18%) or drunk three or more times (4% v. 6%). The remaining five of the seven measures of substance use were also on average lower among Unplugged pupils (for every 100 pupils, 2 to 4 fewer engaged in these use patterns), but these differences did not meet usual criteria for statistical significance. Among them were any cannabis use, use three or more times in the past month (at 4% v. 6%, very nearly statistically significant) and three measures of smoking from any to near daily At the follow-up a year before, near-daily smoking had been significantly lower among Unplugged than control pupils but this did not persist. use.
These results could not include data from the 22% of pupils who did not complete the latest follow-up survey, or who could not be identified as the same individual who completed a baseline survey. On the assumption that they did not change their behaviour or were all non-users, the results remained substantially the same. When instead it was assumed that all the missing pupils were engaging in each substance use pattern assessed, still average use levels were uniformly lower in Unplugged schools, but none were significantly different from those in control schools.
Because (via an anonymous code) individuals could be linked back to their baseline responses, the researchers could identify transitions in substance use patterns. Of the 27 possible transitions, five were significantly more or less likely in Unplugged than control schools, all in a favourable direction. Unplugged pupils were significantly more likely to have remained non-smokers (83% v. 81%) and non-users of cannabis (94% v. 92%), and to continue to say they had not been drunk in past month (87% v. 85%). Among those who had been drunk once or twice in the past month at baseline, significantly fewer Unplugged pupils progressed to more frequent drunkenness (16% v. 33%), and more reverted to not being drunk at all (59% v. 39%). Though these were the only statistically significant differences, across all 27 Three (continued sporadic or occasional use) were not clearly favourable one way or the other. comparisons, 20 favoured Unplugged pupils and four control pupils. In line with findings on use levels, transitions in smoking were least likely Five for and three against. to favour Unplugged pupils.
A report on the study based on the first follow-up three months after lessons ended found that significant impacts were confined to the boys. As well as the smoking, drunkenness and cannabis use measures in the featured report, this sub-study assessed the prevalence of any illicit drug use over the past 30 days, including cannabis. Of these eight measures, all moderately or substantially favoured Unplugged boys over boys in control schools, and six of the differences were statistically significant. In contrast, none of the findings for girls were significant and half were in the 'wrong' direction, including all three measuring or mainly reflecting cannabis use. The findings were due to more boys remaining at or moving to lower use levels, and fewer starting to use or increasing the frequency of use. Unplugged was particularly ineffective (and there were some unfavourable impacts) among the minority of girls with relatively low self-esteem and among older (13–14-year-old) girls.
The authors concluded that comprehensive social influence curricula can effectively be delivered in the European school setting and help delay onset of substance use. Fifteen months after the lessons ended, the approach tested in the study still exerted a restraining influence on drunkenness and cannabis use. Overall effects on smoking seen earlier were no longer apparent, though the curriculum may have helped non-smokers resist starting. Implementing Unplugged in one or two classes could prevent one case of alcohol abuse and one case of cannabis use. The curriculum both hindered progression to use or higher levels of use and facilitated reversion to less intensive patterns of use. Lack of impact among girls may have been due to their more advanced (and therefore less easily altered) social and emotional development, and the fact that the programme did not focus on bolstering the low self-esteem more common and perhaps also more damaging among the girls.
commentary By design, at entry to the study none of the schools were implementing specific drug prevention interventions with strong packages targeted at the relevant school years, a situation which presumably persisted in most control schools. This should have given Unplugged a weak comparator against which what was intended to be a strong package could display its advantages in a study large enough to detect these. What emerged was a pattern of generally positive but modest and usually not statistically significant benefits relative to control schools. It seems probable that Unplugged was indeed preferable to doing nothing very much specifically to prevent substance use. However, if this was the case, the benefits were quite limited.
Among other things, the hope was that the intervention would reduce how many children of a given age had started smoking, drinking, or using cannabis and other drugs. In the event, none of the reported measures of whether substances had been used in the past month were significantly reduced. Reducing regular use was another objective, one as far as we know achieved only in respect of smoking, and then only fleetingly. In respect of drunkenness, however, there were consistent, statistically significant and lasting positive impacts. A later report from the study reveals that this effect was accompanied by a reduction in problems Quarrel or argument; fighting; accident or injury; loss of money or other valuable items; damage to object or clothing; relationship problems; poor school performance; victimisation by robbery; or hospitalisation. related to drinking but not by any significant reductions in drinking as such or in regular drinking. Together these reports suggest that the lessons did retard the age-related growth in problem drinking but (despite the positive direction of most findings) had no lasting significant impacts on the prevalence or intensity of substance use as such. At a more microscopic level, of 45 possible transitions between use or problem levels, just eight were significantly more or less likely in Unplugged schools, all in a favourable direction. Of these, all but two concerned alcohol. How much these findings can be relied on is questionable. The favourable direction of most other transitions attenuates but does not eliminate concern that among so many tests, some would have thrown up statistically significant differences purely by chance.
What the study intended to test and by which methods was well set out in advance and the results have been or will be comprehensively reported according to that protocol, a major methodological advance on some other studies. Still, several methodological issues mean that such positive findings as there were may have been artefacts of the implementation and analysis of the study rather than reflecting a real impact of the interventions.
Criticisms of the way they conducted the study and analysed and presented the results were put to the authors in comments on an earlier report. In response to the point that statistically significant differences were few, the authors argued that the "overall tendency" of the results was encouraging. However, the purpose of randomising schools to Unplugged and testing results for statistical significance is to help eliminate reasonable doubt that such tendencies are due to something other than the interventions. On this yardstick, reasonable doubt remains for most of the measures reported so far. It also appears that many schools were unwilling or unable to implement Unplugged, casting doubt on the implementation feasibility the authors tried to achieve. Summary below; details in the background notes.
Randomisation was compromised when, faced with the burden of implementing the interventions, 24 of the 102 schools allocated to Unplugged pulled out of the study, possibly leaving a set of schools keener on drug misuse prevention than those which remained in the control arm. Given how generally small they were, this could conceivably account for differences in outcomes between the two arms.
Before this stage another 120 schools declined the study, mainly because they were unable to schedule the intervention during the next school year, casting doubt over whether Unplugged really was as feasible for schools to implement as its creators had intended. This plus the loss of pupils from the study mean the findings can only be considered applicable to the roughly half of schools prepared to take on the burden of the research and interventions, and to the minority of the entire pupil population taught in such schools and who complete the surveys required by research projects. Among those which did take on the intervention, the parental and peer-leader supplements did not prove feasible and implementation of the core curriculum itself was "just moderate".
In the featured report, Unplugged schools had a significant advantage on (probably) two out of eight measures of the prevalence of different types of substance use and just five of the 27 possible transitions between these types of substance use. Some findings which did meet criteria for statistical significance might not have done so using alternative ways of testing the results, such as adjusting for the risk of finding some positive results purely by chance among the many comparisons made between Unplugged and control schools. Assumptions made about the development of substance misuse among the 1538 children missing from the analysis could also have eliminated some or all of the statistically significant differences.
The findings for boys when the sample was divided by sex strongly suggest a consistently beneficial programme impact, but 'suggest' is all they can do, because the analysis was not planned in advance. Such post hoc analyses can capitalise Post-hoc subsample analyses of this kind are best seen as generating hypotheses for testing in a study specially designed for this purpose. The main problems (there is no implication that these all applied in this case) are that they rob the results of the reassurance of the level playing field created by randomising patients to different treatments, build on what may be chance variation in the effectiveness of the intervention between different subsamples, test effects not derived from the theory of how the intervention is supposed to work, and can capitalise on the fact that samples can be sub-sampled in any number of ways until one (perhaps purely by chance) results in a significant finding. As a result, "any conclusion of treatment efficacy (or lack thereof ) or safety based solely on exploratory subgroup analyses are unlikely to be accepted" (Lewis J.A. "Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline." Statistics in Medicine: 1999, 18, p. 1903–1904. These risks are eliminated or reduced by specifying the subsamples in advance at the time the trial is designed but often this is not the case (Al-Marzouki S., Roberts I. "Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet." The Lancet: 2008, 372, 19 July, p. 201). on the likelihood that, purely by chance, one of the many ways a sample can be divided up will produce a significant finding in one of the sub-samples.
Despite some encouraging findings in the featured study, these are not strong enough to alter the view that drug education in secondary schools makes little contribution to the prevention of problems related to drinking and illegal drug use (NOTE), The cited review supports the modest effectiveness of school programmes in preventing cannabis use. However, of the four studies on which this verdict was based, one was a primary school programme not focused on substance use at all but on classroom management, education and parenting, another was conducted only among pupils for some reason excluded from mainstream education, and the programme studied in a third has since failed in a more real-world study conducted by researchers not associated with its development. The remaining study was conducted in secondary schools but the impact on cannabis use was not statistically significant. though the evidence in respect of smoking is stronger.
Mixed and generally inconclusive findings of a prevention impact from school programmes targeting substance use do not negate the possibility that general attempts to create schools conducive to healthy development will affect substance use along with other behaviours, nor do they relieve schools of the obligation to educate their pupils on this important aspect of our society. Arguably too, while less or safer substance use may be a desirable side-effect, drug education should be assessed against educational and youth development criteria to do with being relevant and useful as assessed by the young people themselves, rather than pre-set behaviour change objectives.
Thanks for their comments on this entry to Adrian King of InForm. Commentators bear no responsibility for the text including the interpretations and any remaining errors.
Last revised 18 February 2011
Background notes to this entry
Comment/query to editor
Give us your feedback on the site (one-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates
REVIEW 2015 Prevention of addictive behaviours
STUDY 2000 Education's uncertain saviour
NASTY SURPRISES 2004 Confident kids ... like to party
STUDY 2003 Family programme improves on school lessons