Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials
Effectiveness bank home page. Opens new windowReview analysis

This entry is our analysis of a review or synthesis of research findings considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original review was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the review. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials.

Humphreys K., Blodgett J.C., Wagner T.H.
Alcoholism: Clinical and Experimental Research: 2014, 38(11), p. 2688–2694.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Humphreys at Keith.Humphreys@stanford.edu. You could also try this alternative source.

12-step fellowships offer a way to reconcile shrunken resources with the desire to get more patients safely out of treatment. Accounting for the self-selection bias which has obscured AA’s impacts, this synthesis of US trials finds that attending more meetings after treatment boosts abstinence. Why then is research equivocal on whether promoting attendance improves drink-related outcomes?

Summary Involvement with Alcoholics Anonymous (AA) mutual aid groups is associated with better outcomes on alcohol-related, psychological, and social measures, but it has been unclear whether this correlation reflected AA’s effectiveness, or was merely an artefact of AA attracting drinkers who are more motivated to change, less troubled, or more socially stable, who would in any event have done better. The case for AA truly being effective in helping to control dependent drinking received a major boost when trials which randomly allocated patients to professional interventions to facilitate and promote AA attendance found these improved outcomes. These trials still cannot rule out the possibility that self-selection bias inflates estimates of the impact of AA participation on drinking, because they test the effectiveness of AA facilitation interventions, not AA as such. However, such trials do provide data which can be re-analysed to generate an estimate of AA’s effectiveness free of self-selection bias, using a statistical approach known as an ‘instrumental variables’ analysis. The featured study was the first to take advantage of this opportunity.

Key points icon

Key points
From summary and commentary

This synthesis of US trials found that the more AA meetings problem drinkers attend after treatment, the fewer the days on which they drink, even after accounting for the possibility that more promising patients self-select into AA.

However, research is equivocal on whether systematically and intensively encouraging attendance at 12-step groups significantly improves drink-related outcomes; heavy drinking and adverse consequences are usually unaffected even if abstinence is modestly boosted.

A trial in England found no statistically significant improvements in abstinence after more active referral to 12-step groups, but was not a definitive test of these types of interventions.

Among patients allocated to interventions to promote AA attendance, the aim is to mathematically isolate the portion of that attendance due to the intervention over and above the patients’ ‘natural’ attendance rate. The degree to which that portion alone can be shown to have been associated with drinking outcomes then becomes a relatively ‘pure’ estimate of the impact of attending AA, stripped of selection bias. The instrumental variables method makes such estimates possible. In the featured analysis the ‘instrument’ was randomised allocation to a more AA-promoting treatment versus another treatment. This ‘instrument’ is suitable because while it affects AA attendance, randomisation should mean it is entirely unrelated to the patient characteristics which might make one person more likely than another to attend.

Trials were identified by searching a database provided by the US National Institutes of Health, limited to studies which it has supported. The aim was to find studies which had randomly allocated patients with alcohol use disorders (dependence or abuse) to at least one intervention intended to bolster participation in AA versus another intervention which did not, or not to the same degree. The search was limited to peer-reviewed reports published no later than 2010.

Five US studies were found including Project MATCH. It consisted of two trials, one of patients leaving prior intensive outpatient or inpatient treatment (the aftercare arm), and one of patients starting treatment afresh (the outpatient arm), making six comparisons in all between AA and other approaches. The intention was to pool their results, but there was evidence that the Project MATCH aftercare arm had recruited patients who significantly differed, so results from the 1582 patients in the other five trials (including Project MATCH’s outpatient arm) were pooled, while those from the 774 patients in Project MATCH’s aftercare arm were separately analysed. Across the trials the patients were typically middle-aged white men working full time and either currently or formerly married.

At issue was the relationship between the change from baseline to post-treatment follow-up in the proportion of days patients were abstinent from alcohol on the one hand, and on the other the proportion of days during the follow-up periods on which the patients had attended AA mutual aid meetings. These variables were chosen because they had been recorded in each of the studies. For similar reasons, the follow-up points chosen for the analysis were three and 15 months after the start of treatment. If a relationship remained after the instrumental variable analysis had partialled out extraneous influences, it would be evidence that attending AA was not just associated with abstinence, but actively promoted it.

Main findings

A prerequisite for the analysis was that allocation to interventions intended to increase AA attendance strongly did so; this was found to be the case. Across the five pooled trials, the effect was to raise attendance from 6% of days to 18% and 16% respectively over the first three and 15 months of the trials. In the Project MATCH aftercare trial, over the first three months attendance rose from 28% to 40% of days, but over the entire 15 months the increase was just 6% – from 20% to 26%.

After randomised allocation to an AA-promoting intervention had been used as the ‘instrument’ to control for extraneous influences, still across the five pooled trials and at both three- and 15-month follow-ups there remained a statistically significant relationship between greater AA attendance and more days of abstinence. Though high rates of abstinence left constricted room for improvement, at the three-month point the relationship equated to roughly a further two meetings a week being associated with 3.3 extra non-drinking days per month. The relationship not only persisted to 15 months, but became slightly stronger. However, there were no such statistically significant relationships in the Project MATCH aftercare arm, analysed separately for reasons explained above.

The authors’ conclusions

It is encouraging that the results are broadly consistent with what prior research has found: AA appears actually to benefit people with drinking problems rather than simply acting as a gathering place for individuals who would have improved without it. The impacts on abstinence assessed across the five pooled trials were additional to those generated by the AA-promoting interventions themselves, and the effect persisted to 15 months after treatment ended.

The exception was the Project MATCH aftercare trial. Here the AA-promoting intervention did stimulate greater attendance, but that was not associated with any significant increase in days of abstinence. The likely explanation is a ‘ceiling effect’: in the other trials the interventions shifted attendance from zero or infrequent to steady, more of a step change than the frequent to yet more frequent shift in the Project MATCH aftercare trial.


Findings logo commentary An Effectiveness Bank hot topic introduces mutual aid groups and the evidence on their effectiveness, providing further information on the developments and studies mentioned below.

For UK commissioners, mutual aid spearheaded by 12-step fellowships offers a way to reconcile diminished resources with the desire to get more patients out of treatment without increasing the risk of relapse. The interest at national level can be judged from the resources made available since around 2010 to aid implementation. How much has trickled down to local service level is unclear.

One blockage to professionals in the UK embracing 12-step groups as part of their practice is lack of faith in their effectiveness, which in turn may be due to the lack of rigorous studies, especially from the UK. With just three randomised trials (all involving coerced attendees) among its collection of studies, a review published in 1999 synthesised the results of trials comparing AA groups against other approaches or no treatment at all. Findings from the three randomised trials suggested that people forced to attend AA do no better and possibly worse than when coerced instead into professionally run treatments or left to sort out their own ways of overcoming their problems. In contrast, the non-randomised studies, in which (with one partial exception) alcoholics chose whether or not to attend AA meetings, recorded statistically significant advantages over other approaches. This pattern of results suggests that AA looks better in some studies because those who attend are more motivated, and that people coerced into attending AA meetings might do worse than those coerced into other treatments. However, the three randomised trials were deeply flawed as assessments of AA as usually accessed and attended, and in two of the trials methodological features meant they were poor indicators of relative impacts on drinking.

Nearly all the research on 12-step groups and allied treatments comes from the USA, but the US record – where the 12 steps are deeply engrained and widely accepted – is not necessarily a guide to their impact in other societies. The featured study added to this literature by re-analysing data from a restricted set of US studies funded by the US National Institutes of Health. With this study, so far attempts to disentangle competing explanations of links between AA and abstinence have suggested that in the US context, attending AA groups promotes abstinence except after prior intensive (and especially inpatient) treatment. From the UK, a study of interventions which substantially increased attendance at 12-step groups among patients leaving inpatient treatment found no significant effect on abstinence. However, abstinence rates did tend to be higher among patients encouraged to attend 12-step groups, and the study was not sufficiently large and long-term, and the 12-step encouragement not sufficiently intensive, for the results to be considered a definitive verdict on the groups, or on the usefulness of treatment services encouraging patients to join them.

The UK study exemplified the fact that though 12-step groups themselves are not amenable to randomised trial, patients can randomly be allocated to ‘12-step facilitation’ interventions which promote attendance at the groups. Unlike actually attending the groups, these are in the hands of services to choose to deliver or not. Despite substantially increasing 12-step group attendance, relative to alternative treatments these interventions have generated inconsistent impacts on drinking. Some studies have found abstinence modestly boosted relative to alternative approaches, but generally without any extra impacts on heavy drinking or its adverse consequences, results which seem to cast doubt on the relationship between AA attendance and abstinence or less damaging drinking; if those links were strong and consistent, we would expect an intervention which promoted attendance also to beneficially affect drinking. However, much may depend on the make-up the treatment sample. Promoting 12-step groups cannot be expected to have much impact among patients already committed to attending or who have already attended sufficiently to gain whatever benefit they can. Maximum impact can be expected among patients not already committed to the groups, but who would try them if encouraged to do so by their counsellors or clinicians. Impact may also depend on the treatment programme which 12-step facilitation follows or forms part of; there are several indications that impacts are greatest when this too is 12-step based.

These general conclusions are substantiated below by analysing the featured study and similar studies which have attempted to expose the impact of 12-step groups by stripping away self-selection bias, and by detailing results from studies of treatment interventions which have promoted engagement with AA groups.

The featured study was an attempt to overcome what we have termed the ‘methodological Catch 22s’ which have left the issue of AA’s effectiveness open to debate. The difficulty is that the classic randomised trial format fits mutual aid badly. Most fundamentally, participating in mutual aid groups is something someone does, not something done to them which can be expected to work regardless of whether they chose that route to recovery and subsequently actively participated. Unlike ‘gated’ professional services, it is impossible to deny someone access to a free and open-access mutual aid network. Though randomisation is either impossible or violates the essence of mutual aid, without it results are vulnerable to the possibility that people who choose to attend and then become engaged in mutual aid groups do better than those who choose not to just because they are keener to achieve abstinence, not due to any impact of the groups – the so-called ‘self-selection’ bias which the featured study sought to overcome. Self-selection could also work the other way, drinkers selecting not to attend AA. Especially where this recovery strategy is pervasive, those prepared to deny themselves access to it are not necessarily typical dependent drinkers.

For the first time the featured analysis attempted to eliminate self-selection bias by capitalising on studies which had randomly allocated patients not to AA meetings, but to treatment interventions which did versus did not (or not so strongly) systematically promote AA attendance. The thinking was that extra attendance promoted in this way could not be due to the greater motivation or resources of the patients, so would offer an unbiased estimate of the impact of attendance on abstinence. As in other studies, in Project MATCH when AA followed inpatient treatment, attendance made no statistically significant extra contribution to abstinence. But across the remaining studies, the results implied that going to an additional two meetings each week would be associated with an additional three to four days of abstinence per month. Though in some ways an advance on previous estimates, it seems possible that the presumed impact of attending more meetings was in fact a gradient reflecting how well patients responded to the AA-promoting intervention itself. The better they responded to it, the more meetings they would attend and the more they would remain abstinent, making it look as if meeting attendance caused the extra abstinence, when in fact both were caused by the professional intervention.

If we accept that the featured study has shown that attending AA groups can promote abstinence, what are its specific implications for practice? The demonstration is in the context of prior professional encouragement to engage with AA. In itself it does not necessarily mean that abstinence or reduced drinking will also be promoted by choosing to attend outside the context of treatment, though a study described below suggests this can be the case. Nor does it indicate that choosing AA after therapy which is not geared to encouraging this will be similarly effective, an implication of some of the Project MATCH findings. If treatment context is important, so too may be the national context, and similar results might not have been found outside AA’s US homeland. Several studies have also shown that while relative to other approaches, promoting attendance at 12-step groups may be associated with abstinence, that does not necessarily extend to extra reductions in heavy drinking or in drink-related adverse consequences. It has been argued that to be considered truly effective relative to other approaches, a treatment must not only further improve the specific symptom at which it is targeted (in the case of promoting 12-step groups, by fostering abstinence) but also further reduce the burden that symptom places on the patient and ameliorate overlapping comorbidities – in other words, tackle the underlying causes of the patient’s problems (in plural), not just suppress one symptom of those problems.

Other analyses accounting for self-selection bias

Though arguably best placed to find an impact of attending AA groups, the featured study was not the first to seek to expose the impact of the groups by using an ‘instrumental variable’ methodology to account for bias due to drinkers self-selecting into the groups.

Capitalising on the relative unavailability of AA meetings was the strategy adopted in an earlier US attempt. Participants were patients encouraged to join AA groups following 12-step based inpatient treatment. Standard analysis found what looked like a significant positive effect of attending versus not attending the meetings. Even after adjusting for other factors, patients who went on to attend AA in the three months after leaving the inpatient centre were almost four times more likely to have remained abstinent. Though it remained, at 1.7 times more likely to be abstinent, the effect was halved and became statistically insignificant when adjusted for self-selection. The adjustment relied on the fact that most of the sample were not in a position to drive themselves to meetings and a minority did not live in a town with an AA group. Both factors affected whether patients attended the groups but were presumed to have no direct impact on abstinence – the only effect they could have, it was thought, was via influencing attendance. Unfortunately this assumption was not tested by examining the data, and it is not hard to think of ways both factors could be related to drinking – for example, via car owners having greater economic resources and more to lose from continued dependent drinking, or via towns with AA groups having a different drinking culture to those without. In this earlier study there was no ‘ceiling effect’ to restrict the influence of attending the meetings, thought in the featured study to have prevented the Project MATCH aftercare trial from finding an abstinence-promoting effect of AA attendance. The earlier study also found no extra abstinence due to attending more meetings, bolstering the impression that attending AA groups had little effect.

A second study which also used the ‘instrumental variable’ methodology instead recruited previously untreated alcoholics who had contacted the alcoholism treatment system via an information and referral or detoxification centre. Instead of abstinence as an outcome, it averaged the severity of drinking in each of the last six months of a one-year follow-up period, and related this to AA attendance in the previous six months. Drinkers selected for the analysis were those who (apart from detoxification) did not go on to start professional treatment, many of whom nevertheless attended AA groups. In this study the factors relied on to sift AA’s impacts from those of self-selection were how serious a problem the participant considered their drinking, a tendency to cope with problems by seeking information/advice, and the participant’s sex. As hoped, all three were related to whether the participant attended AA meetings, but not to the severity of their drinking as assessed at the one-year follow-up.

In contrast to the study described above, this second study found that using these factors to eliminate bias due to self-selection into AA doubled the strength of its association with reduced severity of drinking. In the first study, self-selection bias had magnified the apparent impact of attending AA groups, possibly because promising clients most engaged with the 12-step inpatient programme continued to engage with 12-step support on leaving. When this second study investigated an untreated sample, self-selection bias had instead obscured the magnitude of AA’s effect. Perhaps appreciating their difficulties, patients least likely to be able to avoid drinking chose AA rather than attempting to go it alone without treatment and without the support of a mutual aid group. As an aside, in the second study it is worth noting that while patients who attended AA groups were a year later much more likely to be abstinent (41% v. 16%) than those who chose neither treatment nor AA, there was no substantial or significant difference in their experience of drink-related problems.

Both these earlier instrumental variable analyses assessed the impacts of AA attendance versus non-attendance, not as in the featured study, the impacts of different degrees of attendance (including non-attendance).

Does encouraging AA involvement improve drinking outcomes?

Treatment services will perhaps be less interested in whether AA attendance promotes abstinence, than in whether intervening to encourage patients to attend AA will lead to more of what patient and service are trying to achieve. That result will be a product not only of the impact of attending groups – subject of the featured analysis – but also the likelihood that patients will attend. A finding that when a (former) patient attends AA they are more likely to be abstinent is of less practical value if very few patients can or will attend. That directs our attention to the AA-promoting interventions mounted as part of the patient’s treatment, which the featured analysis used only incidentally to help estimate the impact of attending the groups. The studies summarised below generally indicate that despite often substantially increasing 12-step group attendance, there have been no or only modest statistically significant impacts on drinking relative to alternative treatments. Some studies have found abstinence modestly boosted relative to alternative approaches, but generally without any extra impacts on heavy drinking or its adverse consequences. These results also cast doubt on the relationship between AA attendance and abstinence or less damaging drinking; if those links were strong and consistent, we would expect an intervention which promoted attendance also to beneficially affect drinking.

Varied results in six trials included in featured analysis

One way to approach this issue is to look at the six US randomised trials which the featured analysis used to generate its estimate that attending AA groups promoted abstinence from drinking. For treatment services, the more directly relevant question is whether this also means intervening to promote attendance will boost abstinence or other desired outcomes. In fact these six studies delivered mixed verdicts on the interventions they tested. Despite substantially increasing 12-step group attendance, significant impacts on drinking were absent or modest.

Summarising their results trial by trial, one found that intensifying efforts to encourage AA involvement increased abstinence but without affecting heavy drinking or its adverse consequences. Similar results emerged from another study, which found abstinence was bolstered by an intervention which effectively encouraged AA involvement compared to case management, but without any extra impacts on heavy drinking or its adverse consequences. A further study (1 2 3) tested whether outcomes were improved by supplementing alcohol-focused behavioural couples therapy with relapse prevention techniques or interventions encouraging involvement in Alcoholics Anonymous. With minor and varying exceptions, neither supplement further increased abstinence or marital happiness. Equality of impact was also the dominant impression in the two arms of the Project MATCH study (1 2 3), though a 12-step based therapy encouraging AA involvement did at times generate slightly more abstinence and less drinking than alternative therapies, effects the researchers thought inconsistent and not clinically meaningful. A final study (1 2) was of cocaine-dependent patients and no alcohol-related results have been published. For more on these studies unfold Unfold supplementary text the supplementary text.

Other similar studies including English trial

To the studies above can be added one conducted in England. It recruited 151 alcohol or drug dependent patients admitted for a 10–14-day inpatient detoxification on a specialist NHS ward in London. Nearly 57% were being detoxified primarily from alcohol and 38% opiates. Patients were randomly allocated to three different referral procedures.

% interviewed patients attended groups or abstinent

The control group were given a list of the 12-step meetings to be held on the ward, and just before they left the unit they were also given a list of meetings in their home areas and offered 12-step literature. Additional to these steps, another set of patients saw a doctor trained by the researchers to systematically encourage 12-step group attendance. A third set of patients were offered a similar discussion, but this time with an active member of a 12-step mutual aid group. Abstinence from the substance in relation to which they were being treated was adopted as the indicator of a positive outcome. The general picture ( chart) was that being encouraged by either a doctor or a peer (especially the latter) substantially improved attendance at 12-step groups, but resulting impacts on abstinence were much smaller and not statistically significant. For example, compared to the control procedure, peer-referral doubled 12-step group attendance but the difference in the abstinence rate (as assessed on average two and a half months later) was a statistically insignificant 44% versus 36%. Though falling short of statistical significance, the small increase in the proportions abstinent might (if further studies suggest it was not a chance finding) be considered clinically worthwhile, especially if associated with broader reductions in substance use and wider quality of life gains. As the authors suggest, it might also have been a harbinger of greater long-term improvements.

A similar pattern of findings – increased attendance but less or no significant impact on drinking – have emerged from other studies, though a US study did find both attendance and substance use and problems significantly reduced after intensive referral to 12-step groups. For more on these studies unfold Unfold supplementary text the supplementary text.

Thanks for their comments on this entry in draft to research author Professor Keith Humphreys of Stanford University in the USA, and David Best of Sheffield Hallam University in England. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 28 February 2017. First uploaded 15 February 2017

Comment/query
Open Effectiveness Bank home page


Top 10 most closely related documents on this site. For more try a subject or free text search

REVIEW 2015 Risks and benefits of nalmefene in the treatment of adult alcohol dependence: a systematic literature review and meta-analysis of published and unpublished double-blind randomized controlled trials

STUDY 2012 Does active referral by a doctor or 12-step peer improve 12-step meeting attendance? Results from a pilot randomised control trial

MATRIX CELL 2020 Alcohol Treatment Matrix cell E4: Treatment systems; Psychosocial therapies

STUDY 2007 A randomized controlled trial of intensive referral to 12-step self-help groups: one-year outcomes

STUDY 2009 Changing network support for drinking: Network Support Project 2-year follow-up

HOT TOPIC 2018 Can 12-step mutual aid bridge recovery resources deficit?

REVIEW 2013 Metaanalysis of naltrexone and acamprosate for treating alcohol use disorders: when are these medications most helpful?

HOT TOPIC 2021 ‘Dangerous data’: drinking after dependence

MATRIX CELL 2020 Alcohol Treatment Matrix cell A4: Interventions; Psychosocial therapies

MATRIX CELL 2020 Drug Treatment Matrix cell E4: Treatment systems; Psychosocial therapies