Send email for updates

About updates
Research analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text The Summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Title and link for copying Comment/query to editor

Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes?

Harris A.H.S , Humphreys K., Bowe T. et al.
Journal of Behavioral Health Services and Research: 2010, 37(1), p. 25–39.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Harris at You could also try this alternative source.

This first major multi-modality test of a treatment engagement indicator widely used as a quality control yardstick in the USA found it was only very weakly related to patient improvement seven months after starting treatment, confirmation that simple measures of what happens during treatment struggle to capture what really makes treatment effective.

Summary Quality indicators constructed from administrative data such as patient attendance or staffing are inexpensive and easy to generate but are often of low or unknown validity in terms of being related to patient outcomes; in other words, they may be feasible to collect but not valid as indicators of performance.

To measure the quality of substance use disorder treatment services, the Washington Circle An organisation supported by the US health service's Center for Substance Abuse Treatment. developed initiation and engagement measures. The National Committee for Quality Assurance (NCQA), a non-profit organisation dedicated to improving the quality of health care, incorporated these measures in to the Health Plan Employer Data and Information Set (HEDIS). Because HEDIS is the most widely used set of quality measures in the US managed health care industry, many health care systems are now tracking these measures.

The measures are:
Initiation The percentage of known substance use disorder patients who after being out of treatment for at least two months 60 days. then attend Either a residential admission or an outpatient visit. for substance use disorder treatment and re-attend within a fortnight.
Engagement The percentage of known substance use disorder patients who within a month 30 days. of initiating treatment as defined above attend at least another two times; for people who have initiated residential or inpatient care, the two visits must be within a month of their being discharged from the residential setting – effectively a measure of continuing care or aftercare.

The featured study drew its data from a nationwide sample Up to 50 new patients were randomly selected from each programme in a randomly selected and representative sample of Veterans Affairs substance use disorder treatment programmes. of 5723 patients who had entered treatment at 118 inpatient, residential and outpatient programmes run by the US health service for former military personnel. All but a few initiated treatment, making it difficult to reach conclusions about initiation v. non-initiation, so the featured study focused on engagement.

The study focused further on new patients who may or may not go on to engage with treatment as defined by HEDIS, not those already in treatment for some time, so based its analyses on the 2789 who had started treatment within the past fortnight after being out of treatment for at least two months. Almost exclusively men, they averaged 48 years of age and at baseline were experiencing significant medical and employment problems. They were more likely to be suffering problems or severer problems related to their drinking than to use of other drugs.

Of these patients, 1820 or about two thirds provided follow-up data on average just over seven months later. The scores missing patients Missing patients tended to be younger, were less likely to be homeless, and had less severe drug and employment problems but more severe psychiatric and legal problems as measured by the Addiction Severity Index. were likely to have supplied had they been contacted were estimated from what was known about them and the data they had provided initially.

Primarily at issue was whether patients who engaged as defined by HEDIS had more favourable trends in their alcohol, drug and legal problems than patients who did not engage. If they did, this would be consistent with engagement truly being indicative of more effective treatment.

Main findings

Generally the picture was that patients who met the HEDIS engagement criterion did experience greater remission in their problems than those who did not, but only if they started their treatment as outpatients, the relationships were very weak, and in terms of concrete outcomes like abstinence, not statistically significant. Details below.

Patients who subsequently engaged in treatment were more likely to be African American and homeless and less likely to be Caucasian than those who did not engage. They had also reported more severe alcohol, psychiatric, and legal problems. Such differences might have affected outcomes regardless of how well they engaged with treatment, so as well as analysing the 'raw' figures, further analyses adjusted for differences between engagers and non-engagers; these adjusted figures are reported below.

Problem scores assessed for the past month by the Addiction Severity Index questionnaire could vary from 0 to 1 at their most severe. At first for drinking they averaged about 0.4. On average this score halved over the seven months of the follow-up. Patients who had engaged with treatment experienced a small but statistically significant further reduction of 0.03 points. When the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, only for the outpatients did the further problem reduction among engagers (0.06 points) remain statistically significant. Though the trend for engagers to do better remained, it was no longer statistically significant when the sample was limited to people actually diagnosed with alcohol problems, or when the outcomes were alcohol abstinence or changes in the number of days on which patients got drunk.

Similarly for problems with drugs other than alcohol. At first problem scores averaged 0.18. On average this score too halved over the seven months of the follow-up. Patients who had engaged with treatment experienced a small but statistically significant further reduction of 0.02 points. Though the trend for engagers to do better remained, it was no longer statistically significant when the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, when it was limited to people actually diagnosed with drug problems, or when the outcome was abstinence from drug use.

Legal problems too on average remitted somewhat from 0.19 to about 0.14. Patients who had engaged with treatment experienced a statistically significant further reduction of 0.04 points. When the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, only for the outpatients did the further problem reduction among engagers (0.04 points) remain statistically significant.

These changes in composite measures of different problem domains were extrapolated to what they might mean for more concrete measures. For example, it would be expected that on average patients who did versus did not engage would be drunk under one day (0.62 days) fewer a month, and would be marginally more likely to be abstinent from drugs (increase from 77.5% to 77.8%) and less likely to be in prison (decrease from 4.8% to 4.6%).

The authors' conclusions

The HEDIS engagement indicator has been widely adopted, but largely in the absence of evidence linking it to patient outcomes. This is the first study to examine the strength of the association between meeting this criterion and patient-level changes in alcohol, drug, and legal symptoms in a nationwide health care system that includes both outpatients and patients treated in inpatient–residential programmes. It found these symptoms did improve more among patients starting treatment at facilities for ex-military personnel who met the engagement criterion, particularly in outpatient settings. Though statistically significant, the extra improvements were clinically modest.

It is important to remember that these results related each individual patient's engagement to remission in their problems. They do not necessarily mean that programmes which on average engage a greater proportion of their treatment starters also on average have better outcomes. In fact this was not the case when almost the same data set was analysed by programme instead of by patient. Together these finding mean that in the featured health care system, the engagement measure adopted by HEDIS modestly predicts which individuals will improve most, but not which programmes generate the greatest improvement across their caseloads. Appreciating this should temper enthusiasm for using these measures in pay-for-performance systems or to choose the most effective treatment facility.

Some further limitations of the study should be noted. The modest relationships found between engagement and outcomes may mean that engagement causes better outcomes, but may also mean that variables not adjusted for in the study (such as the motivations of the patients, their family support, or their involvement in mutual aid groups) generated both better outcomes and deeper engagement. In this latter scenario, engagement would have simply been a non-active by-product of the factors which really generated better outcomes. Engagement was not related to outcomes in residential/inpatient programmes, perhaps because the measure adopted by HEDIS does not reflect engagement with the core treatment, but with follow-on treatment, which (for example) patients may not attend simply because it is hard to access, regardless of their progress. Finally, these results were obtained from a very particular health care system with a distinct caseload and relatively well integrated services. Even then they reflect the engagement-outcome relationship only among patients not in treatment for at least two months and at the start of a new episode of care.

Findings logo commentary Findings of this study confirm a common conclusion: that what makes people come back to a treatment service is not necessarily what makes treatment effective or which leads to the desired changes in substance use. They may overlap, but sometimes not, and sometimes only very little, as in the featured study. Retention or attendance are just two dimensions (the most easily measured) of engagement. Sometimes deepened engagement may actually shorten retention because clients are ready to leave sooner. But generally retention is a sign that clients are actively 'working the programme', attending counselling sessions, talking about the things that matter, forging a therapeutic relationship with their counsellor and/or other clients, getting extra help if needed. Arguably it is what is done during the retained period and during the attendances which makes the difference to outcomes rather than merely attending. Treatment as a whole may also be a relatively minor factor in what patients at least see as contributing to their recovery. The main exception is prescribing substitute drugs like methadone to opiate addicted patients, a modality where staying in treatment is indeed the key to its success.

The featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They observed that incentives to meet targets based on criteria like the HEDIS engagement indicator can lead to dramatic improvements in hitting these targets, yet achieving these targets can bear at best a weak relationship to subsequent patient outcomes. Instead they favoured schemes based on the patients' actual substance use (or other direct measures of progress) assessed during treatment using objective techniques such as urine tests and processed in such a way that the results have consequences for the treatment provider.

Despite the failure of the HEDIS engagement indicator to predict the effectiveness of the programme and only very modestly the progress of an individual patient, other measures tested in the same health care system have proved a better option. In a study confined to alcohol-related outcomes at outpatient programmes – the combination for which the featured study found engagement most closely related to outcomes – the strongest indicator of which agencies had the best average outcomes was the proportion of their patients who attended at least three times in the first month, though how many attended at least twice, four, five or six times were not far behind. These indicators accounted for about a quarter of the variation between agencies in how well their patients did, a substantial relationship. The impetus for the study was the poor performance of an indicator – three-month retention – very similar to the 12-week retention indicator used recently as a benchmark for British drug dependence treatment services, though in relation mainly to the treatment of opiate-addicted patients.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover. Commissioners of services have been offered guidance from the National Treatment Agency for Substance Misuse, England's special health authority tasked to improve the availability, capacity and effectiveness of drug misuse treatment.

Thanks for their comments on this entry in draft to Alex Harris of the US Veterans Affairs health care system. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 28 September 2012

Comment/query to editor
Give us your feedback on the site (one-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates

Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2011 Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes

REVIEW 2017 An evidence review of the outcomes that can be expected of drug misuse treatment in England

STUDY 2009 The Drug Treatment Outcomes Research Study (DTORS): final outcomes report

STUDY 2010 Gender differences in client-provider relationship as active ingredient in substance abuse treatment

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

REVIEW 2009 Continuing care research: what we have learned and where we are going

STUDY 2010 A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers

STUDY 2003 Systematic but simple way to determine who needs residential care

STUDY 2012 Advancing recovery: implementing evidence-based treatment for substance use disorders at the systems level

STUDY 2008 Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment