A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers.

Crits-Christoph P., Ring-Kurtz S., McClure Bridget. et al.
Journal of Substance Abuse Treatment: 2010, 38, p. 251–262.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Crits-Christoph at crits@mail.med.upenn.edu. You could also try this alternative source.

It should have improved relationships between counsellors and clients and between counsellors and their clinics, but a sophisticated system for feeding back client progress made no positive difference on any of these measures. A later study helped establish why: counsellors were not told which patient the feedback related to and what they might do about it.

Summary This US study tested a performance improvement system for group therapy counsellors treating substance use problems, based on offering regular feedback on their patients' progress, in particular their assessment of their relationship with the counsellor (in the form of the therapeutic alliance) and their satisfaction with treatment. Therapeutic alliance was measured because it is related to the outcomes of psychotherapy and counselling in general, and substance use counselling in particular. The patient rated their bond with the counsellor, agreement on tasks, agreement on goals, and the degree to which the counsellor understood them.

This data plus attendance records was collected weekly from all patients in groups run by participating clinicians, regardless of the how long they had been in treatment. Turnover of clients meant that each weekly feedback report could reflect the scores of different individuals. In ways which hid the identity of individual patients and also the identity of individual counsellors from their supervisors, the information was analysed by researchers and made available over the internet to the counsellors as the average responses of the patients they were treating in group therapy sessions as a whole, of those who had been in treatment for different periods, of men v. women, and of different ethnicities. Also available to both supervisors and counsellors was this information aggregated for the entire clinic. As well as absolute scores, the feedback charted trends over the 12 weeks of the study.

Counsellors and supervisors were trained and offered written guidance on how to interpret this information. They were also encouraged to review the clinic report as a team at monthly meetings, using it to stimulate discussion and set goals for improvement. Counsellors were encouraged to examine their own caseload reports. If they wanted, they could discuss these with other counsellors and/or their supervisors. Supervisors could use reports for the clinic as a whole to inform decisions about training, supervision, and resource distribution. The clinic was financially rewarded each time a caseload report was newly downloaded and the counsellor told how much they had earned for their workplace.

Each month the researchers also sent an electronic newsletter to staff to share experiences between the clinics, highlight their innovations and accomplishments, and encourage evidence-based practices.

The system was expected to improve attendance and substance use outcomes by giving counsellors the chance to adjust in the light of the degree to which on average their patients felt they were working well together and were satisfied with the treatment. It might also, it was thought, influence counsellors by signalling that these variables were important to their employers, and by enhancing their satisfaction with the clinic and their views of their employer as progressive and committed to staff development and performance improvement.

For 12 weeks, 20 community-based substance abuse counselling clinics were randomly allocated to implement this feedback system, or simply to collect baseline and study-end data from patients, which was not fed back to staff. Group counselling was the primary modality at all the clinics. Counsellors running groups at least weekly were asked to join the study. Of 123, 118 agreed and all but 20 completed the study.

The patient feedback form also asked for the number of days over the past week that the patient had drunk alcohol or taken drugs, data withheld from the staff, but which constituted the primary yardsticks against which the impact of the performance improvement system was assessed. They key issue was whether patients at the feedback clinics reduced their substance use from baseline to the final 12-week assessment more than those at clinics not in the feedback system.

Main findings

One problem in assessing improvements was that already at the start of the study over three quarters of patients said they had been totally abstinent. To create more room for improvement, the main analyses were conducted on new patients in treatment for a month or less, more of whom would it was thought still be drinking or taking drugs. Still no significant differences were found between trends among patients at feedback and non-feedback clinics. For example, at feedback clinics, from 82% not drinking this rose only slightly to 85%; at non-feedback clinics it remained virtually unchanged at about 75%. For drugs the corresponding figures were 90% and 87%, and 76% and 80%. Similar analyses for all patients regardless of treatment duration also found high levels of abstinence which changed little over the 12 weeks, and no more so at feedback clinics.

Neither were there any significant differences when the analysis was restricted to patients who contributed data at baseline and at 12 weeks, meaning the same individuals were tracked, or when the few 'outlier' clinics with unusually high or low baseline abstinence rates scores were excluded.

Similarly, patients as a whole and the tracked individuals did not develop a deeper alliance with their therapists at services offered feedback, and attendance too was no better.

Turning to the staff, those in services which offered feedback on their performance did not feel a better working relationship had developed between them and their supervisors than in comparison services, nor did job satisfaction differentially improve. On the measures of how prepared staff thought their organisation was to make positive changes, the sole statistically significant difference was that resources like office space and internet access had (relative to other services) actually worsened in the feedback services.

The authors' conclusions

This study found no evidence that the performance feedback system led to the expected improvements in drug or alcohol use, therapeutic alliance, patient attendance, or clinicians' views of their jobs and workplaces. Several factors could explain this lack of impact.

In respect of drug/alcohol use there was little room for improvement by any system. Patients often start treatment abstinent after a legal problem or are mandated to treatment (eg, for drink-driving) even though not dependent. Most can readily stop using (especially after a 'binge') for brief periods; treatment's role is to sustain this. Though also near its ceiling, therapeutic alliance could still have improved, as even more so could attendance. Yet in both cases feedback did not help, suggesting that it was indeed ineffective.

Assuming it was ineffective, one possible reason may have been that administrators and supervisors primarily responsibility for performance could not identify which clinicians were performing relatively poorly, key information for targeting improvement measures. Clinicians too were deprived of what for them may be key information – which individuals were not doing well and might benefit from a change in approach. As long as their job is not threatened, clinicians' motivation for general performance improvements may be marginal, especially among highly experienced counsellors who believe they already perform at an advanced level; without a strong motivation to improve one's performance, clinician-level feedback may not be effective.

Also, team meetings were relied on to generate improvement suggestions in response to feedback. Other mechanisms might have been more successful, such as one-on-one supervisory feedback, independent testing or consultation on difficult cases, or supplementing feedback reports with guidance on what they might mean for practice.

Findings logo commentary The featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They took its findings as an instance of the more general finding that "if there is no risk of reputational damage, information on one's individual or organizational performance is generally disregarded". Only if this information is made public in circles that matter to the clinician (like their employer) or to the organisation (like commissioners and prospective patients) does it exert leverage. They favoured schemes based on the patients' actual substance use (or other direct measure of progress) as assessed during treatment using objective techniques such as urine tests and processed in such a way that the results have consequences for the treatment provider.

Why no impact?

Effectively the featured system reported on the therapist's performance but in ways which had no consequences for them. What it did not do was report on the progress of their individual patients in ways which enabled them to intervene. The researchers' belief that this was a major reason for its lack of impact seemed confirmed by a later study led by the same researchers. In this study, not only did the feedback identify the individual patient, it also reported on their psychiatric wellbeing and functioning and their actual drug and alcohol use. Additionally, it categorised the patient according to how their progress compared to that expected for similar patients, signalling whether they were doing so well that treatment might be ended, were progressing as expected, or were progressing less well than expected. For the latter, the therapist was instructed to consider changing the treatment and given guidance based on a second questionnaire on barriers to progress in the form of problems with therapeutic alliance, poor motivation, inadequate social support, and stressful life events. Given these enhancements, the feedback system did in this case significantly and positively affect all three primary outcomes (drug use, drinking, total problem scores) among the 'off-track' patients who were not progressing as expected, the ones targeted by the system.

This interpretation is in line with general psychotherapy studies of the same system reviewed for the American Psychological Association. The reviewers concluded that real-time feedback to therapists enabling them to monitor patients' responses to psychotherapy and satisfaction with the therapy relationship probably improves psychotherapy outcomes overall, and certainly for clients at risk of deterioration or drop-out – the 'off-track' patients of the later study. They were unsure of the benefits of sharing this feedback with patients unless this could (as in the later study) be done at the therapist's discretion and reframed as they saw fit. Also commended were the additional strategy deployed also in the later study of supplementing feedback with information from off-track patients indicating why things were going wrong and guidance from the feedback system on what therapists might do about it.

The feasibility study which showed that the system tested in the featured study could be implemented exposed one further reason why it might not have worked: with no consequences for doing this or not, and no way to relate the feedback to the individuals they were seeing, the typical clinician accessed their feedback reports only 2.3 times (out of a maximum of eight) over the course of the study.

Particularly disappointing in the featured study was that therapeutic alliance – though a major item in the feedback – did not improve as a result of this feedback. Additional to the factors mentioned by the authors, it is perhaps relevant that group therapy was the modality and that from week to week the therapists may have been seeing different patients. It seems probable that this limited the degree to which they could respond to an individual in the group who felt somewhat alienated from the counsellor, and also that even if they did, this might have less effect than in one-to-one therapy, because much depends on the atmosphere created by the other patients in the group.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover.

Thanks for their comments on this entry in draft to Paul Crits-Christoph of the University of Pennsylvania Medical School in the USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 03 October 2012

Open Effectiveness Bank home page

Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

STUDY 2011 Therapist effectiveness: implications for accountability and patient care

REVIEW 2018 Collecting and delivering progress feedback: a meta-analysis of routine outcome monitoring

STUDY 2009 Relating counselor attributes to client engagement in England

STUDY 2009 The alliance in motivational enhancement therapy and counseling as usual for substance use problems

ABSTRACT 2011 Evidence-based therapy relationships: research conclusions and clinical practices

STUDY 2010 Gender differences in client-provider relationship as active ingredient in substance abuse treatment

DOCUMENT 2013 Sometimes best to break the rules

REVIEW 2011 Integration of treatment innovation planning and implementation: strategic process models and organizational challenges

STUDY 2005 How does motivational interviewing work? Therapist skill predicts client involvement within motivational interviewing sessions