Drug and Alcohol Findings home page in a new window EFFECTIVENESS BANK BULLETIN 8 October 2012

The entries below are our accounts of documents collected by Drug and Alcohol Findings as relevant to improving outcomes from drug or alcohol interventions in the UK. The original documents were not published by Findings; click on the Titles to obtain copies. Free reprints may also be available from the authors. If displayed, click prepared e-mail to adapt the pre-prepared e-mail message or compose your own message. The Summary is intended to convey the findings and views expressed in the document. Below may be a commentary from Drug and Alcohol Findings.


Contents

How to generate better outcomes from treatment systems

Starting point of this evidence-based primer on how to – and how not to – implement treatment improvements was a valuable review (first entry) from two leading US scholars with top-level UK experience. Tactics considered included incentives for outcomes and evidence-based practices, regulations, credentialing requirements, and putting the money in the hands of the patient. Next entries are some of the most relevant of the recent studies cited by the review plus a later follow-up study. In the process a research-informed eye is cast over UK payment-by-results schemes.

Expert advice on how to get the treatment system working better ...

Global performance feedback no use to counsellors ...

Feedback on individuals helps counsellors reverse poor progress ...

Common treatment quality yardstick poor reflection of patient progress ...

US payment-by-results scheme did not improve engagement with treatment ...

Acid test for implementing evidence-based treatment ...


A policy-oriented review of strategies for improving the outcomes of services for substance use disorder patients.

Humphreys K., McLellan T.
Addiction: 2011, 106, p. 2058–2066.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Humphreys at knh@stanford.edu. You could also try this alternative source.

Improving performance of substance use disorder treatment systems is no easy matter and one prone to unintended consequences. All the more welcome then is guidance from leading US experts with top-level experience in the UK and the USA; their favourite tactic, rewarding services for patient progress during treatment, is featured in UK payment-by-results schemes.

Summary Two US experts who worked at top levels on drug policy for the current US president and have advised the UK government draw on scientific literature and their experiences to offer policy makers an overview of system-level strategies to improve the outcomes of services for substance use disorder patients. Their aim was to stimulate discussion rather than conduct a systematic and comprehensive review of the evidence.

Though they may be used together, the review divides the strategies in to process-focused quality improvement strategies which change how treatment staff work in the expectation that this will improve patient outcomes, and patient-focused strategies which concentrate on outcomes without specifying how those are to be achieved.

Process-focused quality improvement strategies

Substantial deficits in the quality of substance use disorder care in the USA and the UK include an excess of paperwork, insufficient time with patients, demoralised staff, lack of medically trained staff, dysfunctional organisational dynamics, and underuse of scientific evidence. Strategies under this heading seek to change organisational, financial and clinical practices to redress such deficits. These improvements will it is hoped translate into better patient outcomes, but this cannot be assumed. In general, the longer the time between receipt of services and outcome measurement, the less the outcome can be taken as indicative of the quality of those services.

Increasing licensure/credentialing requirements

Most US addiction treatment programmes have no physicians or nurses and the primary credential of many counsellors (to an extent also in the UK) is themselves being in recovery. It therefore seems logical that increasing credentialing requirements for staff will improve quality.

However, such stiffening would exacerbate staff recruitment and retention problems unless accompanied by inducements to make a career in the field more attractive. Also, credentialing has become an entrepreneurial activity; countless organisations develop and sell credentials which do not always assure quality. Finally, the evidence from hundreds of clinical trials is that counselling outcomes are not predicted at all by the counsellor's type or level of educational degree; a prestigious degree in medicine, for example, does not ensure that a physician can make an emotional connection to an addicted homeless patient or an alcoholic soldier suffering post-traumatic stress disorder.

Despite these caveats, legitimate credentialing and licensure could help weed out unsuitable individuals, some credentials are essential to clinical activities such as prescribing medication or taking blood, and more highly educated staff are particularly receptive to evidence-based practices.

Beyond these rather gross indicators of value, new credentialing policies are a weak lever for improving addiction service outcomes, though one which might be stronger if focused on specific clinical competencies rather than coursework or hours of training.

Measuring and/or incentivising evidence-based clinical practices

Evidence-based clinical practices which have been incentivised in the USA include screening primary care patients for drinking problems and retaining patients in specialist treatment for at least three months. US health-care systems have also begun to monitor the proportion of patients whose substance use disorder is identified, the proportion who engage early in care, and the proportion retained in care over time.

Such incentives can lead to dramatic improvements in hitting these targets. However, achieving them can bear at best a weak relationship to subsequent patient outcomes (see for example this Findings analysis), shown among others by US studies of targets like the three-month retention measure also adopted in the UK. Research has shown that treatment providers respond to incentives (important in itself), but has yet to clarify which provider practices should be changed to improve patient outcomes.

Another issue is whether anyone other than the providers themselves should be made aware of how closely they follow evidence-based guidelines. A recent trial found that giving treatment services feedback on their patients' progress had no effect on clinical performance, confirming that if there are no consequences from doing so, information on one's performance is generally disregarded. Performance assessments are best made public if the goal is to improve outcomes.

Improving managerial capacity and business practices

This strategy involves expert business consultants helping treatment services improve their management skills, knowledge and capacity. The highest-profile US efforts are the Network for the Improvement of Addiction Treatment (NIATx) and the follow-on Advancing Recovery project.

An intriguing strategy employed by NIATx is the 'walkthrough'. Managers attempt to access care in their own services from the patient's point of view – for many, an eye-opening experience which helped explain low rates of treatment entry and retention. Other management practices taught in these initiatives include careful analysis and allocation of funding and better development of a business case for new funding. In response, services reduced waiting times and increased retention, and these gains persisted after intensive organisational consultation had ended.

The Advancing Recovery project generally improved continuity of care and use of evidence-based pharmacotherapies. However, both initiatives probably attracted better-organised and led and more motivated services; effects are likely be less dramatic if such initiatives are applied to all services. Also, as yet changes in care processes have not been related to long-term patient outcomes.

Embedding substance use disorder care in a higher quality care network

An example of this strategy is the Obama administration's policy of medicalising care for substance use disorders. Through funding for screening and brief intervention in primary care settings and changes in public and private insurance, it has begun transplanting substance use care into general health care. This brings with it features which may improve patient outcomes, such as medically trained staff, availability of medications, financial incentives for quality, electronic health record monitoring of patients, and a broad culture of careful inspections and monitoring. Co-location should also facilitate access to supplemental medical services. Finally, integrated care coordinated by one's usual GP may be more accessible and less stigmatising than going to an 'addiction treatment programme', meaning it may also be possible to engage lower severity and more manageable patients.

However, this does not guarantee quality improvement. Medical care systems could re-allocate resources intended for substance use disorder care, and patients may be treated by less knowledgeable practitioners. Integration remains a promising idea in search of rigorous evidence rather than something which can be assumed to work.

Patient-focused strategies

Another class of strategies focus on the patient's actual outcomes rather than organisational or clinical practices. From this perspective, some of quality improvement strategies described above may be criticised for not focusing on what ultimately matters most, a point strengthened by research showing how changing care processes has often not translated into better patient outcomes and can have negative side effects. The patient-focused approaches described under this heading have been implemented less frequently in the addictions than process-focused strategies, so examples are drawn from treatment of other disorders.

Rewarding post-treatment outcomes

In some health care sectors providers are directly incentivised to produce specific long-term patient outcomes. An example from substance use treatment is the payment-by-results programme launched by the UK government, the first in this sector.

One issue is that what happens to a chronically ill individual after treatment becomes less closely linked to care quality over time. Measuring outcomes too long after treatment could demoralise providers held accountable for things over which they have little control. Another risk is that providers will exclude patients with a poor prognosis. Also, the cost of re-assessing patients after treatment can be considerable. When this task is assigned to clinicians, follow-up rates and data are poor and time is diverted from treating patients. It is better to use an independent outcomes monitoring team, but this requires a continuing resource commitment.

Rewarding in-treatment performance

Another approach is to reward services for outcomes attained during treatment. This resolves several problems with the previous strategy: finding the patient is easier, and what the provider does should bear a stronger relation to how the patient is doing. Of the outcome improvement approaches described in this paper, this is among the most promising and feasible, not only because it could improve care, but because it focuses clinicians' attention on something they can and should be responsible for throughout the care process.

One such experiment in the US state of Maine initially generated enthusiasm. Services were funded for raising the proportion of patients who by their final contact with the service had achieved outcomes such as abstinence or major reduction in substance use. Performance appeared to improve, but this seemed due to services treating fewer severely troubled clients. Patients may also have been less candid when their accounts of their substance use affected their clinician's income. The system was updated but the most recent evaluation again yielded disappointing results.

It is not necessarily that this approach cannot work, but that performance contracts need to reward objective outcomes such as urine test results and adjust these rewards in the light of the service's case-mix. Such a system has been trialled in US methadone clinics which were informed of their performance, but with no financial or reputational consequences attached, there is no evidence that this changed clinical practice or improved patient outcomes.

Some clinicians might say such approaches are not feasible because they require regular monitoring of patients' substance use. However, this should happen anyway as good clinical practice. Research is, however, still needed to establish the strength of the relationship between in-treatment progress and longer term, post-treatment outcomes.

Rewarding patients for attaining particular outcomes

Individuals with substance use disorders do respond to incentives and sanctions such as short jail terms or housing linked to positive drug/alcohol tests, or those given by contingency management treatment programmes which systematically levy material rewards or privileges for abstinence or other outcomes.

Paying patients can however be resisted by the public and sometimes care providers too on the basis for example that 'They ought to change for free like everyone else'. Non-financial rewards and stressing the public benefits of these schemes (such as safer neighbourhoods) can help. Changes induced by external rewards sometimes reverse once rewards end. Such schemes may however be useful in the early stages of treatment to promote progress and engagement with care.

Making the patient a customer with purchasing power

A radically different approach tried in the USA gave individuals early in the recovery process vouchers to buy whatever services they thought would aid their recovery. Examples included college classes, transport to work or to self-help group meetings, housing, dental care, work training and clothes for job interviews. In theory this should generate improvements via the mechanisms that drive efficiency and quality in commercial markets. Access to Recovery, as the programme is called, expanded both the number of organisations providing services and the number of recipients. More importantly, outpatients in Washington state who received vouchers stayed longer in treatment and were more likely to be employed than comparison patients.

'Personal health budgets' being piloted for health and social care in the NHS are analogous. In consultation with a health professional, chronically ill individuals are given a fixed pool of funds from which they can buy a range of services, assembling a care package tailored to their needs. Extension to the addictions has been discussed within government, but it is not clear whether this will be tried.

The authors' conclusions

Strategies policy makers can use to improve outcomes of substance use disorder treatment are often poorly developed with weak empirical support. Incentives for particular clinical practices can change what systems do, but it is less clear which changes translate into better patient outcomes; some such schemes have proved literally worse than doing nothing. Cases discussed in this paper in which care utilisation was incentivised, but outcomes did not improve, mean money was spent on care that was apparently not needed, which may have adversely affected other patients due to the diversion of resources.

Despite a small evidence base, bringing market forces targeting quality and effectiveness to bear on treatment systems – including directly rewarding outcomes – has significant practical and logical appeal. Particularly promising are initiatives focused on in-treatment performance rather than long-term post-treatment outcomes. Also promising are vouchers that give patients purchasing power for their chosen services, hopefully to be extended to substance use disorders, perhaps via the British health service's personal health budgets.

When implementing such programmes policy makers serve themselves and the field if they embed careful, realistic evaluations from the start. The required investment and level of collaboration are substantial, but justified by the potential payoff in knowledge about how to enhance care for life-threatening disorders.


Findings logo commentary This advice from two of the world's most respected addiction researchers, each also with top-level experience in government drug policy, usefully and lucidly sets out the options for making improvements to treatment systems. Exactly what those improvements are intended to achieve was left open, so its findings should apply to the ambitions of recovery-focused UK national strategies.

Realistically they observe that quality improvement initiatives often seem to bear little relationship to what really improves outcomes for the patient and society, because outcomes do not in fact improve at all or very much. Treatment is a relationship business; mechanistic pulling of levers may shift the gears but not touch the heart of the matter, resulting in surface changes which do not last, a common experience in contingency management programmes which reward and sanction patients for desired behaviour or non-behaviour.

As the authors explain, in turn this means that credentials and training courses completed may mean little when it comes to forging engagement- and outcome-enhancing relationships. Studies can, for example, isolate personality variables which when high or low mean that on average patients respond best to certain types of counselling styles. However, the complexity of each individual means that what is indicated for one aspect of their situation or character may be counter-indicated by another, leaving the sensitivity of the therapist to sort out the best approach. Tying their hands too tightly through detailed treatment manuals (no matter how expert the author) and supervision to ensure they stick to the script has in the case of motivational interviewing led to worse outcomes, not better.

One implication the reviewers do not touch on is that without appropriate recruitment, much of the quality improvement effort put in to training and supervision will be wasted. This was the message which emerged from a study of motivational interviewing training which found that initial gains in skills had waned two months later. However, this was not the case for the addiction and mental health clinicians who, even before training, had been more proficient than the other trainees would be after training. Not only did these 'natural experts' start from a higher level, they went on to absorb and retain more of what they had learned.

UK payment by results schemes

Though featured under schemes which reward post-treatment outcomes, nationally agreed outcomes for pilot payment by results schemes in England often specify in-treatment and treatment exit measures, placing them partly within the during-treatment payment schemes favoured by the featured review. One of the problems it identifies with post-treatment payment schemes – the resources needed to re-contact and reassess patients – is sidestepped by using routinely collected criminal justice and treatment records which do not require contact with the patient. Reports from the pilots suggest these schemes can both be feasible and generate innovations focused sharply on achieving results.

The British schemes attempt to balance in-treatment and treatment exit measures against longer term crime and relapse indicators, generally choosing to place greater financial weight on the longer term. This means services must wait many months – in respect of some measures nearly two years – to receive much of their funding under the scheme, a cash-flow problem which requires counter-measures if services are to survive. Arguably too, as the featured reviewers comment, services are placed at financial risk for outcomes over which they may have little control because they are so far from the time when they had direct influence over the patient.

The alternative of weighting in favour of during treatment and treatment end measures falls foul of the observation in the featured report that these are often loosely related to the longer term recovery the system is trying to generate. For example, 'successfully' completing treatment free of dependence and of opiate and crack cocaine use did mean that over the next four years more patients in England appeared to have avoided relapse, but the difference of 57% versus 43% who did not successfully complete was not as large as would be expected if successful completion correlated strongly with lasting recovery.

Such schemes are classed by the reviewers as "patient-focused strategies" – not to be confused with patient-centred practice in the sense of basing treatment objectives and methods on the patient's preferences and priorities. Being patient-focused in this sense is threatened by payment by results schemes because these pre-set the treatment destination in detail without reference to what the individual patient wants, and in a way services cannot afford to ignore because their financial survival depends on meeting the criteria for payment. Local schemes could still create a space for the patient's ambitions in their payment criteria, but this is not a required element or one included in the national outcomes schema, nor one which sits easily within a system predicated on observable outcomes the public and their representatives recognise and are willing to pay for.

As a commentator on the featured review pointed out, schemes which make payment contingent on either process or outcomes entail a regulatory overhead which could eat in to whatever efficiencies are achieved at the services concerned. For example, the reviewers rightly caution that "performance contracts need to reward outcomes that are objective (eg, urine testing) and case-mix-adjusted". This means some authority has to assess the severity of the caseload in terms of the resources needed to achieve the intended outcomes, and assess whether those outcomes have been achieved in ways which go beyond merely asking the patient. When funding, jobs and organisational survival ride on these assessments, leaving them entirely to the people and organisations at threat may stretch their integrity too far. This concern spawns new regulatory requirements and possibly new regulatory bodies which must, as the Audit Commission described, be capable of overcoming complexity to deliver valid and meaningful measures if disputes, demoralisation and wastage are to be avoided.

In UK payment by results schemes the most visible result has been the setting up of central assessment centres (or LASARS), which have a key role in setting tariffs based on patient severity and verifying outcomes. These say the Gaming Commission should be independent both of treatment services and the commissioners of those services to make them less vulnerable to pressure to manipulate the figures (or at least the suspicion that this is happening), meaning a new body with its own overheads, which itself may require regulation. An independent assessment centre also places another step in the journey to accessing treatment during which access may falter. The plus side may be more efficient assessment, better treatment placement, and the potential for long-term case management to start at the assessment stage, but these possible advantages could have been achieved by centralising within existing structures.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover. Commissioners of services have been offered guidance from the National Treatment Agency for Substance Misuse, England's special health authority tasked to improve the availability, capacity and effectiveness of drug misuse treatment.

Thanks for their comments on this entry in draft to Keith Humphreys of Stanford University in the USA, crime and drugs consultant Russell Webster, and Annette Dale-Perera of the Central and North West London NHS Foundation Trust in England. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 28 September 2012

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

HOT TOPIC 2016 What about evidence-based commissioning?

STUDY 2011 Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes

DOCUMENT 2013 Delivering recovery. Independent expert review of opioid replacement therapies in Scotland

STUDY 2011 Therapist effectiveness: implications for accountability and patient care

STUDY 2014 Assessing the availability of and need for specialist alcohol treatment services in Scotland

DOCUMENT 2010 The Patel report: Reducing drug-related crime and rehabilitating offenders

REVIEW 2011 Implementing evidence-based psychosocial treatment in specialty substance use disorder care

STUDY 2012 Innovation adoption as facilitated by a change-oriented workplace

STUDY 2010 A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers

DOCUMENT 2011 Scottish Drugs Strategy Delivery Commission: first year report & recommendations to minister





A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers.

Crits-Christoph P., Ring-Kurtz S., McClure Bridget. et al.
Journal of Substance Abuse Treatment: 2010, 38, p. 251–262.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Crits-Christoph at crits@mail.med.upenn.edu. You could also try this alternative source.

It should have improved relationships between counsellors and clients and between counsellors and their clinics, but a sophisticated system for feeding back client progress made no positive difference on any of these measures. A later study helped establish why: counsellors were not told which patient the feedback related to and what they might do about it.

Summary This US study tested a performance improvement system for group therapy counsellors treating substance use problems, based on offering regular feedback on their patients' progress, in particular their assessment of their relationship with the counsellor (in the form of the therapeutic alliance) and their satisfaction with treatment. Therapeutic alliance was measured because it is related to the outcomes of psychotherapy and counselling in general, and substance use counselling in particular. The patient rated their bond with the counsellor, agreement on tasks, agreement on goals, and the degree to which the counsellor understood them.

This data plus attendance records was collected weekly from all patients in groups run by participating clinicians, regardless of the how long they had been in treatment. Turnover of clients meant that each weekly feedback report could reflect the scores of different individuals. In ways which hid the identity of individual patients and also the identity of individual counsellors from their supervisors, the information was analysed by researchers and made available over the internet to the counsellors as the average responses of the patients they were treating in group therapy sessions as a whole, of those who had been in treatment for different periods, of men v. women, and of different ethnicities. Also available to both supervisors and counsellors was this information aggregated for the entire clinic. As well as absolute scores, the feedback charted trends over the 12 weeks of the study.

Counsellors and supervisors were trained and offered written guidance on how to interpret this information. They were also encouraged to review the clinic report as a team at monthly meetings, using it to stimulate discussion and set goals for improvement. Counsellors were encouraged to examine their own caseload reports. If they wanted, they could discuss these with other counsellors and/or their supervisors. Supervisors could use reports for the clinic as a whole to inform decisions about training, supervision, and resource distribution. The clinic was financially rewarded each time a caseload report was newly downloaded and the counsellor told how much they had earned for their workplace.

Each month the researchers also sent an electronic newsletter to staff to share experiences between the clinics, highlight their innovations and accomplishments, and encourage evidence-based practices.

The system was expected to improve attendance and substance use outcomes by giving counsellors the chance to adjust in the light of the degree to which on average their patients felt they were working well together and were satisfied with the treatment. It might also, it was thought, influence counsellors by signalling that these variables were important to their employers, and by enhancing their satisfaction with the clinic and their views of their employer as progressive and committed to staff development and performance improvement.

For 12 weeks, 20 community-based substance abuse counselling clinics were randomly allocated to implement this feedback system, or simply to collect baseline and study-end data from patients, which was not fed back to staff. Group counselling was the primary modality at all the clinics. Counsellors running groups at least weekly were asked to join the study. Of 123, 118 agreed and all but 20 completed the study.

The patient feedback form also asked for the number of days over the past week that the patient had drunk alcohol or taken drugs, data withheld from the staff, but which constituted the primary yardsticks against which the impact of the performance improvement system was assessed. They key issue was whether patients at the feedback clinics reduced their substance use from baseline to the final 12-week assessment more than those at clinics not in the feedback system.

Main findings

One problem in assessing improvements was that already at the start of the study over three quarters of patients said they had been totally abstinent. To create more room for improvement, the main analyses were conducted on new patients in treatment for a month or less, more of whom would it was thought still be drinking or taking drugs. Still no significant differences were found between trends among patients at feedback and non-feedback clinics. For example, at feedback clinics, from 82% not drinking this rose only slightly to 85%; at non-feedback clinics it remained virtually unchanged at about 75%. For drugs the corresponding figures were 90% and 87%, and 76% and 80%. Similar analyses for all patients regardless of treatment duration also found high levels of abstinence which changed little over the 12 weeks, and no more so at feedback clinics.

Neither were there any significant differences when the analysis was restricted to patients who contributed data at baseline and at 12 weeks, meaning the same individuals were tracked, or when the few 'outlier' clinics with unusually high or low baseline abstinence rates scores were excluded.

Similarly, patients as a whole and the tracked individuals did not develop a deeper alliance with their therapists at services offered feedback, and attendance too was no better.

Turning to the staff, those in services which offered feedback on their performance did not feel a better working relationship had developed between them and their supervisors than in comparison services, nor did job satisfaction differentially improve. On the measures of how prepared staff thought their organisation was to make positive changes, the sole statistically significant difference was that resources like office space and internet access had (relative to other services) actually worsened in the feedback services.

The authors' conclusions

This study found no evidence that the performance feedback system led to the expected improvements in drug or alcohol use, therapeutic alliance, patient attendance, or clinicians' views of their jobs and workplaces. Several factors could explain this lack of impact.

In respect of drug/alcohol use there was little room for improvement by any system. Patients often start treatment abstinent after a legal problem or are mandated to treatment (eg, for drink-driving) even though not dependent. Most can readily stop using (especially after a 'binge') for brief periods; treatment's role is to sustain this. Though also near its ceiling, therapeutic alliance could still have improved, as even more so could attendance. Yet in both cases feedback did not help, suggesting that it was indeed ineffective.

Assuming it was ineffective, one possible reason may have been that administrators and supervisors primarily responsibility for performance could not identify which clinicians were performing relatively poorly, key information for targeting improvement measures. Clinicians too were deprived of what for them may be key information – which individuals were not doing well and might benefit from a change in approach. As long as their job is not threatened, clinicians' motivation for general performance improvements may be marginal, especially among highly experienced counsellors who believe they already perform at an advanced level; without a strong motivation to improve one's performance, clinician-level feedback may not be effective.

Also, team meetings were relied on to generate improvement suggestions in response to feedback. Other mechanisms might have been more successful, such as one-on-one supervisory feedback, independent testing or consultation on difficult cases, or supplementing feedback reports with guidance on what they might mean for practice.


Findings logo commentary The featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They took its findings as an instance of the more general finding that "if there is no risk of reputational damage, information on one's individual or organizational performance is generally disregarded". Only if this information is made public in circles that matter to the clinician (like their employer) or to the organisation (like commissioners and prospective patients) does it exert leverage. They favoured schemes based on the patients' actual substance use (or other direct measure of progress) as assessed during treatment using objective techniques such as urine tests and processed in such a way that the results have consequences for the treatment provider.

Why no impact?

Effectively the featured system reported on the therapist's performance but in ways which had no consequences for them. What it did not do was report on the progress of their individual patients in ways which enabled them to intervene. The researchers' belief that this was a major reason for its lack of impact seemed confirmed by a later study led by the same researchers. In this study, not only did the feedback identify the individual patient, it also reported on their psychiatric wellbeing and functioning and their actual drug and alcohol use. Additionally, it categorised the patient according to how their progress compared to that expected for similar patients, signalling whether they were doing so well that treatment might be ended, were progressing as expected, or were progressing less well than expected. For the latter, the therapist was instructed to consider changing the treatment and given guidance based on a second questionnaire on barriers to progress in the form of problems with therapeutic alliance, poor motivation, inadequate social support, and stressful life events. Given these enhancements, the feedback system did in this case significantly and positively affect all three primary outcomes (drug use, drinking, total problem scores) among the 'off-track' patients who were not progressing as expected, the ones targeted by the system.

This interpretation is in line with general psychotherapy studies of the same system reviewed for the American Psychological Association. The reviewers concluded that real-time feedback to therapists enabling them to monitor patients' responses to psychotherapy and satisfaction with the therapy relationship probably improves psychotherapy outcomes overall, and certainly for clients at risk of deterioration or drop-out – the 'off-track' patients of the later study. They were unsure of the benefits of sharing this feedback with patients unless this could (as in the later study) be done at the therapist's discretion and reframed as they saw fit. Also commended were the additional strategy deployed also in the later study of supplementing feedback with information from off-track patients indicating why things were going wrong and guidance from the feedback system on what therapists might do about it.

The feasibility study which showed that the system tested in the featured study could be implemented exposed one further reason why it might not have worked: with no consequences for doing this or not, and no way to relate the feedback to the individuals they were seeing, the typical clinician accessed their feedback reports only 2.3 times (out of a maximum of eight) over the course of the study.

Particularly disappointing in the featured study was that therapeutic alliance – though a major item in the feedback – did not improve as a result of this feedback. Additional to the factors mentioned by the authors, it is perhaps relevant that group therapy was the modality and that from week to week the therapists may have been seeing different patients. It seems probable that this limited the degree to which they could respond to an individual in the group who felt somewhat alienated from the counsellor, and also that even if they did, this might have less effect than in one-to-one therapy, because much depends on the atmosphere created by the other patients in the group.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover.

Thanks for their comments on this entry in draft to Paul Crits-Christoph of the University of Pennsylvania Medical School in the USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 03 October 2012

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

STUDY 2011 Therapist effectiveness: implications for accountability and patient care

STUDY 2009 Relating counselor attributes to client engagement in England

STUDY 2009 The alliance in motivational enhancement therapy and counseling as usual for substance use problems

REVIEW 2011 Evidence-based therapy relationships: research conclusions and clinical practices

STUDY 2010 Gender differences in client–provider relationship as active ingredient in substance abuse treatment

DOCUMENT 2013 Sometimes best to break the rules

REVIEW 2011 Integration of treatment innovation planning and implementation: strategic process models and organizational challenges

STUDY 2005 How does motivational interviewing work? Therapist skill predicts client involvement within motivational interviewing sessions

STUDY 2003 How to identify retention-enhancing alcohol counsellors





A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs.

Crits-Christoph P., Ring-Kurtz S., Hamilton J.L. et al.
Journal of Substance Abuse Treatment: 2012, p. 301–309.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Crits-Christoph at crits@mail.med.upenn.edu. You could also try this alternative source.

Evidence that an earlier study feeding back client progress to counsellors did not find improved outcomes because data was aggregated across a caseload rather than identifying individuals doing poorly and recommending remedial actions. After remedying these deficits, a new system significantly improved mental health and reduced substance use.

Summary The featured study builds on an earlier study led by the same researchers which also offered counsellors at non-methadone substance use disorder counselling services regular feedback on their patients' progress in an attempt to improve outcomes. It found no evidence of the expected improvements in drug or alcohol use or treatment process. One reason it was thought was that clinicians were deprived of what for them may have been key information – which individuals were not doing well and might benefit from a change in approach. Instead they were given average figures for their caseload. The researchers argued that clinicians' intrinsic motivation for general performance improvements may be marginal.

The current study tested changes intended to make feedback more effective. Feedback was provided on each identified individual, and instead of the patient's assessment of the therapeutic alliance and satisfaction with treatment, a questionnaire Developed for mental health patients. assessed their psychiatric wellbeing and functioning and An addition for the study. their actual drug and alcohol use. These assessments were made just before each counselling session at the start of treatment and then for another 12 sessions or six months, whichever came first. Therapists could immediately see the patient's responses as automatically tabulated and charted by the research project. Other differences were that this was applied to patients as they started treatment, not some time in to treatment, and that patients and counsellors were engaged in one-to-one counselling rather than group therapy.

On the basis of their progress compared to that expected for patients with a similar severity of problems, the revised feedback also categorised patients as doing so well that treatment might be ended, progressing as expected, or doing less well than expected – the so-called 'off-track' patients. It was these patients in particular that the feedback system was expected to help retrieve from continued poor progress. For them the therapist was instructed to consider changing the treatment and given guidance based on a second questionnaire these patients only were asked to complete. This rated barriers to progress in the form of a weak therapeutic alliance, poor motivation, inadequate social support, and stressful life events. Counsellors were trained to interpret the web-based reports made available to them and to consult a manual which offered suggestions Some specific examples of treatment interventions for the therapeutic alliance domain are as follows: work with resistance by retreating when necessary and being supportive; allow the client to assert his or her negative feelings about the relationship; and give and ask for feedback on the therapeutic relationship. Examples of interventions for the social support domain are the following: assess client's social network; use progressive desensitisation to aid client in overcoming social anxieties; and encourage pursuit of hobbies, participation in clubs, service projects. Examples of interventions for the motivation domain are the following: assess how important it is for the client to make target changes, having him or her rate the desire to make specific changes on a scale of 1–10; discuss the client's important values, goals, and aspirations; and ask open-ended questions to see if the client is ready to make a commitment to change. For the fourth domain, life events, the manual instructs the therapist to explore the stressful event and to facilitate the development of coping strategies to help the patient establish stability. for how to respond to areas of concern. When these did not seem to account for poor progress, counsellors were trained to consider a different kind of treatment, such medications rather than just counselling.

At first the three clinics in the study simply asked patients to complete the first questionnaire at baseline and for 12 sessions/six months, but counsellors were given no feedback. Then the feedback system was implemented. Scores on the first questionnaire were used to determine whether patients improved further as a result of offering their counsellors feedback. These scores were available only for patients still in treatment; by session 12, nearly two thirds had left.

In the pre-feedback period 165 patients supplied baseline and at least some follow-up data, and 139 in the feedback phase. Typically they were unemployed men with long-standing alcohol and/or drug problems, but over three quarters were no longer using these substances at the start of the study. About 40% were calculated to be doing less well than expected – off-track At the start of the study these off-track patients had significantly higher scores overall indicating greater problems. However, the number of days on which they said they had drunk or used drugs in the past week did not significantly differ from the rest of the patients, though on average they did initially admit to more days of drinking. – at some stage in their treatment and (in the feedback phase) were asked to complete the second questionnaire.

Main findings

On all three measures (drug use, drinking, total problem scores) off-track patients whose counsellors were offered feedback on their progress and suggestions about how to respond progressed better than similar patients before the feedback system had been implemented. In respect both of drinking and drug use, these at first poorly progressing patients ended the study drinking and using drugs as little as the ‘on-track’ patients However, only in respect of days of drinking in the past week did the trends differ to a statistically significant degree.

Instead of trends from baseline, another analysis followed up the off-track patients only from the time counsellors were alerted to their poor progress – an attempt to assess the added value of the assessment of what was holding them back and its feedback to therapists along with suggestions about how to respond. This did seem to trigger positive change. Up to the point when counsellors were alerted there was little difference between feedback and non-feedback patients, but from then on feedback patients progressed much better on all three measures (drug use, drinking, total problem scores), though now it was the results for drug use and total problems which diverged to a statistically significant degree, while days of drinking did not.

Finally, feedback made no apparent difference to how long patients in general or off-track patients in particular stayed in treatment.

The authors' conclusions

This study suggests that outcome assessment/feedback systems may be of value for enhancing treatment outcomes in substance use treatment clinics. Among patients not progressing as expected ('off-track'), feedback to their counsellors improved outcomes relative to no feedback on all three measures of psychiatric symptoms and functioning, drinking, and drug use, albeit at different points in the feedback process. In contrast to drinking, which responded to feedback from the start, additional improvement in drug use and psychiatric health were not evident for the off-track patients until counsellors were alerted to their being off-track and offered suggestions about how to respond.

These encouraging results should be seen in the light of the study's limitations. Patients were not randomly assigned but sought treatment before or after the feedback system started; this time gap may have seen some changes in the type of patients or other factors. Neither was the study able to identify which diagnostic categories of patients benefited most from the feedback system nor precisely how this improved outcomes, because there was no data on which interventions counsellors implemented in response to feedback reports.


Findings logo commentary The implications (supported – see below – by the general psychotherapy literature) of this intriguing study are that treatment services can 'rescue' patients who are not doing well first by identifying them, probing why this is happening, and then offering appropriate responses. One would hope all this was being done routinely as part of good clinical practice, but at these three clinics and with these patients, systematising the process, basing it on evidence about how patients normally progress, and offering scientifically based and/or well worked out responses, did improve outcomes.

However, enthusiasm should be tempered with an appreciation of the clinical as well as statistical significance of the findings. On average patients started the study drinking less than a day a week and using other drugs even less often. Greater improvement from this already low starting point may not signify clinically significant added value. Also, the crunch issue is what the feedback system as a whole achieved, and here the sole statistically significant gain was in reduced drinking, and even without feedback, by the end of the study patients were roughly drinking on average just once a fortnight. Arguably too, a system designed to intercept impending treatment failure is best judged against how many patients met pre-set criteria for failure/success, not average days of substance use.

Why the difference?

Several factors might have helped make the difference between an ineffective feedback system in the previous study and an effective one in the current study. Of these, it seems entirely plausible that (as the authors surmise) the active ingredients included individualisation of feedback, its more concrete nature including actual substance use (not just how patients felt about the therapy), and the guidance given to counsellors on how to respond.

This interpretation would be in line with psychotherapy studies of the same system reviewed for the American Psychological Association. The reviewers concluded that real-time feedback enabling therapists to monitor patients' responses to psychotherapy and satisfaction with the therapy relationship probably improves outcomes overall, and certainly for clients at risk of deterioration or drop-out – the 'off-track' patients of the featured study. They were unsure of the benefits of sharing feedback with patients unless this could (as in the featured study) be done at the therapist's discretion and reframed as they saw fit. Also commended were the additional strategy deployed in the featured study of supplementing feedback with information from off-track patients indicating why things were going wrong and guidance on what therapists might do about it.

If these were indeed the factors which made the difference, it remains to be explained why they did so. The predecessor to the featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They took its findings as an instance of the more general finding that "if there is no risk of reputational damage, information on one's individual or organizational performance is generally disregarded". They argued that only if this information is made public in circles that matter to the clinician (like their employer) or to the organisation (like commissioners and prospective patients) does it exert leverage.

But this new study suggests that when it concerns the individual patient's 'performance', feedback does have an impact, even when As it appears from the report on the featured study. the counsellor is at no direct risk from disregarding it. Possibly the rather stark warnings that "This client may end up with no significant benefit from therapy", or that patients doing even less well "may drop out of treatment prematurely or have a negative treatment outcome", alerted counsellors to the possibility that variables monitored by management like how many of their patients satisfactorily completed treatment would be affected if they did nothing about the feedback. More positively and as the authors imply, it could also be that people who undertake this work because they want to make life better for the individuals they are seeing grasp an opportunity to do so which requires little effort to grasp; feedback and possible responses were 'laid out on a plate' for them; they did not even have to administer the questionnaires which gathered this information.

This likely explanation is however not the only one. If therapists did discuss patients' scores with them, perhaps the revelation that they were doing relatively poorly was itself enough to jerk them in to doing better – or at least submitting questionnaire returns which indicated this. The pattern of the findings and the general psychotherapy literature suggests this is unlikely. It could also be that the experience of completing the second questionnaire exploring why things might be going wrong in treatment had its own impacts, regardless of how the counsellor responded. Other possibly pertinent differences between this and the previous study were that patients had to consent to a system which would identify them to their counsellors, including whether they were still drinking or using drugs. The resulting caseload might have been relatively highly selected and motivated. Outside the context of a research project requiring informed consent, a less promising set of patients might respond less well. Also, the feedback system in the featured study started right at the start of treatment before patterns and relationships were established, not part way through, and the earlier study was concerned with group therapy involving week to week a possibly different set of patients. It seems probable that this limited the degree to which counsellors could respond to poor progress by any individual in the group, and also that even if they did, this might have less effect than in one-to-one therapy. The timeliness of the feedback offered in the featured study may also have been relevant. What might have been a considerable investment in computerisation technology and expertise meant patients could complete feedback questionnaires just before a session started, and that data could be analysed and organised in to an assimilable format and fed back to the counsellor in time for them to use during the session.

Other methodological issues

It is not clear why drinking fell from the start in the feedback phase but the other outcomes only after counsellors were alerted to poorly progressing patients. Apart from the interpretations offered by the authors, it seems possible that an institutional focus on drinking, or the fact that off-track feedback patients were at treatment entry still drinking on average nearly a day a week (in the context of the study, unusually high), meant that all it took to prompt the counsellors to deal with the drinking was to know that it was happening. As the authors speculate, in this substance use clinic, psychiatric problems might not trigger a response until counsellors were told in no uncertain terms that these were among the complex which for off-track patients jeopardised the entire treatment process.

Apart from the caveats cited by the authors is the fact that very few patients were left by the end of 12 sessions. The study did not attempt to assess those who left treatment, leaving its conclusions based on fewer and fewer patients as it progressed. What happened to those who left could substantially alter the impression given by the minority who stayed. A related point is that it remains to be explained why a system designed to identify and intercept impending treatment failure which might lead to premature drop-out did not improve retention. At its best it could be that the patients who were rescued from premature drop-out were counter-balanced by those identified by the feedback system as ready for discharge, or that the system elevated the progress of off-track patients to the point where they could safely leave rather than drop out and relapse.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover.

Thanks for their comments to Paul Crits-Christoph of the University of Pennsylvania Medical School in the USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 03 October 2014. First uploaded 03 October 2012

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2010 A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers

STUDY 2011 Therapist effectiveness: implications for accountability and patient care

STUDY 2010 Gender differences in client–provider relationship as active ingredient in substance abuse treatment

REVIEW 2011 Evidence-based therapy relationships: research conclusions and clinical practices

STUDY 2009 The alliance in motivational enhancement therapy and counseling as usual for substance use problems

REVIEW 2011 Integration of treatment innovation planning and implementation: strategic process models and organizational challenges

DOCUMENT 2013 Sometimes best to break the rules

REVIEW 2011 Evidence-based psychotherapy relationships: Collecting client feedback

STUDY 2005 How does motivational interviewing work? Therapist skill predicts client involvement within motivational interviewing sessions

STUDY 2003 How to identify retention-enhancing alcohol counsellors





Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes?

Harris A.H.S , Humphreys K., Bowe T. et al.
Journal of Behavioral Health Services and Research: 2010, 37(1), p. 25–39.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Harris at alexander.harris2@va.gov. You could also try this alternative source.

This first major multi-modality test of a treatment engagement indicator widely used as a quality control yardstick in the USA found it was only very weakly related to patient improvement seven months after starting treatment, confirmation that simple measures of what happens during treatment struggle to capture what really makes treatment effective.

Summary Quality indicators constructed from administrative data such as patient attendance or staffing are inexpensive and easy to generate but are often of low or unknown validity in terms of being related to patient outcomes; in other words, they may be feasible to collect but not valid as indicators of performance.

To measure the quality of substance use disorder treatment services, the Washington Circle An organisation supported by the US health service's Center for Substance Abuse Treatment. developed initiation and engagement measures. The National Committee for Quality Assurance (NCQA), a non-profit organisation dedicated to improving the quality of health care, incorporated these measures in to the Health Plan Employer Data and Information Set (HEDIS). Because HEDIS is the most widely used set of quality measures in the US managed health care industry, many health care systems are now tracking these measures.

The measures are:
Initiation The percentage of known substance use disorder patients who after being out of treatment for at least two months 60 days. then attend Either a residential admission or an outpatient visit. for substance use disorder treatment and re-attend within a fortnight.
Engagement The percentage of known substance use disorder patients who within a month 30 days. of initiating treatment as defined above attend at least another two times; for people who have initiated residential or inpatient care, the two visits must be within a month of their being discharged from the residential setting – effectively a measure of continuing care or aftercare.

The featured study drew its data from a nationwide sample Up to 50 new patients were randomly selected from each programme in a randomly selected and representative sample of Veterans Affairs substance use disorder treatment programmes. of 5723 patients who had entered treatment at 118 inpatient, residential and outpatient programmes run by the US health service for former military personnel. All but a few initiated treatment, making it difficult to reach conclusions about initiation v. non-initiation, so the featured study focused on engagement.

The study focused further on new patients who may or may not go on to engage with treatment as defined by HEDIS, not those already in treatment for some time, so based its analyses on the 2789 who had started treatment within the past fortnight after being out of treatment for at least two months. Almost exclusively men, they averaged 48 years of age and at baseline were experiencing significant medical and employment problems. They were more likely to be suffering problems or severer problems related to their drinking than to use of other drugs.

Of these patients, 1820 or about two thirds provided follow-up data on average just over seven months later. The scores missing patients Missing patients tended to be younger, were less likely to be homeless, and had less severe drug and employment problems but more severe psychiatric and legal problems as measured by the Addiction Severity Index. were likely to have supplied had they been contacted were estimated from what was known about them and the data they had provided initially.

Primarily at issue was whether patients who engaged as defined by HEDIS had more favourable trends in their alcohol, drug and legal problems than patients who did not engage. If they did, this would be consistent with engagement truly being indicative of more effective treatment.

Main findings

Generally the picture was that patients who met the HEDIS engagement criterion did experience greater remission in their problems than those who did not, but only if they started their treatment as outpatients, the relationships were very weak, and in terms of concrete outcomes like abstinence, not statistically significant. Details below.

Patients who subsequently engaged in treatment were more likely to be African American and homeless and less likely to be Caucasian than those who did not engage. They had also reported more severe alcohol, psychiatric, and legal problems. Such differences might have affected outcomes regardless of how well they engaged with treatment, so as well as analysing the 'raw' figures, further analyses adjusted for differences between engagers and non-engagers; these adjusted figures are reported below.

Problem scores assessed for the past month by the Addiction Severity Index questionnaire could vary from 0 to 1 at their most severe. At first for drinking they averaged about 0.4. On average this score halved over the seven months of the follow-up. Patients who had engaged with treatment experienced a small but statistically significant further reduction of 0.03 points. When the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, only for the outpatients did the further problem reduction among engagers (0.06 points) remain statistically significant. Though the trend for engagers to do better remained, it was no longer statistically significant when the sample was limited to people actually diagnosed with alcohol problems, or when the outcomes were alcohol abstinence or changes in the number of days on which patients got drunk.

Similarly for problems with drugs other than alcohol. At first problem scores averaged 0.18. On average this score too halved over the seven months of the follow-up. Patients who had engaged with treatment experienced a small but statistically significant further reduction of 0.02 points. Though the trend for engagers to do better remained, it was no longer statistically significant when the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, when it was limited to people actually diagnosed with drug problems, or when the outcome was abstinence from drug use.

Legal problems too on average remitted somewhat from 0.19 to about 0.14. Patients who had engaged with treatment experienced a statistically significant further reduction of 0.04 points. When the sample was divided in to those who started treatment in outpatient versus residential or inpatient programmes, only for the outpatients did the further problem reduction among engagers (0.04 points) remain statistically significant.

These changes in composite measures of different problem domains were extrapolated to what they might mean for more concrete measures. For example, it would be expected that on average patients who did versus did not engage would be drunk under one day (0.62 days) fewer a month, and would be marginally more likely to be abstinent from drugs (increase from 77.5% to 77.8%) and less likely to be in prison (decrease from 4.8% to 4.6%).

The authors' conclusions

The HEDIS engagement indicator has been widely adopted, but largely in the absence of evidence linking it to patient outcomes. This is the first study to examine the strength of the association between meeting this criterion and patient-level changes in alcohol, drug, and legal symptoms in a nationwide health care system that includes both outpatients and patients treated in inpatient–residential programmes. It found these symptoms did improve more among patients starting treatment at facilities for ex-military personnel who met the engagement criterion, particularly in outpatient settings. Though statistically significant, the extra improvements were clinically modest.

It is important to remember that these results related each individual patient's engagement to remission in their problems. They do not necessarily mean that programmes which on average engage a greater proportion of their treatment starters also on average have better outcomes. In fact this was not the case when almost the same data set was analysed by programme instead of by patient. Together these finding mean that in the featured health care system, the engagement measure adopted by HEDIS modestly predicts which individuals will improve most, but not which programmes generate the greatest improvement across their caseloads. Appreciating this should temper enthusiasm for using these measures in pay-for-performance systems or to choose the most effective treatment facility.

Some further limitations of the study should be noted. The modest relationships found between engagement and outcomes may mean that engagement causes better outcomes, but may also mean that variables not adjusted for in the study (such as the motivations of the patients, their family support, or their involvement in mutual aid groups) generated both better outcomes and deeper engagement. In this latter scenario, engagement would have simply been a non-active by-product of the factors which really generated better outcomes. Engagement was not related to outcomes in residential/inpatient programmes, perhaps because the measure adopted by HEDIS does not reflect engagement with the core treatment, but with follow-on treatment, which (for example) patients may not attend simply because it is hard to access, regardless of their progress. Finally, these results were obtained from a very particular health care system with a distinct caseload and relatively well integrated services. Even then they reflect the engagement-outcome relationship only among patients not in treatment for at least two months and at the start of a new episode of care.


Findings logo commentary Findings of this study confirm a common conclusion: that what makes people come back to a treatment service is not necessarily what makes treatment effective or which leads to the desired changes in substance use. They may overlap, but sometimes not, and sometimes only very little, as in the featured study. Retention or attendance are just two dimensions (the most easily measured) of engagement. Sometimes deepened engagement may actually shorten retention because clients are ready to leave sooner. But generally retention is a sign that clients are actively 'working the programme', attending counselling sessions, talking about the things that matter, forging a therapeutic relationship with their counsellor and/or other clients, getting extra help if needed. Arguably it is what is done during the retained period and during the attendances which makes the difference to outcomes rather than merely attending. Treatment as a whole may also be a relatively minor factor in what patients at least see as contributing to their recovery. The main exception is prescribing substitute drugs like methadone to opiate addicted patients, a modality where staying in treatment is indeed the key to its success.

The featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They observed that incentives to meet targets based on criteria like the HEDIS engagement indicator can lead to dramatic improvements in hitting these targets, yet achieving these targets can bear at best a weak relationship to subsequent patient outcomes. Instead they favoured schemes based on the patients' actual substance use (or other direct measures of progress) assessed during treatment using objective techniques such as urine tests and processed in such a way that the results have consequences for the treatment provider.

Despite the failure of the HEDIS engagement indicator to predict the effectiveness of the programme and only very modestly the progress of an individual patient, other measures tested in the same health care system have proved a better option. In a study confined to alcohol-related outcomes at outpatient programmes – the combination for which the featured study found engagement most closely related to outcomes – the strongest indicator of which agencies had the best average outcomes was the proportion of their patients who attended at least three times in the first month, though how many attended at least twice, four, five or six times were not far behind. These indicators accounted for about a quarter of the variation between agencies in how well their patients did, a substantial relationship. The impetus for the study was the poor performance of an indicator – three-month retention – very similar to the 12-week retention indicator used recently as a benchmark for British drug dependence treatment services, though in relation mainly to the treatment of opiate-addicted patients.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover. Commissioners of services have been offered guidance from the National Treatment Agency for Substance Misuse, England's special health authority tasked to improve the availability, capacity and effectiveness of drug misuse treatment.

Thanks for their comments on this entry in draft to Alex Harris of the US Veterans Affairs health care system. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 28 September 2012

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2011 Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes

STUDY 2009 The Drug Treatment Outcomes Research Study (DTORS): final outcomes report

STUDY 2010 Gender differences in client–provider relationship as active ingredient in substance abuse treatment

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

REVIEW 2009 Continuing care research: what we have learned and where we are going

STUDY 2010 A randomized controlled study of a web-based performance improvement system for substance abuse treatment providers

STUDY 2003 Systematic but simple way to determine who needs residential care

STUDY 2012 Advancing recovery: implementing evidence-based treatment for substance use disorders at the systems level

STUDY 2008 Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment

STUDY 2011 Therapist effectiveness: implications for accountability and patient care





Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes.

Brucker D.L., Stewart M.
Journal of Behavioral Health Services and Research: 2011, 38(3), p. 383–397.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Brucker at debra.brucker@gmail.com. You could also try this alternative source.

In 2007–08 the US state of Maine introduced a new scheme directly linking funding for outpatient treatment services to performance in terms of waiting times and retention, but financial and service delivery impacts were negligible. Were the incentives too weak, or were services already doing as well as they could?

Summary US single state agencies are public bodies responsible for the coordination of substance abuse services in states and territories, in particular for clients who cannot fund their own care. Most do not provide services directly, but purchase from community-based systems of care. Performance-based contracting is one way they can try to improve quality. It involves offering direct financial incentives to service providers as long as they meet pre-determined levels of performance on defined indicators.

Maine was one of the first states to implement such a system, providing an opportunity to examine the relationship between services being paid in this way and their performance, in this case on measures of access to and retention in treatment. Performance-based contracting was introduced in 1992 to shift the publicly funded treatment system from a focus on outputs to outcomes. Results were mixed; providers reduced certain services, yet reported some better outcomes. Possible 'cherry picking' and 'gaming' were suggested by inconsistencies between client data reported for the payments system and treatment service medical records.

To address these criticisms, Maine restructured its system, with effect for outpatient services in financial year 2008 (1 July 2007 to 30 June 2008). The aim was to foster efficiency and quality of service by tying performance to actual payment level. First agencies were financially rewarded or sanctioned for exceeding or missing the number of 'units of service' they were contracted to provide, an attempt to prevent cuts in patient numbers. Then 'access' or waiting time targets were set at different levels for standard and intensive outpatient services: typical Specified as the median, meaning the middle value in the ranking from greatest to least delay. times from first phone contact to assessment were to be no greater than five and four days, and from assessment to the start of treatment, no more than two and one week.

Retention targets too were set: standard services were to retain at least 50% of new patients for at least four days and 30% for at least three months. Operationalised as 90 days. Corresponding targets for intensive services were 85% for at least four sessions and 50% to complete treatment. For Maine, 'completion' meant that the patient had achieved at least two thirds of their treatment plan Plans were to include objectives specific to the client, for example: abstinent during treatment; significant reduction in problem use; willingness to voluntarily seek continued care as necessary; participation in self-help. before an agreed discharge.

By exceeding or undershooting these targets, services stood to gain/lose 9% of their contracted fee. Assessments were made quarterly and payments adjusted the following quarter.

In financial year 2008, around 5000 adults started treatment at the 17 services in the new payment scheme. Data from them in respect of retention was contrasted with the same services the year before the new scheme was implemented, and in respect of waiting times, with that from around 4000 patients at services not in the scheme. Typically, patients were white single men not in full-time employment with a drink problem, though intensive services saw slightly more patients (around 4 in 10 of the caseload) whose primary problem was opiate use. Over half also had mental illness diagnoses.

Main findings

Generally services were not subject to substantial financial gains or losses as a result of the payments scheme. Over financial year 2008 they were budgeted to receive $3,531,364 and could gain $238,099 by exceeding performance targets, but only 19% of the incentive money was paid out. Across the year, payment adjustments for each agency ranged from about a loss of 7% of their contracted fee to a gain of 7%, averaging virtually zero and typically Median level of payment, meaning the middle value in the ranking from greatest loss to greatest gain. a loss of just 1%.

Based on raw figures unadjusted for caseload and other factors, non-intensive programmes in the payment scheme were significantly more likely to hit their waiting time targets than those outside the scheme: in respect of assessment, 61% versus 52%, and 92% v. 85% in respect of time to treatment. However, the reverse was the case for intensive services: corresponding figures were 69% v. 78%, and 86% v. 93%. The pattern of non-intensive services doing better if in the payment scheme, but intensive services doing worse, was generally similar in respect of average days patients had to wait for assessment or treatment.

Contrasting their pre-scheme (financial year 2007) to post-scheme (2008) performance, after joining the payment scheme both standard and intensive services recorded worse retention on all measures, often substantially and significantly worse. For example, from 40% of patients staying in standard programmes for three months the figure fell to just 24%. From 53% completing intensive programmes, the figure fell to 46%.

But further analyses revealed that these findings were not due to being subject or not to the payment scheme. Once other factors Number of older patients; sex; white v. non-white ethnicity; married/cohabitating or not; full-time employed or not; co-occurring mental illness or not; health insurance v. none (a proxy for other unmeasured socioeconomic factors); referred from criminal justice sources or not; agencies with large versus smaller caseloads. had been taken in to account, on no measure of either waiting time or retention was being in the scheme associated with a statistically significant difference.

While being in the payment scheme made no apparent difference, retention was significantly related to other factors. For example, across both assessed years (2007 and 2008), significantly more likely to stay for three months at non-intensive services were women, criminal justice referrals, those also mentally ill, or employed full time. Less likely to stay three months were patients whose primary drug was cocaine or those at agencies with large caseloads. In the year when the payment scheme was in operation, full-time employees were also more likely to complete their treatments at intensive services than other patients, as were white patients.

The authors' conclusions

As a whole, the results presented here suggest that as implemented in Maine in financial year 2008, performance-based contracting had only minimal effects on agency reimbursement and no effects on time to assessment, time to treatment, patient participation, length of stay, or completion of treatment. Financial and service delivery impacts were negligible.

Why this non-impact? First, whether or not subject to the new payment scheme, agencies as a whole were already doing well on waiting times for assessment and treatment. Also the financial consequences of meeting or not meeting targets were in practice very small and limited even in worst/best cases to 9% of base funding, perhaps insufficient to move agencies to adopt policies and procedures to improve performance.

On the other hand, there was no evidence that (unlike the earlier scheme) the new scheme led agencies to limit admissions to the most promising patients, a particularly important finding. Also, implementing the scheme meant that, with providers, the state agency could improve accountability and reinforce organisational focus on access to and retention in treatment. Just implementing a performance-based contracting system which is both operationally effective and accepted by services is an important achievement.

In interpreting these findings it should be borne in mind that they are limited to the last contact the patient had with the service. Post-treatment data are not collected at state level.


Findings logo commentary It is tempting to say that the Maine scheme was accepted by services because it apparently demanded little of them they were not already achieving, so made little difference financially. Across health care, patients often do not comply with remedies which require significant lifestyle change. Many even fail to regularly take pills which could prolong their lives. Patient resistance sets limits to the degree of compliance the treatment service can achieve, even when this is specifically targeted. Unrealistic targets may either lead services to cheat or to penalties so severe that what may be the only accessible treatment service in an area is curtailed or closed down, an attempt to improve services which would end up making things worse for the patients.

The Maine scheme was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. Despite Maine's unpromising experience, they favoured the same types of schemes based on during and in treatment measures (but including substance use, not just attendance) because these most closely reflect the influence of the treatment, focus the service on patient progress indicators which it should in any event be monitoring, and do not require following up patients.

Maine's attempt included criteria similar to some on which funding has been made partly contingent in Britain, like 12 weeks retention (in Maine, three months) and successful treatment completion, in both jurisdictions entailing a planned discharge, plus other elements which differ. In Britain some criteria are applied at the level of a local treatment system (such as numbers in effective treatment, entailing 12 weeks retention or planned discharge) and others (as in Maine) at the level of an individual service, the latter most notably in the form of pilot payment by results schemes in England.

Payment by results schemes in England

As in Maine, nationally agreed outcomes for the pilots often specify during treatment and treatment exit measures. Partially overcoming a limitation noted in the featured study, British schemes also attempt to balance during/end treatment measures against longer term crime and relapse indicators, generally choosing to place greater financial weight on the longer term. Onerous follow-up requirements are sidestepped by using routinely collected criminal justice and treatment records which do not require contact with the patient. Reports from the pilots suggest these schemes can both be feasible and generate innovations focused on achieving results.

However, this means the services in the pilot schemes must wait many months – in respect of some measures, nearly two years – to receive much of their funding under the scheme, a cash-flow problem which requires counter-measures if services are to survive. Arguably too, as the US reviewers commented, services are placed at financial risk for outcomes over which they have little control because they are so far from the time when they had direct influence over the patient.

The alternative of weighting in favour of during treatment and treatment end measures falls foul of the fact that these are often at best loosely related to the longer recovery the system is trying to generate. For example, 'successfully' completing treatment free of dependence and of opiate and crack cocaine use did mean that over the next four years more patients in England appeared to have avoided relapse, but the difference of 57% versus 43% who did not successfully complete was not as large as would be expected if successful completion correlated strongly with lasting recovery.

Above all, such schemes have left little or no room within their structures for patient-centred practice in the sense of basing treatment objectives on the patient's priorities. Instead they pre-set the treatment destination in detail without reference to what the individual patient wants, and in a way services cannot afford to ignore because their financial survival depends on meeting the criteria for payment. Local schemes could still create a space for the patient's ambitions in their payment criteria, but this is not a required element or one included in the English national outcomes schema, nor one which sits easily within a system predicated on observable outcomes the public and their representatives recognise and are willing to pay for.

Playing the game

Maine at first experienced what was suspected to be 'gaming' in the form of returns from services which did not match their own clinical records. Partly to avoid this, schemes which make payment contingent on client progress or outcomes often entail a regulatory overhead which could eat in to whatever efficiencies are achieved at the services concerned. For example, it has been argued that rewarded outcomes should be objective (eg, urine testing) and case-mix-adjusted. This means some authority has to assess the severity of the caseload in terms of the resources needed to achieve the intended outcomes and assess whether those outcomes have been achieved in ways which go beyond merely asking the patient. When funding, jobs and organisational survival ride on these assessments, leaving them entirely to the people and organisations at threat may stretch their integrity too far. This concern spawns new regulatory requirements and possibly new regulatory bodies which must, as the Audit Commission described, be capable of overcoming complexity to deliver valid and meaningful measures if disputes, demoralisation and wastage are to be avoided.

In English payment by results schemes the most visible result has been the setting up of central assessment centres (or LASARS), which have a key role in setting tariffs based on patient severity and verifying outcomes. These, say the Gaming Commission, should be independent both of treatment services and the commissioners of those services to make them less vulnerable to pressure to manipulate the figures (or at least the suspicion that this is happening), meaning a new body with its own overheads, which itself may require regulation. An independent assessment centre also places another step in the journey to accessing treatment during which access may falter. The plus side may be more efficient assessment, better treatment placement, and the potential for long-term case management to start at the assessment stage, but these possible advantages could have been achieved by centralising within existing structures. Alternatively, as in some drug pilots, the treatment services themselves can be trusted to act professionally in allocating clients to different need levels and treatments. Subject to clinical audit and review, this has been the method adopted by the NHS mental health payment-by-results scheme.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover. Commissioners of services have been offered guidance from the National Treatment Agency for Substance Misuse, England's special health authority tasked to improve the availability, capacity and effectiveness of drug misuse treatment.

Last revised 29 September 2012

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2009 The Drug Treatment Outcomes Research Study (DTORS): final outcomes report

STUDY 2012 Randomized trial of standard methadone treatment compared to initiating methadone without counseling: 12-month findings

STUDY 2011 Interim methadone treatment compared to standard methadone treatment: 4-month findings

STUDY 2009 A practical clinical trial of coordinated care management to treat substance use disorders among public assistance beneficiaries

STUDY 2012 Delivering service quality in alcohol treatment: a qualitative comparison of public and private treatment centres by service users and service providers

STUDY 2008 Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

STUDY 2012 Advancing recovery: implementing evidence-based treatment for substance use disorders at the systems level

STUDY 2010 Gender differences in client–provider relationship as active ingredient in substance abuse treatment

REVIEW 2011 A policy-oriented review of strategies for improving the outcomes of services for substance use disorder patients





Advancing recovery: implementing evidence-based treatment for substance use disorders at the systems level.

Schmidt L.A., Rieckmann T., Abraham A. et al.
Journal of Studies on Alcohol and Drugs: 2012, 73(3), p. 413–422.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Schmidt at laura.schmidt@ucsf.edu. You could also try this alternative source.

In the US homeland of competition and private health care, it was cooperation and coordination which led to the introduction of new medications and innovations to promote continuing care – plus the exercise of regulatory and financial muscle and the salutary experience of senior staff who placed themselves in the patient's shoes.

Summary The study addressed what many believe is the key issue in advancing evidence-based practice – not establishing what those practices are, but how to get them implemented in the 'real world' by commissioners and services. It did so in what in US terms the researchers saw as a best-case scenario. The implication is that if implementation proved difficult in these circumstances, it would be even more so without the support offered by the research project and in less promising jurisdictions.

The commissioners in this case and the main players were US single state agencies. These coordinate substance abuse services in states and territories, in particular for clients who cannot fund their own treatment. Most purchase services from community-based systems of care and exercise added influence via licensing and credentialing regulations applied even to fee-charging private centres.

The project evaluated by the featured study started with an invitation to single state agencies to partner with local treatment centres to test one or two strategies for promoting adoption of science-based treatments selected from a national short list. A competitive application process resulted in 12 jurisdictions The states of Alabama, Arkansas, Colorado, Delaware, Florida, Kentucky, Maine, Missouri, Rhode Island and West Virginia, and the counties of Baltimore and Dallas. joining the project, each partnered by (generally) three or four treatment services, with the ultimate goal of system-wide adoption. All but one jurisdiction had little prior experience of implementing the targeted practices.

The practices they chose were acamprosate and naltrexone for alcohol problems and buprenorphine for opiate addiction, and psychosocial approaches to promoting continuing care or aftercare on an outpatient basis following more intensive, residential or specialist care.

To promote adoption of such practices the project identified five change 'levers':
1 Financing analysis reviewing budgets, costs, and reimbursement mechanisms at the systems level. Adopting new therapies entails costs for technology, training, supervision, and productivity losses during training. Understanding where these costs lie helps government officials advocate for funding and changes in payment arrangements that incentivize new treatment modalities.
2 Regulatory and policy analysis of regulations and accreditation rules which govern practitioner qualifications and the operation of treatment services.
3 Inter-organisational relationship analysis to assess roles and relationships among stakeholders, the system's structure, and how such relations can be changed or used to maximally support innovation.
4 Operations analysis to identify service delivery problems and makes changes to the clinical process.
5 Customer analysis drawing on patient experiences of the care process as a crucial tool for identifying ways to improve it.

The adoption process was aided by national learning sessions and expert coaching provided by the research project, offering participating sites opportunities for face-to-face collaboration and technical assistance. How far this worked was assessed mainly through data collected by participating sites Data from Florida was too incomplete and inconsistent for analysis. Delaware was excluded because it was the only state to promote the use of motivational incentives, so not comparable to other sites. on the total number of patients newly admitted to (or discharged from) treatment who were prescribed the targeted medications or received continuing care of the kind being promoted at that site. The adoption process – the working of these levers – was described by drawing on interviews and focus groups with staff six, 12, and 18 months after baseline, related documentation, and research field notes taken during project planning, coaching calls, and other events.

Main findings

Adoption rates

Most but not all sites reported some success as measured by the number of newly admitted or discharged patients treated with the targeted approaches.

Five states aimed to promote medication-assisted treatments for alcohol or opioid Drugs with opiate-type effects (including analgesia and the capacity for producing euphoria and dependence) derived from the opium poppy like opium, heroin, and morphine (know as opiates) and synthetic drugs with similar effects like methadone and buprenorphine. use disorders. Maine experienced rapid and sustained success in promoting buprenorphine for opioid dependence, numbers rising from 20 patients a quarter to over 80. In respect of the same treatment, in West Virginia numbers rose slowly from 57 to 76, while in Dallas physicians were reluctant to prescribe buprenorphine for under 30-year-olds and lack of public health insurance coverage inhibited implementation. Missouri promoted naltrexone and acamprosate for alcohol dependence – after a slow start, prescribed by the end of year one by all participating clinics. Donated extended-release naltrexone from the manufacturer stimulated its use for alcohol dependence in Colorado, but subsequent shortages led numbers to decline.

Though varied, the six continuing care initiatives all addressed transitions in care facilitated by face-to-face or telephone counselling. In Alabama, from 30% of adolescents entering outpatient care following a stay at two residential treatment centres, the proportion rose to 65% in the sixth quarter. Baltimore aimed to transfer stabilised buprenorphine patients from drug treatment centres to health centres; during the first year the proportion doubled, freeing more slots for buprenorphine patients, as evidenced by numbers rising from 315 during the fourth quarter to 451 by the sixth. Steady progress in Colorado on continuity between detoxification and outpatient care was facilitated by financial incentives for the receiving programmes. Kentucky successfully addressed the same issue but in a rural community and via patient video presentations, 'navigators', and case managers. Because of its large rural areas, Arkansas implemented telephone-based continuing care for adults discharged from residential care. The number of patients served nearly doubled over the first year but then declined due to staff turnover. Rhode Island also struggled to implement telephone-based continuing care following outpatient treatment. Patients were reluctant to comply and counsellors sceptical of the value of a telephone intervention.

The change process

To finance the innovations sites generally sought new money. but this was forthcoming only in Maine and Baltimore. Maine's legislature allocated $500,000 to purchase medications and its public health insurance office added buprenorphine to its pharmaceutical formulary, successes which seemed attributable to close relationships and trust between the single state agency and other decision-makers in the small state bureaucracy.

When such efforts failed, most partnerships defaulted to reallocating existing funds and/or to increasing flexibility in contractual arrangements for paying treatment services. Rhode Island converted some of its outpatient slots into slots for continuing care management, amended provider contracts to permit these new expenditures, and approved a new billing code. Missouri also restructured existing contracts and allowed treatment centres to purchase physician time and medications. Notably, this step took on its own momentum; after it was shown that alcohol medications reduced treatment readmissions and improved outcomes, Missouri's Department of Corrections allocated $500,000 for medications for offenders on probation and parole.

Regulatory and policy changes were among the most common and successful levers, often in tandem with financing changes. Missouri and Maine changed certification standards to require centres to have staff physicians, meaning they now had access to a prescriber, a requirement later embedded in contracts. Several single state agencies arrived at complementary licensing and contract changes after alternatives had failed. Maine, for example, initially encountered opposition to medications from some 12-step-oriented counsellors. Education and feedback sessions to negotiate a compromise failed, so government officials turned to licensing and contract requirements, effectively to force change on services which wished to stay in business.

Advancing Recovery partnerships actively brokered inter-organisational relationships with other state agencies and supported quality improvement collaborations that brought stakeholder groups together to support clinical innovation. A general theme was the importance of the single state agency's place in the state bureaucracy. Small governments seemed to facilitate autonomy and trust, while complications arose in more complex bureaucracies where authority was fragmented across multiple, loosely coupled divisions. West Virginia illustrated the challenges when its licensing authority ordered the closure of an "unlicensed" buprenorphine programme at the largest treatment centre. The single state agency intervened and eventually convinced officials that the programme was legal and the matter was resolved, but only after uncertainty and turmoil.

Understanding the inter-organisational layout had benefits for building stronger coalitions and regional provider networks which, among other things, could offer periodic training and technical assistance on evidence-based practices. For example, the Texas single state agency supported training on motivational interviewing so Dallas providers could encourage opioid-dependent patients to use buprenorphine, stronger ties among Alabama treatment services facilitated transfer of adolescents from residential to outpatient services, and Baltimore transferred buprenorphine patients to federally qualified health centres, increasing capacity for treating opioid dependence and facilitating integration with primary care.

In operations analyses a key tactic was the 'walkthrough'. Senior staff pretended to be patients and experienced the process of being admitted and treated from the patient's point of view, helping identify and address inefficiencies related to patient flow and administrative procedures such as scheduling, billing, charting, and patient follow-up. Flow charts mapping the process of implementing medication-assisted treatment enabled the Missouri partnership to identify ways to increase the use of medications, including a more private and confidential setting for intakes, a new billing code, and training staff on the use of medications. Missouri's single state agency then developed policies to support the suggested changes, adding billing codes, issuing treatment guidelines, changing contracts to permit purchase of medications, and setting up a central medication purchasing capacity, all of which required buy-in from state officials at higher levels of government.

Sites also embraced the concept of piloting changes in care, for example, fine-tuning new procedures with one counsellor who then becomes an advocate for the rolling this out to their colleagues, rather than immediate across-the-board implementation.

Customer analysis encouraged treatment services to understand the experience of their patients as 'customers'. Walkthroughs again were important. Such analyses in West Virginia revealed that buprenorphine patients were not welcomed at 12-step meetings, so the partnership developed alternative support groups. The concept spread state-wide and the groups built a more active consumer constituency. During the legal crisis that threatened closure of buprenorphine services, this proved instrumental in preserving services.

The authors' conclusions

The study showed that partnerships in diverse treatment systems could achieve meaningful gains in the adoption of medications and continuing care, although the number of new patients served by some sites remained small. Though some changes were imposed 'from above' by policymakers, and others started 'from below' with providers piloting new approaches, the greatest successes emerged largely due to coordination of efforts between policymakers and providers.

The process of implementing change could be characterised as trial-and-error adaptation and incremental learning as sites attempted to overcome barriers. No partnership achieved success through a single formula or discrete policy change; no single tool worked equally well everywhere and for each innovation. It might be assumed that reforms in health care systems unfold in discrete stages: policy development, government debate, a new law, and ultimately implementation. These observations suggest this process is far less tidy and linear. For example, when new funding could not be generated, some partnerships found there were still ways to pull financing levers by reallocation and/or payment incentives, bolstered by regulatory changes. The implication is that implementing new treatments requires a flexible menu of tools that can accommodate the specific treatment modalities and the contours of the existing treatment system.

Barriers to clinical innovation were substantial across all levels of treatment systems, including lack of special funding, no insurance coverage, limited single state agency regulatory powers, complexity and fragmentation within state bureaucracies, provider resistance, staff turnover and lack of training, limited treatment slots, weak data systems for tracking change, communication problems, and coordinating change in large states and dispersed rural areas.

The nature of the new treatments also determined which implementation strategies were needed and most useful. Medication-assisted therapies were most readily adopted, mainly through regulatory, financing and contracting tools, which overcame philosophical resistance among some staff by requiring patient access to medications. In contrast, implementing continuing care was more complex and involved coordinating fragmented systems through inter-organisational and operational analyses that forged stronger provider networks and identified gaps in the continuum of care. Limited availability of treatment slots could obstruct these efforts, as did staff turnover requiring retraining. In Rhode Island, for example, counsellors and patients often saw outpatient discharge as the end of care and were reluctant to make telephone calls and participate in continuing care. These struggles illustrate the need for building consensus in which both practitioners and patients embraced the value of the service.

Successful systems change arose from a cooperative division of labour between policymakers and treatment services. Single state agency officials were usually best placed to undertake financial, regulatory and inter-organisational analyses, while services could exploit operations and customer impact analyses. The most successful partnerships involved coordinated and complementary changes across multiple levels of the system all at once, nowhere better illustrated than in Missouri.

It should be remembered, however, that the partnerships in this study were selected through competitive applications and provided with added funding and technical assistance to support their change efforts – a best-case scenario of what can be achieved. Although meaningful change can be achieved without these supports, it is likely to be slower and even more incremental.


Findings logo commentary These best-case scenario attempts resulted in major advances but also slow progress, reversals and resistance. In the end, the exercise of power in the form of regulation, policy change, and financial incentives/threats were often needed.

The featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. Along with its predecessor, the Advancing Recovery project was the main example of attempts to foster better treatment by improving managerial capacity and business practices. The review acknowledged the gains made in the treatment process but cautioned that these processes had yet to be shown to improve long-term patient outcomes. As the evidence stood, the reviewers favoured schemes based on the patients' actual substance use (or other direct measures of progress) assessed during treatment using objective techniques such as urine tests, and processed in such a way that the results have consequences for the treatment provider. However, such schemes must themselves be implemented using levers such as those tested in the featured study.

British practitioners and managers seeking to improve their practice have available to them the web site of the Substance Misuse Skills Consortium, an independent initiative led by treatment providers to harness the ideas, energy and talent within the substance misuse treatment field, to maximise the ability of the workforce, and to help more drug and alcohol misusers recover. Commissioners of services have been offered guidance from the National Treatment Agency for Substance Misuse, England's special health authority tasked to improve the availability, capacity and effectiveness of drug misuse treatment.

Thanks for their comments on this entry in draft to Laura Schmidt of the University of California at San Francisco in the USA and John Witton of the National Addiction Centre in London, England. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 01 October 2012. First uploaded

Comment/query to editor
Back to contents list at top of page
Give us your feedback on the site (two-minute survey)
Open Effectiveness Bank home page
Add your name to the mailing list to be alerted to new studies and other site updates


Top 10 most closely related documents on this site. For more try a subject or free text search

REVIEW 2011 Alcohol-use disorders: diagnosis, assessment and management of harmful drinking and alcohol dependence

DOCUMENT 2013 Delivering recovery. Independent expert review of opioid replacement therapies in Scotland

DOCUMENT 2009 Guidelines for the psychosocially assisted pharmacological treatment of opioid dependence

STUDY 2011 Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes

DOCUMENT 2015 American Society of Addiction Medicine (ASAM) national practice guideline for the use of medications in the treatment of addiction involving opioid use

REVIEW 2012 BAP updated guidelines: evidence-based guidelines for the pharmacological management of substance abuse, harmful use, addiction and comorbidity: recommendations from BAP

STUDY 2015 Extended-release naltrexone for alcohol and opioid problems in Missouri parolees and probationers

DOCUMENT 2014 Consolidated guidelines on HIV prevention, diagnosis, treatment and care for key populations

STUDY 2002 The grand design: lessons from DATOS

STUDY 2011 Scoping study of interventions for offenders with alcohol problems in community justice settings





L10 Web Stats Reporter 3.15 LevelTen Hit Counter - Free PHP Web Analytics Script
LevelTen dallas web development firm - website design, flash, graphics & marketing