Using pay for performance to improve treatment implementation for adolescent substance use disorders
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

Using pay for performance to improve treatment implementation for adolescent substance use disorders.

Garner B.R., Godley S.H., Dennis M.L. et al.
Archives of Pediatric and Adolescent Medicine: 2012, 166(10), p. 938–944.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Garner at brgarner@chestnut.org.

How can it be that incentives to therapists improve implementation of a therapy without further helping patients overcome substance use problems? In this US study of young substance users, disappointing results may reflect the inability of time-limited programmes to make an impression on the lives of youngsters subject to powerful influences, including criminal justice intervention.

Summary Pay-for-performance schemes in health care systems tie financial incentives to the achievement of predefined quality/activity criteria such as the delivery of a particular intervention to a certain standard. They attempt to influence the health service provider to make greater use of what the funder sees as effective interventions. Despite the rapid spread of such schemes, there have been few randomised trials to test whether they improve outcomes for patients.

This US study tested a pay-for-performance scheme as way of promoting use of the Adolescent Community Reinforcement Approach (A-CRA) for the treatment of teenage problem substance users. Engaging also parents/carers, the approach is designed to increase adolescents’ access to and to reinforce rewarding experiences which do not involve substance use. It was manualised and tested in the US Cannabis Youth Treatment Study panel for more on the approach and on the study.

The Cannabis Youth Treatment Study

This account is based on the description (study 7) in these Finding background notes and this report on the study.

The Adolescent Community Reinforcement Approach (A-CRA) was manualised and tested in the US Cannabis Youth Treatment Study. At four treatment sites, this recruited 600 12–18-year-olds diagnosed as abusing or dependent on cannabis. Together with their families they were randomly allocated to one of five outpatient treatment approaches.

The basic treatment consisted of two one-on-one motivational enhancement sessions followed by three cognitive-behavioural therapy sessions conducted in small groups of five or six children. Running over six weeks, it was intended to be a brief, low-cost initial treatment which could be widely adopted, even in non-clinical settings such as school welfare services.

All four clinics in the trial provided this basic treatment to some patients, and to others one of two more extensive therapies lasting 12 weeks or more. At two clinics the more extensive therapies built on the basic regimen, adding seven further cognitive-behavioural sessions, or these plus family support and parent education. In the other two clinics, rather than building on the basic sessions, two alternative approaches were tried.

The Adolescent Community Reinforcement Approach was one of these alternative approaches. It occupied 10 individual sessions with each youngster and another four with their parents or other carers, aiming to develop rewarding non-drug using activities for the child. During sessions with the child therapists helped identify the causes and consequences of the child’s substance use, set and reviewed clear, simple and obtainable counselling goals, and tracked the child’s satisfaction in multiple life areas to inform further goal planning. Other procedures included identifying and reinforcing pro-social behaviours that compete with substance use and skills training related to relapse prevention and problem solving. With parents, therapists reviewed important parenting practices for helping adolescents stay alcohol- and drug-free, and sought to improve problem-solving and positive communication in the family. In two of the four parent sessions the child joined them to practice communication and problem-solving.

The other alternative was Multi-Dimensional Family Therapy, which sought to engage not just the family but other significant figures in the child’s life (such as teachers and probation officers) to establish a social environment conducive to healthy development.

None of these more extensive or more elaborate alternatives significantly improved on the basic approach. Over the next 30 months, all were followed by worthwhile if limited improvements in substance use and related problems, but these could not all be attributed to the treatments. Most clients were involved with the criminal justice system and a fifth spent long periods in detention, hospital or other controlled environments. By definition, all the youngsters started treatment with substance use problems; on this count they could not get worse, but ‘natural’ turbulence in their substance use and problems (which was considerable) could mean that, even without treatment, at any later point some would get better.

Though not to a statistically significant degree, the Adolescent Community Reinforcement Approach did have the edge in remission and days abstinent over the other family therapy and the basic treatment at the two sites where it was implemented. Costing least, it also had some significant advantages in cost-effectiveness.

This at best partly encouraging picture should be seen in the light of the treatments and the populations being served. Some treatments were longer and more expensive than others but all were relatively brief, cheap and non-intensive. And though some did involve family and others, they were limited in the leverage they could exert or the resources they could provide to change the child’s life.

For about a third this may have been enough, but for others with multiple and severe difficulties in their lives, all the treatments may have been inadequate and perhaps misdirected at cannabis use, when much of the study’s sample was characterised by criminality, criminalisation, school problems, psychological disturbance, broken families, and, in the US context, atypically high rates of alienation from religious affiliation.

The opportunity for the study was provided by government funding of 34 treatment organisations to implement the Adolescent Community Reinforcement Approach, of which 29 joined the featured study. Each implementing therapist at the organisations was trained and expertly supervised by the approach’s developers, and provided feedback on the quality of their implementation.

For the featured study, additionally each organisation was randomly allocated carry on as usual (the control group) or to implement a pay-for-performance scheme which meant therapists could over a year earn an extra 4–7% of their salaries. The scheme rewarded them for each patient they delivered the approach to completely enough to meet the study’s standard (at least seven sessions over the first 14 weeks of treatment which delivered at least 10 of the 12 intended components of the programme), and also offered payment for quality of implementation, in both cases based on expert assessments of recorded sessions.

All but 15 of the 120 therapists at the 29 services agreed to join the study. Outcome data for their patients was available for only 81 therapists, amounting to 600 patients out of the 1173 at the services. Two-thirds of the youngsters were involved in the criminal justice system. Despite randomisation, there remained some differences between therapists and patients (the latter significantly related to substance use outcomes) offered incentives and those not. These were adjusted for in the analysis.

Main findings

As expected, offering incentives for these improved both the competence with which therapists delivered the therapy (24% v. 9% were assessed as competent) and the number of patients who received a programme which met the study’s completion standard (17% v. just under 3%). However, these improvements did not result in a greater proportion of patients who six months later had not in the past month experienced substance use problems and had not been in prison, hospitalised or held in some other controlled environment. This yardstick of remission was reached by 42% of the children seen by incentivised therapists but by 51% of the remainder, a slight and not statistically significant advantage for the non-incentivised therapists.

Regardless of incentives, patients of therapists assessed as competent in the therapy did not as a result do significantly better, but those who received a programme which met the study’s completion standard were significantly more likely to be in remission.

The authors’ conclusions

Findings suggest pay-for-performance can improve implementation of evidence-based treatments in routine practice settings. As expected, offering monetary bonuses to therapists substantially increased their competence in the Adolescent Community Reinforcement Approach and the completeness of their delivery as assessed directly by their interactions with patients, though there remained considerable room for improvement. There was no resultant impact on remission from substance use problems, possibly because remission rates were so high across the caseload, leaving little room for extra gains. Any impacts might have been obscured by the low overall patient follow-up rate of 61%.


Findings logo commentary The study exemplifies the not unusual pattern of this kind of implementation drive having the targeted impacts on the process of treatment, without this bearing any subsequent relation to how well patients do. As the authors point out, we can have particular confidence in this finding because the process measures were based on expert assessments of session recordings.

Part of the reason may be that (especially as routinely implemented) one bona fide treatment is as effective as another for young caseloads as they are for older ones (1 2). Also, in the featured study two-thirds of patients were involved with the criminal justice system. The leverage this exercised and the other inputs in to the children’s lives may have overwhelmed any extra gains from adequately implementing the Adolescent Community Reinforcement Approach for what was still a minority of the caseload. Possibly too, incentives led therapists to inappropriately accelerate the programme or neglect other ways of dealing with the youngsters which might have been more effective. Further considerations below.

More of the youngsters seen by incentivised than non-incentivised therapists participated in a well-delivered version of the Adolescent Community Reinforcement Approach, but according to the study’s remission yardstick, receiving this instead of a poorer quality approach or the alternative practices at the clinics did not help them stay free of substance use problems. It remains possible that a less absolute and more sensitive measure of improvement (such as reduction in the frequency of substance use) would have exposed a significant impact. Another possibility is that the well-delivered approach was indeed more effective, but so few youngsters got this (under 1 in 5) that spread over the whole caseload it did not make a significant difference. However, this seems an unlikely explanation for the negative results, because remission was actually less likely among the patients of incentivised therapists. There is no indication here that had more therapists reacted as intended to the incentives, we would have seen the expected improvements in their patients.

It might be objected that across all the therapists there was a relationship between completeness of implementation and remission, suggesting that the better the therapy was done, the better the outcomes. However, completion could largely have reflected how cooperative the children and families were and how motivated to receive help. To meet the completion criterion, not only would therapists have had to deliver seven sessions in 14 weeks, but patients attend them and make sufficient progress for the therapist to have moved through 10 of the 12 components of the programme.

For an account of the Cannabis Youth Treatment trial which first tested the Adolescent Community Reinforcement Approach and a discussion of its findings see study 7 in these Finding background notes. For an assessment of the (generally modest) impacts of treatment for youth cannabis use in general, see this Findings analysis.

Last revised 21 August 2014. First uploaded 14 August 2014

Comment/query
Open Effectiveness Bank home page


Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2013 Multidimensional family therapy lowers the rate of cannabis dependence in adolescents: A randomised controlled trial in Western European outpatient settings

REVIEW 2006 Motivational arm twisting: contradiction in terms?

REVIEW 2011 Alcohol-use disorders: diagnosis, assessment and management of harmful drinking and alcohol dependence

STUDY 2011 Treatment of adolescents with a cannabis use disorder: Main findings of a randomized controlled trial comparing multidimensional family therapy and cognitive behavioral therapy in The Netherlands

STUDY 2009 Multidimensional Family Therapy for young adolescent substance abuse: twelve-month outcomes of a randomized controlled trial

REVIEW 2013 The comparative effectiveness of outpatient treatment for adolescent substance abuse: A meta-analysis

STUDY 2009 Therapist behavior as a predictor of black and white caregiver responsiveness in multisystemic therapy

DOCUMENT 2010 Drug Strategy 2010. Reducing Demand, Restricting Supply, Building Recovery: Supporting People to Live a Drug Free Life

STUDY 2011 Using a cross-study design to assess the efficacy of motivational enhancement therapy-cognitive behavioral therapy 5 (MET/CBT5) in treating adolescents with cannabis-related disorders

REVIEW 2010 Research Review: The effectiveness of multidimensional family therapy in treating adolescents with multiple behavior problems – a meta-analysis