The impact of paying treatment providers for outcomes: difference-in-differences analysis of the ‘payment by results for drugs recovery’ pilot
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

The impact of paying treatment providers for outcomes: difference-in-differences analysis of the ‘payment by results for drugs recovery’ pilot.

Mason T., Sutton M., Whittaker W. et al
Addiction: 2015, 110(7), p. 1120–1128.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Mason at thomas.mason@manchester.ac.uk.

A flagship drug treatment policy initiative appears to have backfired in England, where the government’s pilot payment-by-results schemes seem to have led to fewer successful completions of treatment and more prospective patients declining treatment.

Summary Typically pay-for-performance schemes have linked payments to health service providers to measures of clinical quality rather than outcomes, but recent schemes introduced by the UK government have focused more clearly on paying for ‘outcomes’. These funding arrangements are promoted as paying treatment services for the value they produce, and are intended to incentivise innovation rather than the adoption of pre-set procedures and techniques. However, they can also generate unintended effects and transfer financial risk from the funders to the providers of services. Among these initiatives has been a scheme for funding treatment of drug and alcohol problems, the subject of the featured study.

Key points icon

Key points

In 2012 the Department of Health in England introduced a pilot programme in eight areas under which drug and alcohol treatment services received payments based on their patients achieving ‘recovery-focused’ outcomes.

Compared to other areas, these payment-by-results schemes appear to have led to slightly fewer successful completions of treatment and more patients declining treatment.

These unwanted results may be reversed in the longer term, and it remains possible that performance on other yardsticks of success has been improved by the schemes, particularly post-treatment relapse.

Previously commissioning of drug treatment services had focused on retention in treatment as the principal measure of effectiveness, and service providers were paid using ‘block’ or ‘activity’ contracts. A break from this came in April 2012 when the Department of Health in England introduced a pilot programme under which drug and alcohol treatment services received payments based on their producing ‘recovery-focused’ outcomes. This pilot in eight areas was a pillar of the 2010 drug strategy, which prioritised service users recovering from their dependence on drugs or alcohol and successfully completing and leaving treatment. Primarily against the yardstick of successful completion, the featured study compared the performance of the pilot areas in their first year against those of other areas in England, drawing on data routinely collected by the National Drug Treatment Monitoring System on patients being treated for problems with drugs other than alcohol.

Evaluation yardsticks

Under the pilot scheme drug action teams responsible for organising treatment services in their areas linked provider payments to performance indicators specified (with limited scope for local variations) in a national outcomes framework. Outcomes were grouped under: progression towards abstinence from problem drug(s); reduction in offending; and improved health and wellbeing. The evaluation focused on the first domain, and specifically on successful treatment completion as the key proxy for recovery used by Public Health England. Together with not returning to treatment within the next year, this outcome attracted a large part of the payments to services in the pilot areas. Since the non-return outcome cannot be assessed at least 12 months after discharge, the featured report was confined to the first phase – successful completion of treatment – which triggers the first payment to providers in the abstinence domain. The monitoring system records a successful completion when an individual completes their planned treatment having been judged by a clinician to be free of dependence on the drug in respect of which they were being treated, and also using neither heroin nor crack. The rate of successful completions is the proportion these form of all treatment episodes during the financial year.

Because it seemed possible that treatment changes generated by the schemes might have weakened engagement with treatment services, the evaluation also assessed rates of treatment refusal – patients referred to a treatment service and who met face-to-face with service staff, but decided not to start treatment.

The key measure was not the absolute rates of completions and refusals, but how these changed from before (April to December of 2011) to after (April to December of 2012) the start of the pilot schemes. If after adjusting for other influences, they changed to a significantly different degree in the pilot areas compared to other areas, this would be evidence that introducing the schemes had had an impact. To assess this, before-after trends in the eight pilot areas were benchmarked against trends in:
• all 140 other drug action team areas in England;
• the 42 areas most similar to the pilot areas in the percentages of the local population who used opiates or crack and the areas’ deprivation indices;
• the 90 areas in the same four government-office regions of England (East, North East, South West, West Midlands).

Some non-pilot areas have nevertheless implemented payment-by-results schemes, so similar comparisons were made between trends in successful completions and refusals across all payment-by-results areas versus areas with no such schemes as of July 2012.

Main findings

Whether confined to the pilot areas or across England, and whatever the benchmark, in payment-by-results areas successful treatment completions fell once schemes started and treatment refusals rose relative to other areas. Details below.

Relative to other areas, after the schemes started patients in pilot areas became significantly less likely to successfully complete treatment. After adjusting for other influences, it was calculated that 1.3% fewer treatment episodes ended in successful completion than would have done had the pilot areas followed the same before-after trajectory as other areas. Raw figures were in pilot areas, about 13.7% of episodes ending in successful completion in 2011 falling to 11.3% in 2012, and in all other areas, a smaller fall from 12.5% to 12.2%. The extra drop in the completion rate in pilot areas was comparable when the benchmark was changed to similar areas or to areas in the same regions.

While successful completions appeared to fall as a result of the schemes opening, the proportion of prospective patients who declined treatment rose slightly but significantly relative to all other areas by about 1%. Raw figures were in pilot areas, 0.47% declining treatment in 2011 rising to 1.03% in 2012, while in all other areas there was a slight fall from 0.55% to 0.43%. This extra increase in refusals remained stable when the benchmark was changed to similar areas or to areas in the same regions.

When all payment-by-results areas (ie, not just the pilot areas) were compared with those without such schemes, there remained an extra 1% decrease in the rate of successful completions. At 0.15% the extra increase in treatment refusals was smaller than in the pilot areas, but still statistically significant.

The authors’ conclusions

After the introduction of payment-by-results pilot schemes, patients in those areas were significantly less likely to complete treatment and significantly more likely to decline to commence treatment. It remains to be seen whether these apparently negative impacts would be matched by similar impacts in the other outcome domains rewarded by payment-by-results schemes and on other measures of treatment quality.

Possibly the decrease in successful completions was due to treatment services holding on longer to clients to try to ensure they were sufficiently recovered not to return to treatment in the following year; if they did, the service would lose the single largest element of funding under the schemes. An increase in treatment refusals seems an unintended consequence of the schemes, one which can be expected to undermine recovery from addiction and increase the chances of the drug users concerned having to attend emergency departments and be admitted to hospital.

It should be remembered that these results derive from the first year of the pilot schemes. Rates of treatment completion may improve in the longer term as the considerable change in treatment systems necessitated by the schemes settles down. Also, this study was not able to randomly allocate areas to implement payment-by-results schemes. Despite adjusting for other influences, there may remain differences between the areas which volunteered for and were chosen as pilot areas (or those which chose themselves to implement schemes) and other areas which affected trends in treatment completions and refusals.


Findings logo commentary This study funded by the Department of Health found a strikingly consistent picture of payment-by-results schemes having the opposite effect to that intended on successful completions, weakening performance on what the government and Public Health England see as a critical indicator of successful treatment. Whether in terms of patient welfare and societal benefits this really did mean introducing the schemes was counterproductive could depend on whether fewer successful completions were counteracted by more patients staying in rather than dropping out of treatment, and whether in the longer term the schemes will be shown to have performed better than in the first year of operation. More below.

As the authors speculate, it is possible that services in the pilot areas held on slightly longer to patients who qualified for successful discharge in order to bolster the stability needed to avoid relapse and treatment return, and that in this they were successful. Though being retained in treatment is no longer favoured by government, it has been associated in England with a better crime-reduction record than successful completion, and is the most consistent indicator of desired outcomes for patients and society. If retention did improve, both patients and society more broadly may have gained from the pilot schemes. However, if fewer successful completions meant, not more patients staying in treatment, but more dropping out, the consequences are likely to have been negative. Also as the authors suggest, it seems likely that the results could have been due to a transient disruption consequent on introducing the schemes, and that later years will see improvements.

Another strand to the featured evaluation tried to find out what was happening in the pilot areas by interviewing or holding focus groups with stakeholders, service users and carers. According to an interim report, these sources appears on balance to favour continuing with some element of payment by results in their local funding formulas, with adaptations based on the experience of the pilots, but this picture comes from within areas which had volunteered for the schemes. The clearest negatives seemed to be unwelcome pressure towards abstinence and treatment exit, particularly curtailing the prescribing of medications like methadone intended to alleviate the need for opiate-dependent patients to use illegal heroin.

An interesting aside in the featured report is the almost total lack of competition between services in the pilot areas, meaning patients had little or no choice of which treatment organisation to go to: “Across the eight pilot areas, the average number of providers in 2011–12 was 1.875. In theory, competition existed in only one of these eight areas, and this area had two providers offering distinct, stand-alone services.” This might be happening too in other areas as commissioners opt for the cost-savings and convenience of bundling all local services under one mega-service umbrella, and smaller organisations lose out in the expensive and time-consuming business of tendering for re-commissioned services.

Findings from areas which implemented payment-by-results schemes on their own initiatives are complicated by the lack of clarity over when those schemes came in to operation. What can be made of the small increase in non-commencement of treatment is unclear. The most worrying interpretation would be that services are becoming more choosy about who they take on in the attempt to cherry-pick those most likely to more rapidly be successful, and that this results in more patients being persuaded that the service is not for them. From the featured report, it seems that patients deterred in this way would normally have had no alternative service to go to, and appear on the records as having declined treatment. However, if services were doing this, the strategy would at least in the first year appear to have backfired. Other possibilities are that cherry-picking went the other way, services opting for higher tariff but more difficult patients, and/or that patients reacted badly to the first step in the process being assessment of the tariff they would attract rather than assessment by a treatment service.

In 2017 the same evaluation project produced its final report on the eight pilot schemes, extending the comparison windows to two years before the start of the schemes in April 2012 versus the following two years, and separately reporting on clients whose primary substance use problem was alcohol versus those for whom it was another drug. For patients whose primary drug problem was not alcohol the analysis netted 154,175 new treatment journeys from before the start of the pilot programmes and 148,941 afterwards; for primary alcohol clients, the corresponding figures were 159,153 and 165,833. As in the earlier study, their outcomes were benchmarked against those of areas which had not implemented payment by results schemes.

Again the results were not encouraging: “Overall … the introduction of [payment by results] did not seem to be associated with the desired effects of outcome-based commissioning of drug and alcohol treatment services.” Many of the results were similar for drug and alcohol patients. For both sets the proportion of patients who had successfully completed treatment had fallen more in scheme than non-scheme areas as had the proportion who successfully completed and did not return to treatment for at least a year – counterproductive findings relating to a key metric used as a proxy for rates of recovery from dependence. Perhaps related to these findings was extended retention in treatment in the pilot areas, but greater rates of unplanned discharge indicative of unsuccessful treatment. Death rates had not significantly improved in the pilot areas. In these areas the cost of treatment had increased significantly more than in non-pilot areas.

Not applicable to alcohol clients were the relative improvements in pilot scheme areas in the proportion of patients who were injecting and rates of abstinence during treatment as recorded by the patients’ clinicians, an important ‘result’ which attracted payments. There was some evidence of a beneficial impact on rates of acquisitive crime among the drug clients, a measure also reported for the alcohol patients (it had not changed to a significant degree) but of far less significance and less closely related to the success of treatment. Staff “frequently highlighted” that the payment by results schemes had led to a greater emphasis on reducing the doses of substitute drugs like methadone prescribed to dependent opioid users, an emphasis unlikely to have helped reduce drug-related harm.

There were also some notable differences in how the introduction of payment by results schemes had affected drug versus alcohol patients. Compared to non-pilot areas, more drug clients had to wait over three weeks before they could start treatment, and after assessment, more failed to start treatment, but neither was the case for alcohol clients. These findings reflect the views of commissioners, service managers and practitioners from four of the pilot sites that “the provision of alcohol services [had] improved over the course of the pilot, partly as a result of greater emphasis and availability of funding for this support under [payment by results]”. The new payment mechanism seemed to have levelled up the provision of alcohol treatment to nearer that of treatment for problems with drugs like heroin and crack, reducing a gap long complained of in the alcohol sector.

Earlier the government had produced its own assessment of the initial performance of the pilot schemes based on the 6,582 patients being treated for drug rather than alcohol problems to February 2013. It found consistent gains (compared to what happened in the same areas before and to the national average) only in the proportion of patients who while in treatment said (via forms completed by staff) they had stopped using their problem substance(s). But in line with the featured study, against the same comparators the proportion exiting treatment free of dependence worsened in the pilot areas. Other measures such as housing status and quality of life were largely unaffected by the opening of the schemes. For the 3,081 patients whose problems were mainly with alcohol, things seemed somewhat worse. There was no indication that the pilots had elevated abstinence rates and the proportion exiting treatment free of dependence was lower than in the rest of England and lower than in the same areas before the pilots.

For more on payment-by-results in the UK and on treatment commissioning in general, see this Effectiveness Bank hot topic.

Thanks for their comments on this entry in draft to Russell Webster. Commentators bear no responsibility for the text including the interpretations and any remaining errors.

Last revised 02 April 2020. First uploaded 16 June 2015

Comment/query
Open Effectiveness Bank home page


Top 10 most closely related documents on this site. For more try a subject or free text search

HOT TOPIC 2016 What about evidence-based commissioning?

MATRIX CELL 2018 Drug Treatment Matrix cell E2: Treatment systems; Generic and cross-cutting issues

DOCUMENT 2017 Commissioning impact on drug treatment: The extent to which commissioning structures, the financial environment and wider changes to health and social welfare impact on drug misuse treatment and recovery

REVIEW 2011 A policy-oriented review of strategies for improving the outcomes of services for substance use disorder patients

MATRIX CELL 2020 Alcohol Treatment Matrix cell E2: Treatment systems; Generic and cross-cutting issues

STUDY 2012 Drug system change pilots evaluation: final report

DOCUMENT 2010 Commissioning for recovery. Drug treatment, reintegration and recovery in the community and prisons: a guide for drug partnerships

STUDY 2017 Does paying service providers by results improve recovery outcomes for drug misusers in treatment in England?

DOCUMENT 2010 Drug Strategy 2010. Reducing Demand, Restricting Supply, Building Recovery: Supporting People to Live a Drug Free Life

DOCUMENT 2017 Drug misuse and dependence: UK guidelines on clinical management