Developing and validating process measures of health care quality
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study added to the Effectiveness Bank. The original study was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

Developing and validating process measures of health care quality.

Harris A.H.S., Kivlahan D.R., Bowe T. et al.
Medical Care: 2009, 47(12), p. 1244–1250.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Harris at

Finding that a retention benchmark like that used for years in Britain was only loosely related to patient improvement led a US health service to start a comprehensive search for better indicators. Intensity of contact in the first month best predicted which services most benefited their patients.

Summary Collecting information on the intended outcomes of treatment such as remission of dependence and reduction in substance use is difficult and costly and data is often incomplete and subject to manipulation. An alternative is to use routinely collected information on the process (such as retention, attendance, and staffing) rather than the outcomes of treatment, and to use these to construct quality indicators which predict good outcomes. The problem is that though these may be inexpensive and easy to generate, often they are poorly related to the intended outcomes or the relationship has not been investigated. To address this issue, the US's health service for former military personnel generated a range of candidate indicators and assessed their relationships to the outcomes of alcohol/drug problem treatment.

The process in theory

The researchers reasoned that process quality indicators should meet three criteria:
• at the agency level, across all their patients treatment agencies which score higher should on average have better outcomes;
• at the individual level, individual patients whose treatment embodied these indicators should do relatively well;
• the kind of care implied by the indicators should be supported by research and acceptable to patients and clinical staff.

Many widely implemented indicators do not meet these criteria, largely because they were derived from expert opinion or consensus without being validated against real outcomes. For example, an indicator focused on retention for at least three months in low-intensity treatment has not been found to predict which treatment agencies for former US military personnel have on average the best outcomes, and only modestly to predict which individual patients do well, and then only for some types of patients.

An alternative development model tested in this study begins on the one hand by collecting treatment outcome data from enough agencies to detect clinically significant differences, and on the other, by developing a set of candidate quality indicators (based on expert opinion or some other method) which can be constructed from existing data sources. The relationships between these two are then assessed to discover which potential indicators really do predict good outcomes, both at the level of the agency and the individual patient.

That is as far as the example given by the current report extended ( below), but the process should go further to test the same relationships using a new set of agencies and/or patients. Indicators which survive this double check should be assessed by experts and stakeholders for (among other issues) their compatibility with clinical guidelines or recognised evidence, clinical acceptability, the risk of manipulation, and possible unintended consequences. After this, the indicators can be piloted and re-evaluated in the light of new clinical evidence and/or unintended consequences.

Applied to alcohol treatment

The first part of this procedure was piloted using data from 2701 patients starting treatment for substance use problems at 71 outpatient programmes based at 54 agencies run by the US health service for former military personnel. Indications that quality indicators might differ for different types of substance use problems led the researchers to focus on alcohol-related outcomes. The patients in the study completed assessments of the severity of their problems at treatment entry, and about two thirds For the remainder the scores they would have registered were estimated on the basis of their baseline scores, diagnostic data, age and sex. returned repeat assessments by mail about seven months later. At issue was whether the degree to which they had improved on drink-related measures could have been predicted by what happened during their treatment as reflected in indicators derived from routinely collected administrative and clinical data.

All the candidate indicators considered feasible given the available data, and supported by theory or research, were concerned with intensity and duration of care. Their constituents were composed of retention periods, number of times the patient attended during those periods at different stages (the first month as opposed to the first two or three months), and whether the visits related to their addiction or mental health problems. Over 100 possible indicators were constructed from permutations of these constituents. Each was assessed for the degree to which it was associated with drink-related improvements at the seven-month follow-up both at agency and individual levels.

Main findings

Combining mental health with addiction attendances did not improve any of the possible indicators' abilities to predict outcomes, so only addiction attendances were considered. Just nine of the candidate indicators were significantly linked to improvements in the main measure The Addiction Severity Index's Alcohol Composite score. of patients' drink-related problems at both agency and individual levels. Five of the indicators – the ones most closely related to outcomes – reflected the number of times patients attended in the first month of their treatment. The remainder concerned numbers of attendances per month over the first two or three months of treatment.

At the agency level, the strongest indicator of which agencies had the best average outcomes was the proportion of their patients who attended at least three times in the first month – though how many attended at least twice, four, five or six times were not far behind. These indicators accounted for about a quarter of the variation between agencies in how well their patients did. Also significantly related to patient improvement (but less strongly than first-month indicators) were the average number of months over the first two or three months during which patients attended five, six or seven times.

By design, these same indicators also significantly predicted how well each individual patient did, but these relationships were much weaker, accounting at best for 1% to 2% of the variance in outcomes. Again, attendance in the first month provided stronger indicators than over the first two or three months, and stronger still the more often the patient had attended between the range from two to six times.

This picture was broadly confirmed by other analysis methods and in relation to other drink-related outcomes including the number of days on which a patient was drunk, experienced alcohol-related problem, or did not drink.

The authors' conclusions

The study found nine possible indicators of the quality of outpatient treatment which predicted the degree to which on average an agency's patients improved, and also which individual patients improved most. The strongest were attending from three to six times during the first month of care. Though strongly related to an agency's average performance, these indicators did much less well at predicting which individuals would improve most.

Identifying candidate indicators by their relations to outcomes is (as explained above) just the first part of the process. It is also important to bear in mind that this process for developing indicators cannot determine whether relationships found with outcomes mean these processes actually cause patients to improve. The processes and the outcomes might, for example, both be related to another influence such as the motivation of the patients. In this scenario, raising attendance levels 'artificially' might not improve outcomes unless it also somehow affected the real outcome-generator – the patient's motivation.

Findings logo commentary Accepting the cautions of the authors, the fact that at an agency level, first-month attendance was so strongly related to an agency's performance suggests that treatment services which offer and are able to encourage their patients to accept several visits are the kind of organisations which foster the greatest positive change. It is reasonable to expect these agencies to be welcoming and to rapidly forge relationships with patients which encourage them to return. It also seems to make sense to (within the not very frequent limits found by the study – around weekly) front-load scheduled sessions at the start of treatment when patients' expectations, appetite for treatment, motivation, and need for support may be at their height. It is also of interest that the impetus for the study was the poor performance of an indicator – three-month retention – very similar to the 12-week retention indicator used recently as a benchmark for British drug dependence treatment services, though in relation mainly to the treatment of opiate-addicted patients.

Last revised 30 June 2011. First uploaded

Open Effectiveness Bank home page

Top 10 most closely related documents on this site. For more try a subject or free text search

STUDY 2011 Performance-based contracting within a state substance abuse treatment system: a preliminary exploration of differences in client access and client outcomes

STUDY 2012 Delivering service quality in alcohol treatment: a qualitative comparison of public and private treatment centres by service users and service providers

STUDY 2012 A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs

REVIEW 2011 Alcohol-use disorders: diagnosis, assessment and management of harmful drinking and alcohol dependence

STUDY 2012 A pilot randomised controlled trial of brief versus twice weekly versus standard supervised consumption in patients on opiate maintenance treatment

ABSTRACT 2008 Improving public addiction treatment through performance contracting: the Delaware experiment

STUDY 2010 Does meeting the HEDIS substance abuse treatment engagement criterion predict patient outcomes?

STUDY 2012 Implementation issues in an innovative rural substance misuser treatment program

REVIEW 2018 Collecting and delivering progress feedback: a meta-analysis of routine outcome monitoring

STUDY 2008 Replication and sustainability of improved access and retention within the Network for the Improvement of Addiction Treatment