This entry is our analysis of a review or synthesis of research findings added to the Effectiveness Bank. The original review was not published by Findings; click Title to order a copy. Free reprints may be available from the authors – click prepared e-mail. The summary conveys the findings and views expressed in the review. Below is a commentary from Drug and Alcohol Findings.
Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text
Copy title and link
| Comment/query | Tweet
Collecting and delivering progress feedback: a meta-analysis of routine outcome monitoring.
Lambert M.J., Whipple J.L., Kleinstäuber M.
Psychotherapy: 2018, 55(4), p. 520–537.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Lambert at lambert.michaelphd@gmail.com.
Findings amalgamated for the American Psychological Association show that outcomes usually improve when therapists are provided with real-time feedback from the client on their progress and on factors affecting it such as the client–therapist relationship. Especially among clients (including substance use clients) who would otherwise deteriorate or not improve, these systems are among the most effective ways available to services to improve outcomes.
Summary [Though not specific to clients with drug and alcohol problems, the principles derived from this review of psychotherapy studies have been found applicable to patients treated for substance use problems. Addictions work may not necessarily best be conceptualised as psychotherapy, but there is a therapeutic element to it which makes these findings relevant to keyworkers and counsellors. This review updates an earlier version also in the Effectiveness Bank.]
Findings amalgamated for the American Psychological Association show that outcomes of psychotherapy or counselling are improved when practitioners are systematically and routinely provided feedback on client progress and how to improve it if clients are doing relatively poorly.
The types of studies included in the analyses permit the conclusion that the findings were due to a causal effect of feedback on client progress.
Two studies, one each of the two best-recognised feedback systems, show that such benefits can also be generated among clients treated for substance use problems.
The featured review is one of several in a special issue of the journal Psychotherapy devoted to features of the therapist-client relationship related to effectiveness, based on the work of a task force established by the American Psychological Association. This particular review examined the links between outcomes of psychotherapy and the feedback given to therapists on how well their clients are progressing.
Patchy therapist performance is a major reason for giving feedback. A study of over 6,000 clients treated in routine practice found that only about a third improved or recovered. Some therapists hardly have a single client who deteriorates, while others experience consistently high rates. Failures are usually due not to misapplication of therapeutic techniques, but to problems in the therapeutic relationship rooted in subtle or manifest rejection of the client. Clinicians often fail to identify deterioration and are poor at estimating how far clients ultimately will benefit from therapy, particularly those not improving.
Routine outcome monitoring systems address these shortcomings partly by identifying (in time for the therapist to do something about it) patients who are not doing well, and partly by improving the therapist’s performance by making them more aware (and correcting mistaken impressions) of how well clients are doing, what is working, what is not, and who in which areas needs further attention. Such systems regularly track progress using questionnaires clients themselves complete throughout treatment, information immediately provided to clinicians. The two most widely evaluated systems are described in the panel below.
The two most widely evaluated outcome monitoring systems were recognised in the [now suspended] US Substance Abuse and Mental Health Administration’s National Registry of Evidence-based Programs and Practices.
One is the 45-item Outcome Questionnaire System (1). Its primary aim is to improve outcomes for clients the system predicts would otherwise experience treatment failure. It assesses psychological disturbance (particularly anxiety and depression), interpersonal problems and social functioning. An individual client’s progress is compared against that expected for someone with a similar problem profile based on scores from over 11,000 clients who received routine care. Starting with the second therapy session, the comparison identifies ‘off-track’ clients who are not making the expected progress and are at risk of a poor treatment outcome. Studies have found that the system accurately predicts end-of-treatment deterioration in 85% to 100% of cases and can do so early in treatment, far exceeding clinicians’ predictive abilities. The results are then used to decide corrective action using the system’s flowchart of what to try in certain circumstances. A second scale can be completed by off-track clients to help them and their clinicians work through the flowchart by assessing the therapist–client working relationship and the client’s motivation, social support, and experience of stressful events in their life, scores on which key into the flowchart. Both scales are incorporated in an online software application that facilitates real-time electronic feedback for clinicians and if desired, for clients.
The other best recognised system is the Partners for Change Outcome Management System (1 2) which has two brief scales each consisting of just four items. Like the Outcome Questionnaire System, the first assesses the client’s psychological wellbeing, interpersonal relationships and social functioning, data used to predict their progress and identify at-risk clients based on a large archive of results from the system. Clients are flagged as at risk of therapeutic failure if their scores at the third session fall a set degree below the expected trajectory of similar patients. The second scale assesses the strength of the working relationship the client feels they have with the therapist – the ‘therapeutic alliance’. Normally these scales are completed by the client in the presence of the therapist, affording an opportunity to discuss the results and avert an impending negative outcome.
The featured review focused on these two systems, seeking studies published in English which had used them to provide feedback during psychotherapy of individuals, groups, or couples being treated for mental health problems, who had been assigned to programmes informed by one of the systems versus comparison programmes which did not systematically offer feedback to clinicians or patients. The studies also had to report effects on client outcomes in a way which could be amalgamated with the results of other studies in two separate meta-analyses, one for each system.
These analyses assessed average client progress and also the proportions of clients who definitely or ‘reliably’ deteriorated over the course of therapy versus those who did not, and a similar metric for the proportion who definitely improved.
All but two of the 15 studies of the Outcome Questionnaire assigned clients to either receive psychotherapy as usually delivered, or to receive the same psychotherapy informed by feedback on client progress. The exceptions compared the progress of clients under the feedback system with that of pre-system clients. Some studies also assessed the value of supplementing feedback on client progress with the questionnaire for off-track clients intended to help decide how to get them back on track. 8,649 clients were involved the studies, of whom 1,958 were assessed as off-track.
The nine studies of the Partners for Change system also assigned clients either to usual psychotherapy, or to the same psychotherapy informed by feedback on client progress, and in total involved 2,272 clients. Unlike the Outcome Questionnaire studies, the focus was mainly on results for all clients, not just those doing poorly. Results from these studies are expected to generalise to routine care rather than the selected samples of randomised trials.
Unlike other reviews in the same series, the methodologies of the studies included in these analyses enabled a direct presumption that any difference in outcomes between feedback and no feedback clients was actually caused by the feedback system, not just associated with it.
Across all relevant studies the Outcome Questionnaire System significantly improved client functioning and psychological wellbeing to a small degree across all clients (effect size 0.14), to a greater degree among off-track clients predicted to do poorly (effect size 0.33), and maximally among these clients when therapists were also given information on why they might be doing badly plus guidance on what could be done about it, when the effect size reached a moderate 0.49. Details below.
Compared to non-feedback clients, in 11 of the 15 studies the system’s own assessments recorded statistically significant gains among off-track clients predicted to otherwise have done poorly. Among the same clients and using the same assessments, across the eight studies whose results were amalgamated, feedback clients experienced significant extra improvements in post-treatment functioning and psychological wellbeing, equating to a small to medium effect size of 0.33. Consistent with these findings, across six studies clients predicted to do poorly were significantly more likely to have definitely improved than not (1.89 times more likely) and significantly less likely to have definitely deteriorated than not (0.61 times less likely). On all these measures, the results were consistent across the studies.
Results were similar (but less pronounced) when instead of focusing on off-track clients, the analyses included all the clients. Across 10 studies, the feedback system generated a small but statistically significant greater improvement in post-treatment functioning and psychological wellbeing, equating to an effect size of 0.14. Also, greater proportions of feedback clients definitely improved and fewer definitely deteriorated, but these results from just three studies were not statistically significant, so chance findings could not be ruled out. In the individual studies too, findings were rarely statistically significant.
In eight of the 15 studies, the practitioner was not just provided feedback on their clients’ progress, but also information from the client on why they might be doing badly (including a poor therapist–client relationship, lack of motivation or social support, or stressful events) and guidance on what could be done about it. Results could be amalgamated from six of these studies, across which the gains created by the feedback system were considerably greater than when studies were included which confined their feedback to client progress. Across the six studies, off-track clients on average ended treatment feeling and functioning better than when no feedback was provided at all, equating to a medium effect size of 0.49. Consistent with these findings, over the course of therapy clients predicted to do poorly were well over twice as likely to have actually definitely improved than not (2.40 times more likely) and almost two-thirds less likely to have definitely deteriorated than not (0.37 times less likely). All these findings were statistically significant.
However, the analysts were unable to rule out the possibility that amalgamated findings were affected by studies not found by their searches.
Among the nine studies of the Partners for Change system, compared to treatment without feedback, six found statistically significant gains in psychological wellbeing and social functioning as measured by the system’s own assessments. Across the course of therapy, the advantage equated to a small-to-medium effect size of 0.40. However, the analysts were unable to rule out the possibility that amalgamated findings were affected by studies not found by their searches.
Also, when feedback was provided just over twice as many more clients had definitely improved versus not, a statistically significant gain, though the proportion who had definitely deteriorated versus not was virtually the same with or without feedback. What seemed to be happening was that the feedback system transformed substantial numbers of clients who would otherwise not have deteriorated, but also not improved, into clients who evidenced definite improvement. These results were highly variable across the studies. In particular, three of the four conducted outside the United States did not find that feedback generated a significant advantage.
For both systems, the practical implication of the findings are that in routine practice, clinics and practitioners are highly likely to find that feedback systems help their clients improve and prevent deterioration or no-change, either across all clients or among the roughly a third who would otherwise do poorly. However, results for the Partners for Change system were highly variable for reasons which are unclear.
Across the 24 studies of both systems results did not seem to depend on the type of problem being treated, though this issue has not been well researched. For reasons which are unclear, a number of patients do not respond well to feedback systems.
Despite positive findings, these emerged from relatively few studies evaluating effectiveness conducted by a limited number of researchers. Most studies have been partly conducted by or have consulted the developers of the feedback systems, raising the possibility that the findings are due to the researcher allegiance effect. Almost all the studies used the same measure both to track progress and to evaluate the effect of the feedback system. Ideally, several validated methods would have been used to assess mental health at the beginning and end of treatment. In the absence of this, effect sizes may have been inflated. To illuminate the limits of these systems and clarify the factors that maximise client gains, research is needed across a wider range of treatment settings and client populations.
The research evidence supports routinely and formally monitoring the mental health of psychotherapy clients during therapy using either of the reviewed methods. Specifically, it is recommended that practitioners:
• use either the Partners for Change Outcome Management System or Outcome Questionnaire System with adults across treatment modalities (eg, individual, couple, and group) and clinical settings;
• use electronic versions of these systems which expedite and ease practical difficulties;
• use real-time feedback on client progress with an alert that identifies at-risk cases to compensate for the limited ability of clinicians to accurately detect clients who are deteriorating;
• examine feedback progress reports and alerts as vital signs of patient progress, not a reflection of one’s ability as a practitioner;
• in the Outcome Questionnaire System, use the flowchart of corrective actions and the optional extra information on why clients are doing poorly to elicit discussion with clients and to solve problems with at-risk cases, generating additional clinical benefits beyond feedback on client progress alone.
commentary Though conventionally seen as moderate, the largest effect size (0.49) recorded by the analyses is relatively substantial for substance use treatment research, and much larger than that usually attached to the interventions themselves. Especially among clients who would otherwise deteriorate or not improve, these systems are among the most effective ways available to services to improve outcomes.
Apart from the limitations of the research noted by the authors, their analyses concerned only improvement over the course of therapy during the time clients would have been completing the feedback systems’ assessments. Unknown is whether the extra gains made by feedback patients were maintained once therapy ended. The reviewers warned of possible bias due to researchers’ allegiances to the systems they were evaluating, but the same caution applies also to the review itself. Its lead author developed one of the feedback systems the review evaluated and is a partner at the company which owns and distributes the system’s software. Nevertheless, accumulated evidence from relatively rigorous trials and the face validity of the argument that feedback can improve outcomes, suggests the evaluated systems really are effective.
Both systems require clients to complete questionnaires at sessions and practitioners to examine the results, an extra burden on both. An alternative tried which at least relieves the client of the burden is to record sessions and use those to feed back to practitioners during their supervision.
Another issue is whether to feed back to practitioners only or also to therapists. In 2010 a review of the then six evaluations of the Outcome Questionnaire System described above suggested that providing feedback to the client led more to reliably improve, but also more to reliably deteriorate. This paper also reported results for clients not identified as doing poorly. Because all seemed well, their therapists did not receive extra feedback and guidance, so the only question was whether outcomes were improved by feeding back data on the client’s psychosocial progress. On average the answer was in the affirmative whether or not clients were also provided feedback, and in both cases more clients reliably improved. However, fewer clients reliably deteriorated only when it was just the therapist who received feedback. One interpretation is that some clients can benefit from feedback on their poor progress, while others become yet more demoralised.
Feedback systems have been discussed in the Effectiveness Bank’s Drug Treatment Matrix, and the Bank also includes a key study in the substance use sector which demonstrated the value of the Outcome Questionnaire System – as long as it includes not just feedback on client progress, but also on why off-track clients might be doing badly and guidance on what could be done about it. This and other studies are summarised below.
One reason why spotting and responding to poor patient progress is important is that after psychosocial therapy up to 15% of substance use clients end up worse on the outcomes assessed by studies than before. Spotting and rectifying this is not just an effectiveness issue, but has been argued to be an ethical requirement.
Systematising feedback to practitioners in substance use treatment was tried (article starts on page number 204 as printed) in a simple but effective way in late 1980s’ USA in a small study of six counsellors at a drug-free outpatient clinic, who varied considerably in their engagement of and success with patients. To improve patient participation, the clinic set attendance standards, and each counsellor received monthly written feedback on the performance of each of their clients against these standards. Attendance at group and individual counselling sessions significantly improved and more counsellors had caseloads who on average met the standards.
Published in 2012, a key study was conducted at three US substance use counselling services. It adapted the Outcome Questionnaire System described above by extending client assessments to include two items recording the number of days in the past week they had used alcohol or drugs. Patient progress under this system was compared to that before the it had been implemented. For new patients in individual counselling, assessments were made just before each session and immediately fed back to the therapist.
The full system was implemented, including alerting counsellors to off-track patients and giving them feedback on why they might be lagging due to poor therapeutic relationships, lack of motivation, weak or the wrong kind of social support, or stressful events. This additional step with off-track patients proved vital. Up to the point when counsellors were alerted there was little difference between feedback and non-feedback patients, but from then on feedback patients progressed much better than in pre-system days on all three measures (drug use, drinking, total problem scores). In the end, patients who at first were doing less well than expected ended up with substance use levels no greater than those of more promising patients.
Exemplifying the featured review’s caution about possible researcher allegiance bias, one of the authors of the study had developed the feedback system they evaluated and was, the featured review noted, a partner at the company which owns and distributes the system’s software.
The other system evaluated by the featured review – the Partners for Change Outcome Management System described above – was trialled among US soldiers enrolled in outpatient group therapy at an army substance use treatment programme, typically because they had been referred by their commanding officers after alcohol or drug-related misconduct. All the soldiers met diagnostic criteria for substance abuse or dependence. Apart from the feedback system’s own assessments, each patient’s commander (who did not know which soldiers had been allocated to feedback) and therapist rated their behaviour and conduct both generally and with respect to substance use. There was, however, no direct measure of substance use.
The 263 soldiers were assigned to therapy groups run by 10 therapists. Within each group roughly half the patients had been allocated at random to have their progress fed back to the therapist and half not. The therapists were presumably limited in the extent to which they could (as the system intends) discuss feedback results with patients since others in the group were not subject to this process, and some of the system’s group-related assessments had yet to be developed. Nevertheless, the results closely duplicated those seen in the featured review: small but significantly greater improvement from before to the end of therapy in feedback versus non-feedback patients, gains not focused on those flagged as progressing below expectations but spread across the entire sample.
Specifically, significantly greater improvements were seen in the system’s own assessments of functioning and wellbeing and in commander and therapist ratings. More feedback patients (28% v. 15%) completed treatment having substantially improved and also having reached scores on the system’s assessments no longer associated with clinically significant mental ill-health and poor functioning. Improvers were recruited from patients who would otherwise have registered no improvements rather those who deteriorated; proportions of the latter did not significantly differ among feedback versus no feedback patients. Feedback patients also attended more of the intended five sessions, and were less likely to drop out prematurely before having reached normal assessment scores. However, attendance was not the active ingredient in how feedback improved outcomes; even among those who had attended all five sessions, greater improvement was seen in feedback than non-feedback patients.
Again exemplifying the featured review’s caution about possible researcher allegiance bias, two of the authors of the study are senior staff (including its chief executive) at the organisation which promotes the Partners for Change Outcome Management System.
Listed below are analyses of the other reviews commissioned by the American Psychological Association task force.
Overall conclusions based on the other 16 reviews
The ‘real’, person-to-person relationship
Therapist–client alliance
Alliance in couple and family therapy
Alliance in child and adolescent therapy
Cohesion in group therapy
Therapist empathy
Goal consensus and collaboration
Positive regard
Therapist congruence/genuineness
Therapist self-disclosure and ‘immediacy’
Therapist and client emotional expression
Repairing ruptured alliances between therapists and clients
Managing ‘countertransference’
Treatment credibility
Treatment outcome expectations
Thanks for their comments on this entry in draft to Duncan Raistrick, Clinical Director (retired) of the Leeds Addiction Unit in England. Commentators bear no responsibility for the text including the interpretations and any remaining errors.
Last revised 28 March 2019. First uploaded 14 February 2019
Comment/query
Open Effectiveness Bank home page
Top 10 most closely related documents on this site. For more try a subject or free text search
REVIEW 2011 Evidence-based psychotherapy relationships: Collecting client feedback
ABSTRACT 2011 Evidence-based therapy relationships: research conclusions and clinical practices
REVIEW 2018 Meta-analysis of the alliance–outcome relation in couple and family therapy
STUDY 2011 Therapist effectiveness: implications for accountability and patient care
REVIEW 2018 Psychotherapy relationships that work III
REVIEW 2018 Meta-analyses of the relation of goal consensus and collaboration to psychotherapy outcome
REVIEW 2018 Congruence/genuineness: a meta-analysis
REVIEW 2018 The alliance in adult psychotherapy: a meta-analytic synthesis