Baldwin S.A., Christian S., Berkeljon A. et al.
Journal of Marital and Family Therapy: 2012, 38(1), p. 281–304.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Baldwin at firstname.lastname@example.org.
Review assesses the effectiveness selling points of four largely ‘privatised’ brand-name family therapies for troubled and delinquent teens. Yes, they work better than usual or individualised approaches, but not much and not always, and most of the research has been done by people who stand to gain from positive findings.
Summary The featured review used advanced meta-analytic techniques to amalgamate results from studies of four family therapy approaches which have shown considerable promise for treating youth delinquency and substance use problems. The four approaches (Brief Strategic Family Therapy; Functional Family Therapy; Multidimensional Family Therapy; and Multisystemic Therapy) share a focus on changing dysfunctional family patterns or ‘systems’ which contribute to the onset and maintenance of adolescent delinquency and substance use. In practice, typically they help parents and adolescents communicate better and reduce conflict, improve parenting skills (eg, limit-setting), and help adolescents become better integrated with the systems they are involved in outside the family, such as the school.
Analysts sought published and unpublished reports of the outcomes of trials which randomly allocated children aged 11–19 or their families to these approaches implemented as ‘standalone’ therapies, versus a comparison approach which might be an alternative therapy also organised by the researchers, treatment as usual (eg, court referral to a juvenile delinquency treatment centre), or no particular help. Primarily at issue was whether at the first post-treatment follow-up point (usually when treatment had finished) these approaches improved on the comparators in reducing indicators of delinquency and substance use, the main targets of the interventions, though other outcomes too were assessed. In all 24 such studies were found, half testing Multisystemic Therapy and from three to five each testing the other three approaches.
All but one of the 11 studies where the comparator was treatment as usual concerned Multisystemic Therapy. Across all 11, family therapies registered modestly but significantly better outcomes, amounting to a small effect size of 0.21, though results were heavily influenced by one study in which Multisystemic Therapy performed unusually well. Excluding this, the extra improvement in outcomes was even more modest (effect size 0.12). The advantage gained by Multisystemic Therapy was similarly reduced, from an effect size of 0.22 to 0.13. In both cases, though reduced the impact remained statistically significant. Specifically in respect of delinquency and substance use combined, outcomes were significantly better than treatment as usual, amounting to a statistically significant effect size of around 0.30 regardless of whether the ‘outlier’ study was included.
Another 11 studies compared outcomes from the four family therapies versus alternative research-organised treatments not based on these approaches, such as group therapy, individual counselling for the child, or groups for parents. Across all the approaches, family therapies led to statistically significant extra benefits, amounting to a small effect size of 0.26. Each of the four approaches accounted for from two to four studies. With so few studies, the extra gains made by each of the approaches did not differ to a statistically significant degree. In the lead though was Multisystemic Therapy with an effect size of 0.57, but this was possibly due to chance and derived from just two studies led by the same researcher, and involving the programme developer. [Editor’s note: The most important of the two seems best seen as a contrast between the well structured family therapy and a comparator “selected to represent the usual community treatment for juvenile offenders”. Rather than an equally well structured alternative approach, this is described as an eclectic variety of interventions dependent on the preferences of the therapists, sharing only that they focused on the individual adolescent rather than the family system.] Delinquency and substance use were the main targets of the interventions. When these outcomes were combined, they registered a statistically significant aggregate effect size of 0.43, verging on what is conventionally considered a medium-sized impact, and one greater than their impacts on more secondary outcomes. However, studies focused on treating substance use problems had significantly worse outcomes than studies that focused on treating delinquency – a result possibly influenced by the fact some of the delinquency studies recruited their samples entirely from the justice system, among whom outcomes were relatively good.
Just four studies compared outcomes from one of the family therapies against no particular help, registering a relatively large aggregate effect size of 0.70, greater than when the comparison was against an active treatment.
Confirming earlier research syntheses, for 11–19-year-olds impacts of the four family therapies included in this analysis appear to modestly improve on those from treatment as usual or from other therapies in respect of substance use and delinquency, and to a lesser degree other outcomes too. However, with few studies it was not possible to securely identify which approach had the better record. Among the limitations of the analysis was that it analysed only outcomes from the first follow-up point, typically at the end of treatment.
commentary Britain’s National Institute for Health and Clinical Excellence (NICE) has recommended this family of programmes, all of which integrate intervention in to several aspects of a child’s life and environment, for children and young people who misuse alcohol and who also have other major problems and/or limited social support. A later meta-analytic synthesis of research on the treatment of substance use in young people (in this case, aged 12–20) judged, “The most convincing and consistent comparative effectiveness finding was for family therapy, which showed relatively large positive effects relative to other treatments”. Though all sorts of family therapies were included, most studies were of ‘brand-name’ programmes, including those investigated in the featured review.
There seems no doubt that these approaches can improve on typically less well organised and less extensive usual practices, but it also seems this is not always the case, that evaluations conducted independently of the programme developers have been (but not always) unconvincing, and overall results are not as impressive as the investment in these programmes might be seen to require, especially if they must be costed as add-ons to legally or socially required procedures rather than replacements. A major impediment to their use is the expensive training and supervision and considerable skills required to implement them in ways which have been associated with good outcomes. See below for more on these themes.
Within the article, a clinician from the same US institution as the lead author commented that the findings seemed to confirm the superiority of treating young people in the context of their family systems. With no statistically significant differences between the approaches in their degrees of superiority, decisions on which to implement can be taken on practical grounds. Benefits stretch across behavioural problems (eg, sexual offences, serious drug use, bullying), an advantage for real-world services which try to help youngsters who typically present with multiple problems.
The main impediment to extending these approaches is, he suggested, that training is not readily accessible. It is expensive, time-consuming, and usually not available for individuals as opposed to teams. Making this considerable investment is deterred by high staff turnover. In a vicious circle, training organisations have little incentive to invest in training for these approaches when there are few job opportunities for their trainees. Once trained, practitioners are expected to work intensively with small caseloads and to go beyond the focal client to the wider systems they interact with, extensions some funders do not recognise. When money is tight, there is a temptation to cut back on these high-cost approaches.
In a rational system cost would be weighed against results to test whether extra cost was worth the investment. Such analyses have however not found that the extra costs of the types of programmes included in the featured analysis warrant their returns in terms of impacts on substance use and delinquency or extra benefits overall for society (1 2 3).
The commentator dealt at length with a controversial issue – that researchers in to these family approaches are usually not disinterested examiners, but have themselves developed the approaches, and often have a financial stake in their success as well as an intellectual affiliation to the approach.
Three of the four models have, as he put it, been ‘privatised’, training being provided on a for-profit basis which allows retention and ownership of intellectual property by programme developers. While this enforces the developer’s stipulations for how the programme should be implemented, it means that the fruits of the “millions of taxpayer dollars ... used directly and indirectly in the discovery, development, and refinement of these approaches ... are not readily accessible to clinicians and educators in the public arena”.
There is, he observed, little evidence that these approaches work well outside the context of a tightly controlled trial usually overseen by the experts who developed the approach – a reference to the so-called ‘researcher allegiance’ effect of concern in several social research areas (1 2 3), where programme developers and other researchers with an interest in the programme’s success have been found to record more positive findings than fully independent researchers.
Why this happens is unclear. A possibility is the general finding that expectations (eg, of teachers of their pupils) affect performance via unintended influences on how research participants are treated; programme developers and other researchers affiliated to the intervention may transmit such expectations to participants via their influence on trainers and practitioners. Another is that investigators committed to the intervention are less rigorous in their testing and analysis methods and effectively look for ways to show it works – for example, by assessing many outcomes, gifting the programme they developed many chances to succeed, but not correspondingly raising the bar for what counts as a statistically significant success. It also seems likely that the involvement of the developer in a study will permit an implementation of the programme which adheres to its principles and methods more thoroughly than could be expected in routine practice.
Such issues came to a head recently in respect of Multisystemic Therapy, which dominates the research record. At the heart of the dispute between the programme’s developer and reviewers for the Cochrane Collaboration, known for the rigour of its analyses, were results from a Canadian trial which the reviewers saw as particularly important, because it was the only one of eight trials conducted fully independently of the developers of the approach, and for which results from all the randomised participants could be included over a defined follow-up period. It found no significant extra benefits from Multisystemic Therapy compared to usual juvenile justice services. Where similar outcome measures were available from this and from other trials, generally impacts in Canada were not just non-significant, but virtually zero, a contrast with trials in which the programme developers had been involved.
Including this study, Cochrane’s reviewers concluded that Multisystemic Therapy has not been shown to have clinically significant advantages over usual services or other interventions for youngsters with social, emotional or behavioural problems. Though tending to favour the approach, results from the eight trials did not aggregate to statistically significant advantages in terms of children being removed from home, crime, arrests or convictions, child psychiatric symptoms or family functioning. When all the sample was included in the analysis, no study found significant differences in substance use.
Based on this review, its lead author constructed a methodological critique of Multisystemic Therapy studies and previous more positive reviews. For scientific discourse, it elicited a vitriolic response from the approach’s developers and researchers, who took it as evidence of a “particularly insidious strategy ... to camouflage the commitment to the status quo with what appears to be a methodological and statistical critique of the forces challenging it”. The critic’s motivation was, they suspected, to sustain the “‘cottage industry’ of mental health treatment and services” which Multisystemic Therapy among others challenges with a view to improving the prospects of troubled youth. In turn a rejoinder from the reviewer contested the criticism of her work in detail and denied any such motivations, beginning with the implication that if anyone had a reason for bias, it was the programme’s developers, who “apart from their hard-won professional pride in their achievements ... have a financial interest” in the programme. She ended by pointedly querying whether the evidence for the programme is free of allegiance bias, is replicable in independent evaluations, or is “built on selective use of evidence by people who developed and profit from it ... has there been some mixture of science and sleight of hand in the making and successful marketing of [Multisystemic Therapy]?”
The featured review referred to this controversy, and made its own contribution by asserting that both in terms of its methodology, and how its results compared with other trials, there was no reason to single out the Canadian trial as a particularly poor or atypical test of the programme, and that even if implementation was poor compared to developer-led trials, this was to be expected in normal practice. Also relevant is that in Canada, how well the programme was implemented bore no relation to its results. The trialists there also pointed to the risk of bias in developer-led trials, asking, “When profit is at stake, is it safe to assume that research is value neutral? The potential for conflict of interest should be considered when selecting evaluators and when interpreting results”.
Thanks for their comments on this entry in draft to Michael L. Dennis of Chestnut Health Systems in the USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.
Last revised 29 May 2014. First uploaded 25 May 2014