Implementation frameworks suggest that modifications should be carefully considered and, if made in the absence of empirical support, that their impact should be measured [6, 12, 35, 36]. Evidence and practical implementation experience is mounting to suggest that clinicians modify EBPs in community mental health settings but that these modifications do not appear to be consistent with published definitions of adaptation, as they are not made in conjunction with a process of using research or practice-level data to determine their necessity or to examine their impact [9–11]. Importantly, there is a dearth of literature around identification of modifiable predictors of fidelity-inconsistent modifications, which have either been associated with poorer clinical outcomes in previous research or have not improved clinical outcomes [34, 35, 37–39]. This study contributes to the literature by examining this important question in a sample of clinicians who were trained to provide CBT to adults or children in an urban mental health system. We examined whether individual-level clinician factors, including training success and attitudes about EBPs, were related to subsequent modification of EBPs. Training success was not associated with subsequent fidelity-consistent modifications, but it was associated with fidelity-inconsistent modifications in an unexpected direction. The EBPAS appeal scale, which indicates the extent to which clinicians endorsed willingness to adopt EBPs if they found them to be intuitively appealing, believed they could be used correctly, or had colleagues who used them successfully, was uniquely associated with subsequent fidelity-inconsistent modifications, and openness to the use of EBPs was associated with fidelity-consistent modifications.
Being trained to criteria for competence in delivering CBT emerged as an important factor associated with clinician reports of fidelity-inconsistent modifications, contributing the greatest unique variance to the model we tested. Although our data do not allow us to make a definitive conclusion, a potential explanation is that clinicians who attained the competencies necessary to implement CBT were better able to recognize when they were departing from the protocol. It is important to note that in this sample, the reported number of modification types was relatively low. It is possible that clinicians underreport modifications (i.e., modify in more ways than they report doing so) and that clinicians who are trained to criterion have a more complete and comprehensive recognition of what is consistent with the protocol (cf., [22]). If this were the case, they may be more able to report accurately when they make modifications that are inconsistent with the protocol. Because clinicians were asked to describe the types of modifications and adaptations that they make in practice, rather than provide an accounting of how many modifications they make in a particular period of time, our findings suggest that clinicians may gravitate toward relatively few specific modifications rather than making a wider variety of modifications or that specific modifications are more salient and easier to recall or identify than others they made have made. Previous research has also indicated that therapists can provide accurate reports of therapeutic practices they deliver but that more training and support may be needed to improve their ability to report more subtle aspects of implementation [20]. In light of this previous research, our preliminary findings encourage further research to better understand the relationship between training and clinician report of modifications.
Some clinician attitudes that have been identified as individual-level predictors of training outcomes and implementation (e.g., [21, 22]) were also found to be associated with modifications to CBT in this study. Clinicians who were open to using evidence-based practices were more likely to make fidelity-consistent modifications, suggesting that clinicians who are open to EBPs may be more likely to try to find ways to make them work in routine care while delivering the intervention as closely to the protocol as possible. Baseline scores on the EBPAS appeal scale, which assesses whether willingness to adopt EBPs is contingent on finding an intervention appealing or having positive experiences with it, were associated with fidelity-inconsistent modifications at follow-up. This finding raises the possibility that clinicians who made more fidelity-inconsistent modifications may have had less positive experiences with CBT or found it less appealing when actually implementing it, and may therefore have seen less value in maintaining fidelity. While these findings are not entirely consistent with other studies that have identified either no association [40], or a positive relationship between clinician attitudes and training outcomes [22], differences in strategies for assessing use of and fidelity to EBPs may account for these discrepancies. Taken together, though, our results reinforce previous findings indicating that clinicians may need to have favorable attitudes toward EBPs to deliver them with fidelity [41]. Collectively, these studies indicate that training is a clear target for dissemination and implementation efforts to focus upon. In particular, ensuring that therapists are trained to deliver EBPs with fidelity and able to identify modifications that are consistent and inconsistent with fidelity while considering their impact on clinical outcomes may result in higher fidelity or more planful adaptation. It may also be important to provide specific guidance around adaptation and modifications as a component of training and consultation programs, and to identify ability to utilize different data sources to guide the process of adaptation as a competency goal for training. Furthermore, training programs may target clinician attitudes by presenting compelling empirical and experiential evidence of benefits to adoption and providing support around overcoming challenges to implementation at the individual level. Finally, training in and possibly requirement of ongoing fidelity monitoring and the use of outcome monitoring may improve fidelity or facilitate planned adaptation while reducing fidelity-inconsistent modifications.
Although this study has a number of strengths, namely being the first prospective study to focus on individual-level factors associated with subsequent modifications to EBPs, there are limitations. First, we relied on interviews to obtain information about the types of modifications that clinicians make in usual care, introducing the potential for recall bias or failure to recognize specific modifications in their own practice. It is likely that modification is better captured when observed in audiotaped sessions by expert raters and in conjunction with fidelity ratings [6]. It will be important in future research to objectively identify modifications, assess reasons for making specific changes, and to examine the impact of the modifications on clinical outcomes. Second, while organizational-level factors may also influence decisions to modify EBPs, our sample size precluded the introduction of additional variables into our model. Third, although roughly comparable, we combined two samples that participated in different training programs, which could have resulted in cohort effects. Furthermore, while the core interventions of the two CBT treatments are very similar, some of the differences in modifications found at the child vs. adult level may also have been accounted for by differences in the populations or aspects of the protocols. While we did control for some of these differences in our regression models, additional research to replicate these findings will be necessary, particularly with sample sizes that are adequate for an exploration of potential interactions between cohorts, attitudes, and modifications. Finally, the clinicians who participated in this study were only a subgroup of all clinicians who received training, although comparisons revealed very few differences between the full sample of trained clinicians and those who participated in the follow-up study.
There is little doubt that clinicians modify EBPs in routine care settings. However, there are potential implications of modifying EBPs, and these are the subject of considerable debate in the literature [6, 35]. Possibilities include improved clinical outcomes and sustainment of EBPs in routine care [35], but there is also the potential for decreased fidelity to essential treatment elements and poorer treatment outcomes [42]. Rather than approaching the modification of EBPs as uniformly positive or undesirable, it is important to empirically and conceptually differentiate between fidelity-consistent and fidelity-inconsistent modifications and planned adaptations. To fully understand their impact, improved measurement of different types of modifications (e.g., via observation) and monitoring of outcomes must occur. Future research is also necessary to elucidate whether specific modifications or planned adaptation result in a “voltage drop” [35] or in similar or even enhanced outcomes for clients. Studies are underway to shed light on this important question, and practice-based research can further this goal [35, 43].
Until more is known about the ways in which modifications impact clinical outcomes in different contexts, developers of complex psychosocial interventions, implementation facilitators, and training consultants can either allow modifications to occur naturally and hope for good outcomes or attempt to facilitate an effective process of adaptation. The development and testing of highly flexible protocols that include guidance on selection of appropriate modules or interventions is a promising direction toward this goal [44]. The results of this study also suggest that training programs should focus on training clinicians to competence, fostering positive attitudes and experiences with EBPs, and facilitating a thoughtful process of outcomes monitoring to guide decision-making and adaptation [35]. In-depth discussions about the relationship between fidelity and modifications, along with ongoing consultation to provide assistance around appropriate adaptation to core and peripheral elements can further enhance implementation efforts.
