Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and... Adm Policy Ment Health (2011) 38:65–76 DOI 10.1007/s10488-010-0319-7 OR IGINAL PAPER Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda • • • Enola Proctor Hiie Silmere Ramesh Raghavan • • • Peter Hovmand Greg Aarons Alicia Bunger Richard Griffey Melissa Hensley Published online: 19 October 2010 The Author(s) 2010. This article is published with open access at Springerlink.com Abstract An unresolved issue in the field of implemen- Keywords Implementation  Outcomes  Evaluation tation research is how to conceptualize and evaluate suc- Research methods cessful implementation. This paper advances the concept of ‘‘implementation outcomes’’ distinct from service system and clinical treatment outcomes. This paper proposes a Background heuristic, working ‘‘taxonomy’’ of eight conceptually dis- tinct implementation outcomes—acceptability, adoption, A critical yet unresolved issue in the field of implementa- appropriateness, feasibility, fidelity, implementation cost, tion science is how to conceptualize and evaluate success. penetration, and sustainability—along with their nominal Studies of implementation use widely varying approaches definitions. We propose a two-pronged agenda for research to measure how well a new mental health treatment, pro- on implementation outcomes. Conceptualizing and mea- gram, or service is implemented. Some infer implementa- suring implementation outcomes will advance under- tion success by measuring clinical outcomes at the client or standing of implementation processes, enhance efficiency patient level while other studies measure the actual targets in implementation research, and pave the way for studies of of the implementation, quantifying for example the desired the comparative effectiveness of implementation strategies. provider behaviors associated with delivering the newly implemented treatment. While some studies of implemen- tation strategies assess outcomes in terms of improvement in process of care, Grimshaw et al. (2006) report that meta- E. Proctor (&)  R. Raghavan  P. Hovmand  A. Bunger M. Hensley analyses of their effectiveness has been thwarted by lack of George Warren Brown School of Social Work, Washington detailed information about outcomes, use of widely varying University in St. Louis, Campus Box 1196, One Brookings constructs, reliance on dichotomous rather than continuous Drive, St. Louis, MO 63130, USA measures, and unit of analysis errors. e-mail: ekp@wustl.edu This paper advances the concept of ‘‘implementation H. Silmere outcomes’’ distinct from service system outcomes and Division of Social Work, Roberts Wesleyan College, clinical treatment outcomes (Proctor et al. 2009; Fixsen Rochester, USA et al. 2005; Glasgow 2007a). We define implementation R. Raghavan outcomes as the effects of deliberate and purposive actions Department of Psychiatry, Washington University School to implement new treatments, practices, and services. of Medicine, St. Louis, MO, USA Implementation outcomes have three important functions. First, they serve as indicators of the implementation suc- G. Aarons Department of Psychiatry, University of California, cess. Second, they are proximal indicators of implemen- San Diego, La Jolla, CA, USA tation processes. And third, they are key intermediate outcomes (Rosen and Proctor 1981) in relation to service R. Griffey system or clinical outcomes in treatment effectiveness Department of Emergency Medicine, Washington University and quality of care research. Because an intervention or School of Medicine, St. Louis, MO, USA 123 66 Adm Policy Ment Health (2011) 38:65–76 treatment will not be effective if it is not implemented well, Implementation Service Client implementation outcomes serve as necessary preconditions Outcomes Outcomes* Outcomes for attaining subsequent desired changes in clinical or Efficiency Satisfaction Acceptability service outcomes. Safety Function Adoption Symptomatology Effectiveness Distinguishing implementation effectiveness from treat- Appropriateness Equity Costs ment effectiveness is critical for transporting interventions Patient- Feasibility from laboratory settings to community health and mental centeredness Fidelity Timeliness Penetration health venues. When such efforts fail, as they often do, it is Sustainability important to know if the failure occurred because the intervention was ineffective in the new setting (interven- *IOM Standards of Care tion failure), or if a good intervention was deployed incorrectly (implementation failure). Our current knowl- Fig. 1 Types of outcomes in implementation research edge of implementation is thwarted by lack of theoretical understanding of the processes involved (Michie et al. system) are examined in implementation research. How- 2009). Conceptualizing and measuring implementation ever, as we argued above, implementation research requires outcomes that are conceptually and empirically distinct outcomes will advance understanding of implementation processes, enable studies of the comparative effectiveness from those of service and clinical effectiveness. For heuristic purposes, our model positions implemen- of implementation strategies, and enhance efficiency in implementation research. tation outcomes as preceding both service outcomes and This paper aims to advance the ‘‘vocabulary’’ of imple- client outcomes, with the latter sets of outcomes being mentation science around implementation outcomes through impacted by the implementation outcomes. As we discuss four specific objectives: (1) to advance conceptualization of later in this paper, interrelationships among these outcomes implementation outcomes by distinguishing implementation require conceptual mapping and empirical tests. For outcomes from service and clinical outcomes; (2) to advance example, one would expect to see a treatment’s strongest impact on client outcomes as an empirically supported clarity of terminology currently used in implementation science by nominating heuristic definitions of implementa- treatment’s (EST) penetration increases in a service set- ting—but this hypothesis requires testing. Our model tion outcomes, yielding a working ‘‘taxonomy’’ of imple- mentation outcomes; (3) to reflect the field’s current derives service outcomes from the six quality improvement aims set out in the reports on crossing the quality chasm: language, conceptual definitions, and approaches to opera- tionalizing implementation outcomes; and (4) to propose the extent to which services are safe, effective, patient- centered, timely, efficient, and equitable (Institute of directions for further research to advance knowledge on these key constructs and their interrelationships. Medicine Committee on Crossing the Quality Chasm 2006; Our objective of advancing a taxonomy of implemen- Institute of Medicine Committee on Quality of Health Care tation outcomes is comparable to the work of Michie et al. in America 2001). (2005, 2009), Grimshaw et al. (2006), the Cochrane group, and others who are working to develop taxonomies and common nomenclature for implementation strategies. Our Methods work is complementary to these efforts because imple- mentation outcomes will provide researchers with a The paper’s methods were shaped around its overall aim: to advance clarity in the language used to describe outcomes framework for evaluating implementation strategies. of implementation. We convened a working group of Conceptual Framework for Implementation Outcomes implementation researchers to identify concepts for label- ing and assessing outcomes of implementation processes. Our understanding of implementation outcomes is lodged One member of the group was a doctoral student RA who within a previously published conceptual framework coordinated, conducted, and reported on the literature (Proctor et al. 2009) as shown in Fig. 1. The framework search and constructed tables reflecting various iterations of the heuristic taxonomy. The RA conducted literature distinguishes between three distinct but interrelated types of outcomes—implementation, service, and client outcomes. searches using key words and search programs to identify literature on the current state of conceptualization and Improvements in consumer well-being provide the most important criteria for evaluating both treatment and imple- measurement of these outcomes, primarily in the health and behavioral sciences. We searched in a number of mentation strategies—for treatment research, improvements are examined at the individual client level whereas databases with a particular focus on MEDLINE, CINAHL improvements at the population-level (within the providing Plus, and PsycINFO. Key search terms included the name 123 Adm Policy Ment Health (2011) 38:65–76 67 of the implementation outcome (e.g., ‘‘acceptability,’’ conceptualization of terms used to assess implementation ‘‘sustainability,’’ etc.) along with relevant synonyms com- outcomes. bined with any of the following: innovation, EBP, evidence based practice, and EST. We scanned the titles and Taxonomy of Implementation Outcomes abstracts of the identified sources and read the methods and background sections of the studies that measured or Through our process of iterative reading and discussion of attempted to measure implementation outcomes. We also the literature, we worked to nominate definitions that (1) included information from relevant conceptual articles in achieve as much consistency as possible with any existing the development of nominal definitions. Whereas our pri- definitions (including multiple definitions we found for a mary focus was on the implementation of evidence based single construct), yet (2) serve to sharpen distinctions practices in the health and behavioral sciences, the key- between constructs that might be similar. For several of the word ‘‘innovation’’ broadened this scope by also identify- outcomes, the literature did not offer one clear nominal ing studies that focused on other areas such as physical definition. health that may inform implementation of mental health Table 1 depicts the resultant working taxonomy of treatments. Because terminology in this field currently implementation outcomes. For each implementation out- reflects widespread inconsistency, we followed leads come, the table nominates a level of analysis, identifies the beyond what our keyword searches ‘‘hit’’ upon. Thus we theoretical basis to the construct from implementation lit- read additional articles that we found cited by authors erature, shows different terms that are used for the con- whose work we found through our electronic searches. We struct in the literature, suggests the point or stage within also conducted searches of CRISP, TAGG, and NIH implementation processes at which the outcome may be reporter and studies to identify funded mental health most salient, and lists the types of existing measures for the research studies with ‘‘implementation’’ in their titles or construct that our search identified. The implementation abstracts, to identify examples of outcomes pursued in outcomes listed in Table 1 are probably only the ‘‘more current research. obvious,’’ and we expect that other concepts may emerge We used a narrative review approach (Educational from further analysis of the literature and from the kind of Research Review), which is appropriate for summarizing empirical work we call for in our discussion below. Many different primary studies and drawing conclusions and of the implementation outcomes can be inferred or mea- interpretation about ‘‘what we know,’’ informed by sured in terms of expressed attitudes and opinions, inten- reviewers’ experiences and existing theories (McPheeters tions, or reported or observed behaviors. We now list and et al. 2006; Kirkevoid 1997). Narrative reviews yield discuss our nominated conceptual definitions for each implementation outcome in our proposed taxonomy. We qualitative results, with strengths in capturing diversities and pluralities of understanding (Jones 1997). According to reference similar definitions from the literature, and also McPheeters et al. (2006), narrative reviews are best con- comment on marked differences between our definitions ducted by a team. Members of the working group read and and others proposed for the term. reviewed conceptual and theoretical pieces as well as Acceptability is the perception among implementation published reports of implementation research. As a team, stakeholders that a given treatment, service, practice, or we convened recurring meetings to discuss the similarities innovation is agreeable, palatable, or satisfactory. Lack of and dissimilarities. We audio-taped and transcribed meet- acceptability has long been noted as a challenge in ing discussions, and a designated individual took thorough implementation (Davis 1993). The referent of the imple- notes. Transcriptions and notes were posted on a shared mentation outcome ‘‘acceptability’’ (or the ‘‘what’’ is computer file for member review, revision, and correction. acceptable) may be a specific intervention, practice, tech- Group processes included iterative discussion, checking nology, or service within a particular setting of care. additional literature for clarification, and subsequent dis- Acceptability should be assessed based on the stake- cussion. The aim was to collect and portray, from extant holder’s knowledge of or direct experience with various literature, the similarities and differences across investi- dimensions of the treatment to be implemented, such as its gators’ use of various implementation outcomes and defi- content, complexity, or comfort. Acceptability is different nitions for those outcomes. Discussions often led us to from the larger construct of service satisfaction, as typi- preserve distinctions between terms by maintaining in our cally measured through consumer surveys. Acceptability is ‘‘nominated’’ taxonomy two different implementation more specific, referencing a particular treatment or set of outcomes because the literature or our own research treatments, while satisfaction typically references the revealed possible conceptual distinctions. We assembled general service experience, including such features as the identified constructs in the proposed heuristic taxon- waiting times, scheduling, and office environment. omy to portray the current state of vocabulary and Acceptability may be measured from the perspective of 123 68 Adm Policy Ment Health (2011) 38:65–76 Table 1 Taxonomy of implementation outcomes Implementation Level of analysis Theoretical basis Other terms in literature Salience by implementation Available measurement outcome stage Acceptability Individual provider Rogers: ‘‘complexity’’ and to a Satisfaction with various aspects Early for adoption Survey certain extent ‘‘relative of the innovation (e.g. content, Individual consumer Ongoing for penetration Qualitative or semi-structured advantage’’ complexity, comfort, delivery, interviews Late for sustainability and credibility) Administrative data Refused/blank Adoption Individual provider RE-AIM: ‘‘adoption’’ Rogers: Uptake; utilization; initial Early to mid Administrative data ‘‘trialability’’ (particularly for implementation; intention to try Organization or setting Observation early adopters) Qualitative or semi-structured interviews Survey Appropriateness Individual provider Rogers: ‘‘compatibility’’ Perceived fit; relevance; Early (prior to adoption) Survey compatibility; suitability; Individual consumer Qualitative or semi-structured usefulness; practicability interviews Organization or setting Focus groups Feasibility Individual providers Rogers: ‘‘compatibility’’ and Actual fit or utility; suitability for Early (during adoption) Survey ‘‘trialability’’ everyday use; practicability Organization or setting Administrative data Fidelity Individual provider RE-AIM: part of Delivered as intended; adherence; Early to mid Observation ‘‘implementation’’ integrity; quality of program Checklists delivery Self-report Implementation Cost Provider or providing TCU Program Change Model: Marginal cost; cost-effectiveness; Early for adoption and Administrative data institution ‘‘costs’’ and ‘‘resources’’ cost-benefit feasibility Mid for penetration Late for sustainability Penetration Organization or setting RE-AIM: necessary for ‘‘reach’’ Level of institutionalization? Mid to late Case audit Spread? Service access? Checklists Sustainability Administrators RE-AIM: ‘‘maintenance’’ Maintenance; continuation; Late Case audit Rogers: ‘‘confirmation’’ durability; incorporation; Organization or setting Semi-structured interviews integration; institutionalization; Questionnaires sustained use; routinization; Checklists Adm Policy Ment Health (2011) 38:65–76 69 various stakeholders, such as administrators, payers, pro- Bartholomew et al. (2007) describe a rating scale for viders, and consumers. We presume rated acceptability to capturing appropriateness of training among substance be dynamic, changing with experience. Thus ratings of abuse counselors who attended training in dual diagnosis acceptability may be different when taken, for example, and therapeutic alliance. pre-implementation and later throughout various stages of Cost (incremental or implementation cost) is defined as implementation. The literature reflects several examples of the cost impact of an implementation effort. Implementa- measuring provider and patient acceptability. Aarons’ tion costs vary according to three components. First, Evidence-Based Practice Attitude Scale (EBPAS) captures because treatments vary widely in their complexity, the the acceptability of evidence-based mental health treat- costs of delivering them will also vary. Second, the costs of ments among mental health providers (Aarons 2004). implementation will vary depending upon the complexity Aarons and Palinkas (2007) used semi-structured inter- of the particular implementation strategy used. Finally, views to assess case managers’ acceptance of evidence- because treatments are delivered in settings of varying based practices in a child welfare setting. Karlsson and complexity and overheads (ranging from a solo practi- Bendtsen (2005) measured patients’ acceptance of alcohol tioner’s office to a tertiary care facility), the overall costs of screening in an emergency department setting using a delivery will vary by the setting. The true cost of imple- 12-item questionnaire. menting a treatment, therefore, depends upon the costs of Adoption is defined as the intention, initial decision, or the particular intervention, the implementation strategy action to try or employ an innovation or evidence-based used, and the location of service delivery. practice. Adoption also may be referred to as ‘‘uptake.’’ Much of the work to date has focused on quantifying Our definition is consistent with those proposed by Rabin intervention costs, e.g., identifying the components of a et al. (2008) and Rye and Kimberly (2007). Adoption could community-based heart health program and attaching costs be measured from the perspective of provider or organi- to these components (Ronckers et al. 2006). These cost zation. Haug et al. (2008) used pre-post items to capture estimations are combined with patient outcomes and used substance abuse providers’ adoption of evidence-based in cost-effectiveness studies (McHugh et al. 2007). A practices, while Henggeler et al. (2008) report interview review of literature on guideline implementation in pro- techniques to measure therapists’ adoption of contingency fessions allied to medicine notes that few studies report management. anything about the costs of guideline implementation Appropriateness is the perceived fit, relevance, or (Callum et al. 2010). Implementing processes that do not compatibility of the innovation or evidence based practice require ongoing supervision or consultation, such as com- for a given practice setting, provider, or consumer; and/or puterized medical record systems, may carry lower costs than implementing new psychosocial treatments. Direct perceived fit of the innovation to address a particular issue or problem. ‘‘Appropriateness’’ is conceptually similar to measures of implementation cost are essential for studies ‘‘acceptability,’’ and the literature reflects overlapping and comparing the costs of implementing alternative treatments sometimes inconsistent terms when discussing these con- and of various implementation strategies. structs. We preserve a distinction because a given treat- Feasibility is defined as the extent to which a new ment may be perceived as appropriate but not acceptable, treatment, or an innovation, can be successfully used or and vice versa. For example, a treatment might be con- carried out within a given agency or setting (Karsh 2004). sidered a good fit for treating a given condition but its Typically, the concept of feasibility is invoked retrospec- features (for example, rigid protocol) may render it unac- tively as a potential explanation of an initiative’s success or ceptable to the provider. The construct ‘‘appropriateness’’ failure, as reflected in poor recruitment, retention, or par- is deemed important for its potential to capture some ticipation rates. While feasibility is related to appropriate- ‘‘pushback’’ to implementation efforts, as is seen when ness, the two constructs are conceptually distinct. For providers feel a new program is a ‘‘stretch’’ from the example, a program may be appropriate for a service set- mission of the health care setting, or is not consistent with ting—in that it is compatible with the setting’s mission or providers’ skill set, role, or job expectations. For example, service mandate, but may not be feasible due to resource or providers may vary in their perceptions of the appropri- training requirements. Hides et al. (2007) tapped aspects of ateness of programs that co-locate mental health services feasibility of using a screening tool for co-occurring mental within primary medical, social service, or school settings. health and substance use disorders. Again, a variety of stakeholders will likely have percep- Fidelity is defined as the degree to which an intervention tions about a new treatment’s or program’s appropriateness was implemented as it was prescribed in the original pro- to a particular service setting, mission, providers, and cli- tocol or as it was intended by the program developers entele. These perceptions may be function of the organi- (Dusenbury et al. 2003; Rabin et al. 2008). Fidelity has zation’s culture or climate (Klein and Sorra 1996). been measured more often than the other implementation 123 70 Adm Policy Ment Health (2011) 38:65–76 outcomes, typically by comparing the original evidence- construct with terms such a given treatment’s level of based intervention and the disseminated/implemented institutionalization. intervention in terms of (1) adherence to the program pro- Sustainability is defined as the extent to which a newly tocol, (2) dose or amount of program delivered, and (3) implemented treatment is maintained or institutionalized quality of program delivery. Fidelity has been the over- within a service setting’s ongoing, stable operations. The riding concern of treatment researchers who strive to move literature reflects quite varied uses of the term ‘‘sustain- their treatments from the clinical lab (efficacy studies) to ability,’’ but our proposed definition incorporates aspects of real-world delivery systems. The literature identifies five those offered by Johnson et al. (2004), Turner and Sanders implementation fidelity dimensions including adherence, (2006), Glasgow et al. (1999), Goodman et al. (1993), and quality of delivery, program component differentiation, Rabin et al. (2008). Rabin et al. (2008) emphasizes the exposure to the intervention, and participant responsive- integration of a given program within an organization’s ness or involvement (Mihalic 2004; Dane and Schneider culture through policies and practices, and distinguishes 1998). Adherence, or the extent to which the therapy three stages that determine institutionalization: (1) passage occurred as intended, is frequently examined in psycho- (a single event such as transition from temporary to per- therapy process and outcomes research and is distinguished manent funding), (2) cycle or routine (i.e., repetitive from other potentially pertinent implementation factors reinforcement of the importance of the evidence-based such as provider skill or competence (Hogue et al. 1996). intervention through including it into organizational or Fidelity is measured through self-report, ratings, and direct community procedures and behaviors, such as the annual observation and coding of audio- and videotapes of actual budget and evaluation criteria), and (3) niche saturation encounters, or provider-client/patient interaction. Achiev- (the extent to which an evidence-based intervention is ing and measuring fidelity in usual care is beset by a integrated into all subsystems of an organization). Thus the number of challenges (Proctor et al. 2009; Mihalic 2004; outcomes of ‘‘penetration’’ and ‘‘sustainability’’ may be Schoenwald et al. 2005). The foremost challenge may be related conceptually and empirically, in that higher pene- measuring implementation fidelity quickly and efficiently tration may contribute to long-term sustainability. Such (Hayes 1998). relationships require empirical test, as we elaborate below. Schoenwald and colleagues (2005) have developed three Indeed Steckler et al. (1992) emphasize sustainability in 26–45-item measures of adherence at the therapist, super- terms of attaining long-term viability, as the final stage of visor and consultant level of implementation (available the diffusion process during which innovations settle into from the MST Institute www.mstinstitute.org). Ratings are organizations. To date, the term sustainability appears obtained at regular intervals, enabling examination of the more frequently in conceptual papers than actual empirical provider, clinical supervisor, and consultant. Other exam- articles measuring sustainability of innovations. As we ples from the mental health literature include Bond et al. discuss below, the literature often uses the same term (2008) 15-item Supported Employment Fidelity Scale (SE (niche saturation, for example) to reference multiple Fidelity Scale) and Hogue et al. (2008) Therapist Behavior implementation outcomes, underscoring the need for con- Rating Scale-Competence (TBRS-C), an observational ceptual clarity as we seek to advance in this paper. measure of fidelity in evidence based practices for ado- lescent substance abuse treatment. Penetration is defined as the integration of a practice Research Agenda to Advance Implementation within a service setting and its subsystems. This definition Outcomes is similar to (Stiles et al. 2002) notion of service penetra- tion and to Rabin et al.s’ (2008) notion of niche saturation. Advancing the conceptualization, measurement, and empir– Studying services for persons with severe mental illness, ical understanding of implementation outcomes requires Stiles et al. (2002) apply the concept of service penetration research on several critical issues. We propose two major to service recipients (the number of eligible persons who themes for this research—(1) conceptualization and mea- use a service, divided by the total number of persons eli- surement, and (2) theory building—and identify important gible for the service). Penetration also can be calculated in issues within each of these themes. terms of the number of providers who deliver a given service or treatment, divided by the total number of pro- Research on Conceptualization and Measurement viders trained in or expected to deliver the service. From a of Implementation Outcomes service system perspective, the construct is also similar to ‘‘reach’’ in the RE-AIM framework (Glasgow 2007b). We Research on several fronts is required to advance the found infrequent use of the term penetration in the imple- conceptual and measurement properties of implementation mentation literature; though studies seemed to tap into this outcomes, five of which we identify and discuss. 123 Adm Policy Ment Health (2011) 38:65–76 71 Consistency of Terminology setting; or (3) a broad effort to implement several new treatments at once. A lingering issue for the field is whether For each outcome listed in Table 1, we found literature implementation processes should be tackled and studied using different and sometimes inconsistent terminology. specifically (one new treatment) or in a more generalized Sometimes studies used different labels for what appear to way (the extent to which a system’s care is evidence-based be the same construct. In other cases, studies used one term or guideline congruent). Understanding the optimal speci- for a label or nominal definition but a different term for ficity of the referent for a given implementation outcome is operationalizing or measuring the same construct. This critical for measurement. As a beginning step, researchers problem was pronounced for three implementation out- should report the referent for all implementation outcomes comes—acceptability, appropriateness, and feasibility. measured. These constructs were frequently used interchangeably or measured under the common generic label as client or Level of Analysis for Outcomes provider perceptions, reactions, and attitudes toward, or satisfaction with various aspects of the innovation, EST, Implementation of new treatments is an inherently multi- or clinical practice guidelines. For example, Graham et al. level enterprise, involving provider behavior, care organi- (2007) assessed doctors’ attitudes and perceptions toward zation, and policy (Proctor et al. 2009; Raghavan et al. clinical practice guidelines with a survey that tapped all 2008). Implementation outcomes are important at each three of these outcomes, although none of them were level of change, but the research has yet to determine explicitly labeled as such: acceptability (e.g. perceived which level or unit of analysis is most appropriate for quality of and confidence in guidelines), appropriateness particular implementation outcomes. Certain outcomes, (e.g. perceived usefulness of guidelines), and feasibility such as acceptability, may be most appropriate for indi- (e.g. these guidelines provide recommendations that are vidual level analysis (for example, providers, consumers), implementable). Other studies interchanged the terms for while others, such as penetration may be more appropriate acceptability and feasibility within the same article. For for aggregate analysis, at the level of the health care example, Wilkie et al. (2003) begin by describing the organization. Currently, very few studies reporting imple- measurement of ‘‘usability’’ (of a computerized innova- mentation outcomes specify the level of measurement, nor tion), including its ‘‘acceptability’’ to clients but later use do they address issues of aggregation within or across the findings to conclude that the innovation was feasible. levels. While language inconsistency is typical in most still- Construct validity. The constructs reflected in Table 1 developing fields, implementation research may be partic- and the terms employed in our taxonomy of implementation ularly susceptible to this problem. No one discipline is outcomes derive largely from the research literature. Yet it is ‘‘home’’ to implementation research. Studies are conducted important to also understand outcome perceptions and across a broad range of disciplines, published in a scattered preferences through the voice of those who design and set of journals, and consequently are rarely cross refer- deliver health care. Qualitative data, reflecting language enced. Beyond mental health, we found articles referencing used by various stakeholders as they think and talk these implementation outcomes in physical health, smok- about implementation processes, is important for validat- ing cessation, cancer, and substance abuse literatures, ing implementation outcome constructs. Through in-depth addressing a wide variety of topics. interviews, stakeholders’ cognitive representations and Clearly, the field of implementation science now has mental models of outcomes can be analyzed through such only the beginnings of a common language to characterize methods as cultural domain analysis (CDA). A ‘‘cultural implementation outcomes, a situation that thwarts the domain’’ refers to a set of words, phrases, and/or concepts conceptual and empirical advancement of the field but that link together to form a single conceptual subject (Luke could be overcome by use of a common lexicon. Just as 2004; Bates and Sarkar 2007), and methods for CDA, such Michie et al. (2009) state the ‘‘imperative that there be a as free-listing and pile-sorting, have been used since the consensual, common language’’ (p. 4) to describe behavior 1970s (Bates and Sarkar 2007). While primarily used in change techniques, so is common language needed for anthropology, CDA is aptly suited for health services implementation outcomes. research that endeavors to understand how stakeholders conceptualize implementation outcomes, informing the Referent for Rating the Outcome generation of definitions of implementation outcomes. The actual words used by stakeholders may or may not reflect the Several of the proposed implementation outcomes could be terms used in academic literature and reflected in our pro- used to rate (1) a specific treatment; (2) the implementation posed taxonomy (acceptability, appropriateness, feasibility, strategy used to introduce that treatment into the care adoption, fidelity, penetration, sustainability and costs). But 123 72 Adm Policy Ment Health (2011) 38:65–76 such research can identify the terms and distinctions that are Salience of Implementation Outcomes by Point meaningful to implementation stakeholders. in the Implementation Process Measurement Properties of Implementation Outcomes The implementation of any new treatment or service is widely recognized as a process, involving a sequence of The literature reflects a wide array of approaches for activities, beginning with initial considerations of what and how to change current care. Chamberlain has identified ten measuring implementation outcomes, ranging from quali- tative, quantitative survey, and record archival. Michie steps for the implementation of an evidence-based treat- ment, Multidimensional Treatment Foster Care (MTFC), et al. (2007) studied perceived difficulties implementing a mental health guideline, coding respondent descriptions of beginning with consideration of adopting MTFC and con- implementation difficulties as 0, 0.5, or 1. Much mea- cluding when a service site meets certification criteria for surement has been ‘‘home-grown,’’ with virtually no work delivering the treatment (Chamberlain et al. 2008). As we on the psychometric properties or measurement rigor. suggest in Table 1, certain implementation outcomes may Measurement development is needed to enhance the por- be more important at some phases of implementation tability and usefulness of implementation outcomes in real process than at other phases. For example, feasibility may be most important once organizations and providers try world settings of care. Measures used in efficacy research will likely prove too cumbersome for real-world studies new treatments. Later, it may be a ‘‘moot point,’’ once the treatment—initially considered novel or unknown—has of implementation. For example, detailed assessment of fidelity through coding of encounter videotapes would be become part of normal routine. too time-intensive for a multi-agency study assessing The literature suggests that studies usually capture fidelity of treatment implementation. fidelity during initial implementation, while adoption is often assessed at 6 (Waldorff et al. 2008), 12 (Adily et al. Theory-Building Research 2004; Fischer et al. 2008), or 18 months (Cooke et al. 2001) after initial implementation. But most studies fail to specify a timeframe or are inconsistent in choice of a Research is also needed to advance our theoretical under- standing of the implementation process. Empirical studies of time point in the implementation process for measuring outcomes. Research is needed to explore these issues, the five issues we list here will inform theory, illuminate the ‘‘black box’’ of implementation processes, and help particularly longitudinal studies that measure multiple implementation outcomes before, during, and after imple- shape models for developing and testing implementation strategies. mentation of a new treatment. Such research may reveal ‘‘leading’’ and ‘‘lagging’’ indicators of implementation Salience of Implementation Outcomes to Stakeholders success. For example, if acceptability increases for several months, following which penetration increases, then we Any effort to implement change in care involves a range of may view acceptability as a leading indicator of penetra- stakeholders, including the treatment developers who design tion. Leading indicators can be useful for managing the and test the effectiveness of ESTs, policy makers who design implementation process as they signal future trends. and pay for service, administrators who shape program Where leading indicators may identify future trends, lagging indicators reflect delays between when changes direction, providers and supervisors, patients/clients/con- sumers and their family members, and interested community happen and when they can be observed. For example, sustainability may be observed only well into, or even after members and advocates. The success of efforts to implement evidence-based treatment may rest on their congruence with the implementation process. Being aware of lagging indi- the preferences and priorities of those who shape, deliver, cators of implementation success may help managers avoid and participate in care. Implementation outcomes may be over-reacting to slow change and wait for evidence of what differentially salient to various stakeholders, just as the may soon prove to be successful implementation. salience of clinical outcomes varies across stakeholders (Shumway et al. 2003). For example, implementation cost Modeling Interrelationships Among Implementation Outcomes may be most important to policy makers and program directors, feasibility may be most important to direct service Our team’s observations of implementation suggest that providers, and fidelity may be most important to treatment developers. To ensure applicability of implementation out- implementation outcomes are themselves interrelated in dynamic and complex ways (Woolf 2008; Repenning 2002; comes across a range of settings and to maximize their external validity, all stakeholder groups and priorities should Hovmand and Gillespie 2010; Klein and Knight 2005) and be represented in this research. are likely to change throughout an agency’s process to 123 Adm Policy Ment Health (2011) 38:65–76 73 adopt and implement ESTs. For example, the perceived implementation, as well as the effects of contextual factors appropriateness, feasibility, and implementation cost asso- that must be addressed and that are captured in imple- ciated with an intervention will likely bear on ratings of mentation outcomes. the intervention’s acceptability. Acceptability, in turn, Established evidence for a ‘‘proven’’ treatment does not will likely affect adoption, penetration, and sustainability. ensure successful implementation. Implementation also Similarly, consistent with Rogers’ theory of the diffusion requires addressing a number of important contextual fac- of innovation, the ability to adopt or adapt an innovation tors, such as provider attitudes, professional behavior, and for local use may increase its acceptability (Rogers 1995). the service system. Constructs in the proposed taxonomy of This suggests that when providers believe they do not have implementation outcomes have potential to capture those to implement a treatment ‘‘by the book’’ (or with precise provider attitudes (acceptability) and behaviors (adoption, fidelity), they may rate the treatment as more acceptable. uptake) as well as contextual factors (system penetration, Modeling the interrelationships between implementation appropriateness, implementation cost). outcomes will also inform their definitional boundaries and For purposes of stimulating debate and future research, thus shape the taxonomy. For example, if two outcomes we suggest that successful implementation be considered in which we now define as distinct concepts are shown light of a ‘‘portfolio’’ of factors, including the effectiveness through research to always occur together, the empirical of the treatment to be implemented and implementation evidence would suggest that the concepts are really the outcomes such as included in our taxonomy. For example, same thing and should be combined. Similarly, if two of implementation success (I, in the equation below) could be the outcomes are shown to have different empirical pat- modeled to reflect (1) the effectiveness (E) of the treatment terns, evidence would confirm their conceptual distinction. being implemented, plus (2) implementation factors (IO’s), which heretofore have been insufficiently conceptualized, Modeling Attainment of Implementation Outcomes distinguished, and measured and rarely used to guide implementation decisions. Once researchers have advanced consistent, valid, and I ¼ fE þ IO’s efficient measures for implementation outcomes, the field will be equipped to conduct important research treating For example, in situation ‘‘A’’, an evidence-based treatment may be highly effective but given its high cost, these constructs as dependent variables, in order to identify correlates or predictors of their attainment. Their only mildly acceptable to key stakeholders and low in sustainability. The overall potential success of implemen- measurement will enable research to determine which features of a treatment itself or which implementation tation in this case might be modeled as follows: strategies help make new treatments acceptable, feasible Implementation success = f of effectivenessðÞ = high to implement, or sustainable over time. The diffusion of + acceptabilityðÞ = moderate innovation literature posits that the implementation out- + sustainabilityðÞ low : come, adoption of an EST, is a function of such factors as perceived need to do things differently (Rogers 1995) In situation ‘‘B’’, a given treatment might be only perception of the new treatment’s comparative advantage moderately effective but highly acceptable to stake- (Frambach and Schillewaert 2002; Henggeler et al. 2002) holders because current care is poor, the treatment is and as easy to understand (Berwick 2003). Such suppo- inexpensive, and current training protocols ensure high sitions require empirical test using measures of imple- penetration through providers. This treatment’s potential mentation outcomes. might be modeled in the following equation: Using Implementation Outcomes to Model Implementation success = Implementation Success f of treatment effectivenessðÞ moderate + acceptabilityðÞ high + potential to improve careðÞ high Reliable, valid measures of implementation outcomes will enable empirical testing of the success of efforts to + penetrationðÞ high : implement new treatments, and pave the way for compar- ative effectiveness research on implementation strategies. Thus using implementation outcomes, the success of In most current initiatives to move evidence-based treat- implementation may be modeled and tested, thereby ments into community care settings, the success of the making decisions about what to implement more explicit implementation is assumed and evaluated from data on and transparent. clinical outcomes. We believe that an exclusive focus on To increase the success of implementation, implemen- clinical outcomes thwarts understanding the process of tation strategies need to be employed strategically. For 123 74 Adm Policy Ment Health (2011) 38:65–76 Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, example, implementation strategies could be employed to R. (2008). Fidelity of supported employment: Lessons learned increase provider acceptance, improve penetration, reduce from the national evidence-based practice project. Psychiatric implementation costs, and achieve sustainability of the Rehabilitation Journal, 31(4), 300–305. treatment being implemented. Understanding how to Callum, P., Iger, J., Ray, M., Sims, C. A., & Falk, R. E. (2010). Outcome and experience of implementing spinal muscular achieve implementation outcomes requires the kind of work atrophy carrier screening on sperm donors. Fertility and Sterility, now underway by Michie et al. (2009) advance a taxonomy 94(5), 1912–1914. of implementation strategies and reflect their demonstrated Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., effects. Marsenich, L., et al. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy In Mental Health, 35, 250–260. Summary and Implications Cooke, M., Mattick, R. P., & Walsh, R. A. (2001). Implementation of the ‘fresh start’ smoking cessation programme to 23 antenatal clinics: A randomized controlled trial investigating two methods The science of implementation cannot be advanced with- of dissemination. Drug and Alcohol Review, 20, 19–28. out attention to implementation outcomes. All studies of Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary implementation should explicate and measure implemen- and early secondary prevention: Are implementation effects out tation outcomes. Given the rudimentary state of the field, of control? Clinical Psychology Review, 18, 23–45. Davis, F. (1993). User acceptance of information technology: System we chose a narrative approach to reviewing the literature characteristics, user perceptions and behavioral impacts. Inter- and constructing a taxonomy. Our purpose is to advance national Journal of Man-Machine Studies, 38, 475–487. the clarity of language, provoke debate, and stimulate more Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A systematic work toward the aims of advancing the con- review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education ceptual, linguistic, and methodological clarity in the field. Research, 18(2), 237–256. A taxonomy of implementation outcomes can help orga- Fischer, M. A., Vogeli, C., Stedman, M. R., Ferris, T. G., & nize the key variables and frame research questions Weissman, J. S. (2008). Uptake of electronic prescribing in required to advance implementation science. Their mea- community-based practices. Journal of General Internal Med- icine, 23(4), 358–363. surement and empirical test can help specify the mecha- Fixsen, D. L., Naoom, S. F., Blase ´, K. A., Friedman, R. M., & nisms and causal relationships within implementation Wallace, F. (2005). Implementation research: A synthesis of the processes and advance an evidence base around successful literature (No. FMHI Publication #231). Tampa, FL: University implementation. of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network. Frambach, R. T., & Schillewaert, N. (2002). Organizational innova- Acknowledgments Preparation of this manuscript was supported by tion adoption: A multi-level framework of determinants and grants P30 MH068579 and UL1RR024992. opportunities for future research. Journal of Business Research Special Issue: Marketing Theory in the Next Millennium, 55, Open Access This article is distributed under the terms of the 163–176. Creative Commons Attribution Noncommercial License which per- Glasgow, R. E. (2007a). eHealth evaluation and dissemination mits any noncommercial use, distribution, and reproduction in any research. American Journal of Preventive Medicine, 32(5S), medium, provided the original author(s) and source are credited. S119–S126. Glasgow, R. E. (2007b). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science References of Dissemination and Implementation in the Service of Public Health: Bethesda, MD. Aarons, G. A. (2004). Mental health provider attitudes toward adoption Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating of evidence-based practice: The evidence-based practice attitude the public health impact of health promotion interventions: The scale (EBPAS). Mental Health Service Research, 6(2), 61–74. RE-AIM framework. American Journal of Public Health, 89, Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence- 1322–1327. based practice in child welfare: Service provider perspectives. Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. Mental Health Services Research, 34, 411–419. (1993). Development of level of institutionalization scales for Adily, A., Westbrook, J., Coiera, E., & Ward, J. (2004). Use of on- health promotion programs. Health Education Quarterly, 20(2), line evidence databases by Australian public health practitioners. 161–179. Medical Informatics & the Internet in Medicine, 29(2), 127–136. Graham, I. D., Brouwers, M., Davies, C., & Tetroe, J. (2007). Ontario Bartholomew, N. G., Joe, G. W., Rowan-Szal, G. A., & Simpson, D. doctors’ attitudes toward and use of clinical practice guidelines D. (2007). Counselor assessments of training and adoption in oncology. Journal of Evaluation in Clinical Practice, 13(4), barriers. Journal of Substance Abuse Treatment, 33, 193–199. 607–615. Bates, D. M., & Sarkar, D. (2007). lme4: Linear mixed-effects models Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., using S4 classes. R Package Version, 0, 99875–99876. Fraser, C., et al. (2006). Toward evidence-based quality Berwick, D. M. (2003). Disseminating innovations in health care. improvement: Evidence (and its limitations) of the effectiveness Journal of the American Medical Association, 289(15), 1969– of guideline dissemination and implementation strategies 1975. 1966–1998. Journal of General Internal Medicine, 21, S14–S20. 123 Adm Policy Ment Health (2011) 38:65–76 75 Haug, N. A., Shopshire, M., Tajima, B., Gruber, V., & Guydish, J. McPheeters, M. L., Briss, P., Teutsch, S. J., & Truman, B. (2006). (2008). Adoption of evidence-based practices among substance Systematic reviews in public health. In R. C. Brownson & D. abuse treatment providers. Journal of Drug Education, 38(2), B. Petitti (Eds.), Applied epidemiology: Theory to practice (2nd 181–192. ed., pp. 99–124). New York: Oxford University Press. Hayes, S. C. (1998). President’s message: Dissemination research Michie, S., Johnson, M., Abraham, C., Lawton, R., Parker, D., & now. The Behavior Therapist, 21(9), 166–169. Walker, A. (2005). Making psychological theory useful for Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after implementing evidence based practice: A consensus approach. the innovation is identified? Clinical Psychology: Science & Quality and Safety in Health Care, 14(1), 26–33. Practice, 9(2), 191–194. Michie, S., Pilling, S., Garety, P., Whitty, P., Eccles, M. P., Johnston, Henggeler, S. W., Chapman, J. E., Rowland, M. D., Halliday- M., et al. (2007). Difficulties implementing a mental health Boykinds, C. A., Randall, J., Shackelford, J., et al. (2008). guideline: An exploratory investigation using psychosocial Statewide adoption and initial implementation of contingency theory. Implementation Science, 2(8), 1–8. management for substance-abusing adolescents. Journal of Michie, S., Fixsen, D., Grimshaw, J., & Eccles, M. (2009). Specifying Consulting and Clinical Psychology, 76(4), 556–567. and reporting complex behavior change interventions: The need Hides, L., Lubman, D. I., Elkins, K., Catania, L. S., & Rogers, N. for a scientific method. Implementation Science, 4(1), 40. (2007). Feasibility and acceptability of a mental health screening Mihalic, S. (2004). The importance of implementation fidelity. tool and training programme in the youth alcohol and other drug Emotional and Behavioral Disorders in Youth, 4, 83–86. (AOD) sector. Drug and Alcohol Review, 26, 509–515. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence & Mittman, B. (2009). Implementation research in mental health process research in family therapy: A rationale and some services: An emerging science with conceptual, methodological, practical guidelines. Psychotherapy, 33(2), 332–345. and training challenges. Administration and Policy in Mental Hogue, A., Dauber, S., Chincilla, P., Fried, A., Henderson, C., Inclan, Health and Mental Health Services Research, 36(1), 24–34. J., et al. (2008). Assessing fidelity in individual and family Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & therapy for adolescent substance abuse. Journal of Substance Weaver, N. L. (2008). A glossary for dissemination and Abuse Treatment, 35, 137–147. implementation research in health. Journal of Public Health Hovmand, P. S., & Gillespie, D. F. (2010). Implementation of Management and Practice, 14(2), 117–123. evidence based practice and organizational performance. Journal Raghavan, R., Bright, C., & Shadoin, A. (2008). Toward a policy of Behavioral Health Services & Research, 37(1), 79–94. ecology of implementation of evidence-based practices in public Institute of Medicine Committee on Crossing the Quality Chasm. mental health settings. Implementation Science, 3, 26. (2006). Adaption to mental health and addictive disorder: Repenning, N. P. (2002). A simulation-based approach to under- Improving the quality of health care for mental and substance- standing the dynamics of innovation implementation. Organiza- use conditions. Washington, DC: Institute of Medicine, National tion Science, 13(2), 109–127. Academies Press. Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: Institute of Medicine Committee on Quality of Health Care in The Free Press. America. (2001). Crossing the quality chasm: A new health Ronckers, E., Groot, W., Steenbakkers, M., Ruland, E., & Ament, A. system for the 21st century. Washington, DC: Institute of (2006). Costs of the ‘hartslag limburg’ community heart health Medicine, National Academy Press. intervention. BMC Public Health, 6(1), 51. Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building Rosen, A., & Proctor, E. K. (1981). Distinctions between treatment capacity and sustainable prevention innovations: A sustainability outcomes and their implications for treatment evaluation. planning model. Evaluation Program Planning, 27, 135–149. Journal of Consulting and Clinical Psychology, 49(3), 418–425. Jones, K. (1997). Mission drift in qualitative research, or moving Rye, C. B., & Kimberly, J. R. (2007). The adoption of innovations by toward a systematic review of qualitative studies. The Qualita- provider organizations in health care. Medical Care Research tive Report, 9(1), 95–112. and Review, 64(3), 235–278. Karlsson, A., & Bendtsen, P. (2005). Acceptability of a computerized Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. A. alcohol screening and advice routine in an emergency depart- (2005). Predicting therapist adherence to a transported family- ment setting ¯ a patient perspective. Addictive Behaviors, 30, based treatment for youth. Journal of Clinical Child & 767–776. Adolescent Psychology, 34, 658–670. Karsh, B. T. (2004). Beyond usability: Designing effective technol- Shumway, M., Saunders, T., Shern, D., Pines, E., Downs, A., Burbine, ogy implementation systems to promote patient safety. Quality T., et al. (2003). Preferences for schizophrenia treatment and Safety in Health Care, 13, 388–394. outcomes among public policy makers, consumers, families, Kirkevoid, M. (1997). Integrative nursing research—an important and provides. Psychiatric Services, 54, 1124–1128. strategy to further the development of nursing science and Steckler, A., Goodman, R., & McElory, K. (1992). Measuring the practice. Journal of Advanced Nursing, 25, 977–984. diffusion of innovative health promotion programs. American Klein, K. J., & Knight, A. P. (2005). Innovation implementation: Journal of Health Promotion, 6, 214–224. Overcoming the challenge. Current Directions in Psychological Stiles, P. G., Boothroyd, R. A., Snyder, K., & Zong, X. (2002). Science, 14(5), 243–246. Service penetration by persons with severe mental illness: How Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation should it be measured? Journal of Behavioral Health Services & implementation. (Special topic forum on the management of Research, 29(2), 198. innovation). Academy of Management Review, 21(4), 1055–1081. Turner, K. M. T., & Sanders, M. R. (2006). Dissemination of Luke, D. A. (2004). Multilevel modeling. Sage University Papers evidence-based parenting and family support strategies: Learn- Series on Quantitative Applications in the Social Sciences. ing from the triple p-positive parenting program system Thousand Oaks, CA: Sage. approach. Aggression and Violent Behavior, 11(2), 176–193. McHugh, R. K., Otto, M. W., Barlow, D. H., Gorman, J. M., Shear, Waldorff, F., Steenstrup, A., Nielsen, B., Rubak, J., & Bro, F. (2008). M. K., & Woods, S. W. (2007). Cost-efficacy of individual and Diffusion of an e-learning programme among Danish general combined treatments for panic disorder. The Journal of Clinical practitioners: A nation-wide prospective survey. BMC Family Psychiatry, 68(7), 1038–1044. Practice, 9(1), 24. 123 76 Adm Policy Ment Health (2011) 38:65–76 Wilkie, D. J., Judge, M. K. M., Berry, D. L., Dell, J., Zong, S., Woolf, S. H. (2008). The meaning of translational research and why it Gilespie, R., et al. (2003). Usability of a computerized PAIN- matters. Journal of the American Medical Association, 299(2), ReportIt in the general public with pain and people with cancer 211–213. pain. Journal of Pain and Symptom Management, 25(3), 213–224. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Administration and Policy in Mental Health Pubmed Central

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda

Administration and Policy in Mental Health , Volume 38 (2) – Oct 19, 2010

Loading next page...
 
/lp/pubmed-central/outcomes-for-implementation-research-conceptual-distinctions-HfThZy8EXF

References (82)

Publisher
Pubmed Central
Copyright
© The Author(s) 2010
ISSN
0894-587X
eISSN
1573-3289
DOI
10.1007/s10488-010-0319-7
Publisher site
See Article on Publisher Site

Abstract

Adm Policy Ment Health (2011) 38:65–76 DOI 10.1007/s10488-010-0319-7 OR IGINAL PAPER Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda • • • Enola Proctor Hiie Silmere Ramesh Raghavan • • • Peter Hovmand Greg Aarons Alicia Bunger Richard Griffey Melissa Hensley Published online: 19 October 2010 The Author(s) 2010. This article is published with open access at Springerlink.com Abstract An unresolved issue in the field of implemen- Keywords Implementation  Outcomes  Evaluation tation research is how to conceptualize and evaluate suc- Research methods cessful implementation. This paper advances the concept of ‘‘implementation outcomes’’ distinct from service system and clinical treatment outcomes. This paper proposes a Background heuristic, working ‘‘taxonomy’’ of eight conceptually dis- tinct implementation outcomes—acceptability, adoption, A critical yet unresolved issue in the field of implementa- appropriateness, feasibility, fidelity, implementation cost, tion science is how to conceptualize and evaluate success. penetration, and sustainability—along with their nominal Studies of implementation use widely varying approaches definitions. We propose a two-pronged agenda for research to measure how well a new mental health treatment, pro- on implementation outcomes. Conceptualizing and mea- gram, or service is implemented. Some infer implementa- suring implementation outcomes will advance under- tion success by measuring clinical outcomes at the client or standing of implementation processes, enhance efficiency patient level while other studies measure the actual targets in implementation research, and pave the way for studies of of the implementation, quantifying for example the desired the comparative effectiveness of implementation strategies. provider behaviors associated with delivering the newly implemented treatment. While some studies of implemen- tation strategies assess outcomes in terms of improvement in process of care, Grimshaw et al. (2006) report that meta- E. Proctor (&)  R. Raghavan  P. Hovmand  A. Bunger M. Hensley analyses of their effectiveness has been thwarted by lack of George Warren Brown School of Social Work, Washington detailed information about outcomes, use of widely varying University in St. Louis, Campus Box 1196, One Brookings constructs, reliance on dichotomous rather than continuous Drive, St. Louis, MO 63130, USA measures, and unit of analysis errors. e-mail: ekp@wustl.edu This paper advances the concept of ‘‘implementation H. Silmere outcomes’’ distinct from service system outcomes and Division of Social Work, Roberts Wesleyan College, clinical treatment outcomes (Proctor et al. 2009; Fixsen Rochester, USA et al. 2005; Glasgow 2007a). We define implementation R. Raghavan outcomes as the effects of deliberate and purposive actions Department of Psychiatry, Washington University School to implement new treatments, practices, and services. of Medicine, St. Louis, MO, USA Implementation outcomes have three important functions. First, they serve as indicators of the implementation suc- G. Aarons Department of Psychiatry, University of California, cess. Second, they are proximal indicators of implemen- San Diego, La Jolla, CA, USA tation processes. And third, they are key intermediate outcomes (Rosen and Proctor 1981) in relation to service R. Griffey system or clinical outcomes in treatment effectiveness Department of Emergency Medicine, Washington University and quality of care research. Because an intervention or School of Medicine, St. Louis, MO, USA 123 66 Adm Policy Ment Health (2011) 38:65–76 treatment will not be effective if it is not implemented well, Implementation Service Client implementation outcomes serve as necessary preconditions Outcomes Outcomes* Outcomes for attaining subsequent desired changes in clinical or Efficiency Satisfaction Acceptability service outcomes. Safety Function Adoption Symptomatology Effectiveness Distinguishing implementation effectiveness from treat- Appropriateness Equity Costs ment effectiveness is critical for transporting interventions Patient- Feasibility from laboratory settings to community health and mental centeredness Fidelity Timeliness Penetration health venues. When such efforts fail, as they often do, it is Sustainability important to know if the failure occurred because the intervention was ineffective in the new setting (interven- *IOM Standards of Care tion failure), or if a good intervention was deployed incorrectly (implementation failure). Our current knowl- Fig. 1 Types of outcomes in implementation research edge of implementation is thwarted by lack of theoretical understanding of the processes involved (Michie et al. system) are examined in implementation research. How- 2009). Conceptualizing and measuring implementation ever, as we argued above, implementation research requires outcomes that are conceptually and empirically distinct outcomes will advance understanding of implementation processes, enable studies of the comparative effectiveness from those of service and clinical effectiveness. For heuristic purposes, our model positions implemen- of implementation strategies, and enhance efficiency in implementation research. tation outcomes as preceding both service outcomes and This paper aims to advance the ‘‘vocabulary’’ of imple- client outcomes, with the latter sets of outcomes being mentation science around implementation outcomes through impacted by the implementation outcomes. As we discuss four specific objectives: (1) to advance conceptualization of later in this paper, interrelationships among these outcomes implementation outcomes by distinguishing implementation require conceptual mapping and empirical tests. For outcomes from service and clinical outcomes; (2) to advance example, one would expect to see a treatment’s strongest impact on client outcomes as an empirically supported clarity of terminology currently used in implementation science by nominating heuristic definitions of implementa- treatment’s (EST) penetration increases in a service set- ting—but this hypothesis requires testing. Our model tion outcomes, yielding a working ‘‘taxonomy’’ of imple- mentation outcomes; (3) to reflect the field’s current derives service outcomes from the six quality improvement aims set out in the reports on crossing the quality chasm: language, conceptual definitions, and approaches to opera- tionalizing implementation outcomes; and (4) to propose the extent to which services are safe, effective, patient- centered, timely, efficient, and equitable (Institute of directions for further research to advance knowledge on these key constructs and their interrelationships. Medicine Committee on Crossing the Quality Chasm 2006; Our objective of advancing a taxonomy of implemen- Institute of Medicine Committee on Quality of Health Care tation outcomes is comparable to the work of Michie et al. in America 2001). (2005, 2009), Grimshaw et al. (2006), the Cochrane group, and others who are working to develop taxonomies and common nomenclature for implementation strategies. Our Methods work is complementary to these efforts because imple- mentation outcomes will provide researchers with a The paper’s methods were shaped around its overall aim: to advance clarity in the language used to describe outcomes framework for evaluating implementation strategies. of implementation. We convened a working group of Conceptual Framework for Implementation Outcomes implementation researchers to identify concepts for label- ing and assessing outcomes of implementation processes. Our understanding of implementation outcomes is lodged One member of the group was a doctoral student RA who within a previously published conceptual framework coordinated, conducted, and reported on the literature (Proctor et al. 2009) as shown in Fig. 1. The framework search and constructed tables reflecting various iterations of the heuristic taxonomy. The RA conducted literature distinguishes between three distinct but interrelated types of outcomes—implementation, service, and client outcomes. searches using key words and search programs to identify literature on the current state of conceptualization and Improvements in consumer well-being provide the most important criteria for evaluating both treatment and imple- measurement of these outcomes, primarily in the health and behavioral sciences. We searched in a number of mentation strategies—for treatment research, improvements are examined at the individual client level whereas databases with a particular focus on MEDLINE, CINAHL improvements at the population-level (within the providing Plus, and PsycINFO. Key search terms included the name 123 Adm Policy Ment Health (2011) 38:65–76 67 of the implementation outcome (e.g., ‘‘acceptability,’’ conceptualization of terms used to assess implementation ‘‘sustainability,’’ etc.) along with relevant synonyms com- outcomes. bined with any of the following: innovation, EBP, evidence based practice, and EST. We scanned the titles and Taxonomy of Implementation Outcomes abstracts of the identified sources and read the methods and background sections of the studies that measured or Through our process of iterative reading and discussion of attempted to measure implementation outcomes. We also the literature, we worked to nominate definitions that (1) included information from relevant conceptual articles in achieve as much consistency as possible with any existing the development of nominal definitions. Whereas our pri- definitions (including multiple definitions we found for a mary focus was on the implementation of evidence based single construct), yet (2) serve to sharpen distinctions practices in the health and behavioral sciences, the key- between constructs that might be similar. For several of the word ‘‘innovation’’ broadened this scope by also identify- outcomes, the literature did not offer one clear nominal ing studies that focused on other areas such as physical definition. health that may inform implementation of mental health Table 1 depicts the resultant working taxonomy of treatments. Because terminology in this field currently implementation outcomes. For each implementation out- reflects widespread inconsistency, we followed leads come, the table nominates a level of analysis, identifies the beyond what our keyword searches ‘‘hit’’ upon. Thus we theoretical basis to the construct from implementation lit- read additional articles that we found cited by authors erature, shows different terms that are used for the con- whose work we found through our electronic searches. We struct in the literature, suggests the point or stage within also conducted searches of CRISP, TAGG, and NIH implementation processes at which the outcome may be reporter and studies to identify funded mental health most salient, and lists the types of existing measures for the research studies with ‘‘implementation’’ in their titles or construct that our search identified. The implementation abstracts, to identify examples of outcomes pursued in outcomes listed in Table 1 are probably only the ‘‘more current research. obvious,’’ and we expect that other concepts may emerge We used a narrative review approach (Educational from further analysis of the literature and from the kind of Research Review), which is appropriate for summarizing empirical work we call for in our discussion below. Many different primary studies and drawing conclusions and of the implementation outcomes can be inferred or mea- interpretation about ‘‘what we know,’’ informed by sured in terms of expressed attitudes and opinions, inten- reviewers’ experiences and existing theories (McPheeters tions, or reported or observed behaviors. We now list and et al. 2006; Kirkevoid 1997). Narrative reviews yield discuss our nominated conceptual definitions for each implementation outcome in our proposed taxonomy. We qualitative results, with strengths in capturing diversities and pluralities of understanding (Jones 1997). According to reference similar definitions from the literature, and also McPheeters et al. (2006), narrative reviews are best con- comment on marked differences between our definitions ducted by a team. Members of the working group read and and others proposed for the term. reviewed conceptual and theoretical pieces as well as Acceptability is the perception among implementation published reports of implementation research. As a team, stakeholders that a given treatment, service, practice, or we convened recurring meetings to discuss the similarities innovation is agreeable, palatable, or satisfactory. Lack of and dissimilarities. We audio-taped and transcribed meet- acceptability has long been noted as a challenge in ing discussions, and a designated individual took thorough implementation (Davis 1993). The referent of the imple- notes. Transcriptions and notes were posted on a shared mentation outcome ‘‘acceptability’’ (or the ‘‘what’’ is computer file for member review, revision, and correction. acceptable) may be a specific intervention, practice, tech- Group processes included iterative discussion, checking nology, or service within a particular setting of care. additional literature for clarification, and subsequent dis- Acceptability should be assessed based on the stake- cussion. The aim was to collect and portray, from extant holder’s knowledge of or direct experience with various literature, the similarities and differences across investi- dimensions of the treatment to be implemented, such as its gators’ use of various implementation outcomes and defi- content, complexity, or comfort. Acceptability is different nitions for those outcomes. Discussions often led us to from the larger construct of service satisfaction, as typi- preserve distinctions between terms by maintaining in our cally measured through consumer surveys. Acceptability is ‘‘nominated’’ taxonomy two different implementation more specific, referencing a particular treatment or set of outcomes because the literature or our own research treatments, while satisfaction typically references the revealed possible conceptual distinctions. We assembled general service experience, including such features as the identified constructs in the proposed heuristic taxon- waiting times, scheduling, and office environment. omy to portray the current state of vocabulary and Acceptability may be measured from the perspective of 123 68 Adm Policy Ment Health (2011) 38:65–76 Table 1 Taxonomy of implementation outcomes Implementation Level of analysis Theoretical basis Other terms in literature Salience by implementation Available measurement outcome stage Acceptability Individual provider Rogers: ‘‘complexity’’ and to a Satisfaction with various aspects Early for adoption Survey certain extent ‘‘relative of the innovation (e.g. content, Individual consumer Ongoing for penetration Qualitative or semi-structured advantage’’ complexity, comfort, delivery, interviews Late for sustainability and credibility) Administrative data Refused/blank Adoption Individual provider RE-AIM: ‘‘adoption’’ Rogers: Uptake; utilization; initial Early to mid Administrative data ‘‘trialability’’ (particularly for implementation; intention to try Organization or setting Observation early adopters) Qualitative or semi-structured interviews Survey Appropriateness Individual provider Rogers: ‘‘compatibility’’ Perceived fit; relevance; Early (prior to adoption) Survey compatibility; suitability; Individual consumer Qualitative or semi-structured usefulness; practicability interviews Organization or setting Focus groups Feasibility Individual providers Rogers: ‘‘compatibility’’ and Actual fit or utility; suitability for Early (during adoption) Survey ‘‘trialability’’ everyday use; practicability Organization or setting Administrative data Fidelity Individual provider RE-AIM: part of Delivered as intended; adherence; Early to mid Observation ‘‘implementation’’ integrity; quality of program Checklists delivery Self-report Implementation Cost Provider or providing TCU Program Change Model: Marginal cost; cost-effectiveness; Early for adoption and Administrative data institution ‘‘costs’’ and ‘‘resources’’ cost-benefit feasibility Mid for penetration Late for sustainability Penetration Organization or setting RE-AIM: necessary for ‘‘reach’’ Level of institutionalization? Mid to late Case audit Spread? Service access? Checklists Sustainability Administrators RE-AIM: ‘‘maintenance’’ Maintenance; continuation; Late Case audit Rogers: ‘‘confirmation’’ durability; incorporation; Organization or setting Semi-structured interviews integration; institutionalization; Questionnaires sustained use; routinization; Checklists Adm Policy Ment Health (2011) 38:65–76 69 various stakeholders, such as administrators, payers, pro- Bartholomew et al. (2007) describe a rating scale for viders, and consumers. We presume rated acceptability to capturing appropriateness of training among substance be dynamic, changing with experience. Thus ratings of abuse counselors who attended training in dual diagnosis acceptability may be different when taken, for example, and therapeutic alliance. pre-implementation and later throughout various stages of Cost (incremental or implementation cost) is defined as implementation. The literature reflects several examples of the cost impact of an implementation effort. Implementa- measuring provider and patient acceptability. Aarons’ tion costs vary according to three components. First, Evidence-Based Practice Attitude Scale (EBPAS) captures because treatments vary widely in their complexity, the the acceptability of evidence-based mental health treat- costs of delivering them will also vary. Second, the costs of ments among mental health providers (Aarons 2004). implementation will vary depending upon the complexity Aarons and Palinkas (2007) used semi-structured inter- of the particular implementation strategy used. Finally, views to assess case managers’ acceptance of evidence- because treatments are delivered in settings of varying based practices in a child welfare setting. Karlsson and complexity and overheads (ranging from a solo practi- Bendtsen (2005) measured patients’ acceptance of alcohol tioner’s office to a tertiary care facility), the overall costs of screening in an emergency department setting using a delivery will vary by the setting. The true cost of imple- 12-item questionnaire. menting a treatment, therefore, depends upon the costs of Adoption is defined as the intention, initial decision, or the particular intervention, the implementation strategy action to try or employ an innovation or evidence-based used, and the location of service delivery. practice. Adoption also may be referred to as ‘‘uptake.’’ Much of the work to date has focused on quantifying Our definition is consistent with those proposed by Rabin intervention costs, e.g., identifying the components of a et al. (2008) and Rye and Kimberly (2007). Adoption could community-based heart health program and attaching costs be measured from the perspective of provider or organi- to these components (Ronckers et al. 2006). These cost zation. Haug et al. (2008) used pre-post items to capture estimations are combined with patient outcomes and used substance abuse providers’ adoption of evidence-based in cost-effectiveness studies (McHugh et al. 2007). A practices, while Henggeler et al. (2008) report interview review of literature on guideline implementation in pro- techniques to measure therapists’ adoption of contingency fessions allied to medicine notes that few studies report management. anything about the costs of guideline implementation Appropriateness is the perceived fit, relevance, or (Callum et al. 2010). Implementing processes that do not compatibility of the innovation or evidence based practice require ongoing supervision or consultation, such as com- for a given practice setting, provider, or consumer; and/or puterized medical record systems, may carry lower costs than implementing new psychosocial treatments. Direct perceived fit of the innovation to address a particular issue or problem. ‘‘Appropriateness’’ is conceptually similar to measures of implementation cost are essential for studies ‘‘acceptability,’’ and the literature reflects overlapping and comparing the costs of implementing alternative treatments sometimes inconsistent terms when discussing these con- and of various implementation strategies. structs. We preserve a distinction because a given treat- Feasibility is defined as the extent to which a new ment may be perceived as appropriate but not acceptable, treatment, or an innovation, can be successfully used or and vice versa. For example, a treatment might be con- carried out within a given agency or setting (Karsh 2004). sidered a good fit for treating a given condition but its Typically, the concept of feasibility is invoked retrospec- features (for example, rigid protocol) may render it unac- tively as a potential explanation of an initiative’s success or ceptable to the provider. The construct ‘‘appropriateness’’ failure, as reflected in poor recruitment, retention, or par- is deemed important for its potential to capture some ticipation rates. While feasibility is related to appropriate- ‘‘pushback’’ to implementation efforts, as is seen when ness, the two constructs are conceptually distinct. For providers feel a new program is a ‘‘stretch’’ from the example, a program may be appropriate for a service set- mission of the health care setting, or is not consistent with ting—in that it is compatible with the setting’s mission or providers’ skill set, role, or job expectations. For example, service mandate, but may not be feasible due to resource or providers may vary in their perceptions of the appropri- training requirements. Hides et al. (2007) tapped aspects of ateness of programs that co-locate mental health services feasibility of using a screening tool for co-occurring mental within primary medical, social service, or school settings. health and substance use disorders. Again, a variety of stakeholders will likely have percep- Fidelity is defined as the degree to which an intervention tions about a new treatment’s or program’s appropriateness was implemented as it was prescribed in the original pro- to a particular service setting, mission, providers, and cli- tocol or as it was intended by the program developers entele. These perceptions may be function of the organi- (Dusenbury et al. 2003; Rabin et al. 2008). Fidelity has zation’s culture or climate (Klein and Sorra 1996). been measured more often than the other implementation 123 70 Adm Policy Ment Health (2011) 38:65–76 outcomes, typically by comparing the original evidence- construct with terms such a given treatment’s level of based intervention and the disseminated/implemented institutionalization. intervention in terms of (1) adherence to the program pro- Sustainability is defined as the extent to which a newly tocol, (2) dose or amount of program delivered, and (3) implemented treatment is maintained or institutionalized quality of program delivery. Fidelity has been the over- within a service setting’s ongoing, stable operations. The riding concern of treatment researchers who strive to move literature reflects quite varied uses of the term ‘‘sustain- their treatments from the clinical lab (efficacy studies) to ability,’’ but our proposed definition incorporates aspects of real-world delivery systems. The literature identifies five those offered by Johnson et al. (2004), Turner and Sanders implementation fidelity dimensions including adherence, (2006), Glasgow et al. (1999), Goodman et al. (1993), and quality of delivery, program component differentiation, Rabin et al. (2008). Rabin et al. (2008) emphasizes the exposure to the intervention, and participant responsive- integration of a given program within an organization’s ness or involvement (Mihalic 2004; Dane and Schneider culture through policies and practices, and distinguishes 1998). Adherence, or the extent to which the therapy three stages that determine institutionalization: (1) passage occurred as intended, is frequently examined in psycho- (a single event such as transition from temporary to per- therapy process and outcomes research and is distinguished manent funding), (2) cycle or routine (i.e., repetitive from other potentially pertinent implementation factors reinforcement of the importance of the evidence-based such as provider skill or competence (Hogue et al. 1996). intervention through including it into organizational or Fidelity is measured through self-report, ratings, and direct community procedures and behaviors, such as the annual observation and coding of audio- and videotapes of actual budget and evaluation criteria), and (3) niche saturation encounters, or provider-client/patient interaction. Achiev- (the extent to which an evidence-based intervention is ing and measuring fidelity in usual care is beset by a integrated into all subsystems of an organization). Thus the number of challenges (Proctor et al. 2009; Mihalic 2004; outcomes of ‘‘penetration’’ and ‘‘sustainability’’ may be Schoenwald et al. 2005). The foremost challenge may be related conceptually and empirically, in that higher pene- measuring implementation fidelity quickly and efficiently tration may contribute to long-term sustainability. Such (Hayes 1998). relationships require empirical test, as we elaborate below. Schoenwald and colleagues (2005) have developed three Indeed Steckler et al. (1992) emphasize sustainability in 26–45-item measures of adherence at the therapist, super- terms of attaining long-term viability, as the final stage of visor and consultant level of implementation (available the diffusion process during which innovations settle into from the MST Institute www.mstinstitute.org). Ratings are organizations. To date, the term sustainability appears obtained at regular intervals, enabling examination of the more frequently in conceptual papers than actual empirical provider, clinical supervisor, and consultant. Other exam- articles measuring sustainability of innovations. As we ples from the mental health literature include Bond et al. discuss below, the literature often uses the same term (2008) 15-item Supported Employment Fidelity Scale (SE (niche saturation, for example) to reference multiple Fidelity Scale) and Hogue et al. (2008) Therapist Behavior implementation outcomes, underscoring the need for con- Rating Scale-Competence (TBRS-C), an observational ceptual clarity as we seek to advance in this paper. measure of fidelity in evidence based practices for ado- lescent substance abuse treatment. Penetration is defined as the integration of a practice Research Agenda to Advance Implementation within a service setting and its subsystems. This definition Outcomes is similar to (Stiles et al. 2002) notion of service penetra- tion and to Rabin et al.s’ (2008) notion of niche saturation. Advancing the conceptualization, measurement, and empir– Studying services for persons with severe mental illness, ical understanding of implementation outcomes requires Stiles et al. (2002) apply the concept of service penetration research on several critical issues. We propose two major to service recipients (the number of eligible persons who themes for this research—(1) conceptualization and mea- use a service, divided by the total number of persons eli- surement, and (2) theory building—and identify important gible for the service). Penetration also can be calculated in issues within each of these themes. terms of the number of providers who deliver a given service or treatment, divided by the total number of pro- Research on Conceptualization and Measurement viders trained in or expected to deliver the service. From a of Implementation Outcomes service system perspective, the construct is also similar to ‘‘reach’’ in the RE-AIM framework (Glasgow 2007b). We Research on several fronts is required to advance the found infrequent use of the term penetration in the imple- conceptual and measurement properties of implementation mentation literature; though studies seemed to tap into this outcomes, five of which we identify and discuss. 123 Adm Policy Ment Health (2011) 38:65–76 71 Consistency of Terminology setting; or (3) a broad effort to implement several new treatments at once. A lingering issue for the field is whether For each outcome listed in Table 1, we found literature implementation processes should be tackled and studied using different and sometimes inconsistent terminology. specifically (one new treatment) or in a more generalized Sometimes studies used different labels for what appear to way (the extent to which a system’s care is evidence-based be the same construct. In other cases, studies used one term or guideline congruent). Understanding the optimal speci- for a label or nominal definition but a different term for ficity of the referent for a given implementation outcome is operationalizing or measuring the same construct. This critical for measurement. As a beginning step, researchers problem was pronounced for three implementation out- should report the referent for all implementation outcomes comes—acceptability, appropriateness, and feasibility. measured. These constructs were frequently used interchangeably or measured under the common generic label as client or Level of Analysis for Outcomes provider perceptions, reactions, and attitudes toward, or satisfaction with various aspects of the innovation, EST, Implementation of new treatments is an inherently multi- or clinical practice guidelines. For example, Graham et al. level enterprise, involving provider behavior, care organi- (2007) assessed doctors’ attitudes and perceptions toward zation, and policy (Proctor et al. 2009; Raghavan et al. clinical practice guidelines with a survey that tapped all 2008). Implementation outcomes are important at each three of these outcomes, although none of them were level of change, but the research has yet to determine explicitly labeled as such: acceptability (e.g. perceived which level or unit of analysis is most appropriate for quality of and confidence in guidelines), appropriateness particular implementation outcomes. Certain outcomes, (e.g. perceived usefulness of guidelines), and feasibility such as acceptability, may be most appropriate for indi- (e.g. these guidelines provide recommendations that are vidual level analysis (for example, providers, consumers), implementable). Other studies interchanged the terms for while others, such as penetration may be more appropriate acceptability and feasibility within the same article. For for aggregate analysis, at the level of the health care example, Wilkie et al. (2003) begin by describing the organization. Currently, very few studies reporting imple- measurement of ‘‘usability’’ (of a computerized innova- mentation outcomes specify the level of measurement, nor tion), including its ‘‘acceptability’’ to clients but later use do they address issues of aggregation within or across the findings to conclude that the innovation was feasible. levels. While language inconsistency is typical in most still- Construct validity. The constructs reflected in Table 1 developing fields, implementation research may be partic- and the terms employed in our taxonomy of implementation ularly susceptible to this problem. No one discipline is outcomes derive largely from the research literature. Yet it is ‘‘home’’ to implementation research. Studies are conducted important to also understand outcome perceptions and across a broad range of disciplines, published in a scattered preferences through the voice of those who design and set of journals, and consequently are rarely cross refer- deliver health care. Qualitative data, reflecting language enced. Beyond mental health, we found articles referencing used by various stakeholders as they think and talk these implementation outcomes in physical health, smok- about implementation processes, is important for validat- ing cessation, cancer, and substance abuse literatures, ing implementation outcome constructs. Through in-depth addressing a wide variety of topics. interviews, stakeholders’ cognitive representations and Clearly, the field of implementation science now has mental models of outcomes can be analyzed through such only the beginnings of a common language to characterize methods as cultural domain analysis (CDA). A ‘‘cultural implementation outcomes, a situation that thwarts the domain’’ refers to a set of words, phrases, and/or concepts conceptual and empirical advancement of the field but that link together to form a single conceptual subject (Luke could be overcome by use of a common lexicon. Just as 2004; Bates and Sarkar 2007), and methods for CDA, such Michie et al. (2009) state the ‘‘imperative that there be a as free-listing and pile-sorting, have been used since the consensual, common language’’ (p. 4) to describe behavior 1970s (Bates and Sarkar 2007). While primarily used in change techniques, so is common language needed for anthropology, CDA is aptly suited for health services implementation outcomes. research that endeavors to understand how stakeholders conceptualize implementation outcomes, informing the Referent for Rating the Outcome generation of definitions of implementation outcomes. The actual words used by stakeholders may or may not reflect the Several of the proposed implementation outcomes could be terms used in academic literature and reflected in our pro- used to rate (1) a specific treatment; (2) the implementation posed taxonomy (acceptability, appropriateness, feasibility, strategy used to introduce that treatment into the care adoption, fidelity, penetration, sustainability and costs). But 123 72 Adm Policy Ment Health (2011) 38:65–76 such research can identify the terms and distinctions that are Salience of Implementation Outcomes by Point meaningful to implementation stakeholders. in the Implementation Process Measurement Properties of Implementation Outcomes The implementation of any new treatment or service is widely recognized as a process, involving a sequence of The literature reflects a wide array of approaches for activities, beginning with initial considerations of what and how to change current care. Chamberlain has identified ten measuring implementation outcomes, ranging from quali- tative, quantitative survey, and record archival. Michie steps for the implementation of an evidence-based treat- ment, Multidimensional Treatment Foster Care (MTFC), et al. (2007) studied perceived difficulties implementing a mental health guideline, coding respondent descriptions of beginning with consideration of adopting MTFC and con- implementation difficulties as 0, 0.5, or 1. Much mea- cluding when a service site meets certification criteria for surement has been ‘‘home-grown,’’ with virtually no work delivering the treatment (Chamberlain et al. 2008). As we on the psychometric properties or measurement rigor. suggest in Table 1, certain implementation outcomes may Measurement development is needed to enhance the por- be more important at some phases of implementation tability and usefulness of implementation outcomes in real process than at other phases. For example, feasibility may be most important once organizations and providers try world settings of care. Measures used in efficacy research will likely prove too cumbersome for real-world studies new treatments. Later, it may be a ‘‘moot point,’’ once the treatment—initially considered novel or unknown—has of implementation. For example, detailed assessment of fidelity through coding of encounter videotapes would be become part of normal routine. too time-intensive for a multi-agency study assessing The literature suggests that studies usually capture fidelity of treatment implementation. fidelity during initial implementation, while adoption is often assessed at 6 (Waldorff et al. 2008), 12 (Adily et al. Theory-Building Research 2004; Fischer et al. 2008), or 18 months (Cooke et al. 2001) after initial implementation. But most studies fail to specify a timeframe or are inconsistent in choice of a Research is also needed to advance our theoretical under- standing of the implementation process. Empirical studies of time point in the implementation process for measuring outcomes. Research is needed to explore these issues, the five issues we list here will inform theory, illuminate the ‘‘black box’’ of implementation processes, and help particularly longitudinal studies that measure multiple implementation outcomes before, during, and after imple- shape models for developing and testing implementation strategies. mentation of a new treatment. Such research may reveal ‘‘leading’’ and ‘‘lagging’’ indicators of implementation Salience of Implementation Outcomes to Stakeholders success. For example, if acceptability increases for several months, following which penetration increases, then we Any effort to implement change in care involves a range of may view acceptability as a leading indicator of penetra- stakeholders, including the treatment developers who design tion. Leading indicators can be useful for managing the and test the effectiveness of ESTs, policy makers who design implementation process as they signal future trends. and pay for service, administrators who shape program Where leading indicators may identify future trends, lagging indicators reflect delays between when changes direction, providers and supervisors, patients/clients/con- sumers and their family members, and interested community happen and when they can be observed. For example, sustainability may be observed only well into, or even after members and advocates. The success of efforts to implement evidence-based treatment may rest on their congruence with the implementation process. Being aware of lagging indi- the preferences and priorities of those who shape, deliver, cators of implementation success may help managers avoid and participate in care. Implementation outcomes may be over-reacting to slow change and wait for evidence of what differentially salient to various stakeholders, just as the may soon prove to be successful implementation. salience of clinical outcomes varies across stakeholders (Shumway et al. 2003). For example, implementation cost Modeling Interrelationships Among Implementation Outcomes may be most important to policy makers and program directors, feasibility may be most important to direct service Our team’s observations of implementation suggest that providers, and fidelity may be most important to treatment developers. To ensure applicability of implementation out- implementation outcomes are themselves interrelated in dynamic and complex ways (Woolf 2008; Repenning 2002; comes across a range of settings and to maximize their external validity, all stakeholder groups and priorities should Hovmand and Gillespie 2010; Klein and Knight 2005) and be represented in this research. are likely to change throughout an agency’s process to 123 Adm Policy Ment Health (2011) 38:65–76 73 adopt and implement ESTs. For example, the perceived implementation, as well as the effects of contextual factors appropriateness, feasibility, and implementation cost asso- that must be addressed and that are captured in imple- ciated with an intervention will likely bear on ratings of mentation outcomes. the intervention’s acceptability. Acceptability, in turn, Established evidence for a ‘‘proven’’ treatment does not will likely affect adoption, penetration, and sustainability. ensure successful implementation. Implementation also Similarly, consistent with Rogers’ theory of the diffusion requires addressing a number of important contextual fac- of innovation, the ability to adopt or adapt an innovation tors, such as provider attitudes, professional behavior, and for local use may increase its acceptability (Rogers 1995). the service system. Constructs in the proposed taxonomy of This suggests that when providers believe they do not have implementation outcomes have potential to capture those to implement a treatment ‘‘by the book’’ (or with precise provider attitudes (acceptability) and behaviors (adoption, fidelity), they may rate the treatment as more acceptable. uptake) as well as contextual factors (system penetration, Modeling the interrelationships between implementation appropriateness, implementation cost). outcomes will also inform their definitional boundaries and For purposes of stimulating debate and future research, thus shape the taxonomy. For example, if two outcomes we suggest that successful implementation be considered in which we now define as distinct concepts are shown light of a ‘‘portfolio’’ of factors, including the effectiveness through research to always occur together, the empirical of the treatment to be implemented and implementation evidence would suggest that the concepts are really the outcomes such as included in our taxonomy. For example, same thing and should be combined. Similarly, if two of implementation success (I, in the equation below) could be the outcomes are shown to have different empirical pat- modeled to reflect (1) the effectiveness (E) of the treatment terns, evidence would confirm their conceptual distinction. being implemented, plus (2) implementation factors (IO’s), which heretofore have been insufficiently conceptualized, Modeling Attainment of Implementation Outcomes distinguished, and measured and rarely used to guide implementation decisions. Once researchers have advanced consistent, valid, and I ¼ fE þ IO’s efficient measures for implementation outcomes, the field will be equipped to conduct important research treating For example, in situation ‘‘A’’, an evidence-based treatment may be highly effective but given its high cost, these constructs as dependent variables, in order to identify correlates or predictors of their attainment. Their only mildly acceptable to key stakeholders and low in sustainability. The overall potential success of implemen- measurement will enable research to determine which features of a treatment itself or which implementation tation in this case might be modeled as follows: strategies help make new treatments acceptable, feasible Implementation success = f of effectivenessðÞ = high to implement, or sustainable over time. The diffusion of + acceptabilityðÞ = moderate innovation literature posits that the implementation out- + sustainabilityðÞ low : come, adoption of an EST, is a function of such factors as perceived need to do things differently (Rogers 1995) In situation ‘‘B’’, a given treatment might be only perception of the new treatment’s comparative advantage moderately effective but highly acceptable to stake- (Frambach and Schillewaert 2002; Henggeler et al. 2002) holders because current care is poor, the treatment is and as easy to understand (Berwick 2003). Such suppo- inexpensive, and current training protocols ensure high sitions require empirical test using measures of imple- penetration through providers. This treatment’s potential mentation outcomes. might be modeled in the following equation: Using Implementation Outcomes to Model Implementation success = Implementation Success f of treatment effectivenessðÞ moderate + acceptabilityðÞ high + potential to improve careðÞ high Reliable, valid measures of implementation outcomes will enable empirical testing of the success of efforts to + penetrationðÞ high : implement new treatments, and pave the way for compar- ative effectiveness research on implementation strategies. Thus using implementation outcomes, the success of In most current initiatives to move evidence-based treat- implementation may be modeled and tested, thereby ments into community care settings, the success of the making decisions about what to implement more explicit implementation is assumed and evaluated from data on and transparent. clinical outcomes. We believe that an exclusive focus on To increase the success of implementation, implemen- clinical outcomes thwarts understanding the process of tation strategies need to be employed strategically. For 123 74 Adm Policy Ment Health (2011) 38:65–76 Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, example, implementation strategies could be employed to R. (2008). Fidelity of supported employment: Lessons learned increase provider acceptance, improve penetration, reduce from the national evidence-based practice project. Psychiatric implementation costs, and achieve sustainability of the Rehabilitation Journal, 31(4), 300–305. treatment being implemented. Understanding how to Callum, P., Iger, J., Ray, M., Sims, C. A., & Falk, R. E. (2010). Outcome and experience of implementing spinal muscular achieve implementation outcomes requires the kind of work atrophy carrier screening on sperm donors. Fertility and Sterility, now underway by Michie et al. (2009) advance a taxonomy 94(5), 1912–1914. of implementation strategies and reflect their demonstrated Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., effects. Marsenich, L., et al. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy In Mental Health, 35, 250–260. Summary and Implications Cooke, M., Mattick, R. P., & Walsh, R. A. (2001). Implementation of the ‘fresh start’ smoking cessation programme to 23 antenatal clinics: A randomized controlled trial investigating two methods The science of implementation cannot be advanced with- of dissemination. Drug and Alcohol Review, 20, 19–28. out attention to implementation outcomes. All studies of Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary implementation should explicate and measure implemen- and early secondary prevention: Are implementation effects out tation outcomes. Given the rudimentary state of the field, of control? Clinical Psychology Review, 18, 23–45. Davis, F. (1993). User acceptance of information technology: System we chose a narrative approach to reviewing the literature characteristics, user perceptions and behavioral impacts. Inter- and constructing a taxonomy. Our purpose is to advance national Journal of Man-Machine Studies, 38, 475–487. the clarity of language, provoke debate, and stimulate more Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A systematic work toward the aims of advancing the con- review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education ceptual, linguistic, and methodological clarity in the field. Research, 18(2), 237–256. A taxonomy of implementation outcomes can help orga- Fischer, M. A., Vogeli, C., Stedman, M. R., Ferris, T. G., & nize the key variables and frame research questions Weissman, J. S. (2008). Uptake of electronic prescribing in required to advance implementation science. Their mea- community-based practices. Journal of General Internal Med- icine, 23(4), 358–363. surement and empirical test can help specify the mecha- Fixsen, D. L., Naoom, S. F., Blase ´, K. A., Friedman, R. M., & nisms and causal relationships within implementation Wallace, F. (2005). Implementation research: A synthesis of the processes and advance an evidence base around successful literature (No. FMHI Publication #231). Tampa, FL: University implementation. of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network. Frambach, R. T., & Schillewaert, N. (2002). Organizational innova- Acknowledgments Preparation of this manuscript was supported by tion adoption: A multi-level framework of determinants and grants P30 MH068579 and UL1RR024992. opportunities for future research. Journal of Business Research Special Issue: Marketing Theory in the Next Millennium, 55, Open Access This article is distributed under the terms of the 163–176. Creative Commons Attribution Noncommercial License which per- Glasgow, R. E. (2007a). eHealth evaluation and dissemination mits any noncommercial use, distribution, and reproduction in any research. American Journal of Preventive Medicine, 32(5S), medium, provided the original author(s) and source are credited. S119–S126. Glasgow, R. E. (2007b). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science References of Dissemination and Implementation in the Service of Public Health: Bethesda, MD. Aarons, G. A. (2004). Mental health provider attitudes toward adoption Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating of evidence-based practice: The evidence-based practice attitude the public health impact of health promotion interventions: The scale (EBPAS). Mental Health Service Research, 6(2), 61–74. RE-AIM framework. American Journal of Public Health, 89, Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence- 1322–1327. based practice in child welfare: Service provider perspectives. Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. Mental Health Services Research, 34, 411–419. (1993). Development of level of institutionalization scales for Adily, A., Westbrook, J., Coiera, E., & Ward, J. (2004). Use of on- health promotion programs. Health Education Quarterly, 20(2), line evidence databases by Australian public health practitioners. 161–179. Medical Informatics & the Internet in Medicine, 29(2), 127–136. Graham, I. D., Brouwers, M., Davies, C., & Tetroe, J. (2007). Ontario Bartholomew, N. G., Joe, G. W., Rowan-Szal, G. A., & Simpson, D. doctors’ attitudes toward and use of clinical practice guidelines D. (2007). Counselor assessments of training and adoption in oncology. Journal of Evaluation in Clinical Practice, 13(4), barriers. Journal of Substance Abuse Treatment, 33, 193–199. 607–615. Bates, D. M., & Sarkar, D. (2007). lme4: Linear mixed-effects models Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., using S4 classes. R Package Version, 0, 99875–99876. Fraser, C., et al. (2006). Toward evidence-based quality Berwick, D. M. (2003). Disseminating innovations in health care. improvement: Evidence (and its limitations) of the effectiveness Journal of the American Medical Association, 289(15), 1969– of guideline dissemination and implementation strategies 1975. 1966–1998. Journal of General Internal Medicine, 21, S14–S20. 123 Adm Policy Ment Health (2011) 38:65–76 75 Haug, N. A., Shopshire, M., Tajima, B., Gruber, V., & Guydish, J. McPheeters, M. L., Briss, P., Teutsch, S. J., & Truman, B. (2006). (2008). Adoption of evidence-based practices among substance Systematic reviews in public health. In R. C. Brownson & D. abuse treatment providers. Journal of Drug Education, 38(2), B. Petitti (Eds.), Applied epidemiology: Theory to practice (2nd 181–192. ed., pp. 99–124). New York: Oxford University Press. Hayes, S. C. (1998). President’s message: Dissemination research Michie, S., Johnson, M., Abraham, C., Lawton, R., Parker, D., & now. The Behavior Therapist, 21(9), 166–169. Walker, A. (2005). Making psychological theory useful for Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after implementing evidence based practice: A consensus approach. the innovation is identified? Clinical Psychology: Science & Quality and Safety in Health Care, 14(1), 26–33. Practice, 9(2), 191–194. Michie, S., Pilling, S., Garety, P., Whitty, P., Eccles, M. P., Johnston, Henggeler, S. W., Chapman, J. E., Rowland, M. D., Halliday- M., et al. (2007). Difficulties implementing a mental health Boykinds, C. A., Randall, J., Shackelford, J., et al. (2008). guideline: An exploratory investigation using psychosocial Statewide adoption and initial implementation of contingency theory. Implementation Science, 2(8), 1–8. management for substance-abusing adolescents. Journal of Michie, S., Fixsen, D., Grimshaw, J., & Eccles, M. (2009). Specifying Consulting and Clinical Psychology, 76(4), 556–567. and reporting complex behavior change interventions: The need Hides, L., Lubman, D. I., Elkins, K., Catania, L. S., & Rogers, N. for a scientific method. Implementation Science, 4(1), 40. (2007). Feasibility and acceptability of a mental health screening Mihalic, S. (2004). The importance of implementation fidelity. tool and training programme in the youth alcohol and other drug Emotional and Behavioral Disorders in Youth, 4, 83–86. (AOD) sector. Drug and Alcohol Review, 26, 509–515. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence & Mittman, B. (2009). Implementation research in mental health process research in family therapy: A rationale and some services: An emerging science with conceptual, methodological, practical guidelines. Psychotherapy, 33(2), 332–345. and training challenges. Administration and Policy in Mental Hogue, A., Dauber, S., Chincilla, P., Fried, A., Henderson, C., Inclan, Health and Mental Health Services Research, 36(1), 24–34. J., et al. (2008). Assessing fidelity in individual and family Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & therapy for adolescent substance abuse. Journal of Substance Weaver, N. L. (2008). A glossary for dissemination and Abuse Treatment, 35, 137–147. implementation research in health. Journal of Public Health Hovmand, P. S., & Gillespie, D. F. (2010). Implementation of Management and Practice, 14(2), 117–123. evidence based practice and organizational performance. Journal Raghavan, R., Bright, C., & Shadoin, A. (2008). Toward a policy of Behavioral Health Services & Research, 37(1), 79–94. ecology of implementation of evidence-based practices in public Institute of Medicine Committee on Crossing the Quality Chasm. mental health settings. Implementation Science, 3, 26. (2006). Adaption to mental health and addictive disorder: Repenning, N. P. (2002). A simulation-based approach to under- Improving the quality of health care for mental and substance- standing the dynamics of innovation implementation. Organiza- use conditions. Washington, DC: Institute of Medicine, National tion Science, 13(2), 109–127. Academies Press. Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: Institute of Medicine Committee on Quality of Health Care in The Free Press. America. (2001). Crossing the quality chasm: A new health Ronckers, E., Groot, W., Steenbakkers, M., Ruland, E., & Ament, A. system for the 21st century. Washington, DC: Institute of (2006). Costs of the ‘hartslag limburg’ community heart health Medicine, National Academy Press. intervention. BMC Public Health, 6(1), 51. Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building Rosen, A., & Proctor, E. K. (1981). Distinctions between treatment capacity and sustainable prevention innovations: A sustainability outcomes and their implications for treatment evaluation. planning model. Evaluation Program Planning, 27, 135–149. Journal of Consulting and Clinical Psychology, 49(3), 418–425. Jones, K. (1997). Mission drift in qualitative research, or moving Rye, C. B., & Kimberly, J. R. (2007). The adoption of innovations by toward a systematic review of qualitative studies. The Qualita- provider organizations in health care. Medical Care Research tive Report, 9(1), 95–112. and Review, 64(3), 235–278. Karlsson, A., & Bendtsen, P. (2005). Acceptability of a computerized Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. A. alcohol screening and advice routine in an emergency depart- (2005). Predicting therapist adherence to a transported family- ment setting ¯ a patient perspective. Addictive Behaviors, 30, based treatment for youth. Journal of Clinical Child & 767–776. Adolescent Psychology, 34, 658–670. Karsh, B. T. (2004). Beyond usability: Designing effective technol- Shumway, M., Saunders, T., Shern, D., Pines, E., Downs, A., Burbine, ogy implementation systems to promote patient safety. Quality T., et al. (2003). Preferences for schizophrenia treatment and Safety in Health Care, 13, 388–394. outcomes among public policy makers, consumers, families, Kirkevoid, M. (1997). Integrative nursing research—an important and provides. Psychiatric Services, 54, 1124–1128. strategy to further the development of nursing science and Steckler, A., Goodman, R., & McElory, K. (1992). Measuring the practice. Journal of Advanced Nursing, 25, 977–984. diffusion of innovative health promotion programs. American Klein, K. J., & Knight, A. P. (2005). Innovation implementation: Journal of Health Promotion, 6, 214–224. Overcoming the challenge. Current Directions in Psychological Stiles, P. G., Boothroyd, R. A., Snyder, K., & Zong, X. (2002). Science, 14(5), 243–246. Service penetration by persons with severe mental illness: How Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation should it be measured? Journal of Behavioral Health Services & implementation. (Special topic forum on the management of Research, 29(2), 198. innovation). Academy of Management Review, 21(4), 1055–1081. Turner, K. M. T., & Sanders, M. R. (2006). Dissemination of Luke, D. A. (2004). Multilevel modeling. Sage University Papers evidence-based parenting and family support strategies: Learn- Series on Quantitative Applications in the Social Sciences. ing from the triple p-positive parenting program system Thousand Oaks, CA: Sage. approach. Aggression and Violent Behavior, 11(2), 176–193. McHugh, R. K., Otto, M. W., Barlow, D. H., Gorman, J. M., Shear, Waldorff, F., Steenstrup, A., Nielsen, B., Rubak, J., & Bro, F. (2008). M. K., & Woods, S. W. (2007). Cost-efficacy of individual and Diffusion of an e-learning programme among Danish general combined treatments for panic disorder. The Journal of Clinical practitioners: A nation-wide prospective survey. BMC Family Psychiatry, 68(7), 1038–1044. Practice, 9(1), 24. 123 76 Adm Policy Ment Health (2011) 38:65–76 Wilkie, D. J., Judge, M. K. M., Berry, D. L., Dell, J., Zong, S., Woolf, S. H. (2008). The meaning of translational research and why it Gilespie, R., et al. (2003). Usability of a computerized PAIN- matters. Journal of the American Medical Association, 299(2), ReportIt in the general public with pain and people with cancer 211–213. pain. Journal of Pain and Symptom Management, 25(3), 213–224.

Journal

Administration and Policy in Mental HealthPubmed Central

Published: Oct 19, 2010

There are no references for this article.