Research Project 3. Applying RCT's to Comparing Effects of Communication Interventions on Use of "End of Grant" Knowledge Innovations by Stakeholders

Research Project 3 implements and evaluates KT interventions designed to promote use of NIDRR technology grantee knowledge (K) outputs selected as “innovations”.. Each innovation is an “end-of-grant” output (K) selected randomly (in a previous step in Research Project 2), from a pool of innovations generated by the grantee, classified by an expert panel as innovative and useful to multiple user categories.
Effects of the designed interventions on potential users are investigated through randomized controlled studies. Three randomized controlled studies covering three technology areas addressed by the Center on KT for TT are scheduled for the years 2, 3 and 4 of the project cycle. One intervention is implemented per year, focused on K from an RERC (Rehabilitation Engineering Research Center), and six categories of knowledge users (KUs)are targeted: other researchers (other than researchers who produced the chosen innovation), clinicians/ practitioners, manufacturers/industry, policy makers/implementers, brokers and consumers.
The KT intervention approach involves a tailored and targeted dissemination of K (TTDK), where the selected K is tailored to the specific contexts of pre-identified user audiences and then delivered to them through multiple channels. By selecting appropriate channels in a sequence that informs participants, demonstrates/ trains them, and assists them in the use of K we expect to achieve awareness, interest and use in the participants. Operationally speaking, the TTDK involves:

Developing six tailored information packages - Contextualized Knowledge Packages (CKP)s - which present the selected innovation K (the original published article) tailored to six specific user contexts through additional information and support material needed for its effective use in their context; and
Developing six tailored webcasts focused on demonstrating the use of the selected K by specific users in their specific contexts.
Providing technical assistance to participants who advance to the use level, when requested.

The overall model for implementing and evaluating the TTDK intervention follows the Knowledge to Action (KTA) Model (Logan and Graham, 1998), with the project team playing a facilitator role to promote knowledge use, where the TTDK approach is a facilitating vehicle (Carrier) to accomplish the uptake and utilization of the NIDRR grantee knowledge by specific KU groups. Further, the implementation design compares TTDK with a targeted dissemination of K (TDK) approach, both approaches compared against a control group where participants are assumed to be pre-exposed to the K by traditional dissemination/diffusion methods.

Purpose

The purpose of Research Project 3 is to study the level of utilization of NIDRR grantee’s K outputs by the knowledge user system (KUS) over time, with and without the TTDK (tailoring and targeting) or the TDK (targeting only) approaches, for given levels of K innovativeness and the given parameters for the KU contexts as identified by Research Project 2.

Scope

The coverage of the randomized controlled studies is limited to the pool of K outputs from participating NIDRR grantees. Further, they address only outputs occurring at the “end of their grant”, as opposed to the K outputs generated at the start of the grant (“integrated KT”). Thus they track the progression of outputs from the end of grant stage through the KTA’s Action Cycle, used for intervention implementation.

Design

Research Project 3 procedures represent a “Scientifically Based Systematic Evaluation” (USDE, 2003) of a KT intervention. The systematic nature of the evaluation is inherent in the KTA model that it follows for implementation of intervention, which considerably overlaps with the rationale of “systematic evaluation” proposed in the Context, Input, Process, and Product (CIPP) framework (Stufflebeam, 2001, 2004, 2005; Stufflebeam and Shinkfield, 2007). Choice of the randomized controlled trial (RCT) as its research design for evaluative conclusions lends the “scientific basis” for the project. Utilizing these procedures, Research Project 3 studies the differential effects of the TTDK intervention on targeted users as compared to the TDK approach and to no intervention (control group). Both the TTDK and the TDK approaches target K user audiences and deliver the K in original format (article) by directly communicating with them; the difference however is that additionally, TTDK tailors the chosen K to the user context, and delivers it via written information package (also called contextualized knowledge package – CKP; Stone, 2010), webcast and technical assistance channels. Participants from each of six earlier mentioned KU categories are randomly assigned to the two treatment groups and a control group (Fig R3-1) The project tracks the level of innovation utilization among all three groups of knowledge users, by measuring the level of use of the knowledge through a pretest (baseline) and two post tests (follow ups) spread over 8 months for each participant. The differences in K use among the three groups are analyzed, for effects attributable to TTDK and to TDK regarding the use of K innovation. A level of knowledge use survey (LOKUS), specially developed for Research Project 3 is used for measuring knowledge use. This instrument was initially inspired by the Levels of Use concept by Hall et al. (1975, 2006) although with structural modifications to suit the needs and constraints of the project context including online administration (Stone, 2010). It was later substantially modified in its conceptual framework through psychometric analyses (Tomita, et al, forthcoming).

Figure R3-1 summarizes the above design.

Figure R3-1. Research Design for CKP-based KT Interventions

	Baseline Assessment	Intervention Delivery	Post Test 1	Intervention Delivery	Post Test 2
R T₁	O_{(1 – 6)}	X₁	O _{(1 – 6)}	X₁	O_{(1 – 6)}
R T₂	O_{(1 – 6)}	X₂	O_{(1 – 6)}		O_{(1 – 6)}
R C	O_{(1 – 6)}				O_{(1 – 6)}

Where T₁ and T₂ are the TTDK and TDK treatment groups; C is the control group; O_(1-6) represents Pre and Post tests administered on six types of stakeholder audiences (K users) that are targeted. X₁ and X₂ are the TTDK and TDK intervention activities, respectively. X₁ includes tailoring and delivery of K via multi-channels (written material, training, and technical assistance) as unique to the chosen KT strategy; whereas X₂ involves only the direct dissemination mode of delivering K to the six stakeholder audiences targeted. The control group does not receive any information about the innovation.

Research Questions:

1. Do the innovation users who were provided with TTDK intervention reach more advanced level of K use than the TDK and the control groups?
2. Do the innovation users who were provided with the innovation information through TDK reach more advanced level of K use than the control group?
3. Are there differences in the level of K use among six types of users that received the TTDK intervention?
4. Do individuals who reach more advanced level of K use have demographic characteristics as well as K use process traits different from the individuals who do not
reach advanced levels?

Expected Outcomes:
Findings from Research Project 3 about the effectiveness of the designed KT strategy (TTDK) should:

Enlighten knowledge creators (participant NIDRR grantees) about the use of KT best practices in enhancing the probability of utilization of their innovative outputs. It should encourage them to add value to their innovations through such practices. (short term)
Replication of the KTA model through the three interventions (RCTs) over 3 years should lead to validation of the model as an effective approach to translating end-of-grant innovations for specific user contexts. (short term/ midterm)
Awareness and extent of use of the innovations by the different user categories should reinforce the adoption of innovations by the users and lead to evidence-based practice (longer term)

Research Project 3 Project Status:

Research Project 3 implemented its first KT intervention study in the field of Augmentative and Alternative Communication (AAC) technology, which is just completed. Preparation for this RCT study implementation took place in Year One (Oct 08 –Oct 09), when operational details including design and structure of the TTDK and TDK components were defined. Construction of LOKUS, the measure of K use was initiated.
The study started in Oct 09. Based on expert panel evaluation, the research knowledge by Dr. Diane Bryen at Temple University on Age Appropriate Vocabulary and Symbol Sets (Bryen, 2004) was selected as the target innovation for the RCT study.
The implementation of the KTA model was initiated by “value-mapping” (Lane and Rogers, 2011, accepted) organizations of affiliation of the relevant stakeholders – brokers, clinicians, manufacturers, policymakers, researchers and consumers with complex communication needs. Results on organizational values, needs and expectations about knowledge from research guided the development of the tailored information packages (CKPs) and the tailored webcasts for the different stakeholder groups. Dr. Bryen provided active input into the improvement of the foregoing material. Each CKP was pilot tested by a respective stakeholder expert and revised. Webcasts were created by producing a central video where Bryen demonstrated the use of the vocabulary lists by an AAC user, and then appending narratives tailored to each stakeholder type.
Level of Knowledge Survey (LOKUS) was constructed, content validated, adapted for online administration through the Vovici software, pilot tested and revised. It was then subject to further psychometric analyses. (Telang, Tomita and Stone, RESNA/FICDATT 2011).
Following the University’s Institutional Review Board approval, participant recruitment was done through the active help of the organizations from January 2010 through October 2010. The actual RCT study implementation started with baseline testing using LOKUS in July 2010, after random assignment of participants into the TTDK, TDK and Control groups. Each group answered LOKUS again at 4 months and then at 8 months. The TTDK group received the CKP via land mail and electronic mail 2 weeks after the baseline testing, while the TDK group received Bryen’s journal article by land mail. The TTDK group was sent password protected links to the webcast a month after they answered the Post test 1. The TDK group did not receive anything during this period. The control group received neither CKP nor the webcast link in the study. Of a total of 251 representing the five different stakeholder types, 207 completed the study. The study came to an end in July 2011 when the last of the participants completed the 8 month study period. Data analysis is complete and results are compiled for sharing through the SEDL/KT4TT webcast on September 22. Stay tuned!!

The second RCT study in Environmental Access technology is now in implementation. The innovation has been chosen, organizations have been value mapped and recruitment has started. Instruments and input material are in construction for pilot testing. We expect to launch the baseline testing by October 2011. Stay tuned for further updates.

References:

Bryen D. N. (2008). Vocabulary to Support Socially-Valued Adult Roles. Augmentative and alternative Communication, Vol. 24, No. 4, Pages 294-301.
Hall, G.E., Loucks, S.F., Ruttherford, W.L., and Newlove, B.W. (1975). Levels of use of the innovation: A Framework for analyzing innovation adoption, Journal of Teacher Education, 26, 52-56.
Logan and Graham, 1998. Toward a comprehensive interdisciplinary model of health care research use. Science Communication, 20, 227-246.
USDE. Scientifically Based Evaluation Methods. Notice of proposed priority. FR Doc 03-27699. Federal Register: November 4, 2003 (Volume 68, Number 213). [Notices] [Page 62445-62447] From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr04no03-38].
Stufflebeam, D.L. and Shinkfield, A.J. (2007). Evaluation Theory, Models, & Applications. San Francisco, CA: Jossey-Bass.

| | |