Develop a Self-Evaluation Questionnaire for Evidence-Based Practice Education
1Department of Nursing, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
2Graduate Institute of Clinical Medical Sciences, College of Medicine, Chang Gung University, Taiyuan, Taiwan
3School of Nursing, College of Nursing, Taipei Medical University, Taipei, Taiwan
4Evidence-Based Knowledge Translation Center, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
5Center for Evidence-Based Medicine, Taipei Medical University, Taipei, Taiwan
6Evidence-Based Medicine Center, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
7Department of Physical Medicine and Rehabilitation, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
8School of Nursing, Chang Gung University, Taiyuan, Taiwan
9Department of Public Health, School of Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan
10Division of Plastic Surgery, Department of Surgery, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan
Background: Teaching Evidence-based practice (EBP) should be evaluated and guided by evidence of its own effectiveness. As educators implementing EBP training, they need instruments to evaluate the learning outcomes of trainees. For that reason, to develop a reliable and minimal time-consuming instrument is necessary. Purpose: The aim of this study was to examine the reliability and validity of an EBP evaluation instrument, the Taipei Evidence-Based Practice Questionnaire (TEBPQ). Methods: The content validity index (CVI) and Cronbach's αwere used to analyze the validity and reliability of the questionnaire. After that, a two parallel group (novice and experienced learner) study design for was designed to determine the construct validity of the instrument. Results: Taipei Evidence-Based Practice Questionnaire (TEBPQ) contained 26 self-report questions, which includes domain of ‘Ask’ (5 items), ‘Acquire’ (7 items), ‘Appraisal’ (4 items), ‘Apply’ (6 items) and ‘Attitude’ (4 items) of evidence-based practice, and learners’ basic characteristics. The overall CVI of TEBPQ was 0.9; while Cronbach's α was 0.87. The construct validity showed that all p values were significant (p<0.05) among 5 domains. Conclusions: The results indicated that the TEBPQ is an instrument with good validity and reliability for evaluating the effectiveness of EBP education.
At a glance: Figures
Keywords: evidence-based practice (EBP), continuing medical education, reliability, validity, instrument development
American Journal of Educational Research, 2014 2 (9),
Received July 07, 2014; Revised August 10, 2014; Accepted August 21, 2014Copyright © 2014 Science and Education Publishing. All Rights Reserved.
Cite this article:
- Chen, Kee-Hsin, et al. "Develop a Self-Evaluation Questionnaire for Evidence-Based Practice Education." American Journal of Educational Research 2.9 (2014): 740-744.
- Chen, K. , Tzeng, P. , Chen, T. , Kuo, K. N. , Liu, H. , & Chen, C. (. (2014). Develop a Self-Evaluation Questionnaire for Evidence-Based Practice Education. American Journal of Educational Research, 2(9), 740-744.
- Chen, Kee-Hsin, Pei-Chuan Tzeng, Tzu-Hsuan Chen, Ken N. Kuo, Hsueh-Erh Liu, and Chiehfeng (Cliff) Chen. "Develop a Self-Evaluation Questionnaire for Evidence-Based Practice Education." American Journal of Educational Research 2, no. 9 (2014): 740-744.
|Import into BibTeX||Import into EndNote||Import into RefMan||Import into RefWorks|
Evidence-based practice (EBP) is the integration of the best research evidence, patients' values and clinical circumstances in clinical decision making . Since the last decade, researchers and health care professionals have become increasingly aware of the importance of using EBP in clinical settings. As a new paradigm in health care’s decision making, health professionals have the obligation to access knowledge efficiently, apply properly, and lead others to use it appropriately.
Teaching EBP should be evaluated and guided by evidence of its own effectiveness . As educators implementing EBP training, they need instruments to evaluate the outcome and to document the self-efficacy of individual trainees.
Shaneyfelt et al. analyzed 104 currently used instruments for evaluation of teaching efficacy of evidence-based medicine with a systematic review. According to et al. the Fresno Test and Berlin Questionnaire were the only instruments that evaluate 4 EBP steps. However, the Fresno Test was time-consuming and required expertise to complete for demonstrating applied knowledge and skills of EBP tasks. On the other hand, the multiple-choice format of Berlin Questionnaire was more feasible to implement . Therefore, further development and testing is required to evaluate EBP attitudes, behaviors, and skills.
Straus pointed out that self-evaluation was very important in evaluating EBP teaching and facilitating clinical practice personnel for a spontaneous self-assessment . To develop an instrument with good validity and minimal time-consuming is necessary in clinical teaching and evaluation.
The aim of this study was to examine the reliability and validity of an EBP evaluation instrument, the Taipei Evidence-Based Practice Questionnaire (TEBPQ).
3. Materials and Methods3.1. Phase I: TEBPQ Set-up and Initial Validation
The purpose of the TEBPQ is to measure the self-efficacy on EBP for individual learner. Because few previous single studies had addressed all of the aspects we wanted to examine, the research team adopted the 4-step model of bedside EBM: (i) Ask: ask an answerable question; (ii) Acquire: track down the best evidence; (iii) Appraisal: critically appraise the evidence for validity, impact, and applicability; and (iv) Apply: integrate the results with the patient’s unique biology, circumstances, and values [3, 4] and the concept of evidence-based medicine evaluation  for measuring the EBP learning self-efficacy. In addition, attitude is an important factor which has a great influence on learning motivation and is related to the willingness to practice EBP clinically in the future . Therefore, the aspect of attitude was added in this tool. The TEBPQ includes 26 items which was selected from the item pool according to the questionnaire’s purpose (Appendix). The study has approval of Taipei Medical University institutional review board.
Cronbach's α, Content Validity Index (CVI) and construct validity were used for initial reliability and the validity test. First of all, we invited 2 experts of EBP methodology, 1 professor on education, 3 physicians, 2 nursing staff, and 1 librarian to evaluate the appropriateness and certainty of the questionnaire. Modifications were made to the wording of some items. As to the purpose of research, we investigated the correlation of questionnaire and scored each question individually. CVI value equal or above 0.8 is indicative of good expert validity . Second, we used Cronbach's α for testing internal reliability — the average of all possible split half correlations equal to or above 0.85 are acceptable results. The method described above was summarized as Table 1.
After the TEBPQ was created, we evaluated it in a representative sample and to construct the validity of the tool.
3.2.1. Study Design and Participants
A contrasted-group approach (two parallel group study design), including groups of novice and experienced learners, was designed to determine the construct validity of the instrument. Convenience sampling was performed in six teaching hospitals in Taiwan between 2007 and 2011. Health care professionals who were 20 years of age or older, interested in attending the workshop were included in the study. Participants who never attended or experienced any EBP lectures/courses before were ‘novice learners’. Individuals who were experienced an EBP workshop (at least 4 hours) or certified by the Taiwan Evidence-based Medicine Association (TEBMA) as an EBM trainer/tutor were ‘experienced learners’.
3.2.2. Intervention (EBP Workshop)
The feature of our EBP workshop is scenario-based discussions. Research showed that lecture-based teaching is not able to satisfy the needs of the new generation of students, in the contrary, workshop with group discussion has obvious learning effectiveness in evidence-based medicine teaching [7, 8, 9]. The workshops were designed by the Evidence-Based Practice Education and Research Committee in the Taipei Medical University and the affiliated hospitals. The committee consisted of seven members who participated in the course design, including experts of EBP methodology, university professors, clinical nurses, physicians, administrator, and researchers. All members had practical experience in teaching and leading discussions in the EBP workshop.
The content of the EBP workshop includes the introduction of EBP (50 minutes), hands-on practice and group discussion (90 minutes), and group report/course feedback (50 minutes). There were two experienced facilitators in each group, who can lead the discussion of EBP in order to help learners complete the following: (1) Ask: listing all possible clinical problems from the scenario, and identify the P (population), I (intervention), C (comparison) and O (outcomes); (2) Acquire: formation of PICO related keywords and performing a search in the evidence-based database (such as Cochrane Library or PubMed); (3) Appraisal: selecting proper articles in the topic, appraising them critically, and (4) Apply: applying in the clinical practice and answering the patient’s questions in plain language. The facilitators require at least 8 hours of experience in actual teaching or leading a group discussion about EBP. Before participating in the study, a course involving group discussion skills was arranged for the facilitators in order to achieve consistency in course instruction.
3.2.3. Evaluation & Statistical Analysis
We have done the power analysis by G power. A two parallel group study design would require an overall sample size of 56 participants per group to provide 80% power for a one-sided test at 5% significance.
SPSS version 17.0 was used for all statistical analyses. A value of p< 0.05 was considered statistically significant. Data analysis included the following:
1. Descriptive statistics were used to summarize participants’ characteristics. Mean and standard deviation (SD) were used to summarize continuous data. Categorical data were summarized using count and percentage (%).
2. Internal reliability: The reliability of the instrument was measured by Cronbach’s α, while a value greater than 0.85 represented a tool with internal consistency.
3. Content validity: We used the content validity index (CVI) to test the validity of the questionnaire. Likert 5 scale was used to evaluate the appropriateness and certainty: 1 is “strongly disagree” that the description of the question is very uncertain and the question should be deleted; 2 is “disagree” that most of the questions should be modified but are worth keeping; 3 is “no opinion” and weakly agree to the question; 4 is “agree” that the question requires little modification; 5 is “strongly agree” that the question is very appropriate and does not require any modification.
4. Construct validity: Mean scores of two parallel groups (novice and experienced learner) were compared using the independent t-test.
4. Results4.1. Phase I: TEBPQ Set-up and Initial Validation
There are 26 self-report questions in TEBPQ (Appendix): “Ask (PICO) (5 items)”, “Acquire (7 items)”, “Appraisal (4 items)”, “Apply (6 items)” and “Attitude (4 items)”. All respondents were asked to rate the questions on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). The total CVI of TEBPQ was 0.9; while Cronbach's αvalue was 0.87.4.2. Phase II: TEBPQ Evaluation
4.2.1. Participants’ Characteristics
A total of 145 participants applied to participate in this study. Among them, 3 individuals felt too much pressure from the training program and consequently failed to complete the course. Three people withdrew because of family reasons. Three cases had missing data. Therefore, there were 136 participants (completion rate of 86%) answering the questionnaire, including 17 physicians (12.5%), 97 nurses (71.3%), 6 pharmacologists (4.4%), 12 librarians (8.8%) and 4 administrative personnel (2.9%).
4.2.2. Construct Validity (Contrasted-group approach)
The results for each domain are shown in Table 2 and Figure 1. The highest mean score of novice learner is ‘attitude’ domain (mean: 3.6, SD ± 0.85) and the lowest mean score is ‘appraisal’ domain (mean: 2.1, SD ± 0.89). Figure 1 shows the entire elevation of scores in experienced learners of the five domains. According to inter quartile range (IQR), the peak of the experienced learners is higher and smaller than that of the novice learners. The p-value of each domain is below than 0.001. It indicates that TEBPQ instrument can differentiate the novice and experienced learners effectively.
Table 2. Mean scores of TEBPQ between novice and experienced participants. (n=136; Novice=68, Experienced=68)
We developed a new self-report questionnaire in measuring the effectiveness of evidence-based practice education. The overall content validity index of TEBPQ was 0.9; while Cronbach's α value was 0.87. A Contrasted-group approach for construct validity showed that all p values were significant (p<0.05) among 5 domains. All these results indicated that this questionnaire is an instrument with good validity and reliability. In reviewing all the current evaluation instruments available, most of them (such as the Fresno and Berlin questionnaires) were time-consuming and not feasible to apply in different specialties. In our study, all questionnaires were finished and collected 10 minutes after the end of the each workshop. In addition, the assessors of Fresno and Berlin require special training before grading which may hinder the wider application of them. Comparing to the Fresno and Berlin assessment tools, the TEBPQ only consume 10 minutes to complete. Overall, the TEBPQ is time saving, easy to use and score. In the reality of busy clinical services, it is impractical and costly to develop courses regarding EBP for different specialties. More general courses can be a valuable starting point for preparing staff for EBP. The components of our study are quite similar to those used by employees in hospitals. Therefore, the result is quite practical in the running of continuing medical education in hospitals.
In this study, we found the similar trend about evidence-based practice education in 5 domains between novice and experienced learners. Initially, the highest mean score of TEBPQ is ‘Attitude’ and lowest one is ‘Appraisal’ among all participants. This is possibly because the participants in this study were volunteers, who believed that clinical work required this competence and were motivated to learn. On the other hand, the ‘Appraisal’ domain scored the lowest because of learners in this study mentioned that they didn’t trained how to ‘Appraisal’ very well previously. We found that the mean score for each domain could be improved (above 3 point) after proper evidence-based practice education. The least degree of improvement in attitude may be a result of the ceiling effect.
However, some limitations were inherent in this study. Firstly, the TEBPQ is a self-report questionnaire which is a subjective form of assessment, with participants often factoring other variables that they perceive may have influenced their performance, thereby skewing the actual performance and outcome (i.erecall bias).The results of TEBPQ compare with other objective evidence-based practice evaluation instrument in the future are needed. Secondly, this study was conducted at 6 hospitals in Taiwan, which is a limited geographic region. Thus, researchers who would like to adopt this instrument to their own study should be carefully. Finally, the long-term effects of learners’ ability to apply EBP in their daily practice at various work settings as well as patient outcomes are difficult to evaluate via TEBPQ.
In this study, we found that TEBPQ is an easy and time-saving instrument with good reliability and validity in measuring the effectiveness of multidisciplinary evidence-based practice education.
Funding: This study was funded with National Science Council for sponsoring the research expenses. Key research project No: NSC 100-2511-S-038-001.
Statement of Competing Interests
The authors have no conflicts of interest to disclose.
|||Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W.and Haynes, R.B. Evidence-Based Medicine: How to Practice and Teach EBM. Churchill Livingstone, Edinburgh &New York, 2000.|
|||Shaneyfelt, T., Baum, K.D., Bell, D., Feldstein, D., Houston, T.K., Kaatz, S., Whelan, C.and Green, M. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA, 296, 1116-1127, 2006.|
|||Straus, S.E. and McAlister, F.A. Evidence-based medicine: a commentary on common criticisms. CMAJ, 163, 837-841, 2000.|
|||Straus, S.E., Richardson, W.S., Glasziou, P., and Haynes, R.B. Evidence-based medicine: How to practice and teach EBM.Churchill Livingstone,United States of America, 2005.|
|||Aarons, G.A. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am, 14, 255-271, 2005.|
|||Waltz, C.F., Strickland, O.L. and Lenz, E.R.Reliability and Validity of Norm-referenced Measures. F. A. Davis Company, Philadelphia, 1991.|
|||Feldstein, D.A., Maenner, M.J., Srisurichan, R., Roach, M.A. and Vogelman, B.S, Evidence-based medicine training during residency: a randomized controlled trial of efficacy. BMC Med Educ, 10, 59,2010.|
|||Schreiber, J., Stern, P., Marchetti, G. and Provident, I. Strategies to promote evidence-based practice in pediatric physical therapy: a formative evaluation pilot project. Phys Ther, 89, 918-933, 2009.|
|||Wu, M.S., Chen, C., Tzeng, P.C., Lien, G.S., Wen, M.L. and Chiu, W.T, Group Emulation Improves the Teaching Efficasy in Evidence-Based Medicine. Formosan JMed, 11, 425-430, 2007.|