Assessing the Impact on Medical Students of an Online Curriculum on Opioid Dependence and Treatment

Mary P. Metcalf, PhD, MPH; T. Bradley Tanner, MD; Susan Wilhelm, PhD

Clinical Tools, Inc. 1506 E. Franklin St, #200, Chapel Hill, NC

Objectives: To develop and assess the impact of an online supplemental curriculum for medical students on various medical knowledge/competency, attitude, and performance measures.

Background: Treatment with buprenorphine is effective and can be conducted in a variety of settings by primary care providers. However, a number of biases exist towards treatment of opioid dependence in primary care. Medical school is an opportune time to addresses -and hopefully decrease – these biases with future physicians.

Methods: An online curriculum of 5 modules for medical students was created, focusing on basic skills of opioid abuse detection and treatment. A convenience sample of 24 volunteer medical students (n=24) at different institutions in years 2-4 completed a OSCE style interview of an SP via chat, completed the educational program, and then re-interviewed the SP. Measures assessed impact on medical knowledge, competency, attitude, and self-efficacy/intended behavior. Clinical skills were assessed using a Communication/Interpersonal Skills rating form and a Clinical Encounter Performance Checklist completed by the SP and an observer.

Results: Medical knowledge/competency increased from a pre-experience average of 68% to an post-test average of 83% ( p<0.05). Average self-efficacy scores increased (p<0.05) and Clinical Encounter Performance Checklist score increased for 84% of participants (p<0.05). Two-thirds of students improved Interpersonal Skills/Communication checklist scores, although the improvement was not statistically significant for this measure. Additional analysis are in process.

Conclusion: Medical student knowledge, competency, self-efficacy and performance were shown to be improved by the educational intervention. Attitude towards treatment of opioid dependence and interpersonal skills related to clinical encounters with possibly substance abusing patients did not show the improvement we had hoped. The intervention is being modified to address these issues more directly.

Background: With support from NIDA (#R44-DA12066) we have developed an online curriculum to educate and further clinical skill development in Year 2 to 4 medical students on buprenorphine treatment for opioid use disorders.

Objective: To educate medical students about treatment of patients with opioid dependence using buprenorphine and impact clinical skills in working with patients with addictive disorders.

Program Description: We are performing a summative evaluation of our curriculum. Each medical student completed a pre/post test effectiveness evaluation of our online curriculum, including pre/post evaluations of clinical skills through a remote live standardized patient (SP) experience. The OSCE style evaluation of the learner's performance during the remote, live SP clinical encounter was assessed by the SP actor and a MA level clinical professional. Measurement was done via: 1) a 9 item Likert scale rating sheet, and 2) a 16 item done/not done checklist. Both modifications were reviewed and approved by two external clinical substance abuse professionals.

Program Evaluation Outcome: Preliminary results (n=25) of this evaluation indicate that the modules have a statistically significantly effect on knowledge and intended behavior related to treatment of patients with opioid dependence (p<.001). Average pre-knowledge score was 68% while the average post-test score was 83% (p<0.001). Attitude and self-efficacy also increased significantly for most modules. Students showed the most improvement in the areas of common comorbidities and pharmacology. Student competency was measured in the following areas: drug detection and screening, comorbid conditions, diagnosis and treatment. Competency (p<0.0007) increased quite significantly from pre to post experience. Competency scores increased an average of 44% from pre to post-interview.

Conclusions: Based on the preliminary results, this curriculum is an effective tool for educating students on buprenorphine and opioid use disorder. The addition of a standardized patient component pre and post curriculum allows the students an opportunity to better demonstrate an increase in competencies and clinical skill in a more true to life situation.

Idea: Medical educators interested in online learning have confidence in their ability to effect knowledge and the tools exist to demonstrate success. Yet, online learning is limited by the inability to show that the learning experience effects higher levels in Bloom's Taxonomy. How do we understand if an online course is succeeding beyond its knowledge improvement goal?

Why the idea was necessary: Students took 5 online courses on the topic of addiction that were broken down into Prevalence, Detection and Diagnosis, Comorbidities, PCP Role, and Pharmacology. The courses demonstrated significant changes in knowledge in a very small sample (N=10) with p ranging from .033 to .001 and average knowledge scores improving from 14% to 52%. Self-efficacy too showed statistically significant improvement (p<.001) even with the very small sample. The students rated themselves as being able to effectively screen and detect opioid abuse following the learning experience. There was not enough of a difference in scores to detect a difference in attitudes and intended behavior. Unfortunately the standardized patient interview showed no change in clinical skills.

What was done: This project utilized a remote standardized patient (SP) experience to test, examine, and improve the learning. We planned to use the ratings of the SP experience to identify the weaknesses in student performance and modify the online learning to address them. After the first round of students completed the study, we looked at the data from both the Modified Skills Inventory (a subjective rating tool completed only by the standardized patient) and the Evaluator Rating Sheet (an objective checklist completed by both the standardized patient and an observer).

Evaluation: Results from the Modified Skills Inventory showed general improvement for all participants post-experience. However, the Evaluator Rating Sheet had some problematic areas; clinical areas which were expected to improve post-experience did not improve. Specifically, students consistently failed to screen the standardized patient using a standardized measure, such as the CAGE-AID. Mental status exams were also rarely conducted, and students showed difficulty in screening patients for psychiatric co-morbid conditions. We evaluated our existing curriculum, and found weaknesses in the level of detail paid to these areas. Pages covering these content areas were expanded to further illustrate these teaching points.

Conclusions: A remote SP experience is a useful tool in the development of an online educational intervention. It can ensure that the student is truly mastering skills and integrating learning into their clinical experience framework.

Clinical skills assessment exams using Standardized patients (SPs) are an established component of medical school education, including the current USLME Step 2 CS exam™. The authors have developed a novel method for conducting SP encounters using web-based chat technology (e.g. Google® chat). “Remote Live Standardized Patient” (RLSP) interviews are conducted entirely online. The interview requires active learner participation focused on clinical skills, and represents a unique solution to teaching specific skills and affecting behavior. We are not aware of other educators using chat for this purpose. We have been pilot testing chat-based OSCE-style exams1,2 since 2007. In 2009 we have begun using RLSPs to teach skills, as well as evaluate them. The instructional RLSP key components include an interactive SP actor, an EMR like interface, and a “Hats Off” mode, which allows the student learner to “ask a preceptor” or research information during the encounter. Each learner’s performance is evaluated by standardized measures and by the RLSP actor feedback. This interactive, responsive web environment allows the RLSP interview to simulate a broad spectrum of learner/patient encounters, from initial screening through diagnosis and treatment.

Advantages include 1) decreased SP training cost and time commitment, 2) more flexible SP scheduling and recruiting since all encounters occur online, 3) elimination of face-to-face constraints of verisimilitude errors and correlation of actor physical appearance compared to the “patient”, and 4) instant availability of chat conversation transcripts for review and learner feedback. Potential limitations include missing the “whole person” experience afforded by face-to-face encounters, including the ability to assess body language and demonstrate physical exam skills. Since face-to-face SP encounters also suffer from realism limitations (i.e. the actor does not typically have the “patient's” presenting symptoms), we feel that the advantages of an RLSP balance and actually outweigh the drawbacks.

Use of web-based chat or its equivalent may have significant appeal to the current generation of Internet-savvy medical students. The RLSP experience appears to be a viable alternative to the traditional face-to-face SP interview, allows cost savings, and prepares students for Step 2 CS-style exams.

Medical students need to understand the role of practicing physicians in substance abuse treatment. In addition they may not be aware of complicated multi-disciplinary treatment issues, the biological and social basis of addictions, and how new treatments are developed then moved into practice. The topic of detection and treatment of opioid addiction is an excellent one to teach these basic skills and convey essential knowledge. By teaching medical students we can impact future physicians before potential negative attitudes regarding substance dependence have developed or solidified.

With research funding from NIDA/NIH, we are developing outlines for a suite of online educational modules addressing these issues. Curriculum development included medical school professionals and substance abuse experts. The prototype module, "Prevalence, Trends and Impact on Health: Opioid Addiction" was pilot tested with 22 medical students in their 2nd through 4th years of study. Pre/post-assessments of knowledge, attitudes, self-efficacy, and intended behavior showed significant increases following completion of the module. Satisfaction with the module was high and there was an interest in learning more about substance abuse. During the coming year (2008), we will perform a large-scale evaluation of the project using an innovative virtual standardized patient. As the need for physicians to recognize and treat addiction disorders grows, these online modules hold promise for integrating the necessary knowledge, skills, and attitudes into the medical school curriculum.

Opioid dependence and treatment with buprenorphine are topics well suited to teach medical students about the role of practicing physicians in substance abuse treatment, complicated multi-disciplinary treatment issues, complex biological and social basis of addictions, and how new treatments are developed and moved into practice. During Phase I of the grant, we developed a curriculum plan for a suite of modules on these issues as they relate to opioid abuse and dependence with input from a focus group of medical school professionals with experience in substance abuse education. The prototype module entitled “Prevalence, Trends and Impact on Health: Opioid Addiction” was developed and underwent usability and pilot testing with medical students in their 2nd through 4th years of study. Usability testing revealed that students enjoyed the online format, found the module well-organized and easy to navigate, and showed an interest in learning more about Buprenorphine. More in-depth pilot testing assessed the impact of the pilot module on the attitudes, self-efficacy, and intendedbehavior of medical students, as well as changes in knowledge and satisfaction with the module material. A retrospective pre/post design was regarding detecting and treating opioid misuse or abuse significantly increased with the completion of the pilot module.

Effective continuing medical education (CME) about substance use disorders and treatment can improve physicians' knowledge of the topic and confidence in intervening, which may lead to positive outcomes for patients. With funding from the NIDA and the NIAAA, we have developed and evaluated two Internet-based substance abuse CME programs: one program featuring 12 courses about opioid dependence and buprenorphine treatment (BuprenorphineCME.com), and the other featuring five courses about alcohol abuse and treatment (AlcoholCME.com). Approximately 2000 participants viewed online courses over the past two years and completed a knowledge test, a satisfaction survey, and a self-efficacy, attitude, and intended behavior questionnaire. Preliminary analysis of this data indicates that the educational experience has a statistically significant impact on self-efficacy, attitude, and intended behavior. Also, results show that the CME courses improve knowledge of the topic and that satisfaction with the courses is consistently high (as of 9/15/05, the overall mean satisfaction rating for courses on both sites was 4.5/5.0). An additional upcoming analysis will examine the impact of each entire program on the outcome measures; we will report these findings during the conference presentation. This type of comprehensive evaluation is novel in the world of online CME. We are optimistic that this research will yield useful information about the most effective approach for educating health professionals about substance abuse.