The assessment of clinical competence in medical education has undergone changes over time, leading to new assessment modalities that are of better quality than previous ones. The expected characteristics of any good assessment are objectivity, validity and reliability. A good assessment is also one that ensures pedagogical alignment with the learning objectives. Therefore, it is important to apply assessment principles and methods that have high psychometric properties. The Objective Structured Clinical Examination (OSCE) is one such current assessment method that has gained credibility thanks to its higher quality compared to the traditional clinical examination, especially by reproducing an authentic situation. The university hospital teaching staff in Conservative Dentistry and Endodontics at the Faculty of Dentistry of Casablanca-Morocco tested this method of OSCE evaluation and, in view of the results, embarked on a process of reform by evaluating clinical skills through OSCEs.
The completion of the tertiary training in Conservative Dentistry and Endodontics (CDE) at the Faculty of Dentistry of Casablanca-Morocco leads to the issuance of a National Diploma of Dental Specialty (NDDS). During their 4-year training, the CDE residents will have to acquire, in addition to the fundamental theoretical knowledge, the competencies of data collection, communication, reasoning and problem solving, clinical, diagnostic and therapeutic examination specific to patients suffering from carious and endodontic pathologies and/or consulting for direct restorations.
Regarding technical knowledge, direct and indirect supervision by the teachers as well as the follow-up with the resident teachers' portfolios allow the validation of the foundation phase of the training. However, the clinical evaluation of resident teachers at the end of the specialist training is still done by the presentation of a clinical case chosen at random on the day of the examination (or a few days before) by a member of the jury. Candidates do not necessarily examine the same patient on the day of the examination, and are faced with situations of varying complexity, which may strongly bias the assessment.
Assessing the residents' clinical skills with the Objective Structured Clinical Examination method (OSCE) allows 1:
- Direct observation of clinical skills.
- Assessment of a wide range of skills in a relatively short period of time.
- A more accurate assessment based on a standardized method.
- Minimization of assessment bias using multiple examiners.
In this article, one will detail the steps involved in implementing OSCE in CDE, as well as the feedback from the first OSCE formative assessment.
Evaluation has an inescapable place in the teaching process. It is an integral part of the educational process. There are three main reasons for this 2:
1- To obtain information to make decisions about a training program.
2- To generate feedback through this evaluation.
3- The third reason is promotional.
Indeed, the evaluation will be used to make decisions. M. Nadeau defines evaluation as a value judgment made on a measure to make a decision 3.
This methodological approach applies perfectly to the evaluation of clinical competence in health, which is gaining considerable momentum and has developed rapidly among health professionals in recent years.
There are three types of evaluation: diagnostic evaluation, formative evaluation and summative or certificate evaluation 4:
Diagnostic evaluation is used at the beginning of the training course. Its purpose may be to select students (for a competitive examination, for example), or to measure the level of knowledge of students at the beginning of the course in order to adapt the content.
Formative evaluation supports the learning process by informing the student about his or her strengths and weaknesses (feedback, debriefing), and helps the learner to get closer to the objectives of the training proposed by the teacher.
Summative evaluation, provide a package of results used to assess whether a program works or not. In health, we cannot certify a simple performance, we certify a complex cognitive functioning.
Williams et al. 5 in the "Guide to Dental Student Assessment" describe assessment tools applicable in dentistry and related to the different levels of Miller's learning pyramid 6. They classify the assessment tools as follows:
• Miller Pyramid "Knows" and "Knows How" Level
• Miller's Pyramid "Shows How" Level
• "Does" level of Miller's pyramid
The psychometric attraction of the OSCE is that it assesses the “shows how” level of the Miller’s Pyramid with reliability and validity 7. The OSCE assesses that a candidate is able to ‘show how’ one would perform in a simulated environment. The simulated environment itself can influence the performance of candidates; therefore, the performance of the candidates in the OSCE might not be the same as their performance in the workplace on identical tasks 8.
Implementation of OSCE in Conservative Dentistry and Endodontics:
Background: NDDS training in Conservative Dentistry and Endodontics:
During the 4-year training in Conservative Dentistry and Endodontics (CDE), resident teachers will be required to acquire theoretical knowledge and practical healthcare. In terms of theoretical knowledge, face-to-face and distance courses as well as oral presentations, they are covered by the teachers of the same department. With the help of the portfolio, the resident collects the records of his or her learning during the course. A tutor teacher is responsible for a resident with whom he/she organizes regular meetings to monitor his/her professional project and its evolution until the last semester. The portfolio will be graded and validated by the tutor and the teachers. Combining the OSCE with another assessment format will minimize practical difficulties and produce an acceptable overall reliability for the clinical competency examination 9.
The evaluation of resident teachers at the end of the internship includes until this date:
- An evaluation of the end-of-specialty thesis. This is a research topic conducted during the specialty period with a team of teachers. The results of the study are presented to a jury (internal and external members).
- An evaluation of the most relevant clinical cases received by CDE during the specialty training period and in collaboration with other specialties and,
- The exam itself takes place over 2 days. After a 10-minute presentation of the candidates' papers, the candidates take:
- A written evaluation on the first day: consisting of basic questions asked in the morning and therapeutic questions in the afternoon. The second day is an assessment of clinical skills: a patient is randomly chosen by the examination board; this patient is entrusted to the candidate who examines him/her and then prepares a clinical observation in 30 minutes with a diagnosis and a treatment plan. The candidate then presents the case in 15 minutes to the jury members.
The limitations of this mode of assessment include: lack of fairness; difficulty in assessing skills such as questioning; no direct observation; no standardization; lack of accuracy and content validity.
In view of this observation and to overcome the disadvantages of conventional clinical examinations, the CDE team at the Faculty of Dentistry of Casablanca decided to develop CDE-adapted OSCEs that will help and allow thorough assessment of the clinical skills of the resident in front of a patient.
In the following section, an assessment through Objective Structured Clinical Examination (OSCE) for the resident end-of-specialty examination is planned.
The OSCE implementation project in CDE is being monitored by a committee of 9 university hospital teachers from the Faculty of Dentistry. This committee meets once a month beginning in March 2022 to monitor progress. The first formative OSCE has been scheduled for July 2022.
At first, a fairly exhaustive set of competencies was defined and, after consultation and revision of the educational objectives, a fairly limited list of competencies was selected for assessment.
Thus, eight items were identified. For each item, the competencies to be assessed were defined by an OSCE circuit (Figure 1).
- 8 stations including 1 procedural.
- 7 clinical cases forming as many stations as possible of 7 minutes each.
- 1 minute for the transfer.
OSCE Circuit in CDE:
A selection of topics was chosen based on the following criteria:
Questions from the list of predefined competencies to be assessed.
Reasonable level of expertise that could be acquired during the training course.
Clarity of instructions and scenario for the simulated patient.
These criteria were provided to all potential question writers. An example of a sample question that met these criteria was presented and then shared with all writers.
Station 1: Interview and clinical examination.
Station 2: Dental trauma (Corono-radicular fracture).
Station 3: Radiographic interpretation.
Station 4: Patients at risk (patient management between 2 chemotherapy sessions).
Station 5: Diagnosis of root resorptions in endodontics.
Station 6: Management of root resorptions in endodontics.
Station 7: Management of infectious/inflammatory outbreaks (Prescription).
Station 8: With procedural materials (Vertical hot filling).
The first 7 stations use simulated patients, the 8th is a procedural station without simulated patients.
For practical purposes decisions around test length generally need to balance reliability coefficients with feasibility and resource issues 10. Eight OSCEs stations have been planned in the CDE department.
For each station, all the necessary equipment has been identified and will be assigned, all functional for the day of the exam.
Pedagogical design part:
The learning targets (2 maximum) were defined. For the formative OSCE we chose the interview and the clinical examination as learning targets. The theme of the clinical case was a postoperative complication of a direct composite filling. The diagnosis was defined in the instructions. The bibliographical references for the design of this case were noted in this part of the instructional design. A summary of the clinical case in few lines was also written. Then, the chronological course of the case was also detailed in this part.
Technical part:
The expected duration of the OSCE is 7min. One minute will be devoted to preparation in front of the station. A standardized patient has been recruited for the formative OSCE station. Two examiners will be evaluating for this station. The examination room (within the CDE department) was fully equipped for the station. Additional radiographic examinations were performed in advance to be provided as needed.
Execution of the formative ECOS:
Phase I: Exam Preparation:
Meeting with the learners: Briefing
Briefing participants on the conduct of the OSCEs, the role of the simulated patient, and what is expected of them, helps to mentally prepare them to perceive the clinical cases in a real way, and to adhere to the role they are assigned. It is a guided discussion that helps students make the connection between events and their actions, and thus encourages reflection so that they can learn from the experiment.
As the OSCE is not yet concretely implemented in our institution, a presentation on the organization of the OSCE has been scheduled for the resident teachers one week before the formative OSCE. This was a preparation phase to explain how the exam would be conducted. During this meeting, we described the general functioning of the OSCE for all participants, as well as the role of each person, the expected behaviors, communication and respect for timing.
The briefing took place in the presence of the head of department. A concrete example of OSCE in CDE was presented in order to trigger motivation without fearing exposure to a situation of failure of the residents in front of the assessor.
The list of materials and the instructions for the procedure were communicated to the participants the same day.
The Clinical case and evaluation grids:
The learning targets, clinical case topic, mode of encounter (2nd patient visit), and patient profile were identified. A description of the OSCE process was written.
The role of the simulated patient was performed by a standardized patient. Training of the standardized patient began a few days before the examination. The observation grids were established following the guide written by Harden 11.
The grids include items assessing the resident's clinical skills and behavior. The distribution of items assessed is intended to reflect the usual activity of an endodontist. Clinical examination and therapeutic management strategies were emphasized; there is no evaluation of technical skills. The grids have been validated by expert and specialized university teachers.
Assessment blueprinting with grids in an OSCE is an efficient method for helping with the test construction process 12. It will provide validity evidence for generalization to other test settings 13.
Checking, revision and testing:
To be conducted properly, clinical case study is a process that requires time and effort. It involves careful consideration of the case, checking and reviewing.
Feedback from colleagues in the review phase proved to be critical to improving the quality of the case. The scenario was rehearsed with two teachers to determine if there were any elements that needed to be corrected or completed.
Development of a standardized patient station:
The station was built based on the clinical competencies identified for training in conservative dentistry. To master the competency, the resident must achieve a series of objectives.
The clinical situation is written by a team of 2 teachers responsible for the station. The whole development process of the station has been checked and controlled:
- The adequacy of the chosen situation and its evaluation criteria with the clinical skills and objectives to be evaluated during the examination.
We selected:
- A clinical situation of postoperative pain
- A real standardized patient
- Second consultation.
- Staff attached to the service help on the day of the OSCE.
Station instructions:
The following instructions were established and communicated separately to each of the stakeholders before the start of the OSCE:
- The starting situation for the resident: name of the patient, setting and reason for consultation (Figure 2).
- The role of the simulated patient: the scenario was submitted to the patient, translated into her mother tongue (Arabic) and defining precisely the initial complaint, the history of the disease, the signs and symptoms.
The simulated patient:
The standardized patient was recruited in consultation one week before the formative OSCE.
We organized and planned training sessions. The training of the simulated patient was done jointly with a professor from the department.
Simulated Patient Training Methodology:
At the beginning of the training session, the simulated patient became familiar with his or her role. This was accomplished by having a clinician play the role of the student.
Observation grids will be completed during the exercise to achieve the required reliability and validity 14. These drills ensure that the simulated patient's performance is consistent with the requested role.
Phase II: During the exam:
Due to the small number of resident teachers in CDE, the OSCEs can be conducted in one day. For the formative exam, we have scheduled it for late morning in the residents' office.
- OSCE start time and on time attendance:
• Instructional staff: 1 hour prior.
• Residents: 30 minutes before.
• Teachers: 30 minutes before.
• Standardized patient: 15 minutes before.
Resident Teacher Guidelines:
Residents are expected to adhere to the following guidelines:
- Punctuality and adherence to instructions during the examination,
- Candidates who arrive late will be excluded from the OSCE exam,
- Attempts to divulge content or obtain station information are considered cheating and will result in exclusion from the exam,
- Residents are received at the department by the organizing team. They are given a short briefing session.
- After the order of passage has been drawn, the residents are called by their family name and brought to the station at a fixed time.
- All personal belongings are placed in a designated area. Cell phones and electronic devices of any kind are strictly forbidden.
- Communication with anyone outside the auditorium is prohibited.
The standardized patient (SP) and assessors:
- They must be at the office at the time indicated.
- If necessary, last-minute details can be communicated to the SP by the station managers.
- A sound signal announces the beginning of the examination, and another announces the end. At this point, the resident in the assessment phase must leave the station.
- The formative OSCE station is filmed, with the written consent of the residents and the simulated patient.
Filling out the evaluation grids and grading:
For each station, we established grids based on the nature of the topic covered in the station.
- The assessors completed each observable item on the grid according to the established scale (well done, partially done, not done).
- Once the examination was completed, each station manager gave the complete file of the evaluation grids for each resident and the score obtained by the resident to the assessment supervisor.
 |
 |
Phase III: After the examination
At the end of the exam, a debriefing of the formative OSCE was held with the evaluated residents and the teacher-examiners to review the process and gather feedback. At the end of the debriefing, a small snack break was served to all participants.
Final OSCE Grading:
The OSCE was graded by adding the scores from the assessment grids and the rating scale. The OSCE was scored out of 20 points (medical expertise and non-technical skills).
Final Report and Validation:
A report was written by the academic supervisor with recommendations for improvement of the OSCE.
Evaluation of the formative OSCE by the learners:
Candidate satisfaction was assessed using a Google Form questionnaire. All residents responded to the satisfaction survey and 100% appreciated the OSCE (6 residents rated it better than the classic exam). The clinical situation was described as very realistic and the simulated patient as very believable. More than half of the residents had already experienced this type of consultation during their curriculum. The level of difficulty was considered appropriate overall. 57.1% considered this assessment fair, 28.6% organized, and 14.3% stressful. OSCE-related anxiety appears to have minimal to no influence on student performance.
The duration of the sessions was not convenient for the candidates being considered less than adequate (43%) or not adequate (43%).
The OSCE was a formative assessment that showed all the resident teachers the skills they had mastered, but also some gaps in their knowledge. The debriefing was useful for all the candidates and allowed them to reflect and conduct critical analyses.
In conclusion, the OSCE was a strong instrument for assessing the skills that resident teachers would value in the CDE department.
5.2. Staff Evaluation of the Formative OSCEThe OSCE assessment effectively meets the objectives of assessing a few clinical competencies in CDE (selected for the formative OSCE).
What we observed:
No residents experienced significant difficulties during the exam. Difficulties were with station time management and stress for most students. These difficulties were expected.
- Regarding knowledge:
Some learners made errors in justifying diagnostic or therapeutic reasoning. Others stuck to one therapeutic proposal instead of offering several therapeutic options to the simulated patient.
One resident forgot to ask for the patient's radiographic report and another resident asked for it after making the diagnosis.
- Regarding skills:
We observed aspects that were not expected, namely the residents' behavior in front of the clinical situation. Indeed, we noticed some mistakes during the clinical examination of the simulated patient (position of the resident during temporomandibular joint (TMJ) palpation and probing are some examples) and the clinical examination was not focused primarily on the sector of the tooth concerned.
The steps of the clinical examination were not correctly prioritized for some.
A few aseptic errors were made (very minor but not expected).
- Regarding soft skills:
The clinical exam included most of the essentials, but we noticed inappropriate body language in one resident, something not noticed during indirect supervision of the residents.
This OSCE interested conservative dentistry and endodontics residents in order to evaluate their clinical skills in this specialty. The experiment carried out (formative OSCE) allowed us to quantify clinical skills with a score and to identify the factors of failure/success. Some of the benefits to consider are the variables (simulated patient and examiner) which are better controlled than the traditional examination. The examination is easily reproducible and standards from one specialist class to another can be compared also. It can be also used in other specialties in dentistry, at the end of the years of postgraduate training.
The OSCE is a pedagogical tool for formative and evaluative assessments that can play an important role in the teaching of CDE. The summative assessment provides uniform comparison of comprehension and identified topics requiring improved instruction 15. Formative assessment has a strong influence on the learning process. The immediately feedback is more conducive to learning 16.
Feedback provided by the resident is highly valued and helps in the advancement and improvement of the testing process 17.
The main limitations are the demanding educational preparation and the organizational burden.
The association of OSCE with portfolios and practice exchange groups could allow the teaching team to train future competent and certified endodontists.
Based on this experience, the implementation of OSCE for resident evaluation has been appreciated by both residents and faculty. The strength of our training is the limited number of residents reducing logistical difficulties.
Residents have shown acceptance of the OSCE as a means of assessment and have expressed interest in participating in additional formative OSCE prior to the evaluative OSCE scheduled in December 2023. More in-depth training and debriefing sessions may be associated with these. Within CDE, therefore, OSCE will have a key role to play in the evolution of teaching in the coming years.
The authors declare that there are no conflicts of interest regarding the publication of this article.
• OSCE: objective structured clinical examination
• CDE: conservative dentistry and endodontics
| [1] | Hijazi M, Downing SM. Objective structured clinical examinations as an assessment method in residency training: practical considerations. Ann Saudi Med 2008; 28(3): 192-199. | ||
| In article | View Article PubMed | ||
| [2] | Chimea T, Kanji Z, Schmitz S. Assessment of clinical competence in competency-based education. Can J Dent Hyg. 2020 Jun 1; 54(2): 83-91. | ||
| In article | |||
| [3] | Nadeau, M.A. 1988. L’évaluation de programme: théorie et pratique. Québec: Presses de l’Université Laval, 2end edition. | ||
| In article | |||
| [4] | Pelaccia T, Tardif, J. Comment mieux former et evaluer les etudiants en medecine et en sciences de la sante. De Boeck superieur; 2016. | ||
| In article | |||
| [5] | Williams JS, Metcalfe A, Shelton CM, Spivey CA. Examining the Association of GPA and PCAT Scores on Objective Structured Clinical Examination Scores. Am J Pharm Educ. 2019 May; 83(4): 6608. | ||
| In article | View Article PubMed | ||
| [6] | Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990 Sep; 65(9 Suppl): S63-7. | ||
| In article | View Article PubMed | ||
| [7] | Khan R, Payne MWC, Chahine S. Peer assessment in the objective structured clinical examination: A scoping review. Med Teach. 2017 Jul; 39(7): 745-756. | ||
| In article | View Article PubMed | ||
| [8] | Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 2013 Sep; 35(9): e1437-46. | ||
| In article | View Article PubMed | ||
| [9] | Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004 Feb; 38(2): 199-203. | ||
| In article | View Article PubMed | ||
| [10] | Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Med Teach. 2013 Sep; 35(9): e1447-63. | ||
| In article | View Article PubMed | ||
| [11] | Harden RM, Lilley P, Patricio M. The definitive guide to the OSCE: the Objective Structured Clinical Examination as a performance assessment. 1st Edition - June 15, 2015. | ||
| In article | |||
| [12] | Hamdy H. Blueprinting for the assessment of health care professionals. Clinical Teacher. 2006 sep; 3(3): 175-9. | ||
| In article | View Article | ||
| [13] | Daniels VJ, Pugh D. Twelve tips for developing an OSCE that measures what you want. Med Teach. 2018 Dec; 40(12): 1208-1213. | ||
| In article | View Article PubMed | ||
| [14] | Harden RM. Revisiting 'Assessment of clinical competence using an objective structured clinical examination (OSCE)'. Med Educ. 2016 Apr; 50(4): 376-9. | ||
| In article | View Article PubMed | ||
| [15] | Gullard A. Objective structured clinical examination for predoctoral implant dentistry education. J Dent Educ. 2022 Dec;86 Suppl 3: 1707-1709. | ||
| In article | View Article PubMed | ||
| [16] | Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007 Nov; 29(9): 855-71. | ||
| In article | View Article PubMed | ||
| [17] | Alkhateeb N, Salih AM, Shabila N, Al-Dabbagh A. Objective structured clinical examination: Challenges and opportunities from students' perspective. PLoS One. 2022 Sep 2; 17(9): e0274055. | ||
| In article | View Article PubMed | ||
Published with license by Science and Education Publishing, Copyright © 2023 Drouri Sofia, Laslami Kaoutar and El Ouazzani Amal
This work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit
https://creativecommons.org/licenses/by/4.0/
| [1] | Hijazi M, Downing SM. Objective structured clinical examinations as an assessment method in residency training: practical considerations. Ann Saudi Med 2008; 28(3): 192-199. | ||
| In article | View Article PubMed | ||
| [2] | Chimea T, Kanji Z, Schmitz S. Assessment of clinical competence in competency-based education. Can J Dent Hyg. 2020 Jun 1; 54(2): 83-91. | ||
| In article | |||
| [3] | Nadeau, M.A. 1988. L’évaluation de programme: théorie et pratique. Québec: Presses de l’Université Laval, 2end edition. | ||
| In article | |||
| [4] | Pelaccia T, Tardif, J. Comment mieux former et evaluer les etudiants en medecine et en sciences de la sante. De Boeck superieur; 2016. | ||
| In article | |||
| [5] | Williams JS, Metcalfe A, Shelton CM, Spivey CA. Examining the Association of GPA and PCAT Scores on Objective Structured Clinical Examination Scores. Am J Pharm Educ. 2019 May; 83(4): 6608. | ||
| In article | View Article PubMed | ||
| [6] | Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990 Sep; 65(9 Suppl): S63-7. | ||
| In article | View Article PubMed | ||
| [7] | Khan R, Payne MWC, Chahine S. Peer assessment in the objective structured clinical examination: A scoping review. Med Teach. 2017 Jul; 39(7): 745-756. | ||
| In article | View Article PubMed | ||
| [8] | Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 2013 Sep; 35(9): e1437-46. | ||
| In article | View Article PubMed | ||
| [9] | Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004 Feb; 38(2): 199-203. | ||
| In article | View Article PubMed | ||
| [10] | Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Med Teach. 2013 Sep; 35(9): e1447-63. | ||
| In article | View Article PubMed | ||
| [11] | Harden RM, Lilley P, Patricio M. The definitive guide to the OSCE: the Objective Structured Clinical Examination as a performance assessment. 1st Edition - June 15, 2015. | ||
| In article | |||
| [12] | Hamdy H. Blueprinting for the assessment of health care professionals. Clinical Teacher. 2006 sep; 3(3): 175-9. | ||
| In article | View Article | ||
| [13] | Daniels VJ, Pugh D. Twelve tips for developing an OSCE that measures what you want. Med Teach. 2018 Dec; 40(12): 1208-1213. | ||
| In article | View Article PubMed | ||
| [14] | Harden RM. Revisiting 'Assessment of clinical competence using an objective structured clinical examination (OSCE)'. Med Educ. 2016 Apr; 50(4): 376-9. | ||
| In article | View Article PubMed | ||
| [15] | Gullard A. Objective structured clinical examination for predoctoral implant dentistry education. J Dent Educ. 2022 Dec;86 Suppl 3: 1707-1709. | ||
| In article | View Article PubMed | ||
| [16] | Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007 Nov; 29(9): 855-71. | ||
| In article | View Article PubMed | ||
| [17] | Alkhateeb N, Salih AM, Shabila N, Al-Dabbagh A. Objective structured clinical examination: Challenges and opportunities from students' perspective. PLoS One. 2022 Sep 2; 17(9): e0274055. | ||
| In article | View Article PubMed | ||
