Creation and use of the OSCE Spring 2025 DSOSCE:Creation and use of the OSCE (Spring 2025) DSOSCE:Creation and use of the OSCE (Spring 2025) Learning outcomes ̶ To describe the objective structured clinical examination's essential motivation and main theoretical principles. ̶ To summarise the features and benefits of the SIMUportfolio integration platform regarding formative and summative assessment. ̶ To propose an appropriate topic for station preparation based on objective structured clinical examination principles. ̶ To demonstrate the implementation of a station based on the principles of an objective structured clinical examination. ̶ To formulates feedback after a practical demonstration of the station. DSOSCE:Creation and use of the OSCE (Spring 2025) Course outline ̶ Theoretical principles and their placement in the context of teaching at MED MUNI ̶ Presentation of the SIMUportfolio technology platform and demonstration of functionalities ̶ Station design based on the principles of objective structured clinical examination. ̶ Practical trial through an objective structured clinical examination. ̶ Feedback and evaluation Introduction What is your background? What do you know about OSCE? Modern methods for assessment Basic facts DSOSCE:Creation and use of the OSCE (Spring 2025) Formative vs Summative assessment DSOSCE:Creation and use of the OSCE (Spring 2025) Low-Stakes vs. High-Stakes OSCE stations Masaryk University, Faculty of Medicine Forms of OSCE assessment ̶ Formative assessment ̶ to monitor student learning to provide ongoing feedback that can be used to improve their learning. help students identify their strengths and weaknesses and target areas that need work help faculty recognize where students are struggling and address problems immediately ̶ Formative assessments are generally low stakes, which means that they have low or no point value. ̶ Summative assessment ̶ to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark. ̶ Summative assessments are often high stakes, which means that they have a high point value. DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE perspective Formative vs Summative assessment ̶ Summary ̶ The most common use of OSCEs is in summative assessment, particularly in graduate or licensure education. ̶ The OSCE is also very effective formatively in teaching and training if adequately constructed and supplemented with feedback. ̶ Recommendation ̶ Formative OSCE Recommended for use during study to practice and reinforce skills, with the possibility of repetition. ̶ Summative OSCE Appropriate after prior training, requires standardisation, calibration of raters, and valid metrics. OSCE Theoretical principles DSOSCE:Creation and use of the OSCE (Spring 2025) When OSCE was developed? The decade between 1900 and 2020 ☺ OSCE – definice a základní principy13 Objective Structured Clinical Examination “An assessment tool based on the principles of objectivity and standardisation, in which the candidates move through a series of time-limited stations in a circuit for the purposes of assessment of professional performance in a simulated environment. At each station candidates are assessed and marked against standardised scoring rubrics by trained assessors”. AMEE Guide 81 – The Objective Structured Clinical Examination (OSCE) DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE Objective Structured Clinical Examination A modern, practical approach used in medical and healthcare education to assess clinical competencies in a structured and standardized way. It simulates real-world clinical scenarios, where students rotate through stations to perform specific tasks while being evaluated by trained OSCE observers. DSOSCE:Creation and use of the OSCE (Spring 2025) Key Features OSCE ̶ Realistic patient interactions (often with standardised patients) ̶ Simulators ̶ Actors ̶ Teachers / Students ̶ Focused assessment of clinical skills, communication, and decision-making ̶ Structured stations with objective scoring checklists ̶ Immediate feedback and opportunities for improvement DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE rotation waiting room 5 min 6 1 2 3 45 5 min 5 min 5 min 5 min5 min 5 min (break) DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE stations DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE stations: Krakow DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE stations: MED MUNI DSOSCE:Creation and use of the OSCE (Spring 2025) Why OSCE? It ensures fair, consistent, and comprehensive evaluation of future healthcare professionals, bridging the gap between theoretical knowledge and clinical practice. DSOSCE:Creation and use of the OSCE (Spring 2025) History and evolution of OSCE ̶ The OSCE was first developed in the 1970s as a response to inconsistencies in clinical assessment. It has since become a gold standard for assessing clinical competencies globally. ̶ Unlike traditional long-case exams, OSCEs are standardised, objective, and cover a broad range of skills through multiple stations. DSOSCE:Creation and use of the OSCE (Spring 2025) Levels of clinical competence Theoretical Frameworks: Miller’s Pyramid MILLER, George E. The assessment of clinical skills/competence/performance. Academic medicine, 1990, 65.9: S63-7. Action Performance Competence Knowledge https://openpress.usask.ca/ideabook/chapter/millers-pyramid-of-clinical-competence/ DSOSCE:Creation and use of the OSCE (Spring 2025) Categories for educational goals Theoretical Frameworks: Bloom’s Taxonomy https://tips.uark.edu/using-blooms-taxonomy/#gsc.tab=0 DSOSCE:Creation and use of the OSCE (Spring 2025) References Objective Structured Clinical Examination ̶ HARDEN, Ronald M.; GLEESON, F. A. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical education, 1979, 13.1: 39-54. ̶ KHAN, Kamran Z., et al. The objective structured clinical examination (OSCE): AMEE guide no. 81. Part II: organisation & administration. Medical teacher, 2013, 35.9: e1447-e1463. ̶ PATRÍCIO, Madalena Folque, et al. Is the OSCE a feasible tool to assess competencies in undergraduate medical education?. Medical teacher, 2013, 35.6: 503-514. ̶ SLOAN, David A., et al. The use of the objective structured clinical examination (OSCE) for evaluation and instruction in graduate medical education. Journal of Surgical Research, 1996, 63.1: 225-230. RAT Theoretical principles ̶ Team-based learning support ̶ Main goals ̶ Increase student engagement ̶ Increase the quality of learning RAT Readiness Assurance Tests iRAT 10 min tRAT 20 min DSOSCE:Creation and use of the OSCE (Spring 2025) RAT in practice DSOSCE:Creation and use of the OSCE (Spring 2025) RAT in practice OSCE at MED MUNI Our experience Výjezdní zasedání projektu SIMU, 5. června 201931 Basics OSCE implementation at MED MUNI ̶ Objective ̶ Observers use a predefined form to assess the student ̶ Structured ̶ Students solve the same assignments at the same place and time ̶ Clinical ̶ Topics examined are mostly based on clinical situations Výjezdní zasedání projektu SIMU, 5. června 201932 Looking back OSCE implementation at MED MUNI ̶ Consultations with foreign experts ̶ Karolinska Institutet, Jagiellonian University, Comenius University ̶ Implementation of the OSCE module into the SIMUportfolio ̶ Support for planning, organisation, run and evaluation ̶ Design of a comprehensive methodology ̶ Manual for OSCE guarantors and observers ̶ 1st Piloting: 8th First Aid Simulation Course (May 2019) ̶ OSCE educational workshop (M. Nowakowski) (June 2019) OSCE training June 15–16, 2019 Choose OSCE domains Medical discipline Course Choose OSCE skills Communication skills Technical skills Choose OSCE competence* History taking Exploring symptoms Set up OSCE stations Total number Lenght Choose type of stations With no patient With patient What to do Blueprinting Relations with curriculum description (learning units & learning outcomes) * History taking, exploring symptoms, critical reasoning, communication (explain, ask, educate, write), other Masaryk University, Faculty of Medicine First aid and Propedeutics Blueprinting template example First aid Propedeutics Communication skills Technical skills DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE implementation Step-by-step Define objectives Identify subjects and topics Allocate rooms and staff Plan course Define scenarios Develop scenario detail Establish scoring thresholds Train observers Implement platform Conduct piloting Evaluate pilot results Deploy in teaching DSOSCE:Creation and use of the OSCE (Spring 2025) Looking back: 2018 – OSCE schedule Activity / Month 03 04 05 06 07 08 09 10 11 12 Define objectives Identify subjects and topics Allocate rooms and staff Plan course Define scenarios Develop scenario detail Establish scoring thresholds Train observers Implement platform Conduct piloting Evaluate pilot results June 2019 - OSCE workshop - Meeting with SIMU management September 2019 - Blueprinting of OSCE pilot - Approval by SIMU management January 2020 - OSCE training in Krakow February–March 2020 - Training of simulated pacients April 2020 - OSCE pilot run at MU July–August 2019 - Curriculum description of selected courses - First aid - Propedeutics Preliminary schedule – timeline summary DSOSCE:Creation and use of the OSCE (Spring 2025) MED MUNI Basic characteristics ̶ Clearly defined content and form = scenarios. ̶ Comparison of the student's real ability with the stated learning objectives in the curriculum. ̶ Each student performs comparably challenging tasks. ̶ Interaction between student and observer is structured, including how the final grade is determined. ̶ The level of knowledge from each station is independently assessable. ̶ Guarantee of objectivity and independence (unbiased). DSOSCE:Creation and use of the OSCE (Spring 2025) MED MUNI OSCE stations ̶ Define the tasks relevant to the subject we want to asess ̶ Not peripheral areas and details ̶ Topics frequently encountered in medical practice ̶ Not infrequently ̶ Consistency with defined curriculum and learning outcomes ̶ Appropriate difficulty according to the student's level of education ̶ Year 2 student vs. Year 6 student DSOSCE:Creation and use of the OSCE (Spring 2025) Disadvantages ̶ Organizational complexity with higher numbers of students. ̶ Necessity of the need for “trusted” simulated patients or actors. ̶ Does not replace a real clinical situation = does not test the student's actual ability to deal with a real clinical situation. ̶ Improperly designed stations and assessments may lead to incorrect student evaluation. DSOSCE:Creation and use of the OSCE (Spring 2025) Cost vs. Performance Number of OSCE stations Time period Total time for 12 students Number of employees involved 10 + 2* 5 min 72 min 24 10 + 2* 10 min 132 min 24 5 + 1* 5 min 72 min** 14 5 + 1* 10 min 132 min** 14 * Break time ** Two-run schema DSOSCE:Creation and use of the OSCE (Spring 2025) MED MUNI Questions I. ̶ In what part of the training will OSCE be deployed? ̶ Ideally deploy OSCE across the curriculum ̶ What level of competences will be tested? ̶ Testing skills, not knowledge. ̶ You're a spectator at a tennis match, sitting in the front row, and you see a teammate knock out the other teammate's teeth with a racket during doubles. What do you do? ̶ In what rooms and who will observe? How long will OSCE last? ̶ Efficient use of available rooms and human resources DSOSCE:Creation and use of the OSCE (Spring 2025) MED MUNI Questions II. ̶ What will be included in the OSCE scenarios? ̶ Instructions for the student ̶ Instructions for the observer ̶ Assessment entry form ̶ Instructions for the simulated patient ̶ Equipment list for the OSCE station ̶ How to determine the final grade? ̶ Observer decides (without knowledge of station outcome) ̶ Observer decides (with knowledge of station outcome) ̶ System decides (based on defined metrics) ̶ How to communicate with OSCE observers? DSOSCE:Creation and use of the OSCE (Spring 2025) Writing OSCE stations ̶ Learning outcomes ̶ Start with clear, measurable learning objectives that align with curriculum goals. ̶ Scenarios ̶ Create realistic clinical tasks with clear instructions for students. ̶ Checklists and rating scales ̶ Develop structured tools for evaluation including essential criteria and global rating elements. ̶ Bias and pitfalls ̶ Avoid ambiguous sentences, cultural bias, and ensure equal difficulty across stations. SIMUportfolio Integration platform at MED MUNI About SIMUportfolio platform Integration and innovative online platform Several modules for comprehensive support for teaching Supports modern learning methods (TBL, OSCE) For students, teachers and faculty management Improves students' knowledge and skills for practice In-house project of the Faculty of Medicine (MU) Content management ̶ For parametric description of curriculum (Faculty of Medicine) ̶ Division of curricula into teaching blocks: ̶ Programmes ̶ Disciplines ̶ Courses ̶ … Objective Structured Clinical Examination (OSCE) ̶ Modern way of evaluation of students of healthcare disciplines ̶ First Aid ̶ Propaedeutics ̶ Separate module for OSCE Management: ̶ OSCE designers, OSCE examiners, OSCE guarantors Sketch definition of OSCE exams Execute evaluation of students Report reporting Stats statistics about OSCE DSOSCE:Creation and use of the OSCE (Spring 2025) OSCE Sketch OSCE in practice OSCE in practice OSCE visualization Data processing, validation and preparation necessary Readiness Assurance Test (RAT) ̶ For Team Based Learning (TBL) ̶ Separate module for RAT Management: ̶ RAT designers, RAT guarantors, RAT facilitators and students Sketch definition of RAT lessons Execute running iRAT/tRAT tests completing RAT tests Stats statistics about RAT RAT Stats Statistics are rendered automatically (without the need for processing) Other modules ̶ Schedule ̶ Lessons of First Aid and Propaedeutics in Simulation Centre ̶ Warehouse ̶ Warehouse management in development ̶ Learning activities ̶ Connection to curriculum, schedule, warehouse ̶ Educational and study materials SIMUportfolio & OSCE https://stage-simu.iba.muni.cz DSOSCE:Creation and use of the OSCE (Spring 2025)