HaPerT - Handheld based Performance Test
ABSTRACT:
Background:
How can clinical competence be assessed? Methods like multiple choice questions have their limitations, since they test only the knowledge component. Clinical competence (defined as a complex of knowledge, skills and attitudes) obviously is measured best by the candidates expressed behavior
in a realistic clinical situation. The widely used objective structured clinical examination (OSCE) follows this approach. In an OSCE, medical students circle through stations and perform different practical tasks. Experts observe the students and assess their behavior using checklists. For scoring, checklists data usually are manually entered into a database. Alternatively, optical mark reader sheets can be used to facilitate data entry.
Innovation:
The HaPerT-System uses PalmOS powered handheld computers to replace checklists. Mobile operation and a touch-screen user interface make the handhelds a well suited tool for this task. ue to the small screen size, we changed the structure of our previously used checklists to a branched checklist design (1. Layer: tasks -> 2.Layer: procedural steps -> 3. Layer: actions) and introduced a new marking procedure, in which only mistakes, incomplete or missing actions of a candidate are recorded (our old checklist were linear - the observer had to tick off all items). This approach significantly reduces the number of pen strokes necessary to mark an average candidate. Observers can also enter new items, which serve as suggestions for updates of the checklists. At the end of the examination the data of all participating candidates are synchronized (by "beaming") and transferred to a desktop computer. This process allows immediate data evaluation, like:
•individual feedback printouts showing markings for each station
• group feedback selected by item weight or item frequency
• item analysis (for formative course evaluation)
The HaPerT System is used at our clinical skills center since March 2000. It minimizes the administrative and logistic efforts of our OSCEs. Observers have no difficulty to familiarize with the new marking approach(about 15 min additional training is required).
Development:
The checklist program was developed using the PalmOS database application Satellite Forms 4.0 by Pumatech (www.pumatech.com). Desktop database programming was done with VBA and Access2000.
BENEFIT TO PARTICIPANTS ATTENDING SESSION: For all participants interested in performance-assessment we will offer hands-on experience on how to set up electronic checklists, how to observe and rate students behavior with the handheld (we will provide clinical video material), how to synchronize test data and how to manage the results of the test in terms of feedback, scoring and evaluation.We will welcome all comments on features still missed.
Michael Schmidts
Institut für med. Aus- und Weiterbildung
POB 10
A-1097 Vienna
Austria
Phone: ++41 1 40400 1335
Fax: ++41 1 40400 1994
Email: michael.schmidts@akh-wien.ac.at
CO-AUTHORS:
Markus Kemmerling
Ruth Willnauer
Martin Lischka
Institut für med. Aus- und Weiterbildung
POB 10
A-1097 Vienna
Austria
Phone: ++41 1 40400 5473
Fax: ++41 1 40400 1194
Email: markus.kemmerlin@akh-wien.ac.at
ruth.willnauer@akh-wien.ac.at
Martin.lischka@akh-wien.ac.at
Website:http://www.akh-wien.ac.at/imaw