Chewing Gum and Duct Tape: Creating an Online Survey Application out of
Available Parts
Patrick Burke, MA,
Dr. Sandy Cook, Ph.D. and Kevin Tomczyk, The University of Chicago, Illinois,
USA
ABSTRACT:
Background/Problem: The Pritzker School of Medicine uses quarterly faculty and course ratings completed by students to recognize and reward teacher excellence as well as guide the curriculum development. Pritzker currently uses scanable paper forms administered during quarterly exams. This process, averaged just over 75% return rate over the last two year for courses with an exam, and those without averaged less than 45%. For multiple reasons, transitioning to online administration was desired, however, creating our own system or purchasing a new application was considered prohibitively expensive and duplicated features already in use in other applications.
Method and Tools: We currently use a web-based courseware system (Blackboard), for authentication and creation of simple assessments, and Teleforms (Cardiff), a product that translates scanable forms into web-based PDF files. By combining features of both programs, we were able to leverage existing technologies to create an evaluation system that met all of our needs. We piloted this hybrid system with two spring quarter courses that did not have in-class exams and often had poor return rates. We created a one-question assessment in the courseware product and linked the web-based PDF file. The forms were available the last week of class and finals week. Students were informed of the assessment via class announcements and email. Multiple reminders were sent out via email. Following what was considered a successful pilot (57%), we used this method for all first and second year courses in the Fall Quarter..
Results: The Fall Quarter average return rate was 63%, with an average of 71% for first year courses and 58% for second year courses. Another change was the increased rate of written comments received; from approximately 40% last year to 76% this year. In addition, the depth and extensiveness of the comments increased. Informal comments from the students indicated they liked the opportunity to fill out the forms away from the pressures of the exams. The change of the administration venue allowed students to spend more time on thinking about the comments as well as the quantity of comments. Computer crashes, the main obstacle, were easily resolved using Blackboard's assessment reset feature.
Conclusion: The combination of the two programs fulfilled all of our stated needs: <sum> Flexibility: using existing PDF assessment forms it was flexible, and not limited by HTML or commercial products. <sum> Authentication with anonymity: it effectively limited access to authorized users yet not linked to submitted data. <sum> Resources: significantly decreased the post submission data processing and no need to purchase new program. <sum> Response Rate: With multiple reminders, rates were better than non-exam administrations and close to paper administration. With an expectation that, as students become more facile with the system, the rates will improve. In addition students provided more comprehensive comments.
Novelty/Discussion: This project represents a significant paradigm shift. Before this project, the question was always, "Build or buy?" This project effectively combined two existing, deployed, robust technologies to do what neither could independently. In the future, the success of this small project means the question will change from "Build or Buy?" to "Can we do it , faster, more reliably, and for less cost with what we have already?".
BENEFIT TO PARTICIPANTS ATTENDING SESSION:
Pritzker School of Medicine needed course surveys that were trackable, but random, and we needed a web-based application to deliver them to students. Only problem was, our programmers were backed up more than two years with other projects. However by combining the authentication features of our existing courseware product with a scanning OCR product we already have, we were able to create an application that met our needs quickly, robustly, and relatively easily. The fact that we have an authentication module constantly up and ready to go with all of our students already in it gives us a launching pad to create our own mini-applications in this manner, without traditional "programmers" at all.
Patrick Burke
924 E. 57th St.
#015
Chicago, IL 60637
Phone: 773-702-4596
Fax: 773-834-3693
Email: mailto:p-burke@uchicago.edu
CO-AUTHORS:
Dr. Sandy Cook, PhD. D., Kevin Tomczyk
924 E. 57th St.
Chicago, IL 60637
Phone: 773-834-7828
773-834-8019
Fax: 773-834-0688
773-834-3693
Email: mailto:scook@uchicago.edu