sigcse97

50 %
50 %
Information about sigcse97
Entertainment

Published on December 10, 2007

Author: Edolf

Source: authorstream.com

Online Evaluation in WWW-based Courseware: The QUIZIT Project:  Online Evaluation in WWW-based Courseware: The QUIZIT Project Lúcio Cunha Tinoco Edward A. Fox N. Dwight Barnette Virginia Tech Background and Motivation:  Background and Motivation EI project, Courseware, Interactive Learning with a Digital Library in CS - http://ei.cs.vt.edu Workshop June 15-21 (apply to fox@vt.edu) Local and Distance Learning Evaluation including based on server’s use Large enrollment, grading overhead Personalized System of Instruction (PSI) QUIZIT: Goals, Applications:  QUIZIT: Goals, Applications Fully WWW-based, automated test system supplements other types of evaluation is not viewed as a substitute for other types for open-book or monitored quizzes helps in identifying problem areas focuses on task completion, not on scores For PSI and/or large enrollment courses Reduces administrative and grading overhead QUIZIT: Goals, Applications:  QUIZIT: Goals, Applications Integration w/ WWW-based courseware and digital libraries Reusable content Automatic grading w/ high interactivity Instant feedback Alternative questions Easy to use and to apply Test Systems:  Test Systems Traditional: letter, paper, oral Test formats Turn-around, grading time Electronic submission, manually graded Video conference, e-mail, etc. Reduction in turn-around, same grading time Electronic submission, automatically graded CBTs, WBTs, Hybrids Reduction in turn-around and grading time Existing Automated Test Systems: Denominations:  Existing Automated Test Systems: Denominations CBTs Authorware Toolbook Electronic Trainer WBTs Mklesson LXRTest Hybrids: PC authoring, option of Internet deployment (ex: Toolbook II) Existing Test Systems: Features:  Existing Test Systems: Features Authoring interface Usability Document representation, portability, reusability Execution environment Distributed vs. stand-alone Record-keeping Adaptability based on performance? static (e.g., predefined) vs dynamic (e.g., inference) Existing Test Systems: CBTs, Hybrids:  Existing Test Systems: CBTs, Hybrids Proprietary interfaces powerful, but hard to learn each system represents documents differently hard to integrate w/ digital libraries compromises portability Stand-alone environments Hybrids provide Internet deployment, but do not provide group capabilities usually do not have the concept of a “class” Existing Test Systems: WBTs:  Existing Test Systems: WBTs Interfaces only fair-to-good Limitations in adding adaptability Presentational syntax in document representation (poor logical modeling, ad hoc use of HTML-extensions) Poor record-keeping and monitor capabilities Also do not treat group monitoring, lack concept of “class” QUIZIT System Architecture:  QUIZIT System Architecture QUIZIT: Main Features:  QUIZIT: Main Features Use of standarized authoring language SGML-based QUIZIT Markup Language (QML) But still provides all the power of HTML and Java Tests can be adaptable Complete log of results and actions into a database Monitor interface in the WWW QUIZIT: Modules:  QUIZIT: Modules Authoring Monitor Registration Activity Run-time Registration and authentication Grading and feedback Review The QUIZITAuthoring Module: QML:  The QUIZITAuthoring Module: QML Content-based representation Use of SGML (ISO/IEC 8879) to define QML Avoids presentational elements The QUIZITAuthoring Module: QML (top):  The QUIZITAuthoring Module: QML (top) The QUIZITAuthoring Module: QML (middle):  The QUIZITAuthoring Module: QML (middle) The QUIZITAuthoring Module: QML (end):  The QUIZITAuthoring Module: QML (end) The QUIZIT Authoring Module: QML Translation:  The QUIZIT Authoring Module: QML Translation After instructor prepares the QML file The QUIZIT Monitor Module: Record-keeping:  The QUIZIT Monitor Module: Record-keeping Log student accesses to the system Keep track of individual and class performance on quizzes Can change system state and db at any time if user is authorized to do so (instructor, GTA). Also includes student registration. Instructor mSQL Database Engine:  mSQL Database Engine Stores and retrieves students’ records and accesses to the system IDs and passwords Scores and students’ answers on every quiz (for every unit in the course) Access logs (begin, end time) Last quiz taken Next quiz to be taken The QUIZIT Run-time Module: Registration and Authentication:  The QUIZIT Run-time Module: Registration and Authentication Provides online password registration and distribution (by Tommy Johnson) Passwords are encrypted in the server side (is client side encryption feasible?) Authenticates every user once and then uses session cookie throughout the quiz (use of hidden fields) The QUIZIT Run-time Module: The Grader Engine:  The QUIZIT Run-time Module: The Grader Engine After selecting a course, authenticating, selecting a unit and taking the quiz... Student using a WWW browser • end time • scores • answers • next quiz The QUIZIT Run-time Module: Reviewing Records and Old-Quizzes:  The QUIZIT Run-time Module: Reviewing Records and Old-Quizzes At any given point in time students can: Review old quizzes (student may not retake quizzes for credit) Come back to the system and re-start from where they left Review their records submission times scores (cumulative, per quiz) the next quiz they will have to take QUIZIT in action!:  QUIZIT in action! Graduate-level course: CS5604 (11 students) Personalized System of Instruction (PSI) Course divided into 11 units Each unit has one quiz with two levels. Each level has two remedial quizzes score < cutoff score < cutoff score > = cutoff CS5604: Progress Report:  CS5604: Progress Report CS5604: Progress: Conclusions :  CS5604: Progress: Conclusions Students do procrastinate They tend to procrastinate more in middle units Keep the pace in the beginning Catch up in the end because of deadline Big quiz “rush” in the end of the semester CS5604: Student Performance:  CS5604: Student Performance CS5604: Performance: Conclusions:  CS5604: Performance: Conclusions Most students only go through 2 of 6 levels “Harder” quizzes: IF, MM, KB Some questions? Does procrastination lead to worse performance? Does “deadline fever” lead to worse performance? QUIZIT in action! (cont.):  QUIZIT in action! (cont.) Undergraduate Level Course: CS1604 (about 150 students) Each quiz has just one level Security concerns are much greater Password distribution needs to be automated How can we assure that one student is not taking a quiz for another? CS1604: Progress and Performance:  CS1604: Progress and Performance Students procrastinate even more Less mature? Performance was based on scores rather than task completion Flat quiz structure may lead to poor indicators Evaluation and Conclusion: HCI Aspects:  Evaluation and Conclusion: HCI Aspects Authoring quizzes with QUIZIT takes a little more time than traditional quizzes Huge savings in grading time, though... Security and authentication are critical in large enrollment classes Extensive record keeping and WWW monitor interfaces are very important Change in the role of the GTA Evaluation and Conclusion: Technical Aspects:  Evaluation and Conclusion: Technical Aspects Instructors need training in QML CGI/mSQL is a very nice combination ... but new Java features may complement this model in the future Record keeping can be further extended Further security mechanisms need to be developed (client side encryption, camera monitoring, etc.) for more general applications Evaluation and Conclusion: Pedagogical Aspects:  Evaluation and Conclusion: Pedagogical Aspects QUIZIT was not designed to completely substitute for other types of evaluation Problems w/ multiple-choice exams can’t be negleted Be careful when using scores in adaptable quizzes! Focus should be on task completion Should work well w/ self-evaluation and distance learning Evaluation and Conclusion: Pedagogical Aspects:  Evaluation and Conclusion: Pedagogical Aspects Does QUIZIT/PSI promote procrastination? What can we do to avoid procrastination in PSI? One option: Bonus for tasks completed in a certain time limit How can we explore more in quiz adaptability? Future Developments:  Future Developments Improvements in student feedback Better error report after each test Graphical User Authoring Interface but keep QML representation Improve question modeling Add more pattern matching options for fill-in questions Training program, Packaging, Distribution

Add a comment

Related presentations

Related pages

Undergraduate Women in Computer Science: Experience ...

Undergraduate Women in Computer Science: Experience, Motivation and Culture. Allan Fisher, Jane Margolis and Faye Miller School of Computer Science ...
Read more

www.eas.asu.edu

Hier sollte eine Beschreibung angezeigt werden, diese Seite lässt dies jedoch nicht zu.
Read more

Harvard's Center for Research on Computation and Society

Harvard's Center for Research on Computation and Society, Cambridge, MA. 176 likes · 2 talking about this. Harvard's Center for Research on Computation...
Read more

Online Evaluation in WWW-based Courseware

Online Evaluation in WWW-based Courseware Lúcio C. Tinoco, Edward A. Fox, N. Dwight Barnette Virginia Polytechnic Institute & State University
Read more

Question 5 - George Washington University

Question 5 - George Washington University
Read more

Exploring Martian Planetary Images C++ Exercises for CS1

Exploring Martian Planetary Images C++ Exercises for CS1 Harriet J. Fell and Viera K. Proulx Northeastern University College of Computer Science Boston, MA ...
Read more

Integrating Technology into CS Education, Uppsala

Uppsala University] Click here to get to our US mirror site (All words and images in the frame to the left are clickable) These pages are maintained by ...
Read more

www.cs.vt.edu

... {1997}, month = {Feb-March 1997}, pages = {194-198}, address = {San Jose}, url = {http://ei.cs.vt.edu/papers/sigcse97.pdf}, author = ...
Read more

Monday, October 26th 11:30-1:00PM... - Harvard's Center ...

http://www.cs.cmu.edu/…/g…/www/papers/sigcse97/sigcse97.html. Undergraduate Women in Computer Science: Experience, Motivation and Culture.
Read more