Published on March 4, 2014
iakm-dev2.slis.kent.edu/ Usability Testing Cathy Carr Holly C. Murphy Matthew Pahls December 8, 2009
Table of Contents Statement of Work . . . . . . . . . . . . . . 3 Characterizations of Tasks . . . . . . . . . . 4-5 Data Collection . . . . . . . . . . . . . . . 6 Characterization of Participants. . . . . . . . . 7 Conclusions. . . . . . . . . . . . . . . . 8 Recommendations for Redesign . . . . . . . . 9 Raw Data. . . . . . . . . . . . . . . . 10-22 Page 2
Statement of Work The document provided will describe our research group’s findings pertaining to our usability test of a newly created iteration of the IAKM public facing web site. The testing completed was performed utilizing the ClearView eye tracking software to accurately monitor the user’s interaction on multiple dimensions. The facility allowed us to work directly with the user, and monitor interaction via click tracking, eye movement, audio and video. Combining those attributes, based on a time scale, our research team was able to more accurately identify where the site changes were successful and where they were not. The Information Architecture and Knowledge Management web site, like many other programs in a large university’s environment, utilizes its web presence to interact with a worldly audience. Commonly the web site is first and foremost, a first impression for a prospective student. So having a strong web presence is paramount to sustaining positive enrollment and retention. The IAKM program at Kent State University roots itself in a rapidly evolving field that has strong ties to the web. As the field establishes itself, IAKM must be able to accurately portray the field in a manor such that people with little preexisting knowledge could identify with it. The site recently underwent some structural changes, and our team was tasked with conducting a full usability study on the product. Since the changes are not dramatic, we needed to know if these slight alterations would have a positive or negative impact on the overall experience. The hierarchical changes were based around the main navigation. The intent of the changes was to improve the overall findability of information within the site. Our group’s investigation would prove or disprove the effectiveness of the changes. The IAKM office tasked our team with focusing our efforts on two particular audience segments; the prospective student and the current student. We used a short screener developed in Qualtrics to identify our four user participants that fit either of the two audience segments. From there, we had each of the users come in to the usability lab individually and we conducted a series of task based analysis tests. These tests were targeted at particular tasks that the audience segments should have the capabilities of easily performing. We quickly discovered that the tasks we perceived as relatively simple turned out to be rather difficult, and the users were having issues completing the tasks. Page 3
Characterization of Tasks Two sets of tasks were developed to test the site with two sets of users. The first set of tasks tested the site for prospective students. These tasks were standard site activities that a user researching or preparing for enrollment would conduct. The second set of tasks tested the site for current students. These tasks were intended for students already in the program. The activities supported navigating the site for content that the student would need during the program cycle. Prospective IAKM student Task 1: You are a prospective KM certificate student. What are the core-required courses? Rationale: This is a basic task that a student would perform to locate information vital to proceeding within the program. Task 2: You are a prospective UXD master’s student. How would you apply to the program? Rationale: For many students this may be their first experience with the site. It is imperative that this task be easy to find and perform. Task 3: You are interested in a graduate assistantship. Find information to apply for this position. Rationale: All prospective students may not attempt this task. But, it is an important item for graduate schools that should be easily located on their sites. Current IAKM student Task 1: You are interested in a graduate assistantship. Find information to apply for this position. Rationale: Again, this may not be a task attempted by all current students. But, it is possible for current students to become graduate assistants. Task 2: It is time to schedule for the Spring 2010 semester. Use your current student standing and find offered classes using the IAKM site. Rationale: The IAKM site lists classes offered for every semester. This is very helpful for students when preparing for registration. Task 3: You are approaching graduation and beginning to think about the master’s project. Find information that will help you. Rationale: Every student will need to locate this information, not only to begin the project, but also to submit the appropriate forms. Page 4
Task 4: You wish to take a workshop next semester. Find the form you will need to ensure you receive credit for the workshop. Rationale: This specific task may not be performed by every student. The ease of locating this particular information assures that the user is able to navigate to the forms page of the site, where other vital program information is located. Task Administration The testing took place in the Kent State University Usability Lab. The above tasks were typed, printed, and read by the facilitator to each participant. The facilitator either sat or stood to the side of the participant while the session was in progress. Each task began from the home page. The participant was able to ask for clarification or restatement of the task. Some guidance was given if the participant strayed off track. The participant was asked to think-aloud during the tasks. ClearView was used to capture audio, video, and eye tracking. These recordings were analyzed at a later time for user comments, difficulties, and successes. Page 5
Data Collection Information was collected from our selected participants on the dimensions of auditory, visually, and click tracking. Each axis is powerful in its own right, but combining them and being able to analyze them collaboratively allows us to produce truly accurate analysis of the participant. While tasks were in progress, we elicited verbal feedback so we could record what the user was thinking. As they were talking out the tasks we were able to understand why the user was doing the task and seeing why they had their own understanding on how the site should function. The lab is also equipped with a computer monitor that has the capability to track eye movements (“gaze”). These recordings are accomplished by having the monitor use an infrared signal projected onto the back on the participants eye and the signal then reflects off the eye and recorded at that point. As the user interacts with the site, the monitor tracks the movement and combines that information with the current page of interaction. This allows us to see exactly what the user us looking at in real time. The recorded instances of the visual attribute can be exported to two rather interesting and rather valuable outputs. Gaze plots and heat maps allow us to take the visual recording and map them to the site so we can visually see where the user focused and how long they focused for. Now we add the fourth and final component to our recording attributes and this is time. Combined with the auditory, visual, and click actions, we can accurately track all user interaction in retrospect to the tasks assigned to the user throughout the entire test. These elements empower the research team to effectively capture the data necessary for proper data analysis, and more importantly see how effective the site alterations are on our target user segments. Page 6
Characterization of Participants Not all of the study participants fit the user profiles established in the requirements analysis. The established profiles are as follows: Prospective IAKM student • Age: 26 – 40 • Sex: Female • State: Ohio • Visual impairment: None • Is not involved with the program • Has not used the current IAKM site • Is looking for more information about the IAKM program • Computer experience: average to above average • Uses a PC every day Current IAKM student • Age: 26 – 40 • Sex: Male • State: Ohio • Visual impairment: None • Is involved in the program • Learned of the program from a former or current student • Computer experience: above average to expert • Uses a PC every day For the study, both a male and female prospective student were selected to participate. The female wore glasses. One of the current student participants also wore glasses. The other current student uses a Mac. Page 7
Conclusions • Vocabulary confusion. Two of the participants were confused with the site terminology. More interesting is that these were the current student participants that have used the previous IAKM site. • Student standing. The participants that were chosen as prospective IAKM students consider themselves as current students because they are already enrolled in KSU. • GA task. All four participants were given the task of locating information to apply for a graduate assistantship. Each individual chose a different path to locate this information, and verbalized a different rationale for their path. • Approach. The two current students performed very different approaches to locate the task information. One performed the hunt and peck method, while the other took a few seconds to plot out a course of action. The prospective students conducted trial and error approaches. • Searching Issues. When the users tried to perform a search the results were somewhat inaccurate. • Using a Crutch. When users are confused they turned to the FAQ as a crutch for locating the information. This proves the user is becoming quickly frustrated with the site and tries to find the quickest path to the proper knowledge. All users tested were personally vested in finding the information because they were being tested on it. The general user might not have that investment in finding the information so they will tend to use the path of least resistance. Page 8
Recommendations for Redesign • Evaluate the navigation labels and organization. This was a stumbling block for the participants. The user had to decide where he or she thought the information was located and was incorrect several times. We recommend performing a card sorting exercise on the labels that are already defined, because the information that is present in the site is accurate, it just lacks proper organization. • Relocate the graduate assistantship information. This task was very insightful from both the current and prospective student perspective. This information may warrant multiple locations within the site. • More prominent display for end of study requirements. The Master’s Project information was located by two different methods. When it was located, it was not where the prospective student would have looked, under forms. • Improve the accuracy of the search. Improving the search will elicit a quick fix. The new search will allow the sites user population to find information while they become accommodated to any heuristic alterations. Page 9
Raw Data: Each task and user is broken down into significant actions. The actions are then grouped into navigation paths, with each path illustrated below the table. User 1: Current IAKM student User 1/Task 1: Illustration User 1/Task 1 Begin Task Looks at side menu 1.1 Clicks on Programs 1.2 Clicks on Current Students Clicks on Employment Opportunities 1.3 Clicks on FAQs Advising 1.4 Tries Search Found target Total task time User 1/Task 1 Time :35 1:03 1:22 1:46 2:57 3:27 2:53 IAKM Home Page Side Menu 1.4 Search Target 1.1 Programs 1.2 Current Students 1.3 FAQs Employment Opportunities Advising Page 10
User 1/Task 2: Illustration User 1/Task 2 none Clicked on Courses and found target 4:30 :05 User 1/Task 3: Illustration User 1/Task 3 3.1 3.2 Looks at side menu Looks at concentrations in center of page and clicks Found target User 1/Task 3 5:05 5:22 5:30 :25 IAKM Home Page 3.1 Side Menu 3,2 UXD Concentration Master’s Program link top center of home page Target Page 11
User 1/Task4: Illustration User 1/Task 4 Begin Task 4.1 Clicks on Courses Looks at side menu. Subject seems lost and says, “why don’t they have a button labeled forms?” 4.2 Goes to search Clicks on forms Found target User 1/Task 4 Time 6:20 6:30 6:50 7:20 7:35 7:50 1:30 IAKM Home Page Side Menu 4.1 Courses 4.2 Search Target Page 12
User 2: Current IAKM student User 2/Task 1: User 2/Task 1 Illustration User 2/Task 1 Begin Task 1.1 Looked at side menu Clicked on Prospective Students Looked at sub-menu Clicked on Financial Aid Found target Total time Time :17 :29 :36 :38 :41 :24 IAKM Home Page 1.1 Side Menu Prospective Students Financial Aid Target Page 13
User 2/Task 2: Illustration User 2/Task 2 Begin Task 2.1 Looked at side menu Clicked on Courses Clicked on Schedules Found target Total time User 2/Task 2 Time 1:26 1:30 1:43 :17 IAKM Home Page 2.1 Side Menu Courses Schedules Target Page 14
User 2/ Task 3 Illustration User 2/Task 3 Begin Task Looked at side menu 3.1 Clicked on Programs 3.2 Clicked on Courses 3.3 Clicked on Current Student Clicked on Forms Found Target Total time User 2/Task 3 Time 2:30 2:35 2:37 2:46 3:01 3:18 :48 IAKM Home Page Side Menu 3.1 Programs 3.2 Courses 3.3 Current Students Forms Target Page 15
User 2/ Task 4: User 2/Task 4 Illustration User 2/Task 4 Begin Task Looked at side menu 3.1 Clicked on Current Students 3.2 Clicked on Forms Found Target Total time Time 4:10 4:12 4:17 :07 IAKM Home Page 1.1 Side Menu Current Students Forms Target Page 16
User 3: Prospective IAKM student User 3/Task 1: Illustration User 3/Task 1 Begin Task Looked at center of page 3.1 Clicked on KM Certificate Found Target Total time User 3/Task 1 Time 5:04 5:07 5:15 :11 IAKM Home Page 1.1 KM Concentration Certificate Program link top center of home page Target Page 17
User 3/Task 2: Illustration User 3/Task 2 Begin Task Looked at center of page 2.1 Clicked on UXD Master’s link 2.2 Looked at side menu Clicked on Apply Now Found target Total time Time 5:45 5:48 5:55 6:00 :15 User 2/Task 3 IAKM Home Page 2.1 UXD Certificate Program link top center of home page 2.2 Apply Now Target Page 18
User 3/Task 3: Illustration User 3/Task 3 Begin Task Looked at side menu 3.1 Clicked on Current Students Time 7:02 3.2 7:49 Clicked on Employment Opportunities Looked at Side Menu Clicked FAQs Clicked on Financial Aid Found target Total time User 2/Task 3 7:15 7:34 7:52 :50 IAKM Home Page Side Menu 3.1 Current Students Employment Opportunities 3.2 FAQs Financial Aid Target Page 19
User 4: Prospective IAKM student User 4/Task 1: Illustration User 4/Task 1 Begin Task Looked at side menu 1.1 Clicked on Courses Time 4:25 4:29 4:40 1.2 1.3 1.4 5:05 User 4/Task 1 Clicked on Catalog Clicked on Prospective Students Programs Looks at concentrations in center of page and clicks on KM Found target Total time 4:50 :40 IAKM Home Page Side Menu 1.1 Courses Catalog 1.2 Prospective Students 1.2 Prospective Students 1.3 KM Concntration Certificate Program link top center of home page Target Page 20
User 4/Task 2: Illustration User 4/Task 2 Begin Task 2.1 In Courses sub menu Clicked on Curriculum in page Found target Total time User 4/Task 2 Time 5:55 6:03 :08 IAKM Home Page Side Menu 2.1 Courses Clicked on KM curriculum on page Target Page 21
User 4/Task 3: Illustration User 4/Task 3 Begin Task Looked at side menu 3.1 Clicked Alumni and Professionals Time 3.3 9:14 9:23 9:32 1:42 3.2 Clicked on Current Students Clicked on Employment Opportunities Clicked on Prospective Students Clicked on Financial Aid Found target Total time User 4/Task 3 7:50 7:58 8:17 8:19 IAKM Home Page Side Menu 3.1 Alumni and Professionals 3.3 Prospective Students 3.2 Current Students Forms Employment Opportunities Target Page 22
Matthew Rames. Usability Test for UnionStation.org Test Preparation If you don’t know what a usability test is it’s a technique used in user centered ...
Proceedings of HCI International 2005, Lawrence Erlbaum. Guidelines and Standards for Web Usability Nigel Bevan Serco Usability Services 22 Hand Court
Usability is the ease of use and ... Instead of creating the complete final ... The types of usability prototypes may vary from using paper models ...
2 Executive Summary Summary of Work Performed I worked with Tanya Payne from Vignette Inc. to conduct the user tests on Vignette’s new publish process ...
View Test Prep - Usability Final Paper.docx from ENGL 421 at Purdue. USABILITY TESTING OF Airbnb ANDROID APPLICATION Report by: Wilkens, Alex Iwakami,
View Test Prep - Usability Final Paper EDIT.docx from ENGL 421 at Purdue. Table of Contents USABILTY TESTING OF AIRBNB ANDROID APPLICATION Report
Summary: How to define usability? How, when, and where to improve it? Why should you care? Overview defines key usability concepts and answers basic ...
1 Integrating EDEN and Knowledge-Based Active Help Systems Roberto Silveira Silva Filho Department of Information and Computer Science University of ...
Summary: This paper discusses the iterative, cyclical design process, including the four tenets of user-centered design, two types of the product design ...