Published on March 13, 2014
1122 BBEESSTT PPRRAACCTTIICCEESS FFOORR Successful Remote Unmoderated Usability Testing
2 Unmoderated Remote Usability Testing (URUT) is a research companion that caters to varied research needs.
3 Remote Testing is also known for its economy with time and money in comparison to a lab-based study, hence its growth in popularity and increased adoption in recent years.
4 At UserZoom we have been conducting remote research for over 6 years. We have compiled 12 key tips that will help you make the most out of the URUT methodology, be eﬃcient, avoid pitfalls and gather as much quality data as possible. Here we go.
5 UX & CX Researchers who do not set up their study goals and requirements at the onset, tend to shuﬄe from one remote study type to another: • Deﬁne your study goals well to achieve a tighter study design. • Have a hypothesis you would like to test. • Each question should have a clear purpose. • Have a vision of what your ﬁnal report will look like. Determine your tactical & strategic goals ﬁrst. 1. Define Your Study Goals Well
6 Vague and unclear goals lead to study designs that are not well thought out and consequently lead to less than actionable data. Good To Know
7 Knowing the goal of your study helps you choose the right study type. Each research approach supports a wide range of study goals: o Customer Experience Studies: such as VOC – Voice of the Customer and website feedback surveys. o User Experience Studies: such as task-based Usability Tests, Card Sorting, Tree Testing, Screenshot Click Testing and Timeout Testing. Choose the right approach for the digital touch point you want to test. 2. Select Your Research Approach Live Websites PrototypesMobile Sites Mobile Apps
8 If the goal is to restructure the site’s information architecture, then Card Sorting and Tree Testing along with some survey questions might be the best bet. If you want to compare your site’s performance with competitors, then task-based Usability Testing is the appropriate study type to select. Good to know
10 With Java Script Code, you will have a higher completion rate, but as a UX & CX researcher you rarely have access to the website’s code. A Browser Add-On is simple to execute, but it may reduce the completion rate. Good To Know
11 Invite participants who will help you achieve your study goals. 4. Choose The Right Recruitment Method • PPaanneell:: A panel provider is a third party vendor that specialises in screening and recruitment of participants. • CCuussttoommeerr lliisstt:: If you have a list of customers who have opted in to participate in research or receive communications, these customers can be recruited by email invitations. • IInntteerrcceepptt ffrroomm yyoouurr lliivvee wweebbssiittee:: A percentage of visitors from your website will receive an intercept message and be invited to take part of the study.
12 The biggest beneﬁt of intercepting live users on your webiste is that you can invite actual visitors. There are substantial savings since there are no recruitment costs and incentives may not be necessary. The biggest drawback, however, is to be mindful of disturbing the visitor experience. Intercept studies are best kept simple and short. About Intercept Studies
13 Clearly communicate with your participants from the start. 5. Set Clear Expectations CCrreeaattee aa ppoossiittiivvee ppaarrttiicciippaanntt eexxppeerriieennccee:: Set expectations at the invitation level to prevent surprises & frustration. Inform participants of: o Device requirements. o The time commitment to take the study. o If a browser add-on installation is required. o The number of study tasks. o Notify them about their progress through the study. o Have a few wrap-up questions before completing the study.
14 Let your participants know what they need to do in the study. 6. Create Clear Tasks • WWhhiicchh ttaasskk ddoo tthheeyy nneeeedd ttoo ppeerrffoorrmm?? Send participants on short missions with clear instructions on what they are required to accomplish. • BBrreeaakk aa llaarrggee ttaasskk ddoowwnn iinnttoo ssmmaalllleerr ccoonnccrreettee oonneess.. It will not only ensure that they participants perform each of the tasks well. You will also receive accurate feedback on follow-up questions. • FFoorr BBeenncchhmmaarrkk--SSttuuddiieess -- ccoommppaarree aapppplleess--ttoo--aapppplleess:: if participants are asked to ﬁnd a phone number for online banking on two banking sites, but are asked to ﬁnd the address on the third site because the phone number is not available, the parameters are slightly diﬀerent and not directly comparable.
15 Deﬁne when a participant has successfully completed a task. 7. Determine The Task Success • VVaalliiddaattiioonn QQuueessttiioonn:: To test ﬁndability of information, it is best to conﬁrm by a validation question. Participants are then asked to ﬁnd or remember a relevant piece of information from the website. • VVaalliiddaattiioonn bbyy UURRLL:: When tracking behaviour and clicks, validation by URL may be the right option. Success is based on whether participants reach the designated URL(s). • SSeellff--rreeppoorrtteedd vvaalliiddaattiioonn:: Users indicate whether they felt they succeeded in completing a task. This method of validation is more subjective or is useful when participants are asked to gather information or review content.
16 Ask participants to review the content on a pharmaceutical website: Does the site adequately explain side eﬀect information of a drug? Do participants feel like they are able to gather enough product information to make an informed purchase decisions? Self-Reported Validation – An Example
17 Be succinct and keep the study as fun & interesting as possible 8. Keep Participants Engaged • KKeeeepp iitt sshhoorrtt aanndd ssiimmppllee:: If the task description and study expectation are boring and long-winded, then users are more likely to abandon the task or study before completing it. • HHiigghhlliigghhtt kkeeyy ppaarrttss ooff tthhee ttaasskk ddeessccrriippttiioonn oorr tthhee qquueessttiioonn:: the use of bolding, a diﬀerent color, underlining, italics, bullet points or numbering enables participants to quickly see and understand a question. • IInntteerraacctt wwiitthh yyoouurr uusseerrss:: One nice way to keep users engaged is to communicate directly with them during the study and to acknowledge their eﬀort.
18 The average completion rate for a unmoderated Remote Usability Test is about 75%. Using fun engaging language and making tasks and questions easy to read and easy to understand will increase the completion rate by at least another 10%. About URUT Studies
19 Use quantitative & qualitative question types 9. Create a good mix • UUssee yyoouurr UUXX mmeettrriiccss:: Remote Testing tools capture eﬀectiveness (task completion success) and eﬃciency metrics (time on task) automatically. • AAdddd aa hhuummaann eelleemmeenntt:: IInclude qualitative data such as users´ comments or session videos in your usability report. If it´s all only graphs and statistics it may be Spartan in its tone and tenor. • TTeellll aa ssttoorryy:: Ensure that you ask the right number of questions with a mix of quantitative and qualitative question-types. It will help you create a more compelling story in the report. • BBee pprreeppaarreedd:: Come up with a thorough question list while designing the study. It will make the analysis much easier.
20 Have at least one positive and one negative open- ended question in each task to capture what is working well for participants and what is not. A multiple response question with a list of potential issues encountered is also helpful in drilling down and understanding the problems participants are experiencing on the site. Good To Know
21 Be mindful of participant´s time, and yours 10. Keep An Eye On The Time • BBee aawwaarree ooff tthhee ttiimmiinngg:: A Remote Study should not be more than 20 minutes. A 20-minute study usually entails about 5 tasks that take users ~ 4 minutes each to complete. You may also then add a ﬁnal questionnaire with ~ 10 questions that will take users 2-5 minutes to complete. • SSttaayy ffooccuusseedd:: There is much a UX & CX researcher would like to learn for the amount of time and money invested in a project. Splitting up a study into two or more separate projects might be one option to an ambitious research project. Another option is to have participants perform a subset of tasks rather than all of them.
22 Keep study times short in length to increase the completion rates. You thereby ensure that the data is collected in time. Long studies can lead to participant fatigue compromising the quality and integrity of the data captured. Good To Know
23 Soft-launch your study to a small sample 11. Preview Internally PPrreeppaarree yyoouurr ssttuuddyy llaauunncchh:: • Preview the study to eliminate problems with the spelling, grammar, and logic. • Have another person review the study to help spot grammatical problems and logic issues and to provide feedback on task diﬃculty and the study length. • Conduct a soft-launch or pilot with about 10-20 users, once the study has been thoroughly QAed and tested. This way you identify any problems that may have been missed during the QA and testing process. • Evaluate the responses that come in. If problems are spotted in the soft launch and the study needs to be ﬁxed and re-launched, then only a small number of participant data is lost.
24 Be fair in your incentives.12. Reward Participants LLeett uusseerrss kknnooww yyoouu ccaarree:: • Incentives are a vital part of usability testing projects and it is important to incentivise users fairly for their time and eﬀort. • One important advantage of Remote Testing is that it is cost-eﬀective. While a lab-based study incents anywhere between 50€-100€ per user, a RUT project incents between 10€ and 20€. • Incentives do not necessarily have to be monetary in nature: discount codes, access to exclusive content or information, donation to a charity, or a drawing to win an item are some other possibilities to consider.
25 The incentive amount is dependent on factors like the complexity of the participant proﬁle, study and device type. A mobile study usually has higher completion rates when panel providers introduce higher incentives. For a 15-minute mobile study, you can expect to provide 12€ – 25€ as incentives. Good To Know
www.linkedin.com/company/userzoom @UserZoom_UK www.userzoom.co.uk Contact us: Cheshire (UK) 50a Alderley Road Wilmslow, Cheshire SK9 1NT Phone: +44 (0) 1625 525 650 Any question? firstname.lastname@example.org
Flexible and versatile Remote Unmoderated Usability Testing is a research companion that caters to varied research needs. It is also known for its economy ...
12 Best Practices für ein erfolgreiches Remote User Testing ... als auch zur User Experience (Remote Usability Testing, Card Sorting, Tree Testing, ...
Why is UserZoom’s remote usability testing unique? UserZoom’s remote unmoderated usability testing ... Remote Usability Testing. 12 Best Practices ...
Flexible and versatile Remote Usability Testing is a research companion that caters to varied research needs. Have a look at these 12 key tips that will ...
Remote Usability Tests sind eine ... "Remote Usability Testing ... Diskutieren Sie mit über 40.000 Mitgliedern Best Practices & Trends im Internet ...
12 Best Practices for UX in an ... In the best teams, the UX folks ... I’ve seen many instances of users scheduled for remote usability testing or site ...
12 Best Practices for UX in an Agile ... Jeff shares the remaining best practices he’s uncovered ... test and repair them using remote usability testing.
Remote usability testing ... Remote Usability Tests: Moderated and Unmoderated. by Amy Schade on October 12 , 2013 Topics ...