Published on March 14, 2014
THE FINAL DEBATE ON How to Measure Usability & User Experience
2 Our Speakers Lee Cooper Head of UX Research, UserZoom Limited UK Jeﬀ Sauro Founding Principal, measuringusability.com Tim Bosenick Managing Director, GfK Sirvaluse Consulting Introduction by: Javier Darriba Co Founder & Co CEO, UserZoom #uzwebinar
3 Quick Housekeeping • Chat box is available, if you have any questions • Or tweet your questions using #uzwebinar • There will be time for Q&A at the end of the webinar • We will be recording the webinar for future viewing • All attendees will receive a copy of the slides/ recording #uzwebinar
4 About UserZoom #1 Customer and User Experience Research and Analytics Platform v Remote Usability Testing v Voice of the Customer Studies v Information architecture research tools: v Card Sorting v Tree Testing v Click testing v Survey Tool Our products Our holistic approach Listen Measure Research Test & act
5 Highlighted Customers #uzwebinar
6 Agenda • UX measurement: Performance metrics Vs Satisfaction metrics – 5 mins • 3 Ways of measuring the subjective part of UX 1. Scale usability system (SuS) – 10 mins 2. Supr – Q – 10 mins 3. UX Score – 10 mins • Q&A (10 min)
7 User Experience Measurement Performance Metrics Satisfaction Metrics ? - Success rate - Number of clicks - Time on task - … - SuS - Supr-Q - UX Score - NPS #uzwebinar
8 Measuring Usability & UX: SuS Lee Cooper @userzoom_uk
9 • What is it? • Where did it come from? • Why do I like it? SUS #uzwebinar
10 • It’s a quesAonnaire consisAng of ten statements. • Each has a ﬁve-‐point scale that ranges from Strongly Disagree to Strongly Agree. • There are ﬁve posiAve statements and ﬁve negaAve statements, which alternate. • It yields a single usability score on a scale of 0-‐100. #uzwebinar What is it?
11 • John Brooke 1986, Digital Equipment Co. • “To provide us with a measure of people’s subjecAve percepAons of the usability of a system” • “To allow us to do so in the very short Ame available to us during an evaluaAon session.” • “It was probably something that could be used by other organizaAons…” John Brooke. SUS: A Retrospective (2013) #uzwebinar Where did it come from?
12 • It’s Quick & Not-‐So-‐Dirty • It’s Technology AgnosAc • It’s Free • It Works With Small Sample Sizes • It’s Not Perfect, but… • It’s Old #uzwebinar Why do I like it?
13 • It’s reliable – It has been shown to be more reliable than many homegrown and commercially available quesAonnaires. • It’s valid – It has proven to be eﬀecAve at disAnguishing between usable and unusable systems at least as well as, and o`en beaer than, other quesAonnaires. – It also correlates highly with other quesAonnaire-‐based measurements of usability. Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar It’s Quick & Not-So-Dirty
14 Microwave Bank: NCR Knowledge Lab Symbian: UIQ/UIP Smartphones 3: 3G Services Nokia: Smartphones It’s Technology Agnostic
15 #uzwebinar It’s Free
16 • It works with small and large sample sizes. • It allows you to measure the perceived usability of a system with a small sample and be conﬁdent that you’ve got a good assessment. Tullis & Stetson. A Comparison of Questionnaires for Assessing Website Usability (2004) #uzwebinar It Works With Small Sample Sizes
17 • It’s not diagnosAc – It does not tell you what makes a system usable or not. • It someAmes confuses – Its scores are not percentages, despite returning a value between 0 and 100. • Its alternaAng items might be problemaAc – It can lead to responding and scoring errors. Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) John Brooke. SUS: A Retrospective (2013) #uzwebinar It’s Not Perfect, but…
18 • There's a wealth of informaAon about its use along with a body of normaAve data. • It has stood the test of Ame… Bangor, Kortum, and Miller. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale,”)(2009) Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar It’s Old
19 Measuring Usability & UX: Supr-Q Jeﬀ Sauro @MeasuringU
Usability & Satisfaction SUPR-Q Jeff Sauro | Measuring Usability LLC 20
21 Measuring Usability LLC #uzwebinar ISO 9241 pt 11 Deﬁnition of Usability
22 Usability + Credibility & Trust + Appearance + Loyalty 75 Candidate Items Measuring Usability LLC #uzwebinar What makes an Eﬀective Website?
23 Usability This website is easy to use. It is easy to navigate within the website. Credibility (Trust, Value & Comfort) This website keeps the promises it makes. I feel confident conducting business with this website. Loyalty How likely are you to recommend this website to a friend or colleague? I will likely visit this website in the future. Appearance I found the website to be attractive. The website has a clean and simple presentation. NPS Explains 94% of SUS Standardized User Experience Percentile Rank-Questionnaire Measuring Usability LLC #uzwebinar SUPR-Q Items & Factors
24 Retail Amazon eBay Zappos Apple Target JC Pennys Wal-Mart Hermes Boston Proper Container Store Crate & Barrel Crumpler Oriental Trading Sears LL-Bean Michaels Payless Shoes Pier 1 Imports Macys Netflix BestBuy OfficeMax OfficeDepot Airlines United Airlines Frontier Airlines American Airlines Southwest Jet Blue Delta Frontier 3rd Party Travel Travelocity Expedia Orbitz Kayak Government Illinois Colorado California CDC USA.gov New York State Cell-Phone Carriers AT&T Wireless Verizon Sprint T-Mobile Floral Service FTD 1800 Flowers ProFlowers 3rd Party Automotive Edmunds Kelly Blue Book Cars Autotrader Financial Services PayPal Fidelity TD Ameritrade Vanguard Bellco US Bank Chase eTrade Scottrade Wells-Fargo News & Information Wall Street Journal CNN FoxNews New York Times IMDB Yelp Craigslist Motley Fool Yahoo Social Networking Facebook Twitter LinkedIn Flickr Relationship/Dating Match.com eHarmony PlentyofFish.com Hotels Hyatt Hilton 200+ Websites Total, 10,000 responses (30 to 200 per website)Measuring Usability LLC #uzwebinar Selection of Websites in SUPR-Q Database
25 A score of 50% = average score Measuring Usability LLC #uzwebinar Normalized database
26 Measuring Usability LLC #uzwebinar Scoring
27 30% 41% 92% 38% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 47% SUPR-Q Health Care Industry
28 Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Aetna 30% 22% 35% 33% 34% BCBS 41% 26% 51% 45% 33% Kaiser 92% 70% 91% 98% 79% United Healthcare 38% 25% 48% 39% 33% All Healthcare 47% 31% 55% 52% 40% Overall, scores for the industry are a bit below average, with Kaiser being a clear leader, especially in trust and loyalty Healthcare Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores
29 85%96%66% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 82% Measuring Usability LLC #uzwebinar SUPR-Q Retail Banking
30 Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Bank of America 66% 67% 70% 54% 52% Chase Bank 96% 97% 93% 94% 83% Wells Fargo 85% 88% 81% 84% 66% All Banking 82% 84% 81% 77% 67% Overall, scores for the industry are above average, with Chase Bank leading the pack, especially in usability and appearance Retail Banking Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores
31 Feb 2011 Oct 2011 NPS 73% -‐7% Credibility 99.8% 62% Usability 99.7% 73% Appearance 99.7% 80% Measuring Usability LLC #uzwebinar Netﬂix Case Study
32 Measuring Usability LLC is a usability and quanAtaAve research ﬁrm based in Denver, Colorado USA focusing on the staAsAcal analysis of human behavior and quanAfying the user experience for Fortune 1000 companies. www.MeasuringUsability.com Twitter @MeasuringU August 20-22nd 2014 Denver, CO Sponsored by UserZoom DenverUX.com Measuring Usability LLC #uzwebinar Measuring Usability LLC
33 Measuring Usability & UX: UX Score Tim Bosenick @GfK_UX
34 Development 2011-2014 GfK UX Score #uzwebinar© GfK 2014 | UX Score Development 34
In 2011, GfK SirValUse and the GfK Verein conducted the basic research for the UX Score metric. Desk research Operationalisation & Design Draft model of User Experience Reduction to a core item set Eight different test objects 50 face-to-face interviews each Final examination of the model Four different test objects 400 face-to-face interviews Three qualities: task-oriented, self- oriented, aesthetic Standardised instrument for the measurement of UX Validation study I Validation study IIPreparation Model of UX #uzwebinar© GfK 2014 | UX Score Development 35
Task-oriented qualities Learnability / Operability Self-oriented qualities Product Fit / Inspiration Aesthetic qualities Look & Feel Task-oriented qualities refer to the operation and learnability of the way in which the product works. User-oriented qualities refer to feelings of belonging and the stimulation of the user that the product engenders. Product-related aspects refer to the features of the product such as how it looks, feels and sounds. These are the three dimensions of UX that we found and validated with the ﬁrst and the second validation study in 2011. #uzwebinar© GfK 2014 | UX Score Development 36
In 2013, we conducted a third validation study in Germany, US and in China. Collection of all relevant metrics to put in our questionnaire, like: SUS, NPS, AttraktDiff, Price Challenger Creation of item substitutes for the 'difficult' items. The first part was the validation study in Germany. We conducted 100 face-to-face interviews of 60 minutes in our labs in Hamburg. After the analysis of the German data confirmed the UX Score metric, User Centric and GfK China conducted the identical study in their countries. After all the lab sessions had been conducted, we started with the online validation. The questionnaire was much shorter than the in the lab (see session structure). Validation in Germany Validation in the US and China Preparation Online Validation (2014) #uzwebinar© GfK 2014 | UX Score Development 37
Test Stimuli with a standardized setup for all three countries iPad 4 Samsung Galaxy Note 10.1 #uzwebinar© GfK 2014 | UX Score Development 38
• Introduction • Price Challenger pre-usage • Device 1 (tablets were shown in rotated order) • Tryout part for around 10 minutes • Brand items (including NPS) • UX Score • SUS • AttraktDiff • Device 2 (repeat the tryout and the questions for the second device) • Price Challenger post-usage Session Structure #uzwebinar© GfK 2014 | UX Score Development 39
UX Score: Overall results The UX Score differentiates well between the devices in all three countries. UX Score Learnability Operability Product fit Stimulation Look and feel 5.4 4.1 4.4 4.4 5.0 4.6 USA (N=101) China (N=103)Germany (N=100) 4.6 4.0 4.3 3.9 4.4 4.2 5.3 4.0 4.3 4.5 5.2 4.6 5.2 4.4 4.7 4.6 5.0 4.8 5.2 4.8 4.8 4.9 5.1 5.0 4.5 4.2 4.1 4.3 4.8 4.4 iPad Samsung iPad Samsung iPad Samsung * * * * * * * * * * * * * * * * * * * * * * * * * Significant difference between the groups (significance level: 5%) #uzwebinar© GfK 2014 | UX Score Development 40
UX Score: Factor analysis DE USA CN 1 2 3 1 2 3 1 2 3 LE I understand the terms and icons used by my tablet 0,819 0,145 0,183 0,864 0,023 0,202 0,845 0,251 0,175 LE I can learn how to use my tablet without a great deal of thought 0,803 0,147 0,114 0,755 0,257 0,166 0,839 0,302 0,136 OP When I use my tablet, it always does what I want it to do 0,773 0,312 0,217 0,651 0,523 0,065 0,426 0,761 0,232 OP Features and informaAon are always where I expect them to be 0,824 0,284 0,217 0,770 0,290 0,239 0,455 0,485 0,466 ST Using this tablet is inspiring 0,358 0,771 0,206 0,161 0,790 0,296 0,239 0,726 0,391 ST The tablet has interesAng new features 0,065 0,847 0,169 0,075 0,844 0,072 0,225 0,829 0,189 PF The tablet is right for me 0,459 0,568 0,363 0,410 0,703 0,298 0,420 0,672 0,428 PF The tablet has exactly the features I need 0,450 0,586 0,281 0,379 0,733 0,166 0,354 0,720 0,377 LF I like the look of the tablet 0,215 0,304 0,830 0,120 0,439 0,751 0,150 0,332 0,887 LF The tablet looks like it is a high-‐quality product 0,213 0,189 0,888 0,335 0,091 0,835 0,259 0,641 0,509 Total explained variance 74,5% 74,0% 78,6% Rotated component matrix. Extraction method: principle component analysis (Varimax with Kaiser normalization) • The item set performs very well in all three countries. • The explained variance of the UX Score metric is high. • The factor structure can also be reproduced in all three countries. However, the factor loadings for three items are in China not as clear as they are in the other two countries. 41
Brand metrics: NPS Promoters Passive Detractors 0 1 2 3 4 5 6 7 8 9 10 NPS (Net Promoter Score) = promoters - detractors 47% 19% 29% 36% 45% 45% 36% 48% 25% 26% 38% 26% 18% 33% 47% 39% 18% 30% -15 -27 3 18 14 -31 iPad Samsun g Germany USA China iPad Samsun g iPad Samsung The NPS Score is different in all three countries. While the German and Chinese participants would rather recommend the iPad, the US participants are more into the Samsung Galaxy Note. * Significant difference Lab (level: 5%) ** Significant difference Online (level: 5%) © GfK 2014 | UX Score Development 42
43© GfK 2014 | UX Score Development Explanation of the NPS by other global scales None of the other metrics is able to explain as much of the NPS as the UX Score. R² Dependent NPS DE USA CN UX Score 0,517 0,584 0,565 SUS 0,322 0,422 0,308 AttrakDiff 0,228 0,476 0,356 Dependent: NPS The diagram shows to which extent the Net Promotor Score (NPS) is explained by other global measures. An adjusted R² value of .50 and better can be interpreted as a relevant connection. #uzwebinar 43
44© GfK 2014 | UX Score Development Explanation of other global scales by the UX Score Linear regression. Adjusted R² is diagrammed. The diagram shows to which extent the UX Score explains other global measures. An adjusted R² value of over .50 can be interpreted as a relevant connection. R² Predictor UX Score DE USA CN SUS 0,704 0,545 0,521 AttraktDiff 0,547 0,536 0,514 Intention to buy the brand 0,435 0,566 0,606 Intention to buy the product 0,457 0,572 0,636 Suitability of the brand 0,499 0,585 0,569 Positive feeling about the brand 0,469 0,460 0,568 Predictor: UX Score The UX Score proved to be a valueable predictor for many global measures, like NPS, SUS etc. In all three countries, the UX Score explaines the other global measures to a high extend. #uzwebinar 44
45© GfK 2014 | UX Score Development % Market Share // Results of the Price Challenger This table shows the estimated market share from the Price Challenger tool. It shows the market share of both devices in relation to the height of the UX Score. The results clearly indicate that a good user experience is important for a high market share. The value of the UX Score can be used as a predictor for the market share. The higher the score, the higher the market share. Germany USA China iPad Samsung iPad Samsung iPad Samsung UX Score 0 – 3,9 points 4% 40% 2% 46% 41% 13% 4,0 – 4,4 points 10% 60% 17% 61% 57% 27% 4,5 – 4,9 points 25% 87% 21% 85% 59% 43% 5 – 6 points 51% 92% 39% 86% 72% 57% #uzwebinar 45
www.linkedin.com/company/userzoom @UserZoom_UK www.userzoom.co.uk Contact us: Cheshire (UK) 50a Alderley Road Wilmslow, Cheshire SK9 1NT Phone: +44 (0) 1625 525 650 Any question?
... How UX Research & Usability Testing Can ... Join this webinar with Jeff Sauro ... Jon Innes of UX Innovation & Jeff Sauro of Measuring Usability ...
UX & Usability resource gold mine, sharing is caring....founder of a ux agency based in Melbourne, I was asked recently asked to put together a workshop ...
View 4241 Ux, Usability posts, presentations, experts, and more. Get the professional knowledge you need on LinkedIn.
Online free webinars in usability and user ... online webinars in usability and UX by leading international experts. Measuring User Efficiency Jeff Sauro ...
... Jon Innes of UX Innovation & Jeff Sauro of Measuring Usability ... Free Webinar: Agile UX ... and Jeff Sauro of Measuring Usability uncover the ...
Measuring User Experience; UX ... Join this webinar with Jeff Sauro ... Best Practices for Benchmarking the Website User Experience with Jeff Sauro.
In this webinar we will explain the ... recognized methods for measuring Usability & User ... How to Measure Usability & UX
An interview with Jeff Sauro about getting stakeholder buy-in for UX research. The interview ... Jeff is a Principal at Measuring Usability ...