Webinar with UX Pioneer Jeff Sauro: Measuring UX & Usability

50 %
50 %
Information about Webinar with UX Pioneer Jeff Sauro: Measuring UX & Usability
Technology

Published on March 14, 2014

Author: UserZoom_UK

Source: slideshare.net

Description

In this webinar we will explain the advantages of the 3 most recognized methods for measuring Usability & User Experience of a product or service: SUS (System Usability Scale), Supr – Q and UX Score.
- How do you measure it?
- What do you need to ask users to find out which experience they had?
- Which factors influence their experience?
- What methods allow you to benchmark the experience you offer compared to competitors?

THE FINAL DEBATE ON How to Measure Usability & User Experience

2   Our Speakers Lee Cooper Head of UX Research, UserZoom Limited UK Jeff Sauro Founding Principal, measuringusability.com Tim Bosenick Managing Director, GfK Sirvaluse Consulting Introduction by: Javier Darriba Co Founder & Co CEO, UserZoom #uzwebinar

3   Quick Housekeeping •  Chat box is available, if you have any questions •  Or tweet your questions using #uzwebinar •  There will be time for Q&A at the end of the webinar •  We will be recording the webinar for future viewing •  All attendees will receive a copy of the slides/ recording #uzwebinar

4   About UserZoom #1 Customer and User Experience Research and Analytics Platform v Remote Usability Testing v Voice of the Customer Studies v Information architecture research tools: v Card Sorting v Tree Testing v Click testing v Survey Tool Our products Our holistic approach Listen Measure Research Test & act

5   Highlighted Customers #uzwebinar

6   Agenda •  UX measurement: Performance metrics Vs Satisfaction metrics – 5 mins •  3 Ways of measuring the subjective part of UX 1. Scale usability system (SuS) – 10 mins 2. Supr – Q – 10 mins 3. UX Score – 10 mins •  Q&A (10 min)

7   User Experience Measurement Performance Metrics Satisfaction Metrics ? -  Success rate -  Number of clicks -  Time on task -  … -  SuS -  Supr-Q -  UX Score -  NPS #uzwebinar

8   Measuring Usability & UX: SuS Lee Cooper @userzoom_uk

9   •  What  is  it?   •  Where  did  it  come  from?   •  Why  do  I  like  it?   SUS #uzwebinar

10   •  It’s  a  quesAonnaire   consisAng  of  ten   statements.   •  Each  has  a  five-­‐point  scale   that  ranges  from  Strongly   Disagree  to  Strongly  Agree.   •  There  are  five  posiAve   statements  and  five   negaAve  statements,  which   alternate.   •  It  yields  a  single  usability   score  on  a  scale  of  0-­‐100.   #uzwebinar What is it?

11   •  John  Brooke  1986,  Digital   Equipment  Co.   •  “To  provide  us  with  a   measure  of  people’s   subjecAve  percepAons  of   the  usability  of  a  system”   •  “To  allow  us  to  do  so  in   the  very  short  Ame   available  to  us  during  an   evaluaAon  session.”   •  “It  was  probably  something   that  could  be  used  by  other   organizaAons…”   John Brooke. SUS: A Retrospective (2013) #uzwebinar Where did it come from?

12   •  It’s  Quick  &  Not-­‐So-­‐Dirty   •  It’s  Technology  AgnosAc   •  It’s  Free   •  It  Works  With  Small  Sample  Sizes   •  It’s  Not  Perfect,  but…   •  It’s  Old   #uzwebinar Why do I like it?

13   •  It’s  reliable   –  It  has  been  shown  to  be  more  reliable  than  many  homegrown   and  commercially  available  quesAonnaires.     •  It’s  valid   –  It  has  proven  to  be  effecAve  at  disAnguishing  between  usable   and  unusable  systems  at  least  as  well  as,  and  o`en  beaer  than,   other  quesAonnaires.   –  It  also  correlates  highly  with  other  quesAonnaire-­‐based   measurements  of  usability.   Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar It’s Quick & Not-So-Dirty

14   Microwave Bank: NCR Knowledge Lab Symbian: UIQ/UIP Smartphones 3: 3G Services Nokia: Smartphones It’s Technology Agnostic

15  #uzwebinar It’s Free

16   •  It  works  with  small  and   large  sample  sizes.   •  It  allows  you  to   measure  the  perceived   usability  of  a  system   with  a  small  sample  and   be  confident  that  you’ve   got  a  good  assessment.   Tullis & Stetson. A Comparison of Questionnaires for Assessing Website Usability (2004) #uzwebinar It Works With Small Sample Sizes

17   •  It’s  not  diagnosAc   –  It  does  not  tell  you  what  makes  a  system  usable  or  not.     •  It  someAmes  confuses   –  Its  scores  are  not  percentages,  despite  returning  a  value   between  0  and  100.     •  Its  alternaAng  items  might  be  problemaAc   –  It  can  lead  to  responding  and  scoring  errors.   Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) John Brooke. SUS: A Retrospective (2013) #uzwebinar It’s Not Perfect, but…

18   •  There's  a  wealth  of   informaAon  about   its  use  along  with  a   body  of  normaAve   data.   •  It  has  stood  the   test  of  Ame…   Bangor, Kortum, and Miller. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale,”)(2009) Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar It’s Old

19   Measuring Usability & UX: Supr-Q Jeff Sauro @MeasuringU

Usability & Satisfaction SUPR-Q Jeff Sauro | Measuring Usability LLC 20

21   Measuring Usability LLC #uzwebinar ISO 9241 pt 11 Definition of Usability

22   Usability + Credibility & Trust + Appearance + Loyalty 75 Candidate Items Measuring Usability LLC #uzwebinar What makes an Effective Website?

23   Usability This website is easy to use. It is easy to navigate within the website. Credibility (Trust, Value & Comfort) This website keeps the promises it makes. I feel confident conducting business with this website. Loyalty How likely are you to recommend this website to a friend or colleague? I will likely visit this website in the future. Appearance I found the website to be attractive. The website has a clean and simple presentation. NPS Explains 94% of SUS Standardized User Experience Percentile Rank-Questionnaire Measuring Usability LLC #uzwebinar SUPR-Q Items & Factors

24   Retail Amazon eBay Zappos Apple Target JC Pennys Wal-Mart Hermes Boston Proper Container Store Crate & Barrel Crumpler Oriental Trading Sears LL-Bean Michaels Payless Shoes Pier 1 Imports Macys Netflix BestBuy OfficeMax OfficeDepot Airlines United Airlines Frontier Airlines American Airlines Southwest Jet Blue Delta Frontier 3rd Party Travel Travelocity Expedia Orbitz Kayak Government Illinois Colorado California CDC USA.gov New York State Cell-Phone Carriers AT&T Wireless Verizon Sprint T-Mobile Floral Service FTD 1800 Flowers ProFlowers 3rd Party Automotive Edmunds Kelly Blue Book Cars Autotrader Financial Services PayPal Fidelity TD Ameritrade Vanguard Bellco US Bank Chase eTrade Scottrade Wells-Fargo News & Information Wall Street Journal CNN FoxNews New York Times IMDB Yelp Craigslist Motley Fool Yahoo Social Networking Facebook Twitter LinkedIn Flickr Relationship/Dating Match.com eHarmony PlentyofFish.com Hotels Hyatt Hilton 200+ Websites Total, 10,000 responses (30 to 200 per website)Measuring Usability LLC #uzwebinar Selection of Websites in SUPR-Q Database

25   A score of 50% = average score Measuring Usability LLC #uzwebinar Normalized database

26   Measuring Usability LLC #uzwebinar Scoring

27   30% 41% 92% 38% 0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%   Industry Average = 47% SUPR-Q Health Care Industry

28   Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Aetna 30% 22% 35% 33% 34% BCBS 41% 26% 51% 45% 33% Kaiser 92% 70% 91% 98% 79% United Healthcare 38% 25% 48% 39% 33% All Healthcare 47% 31% 55% 52% 40% Overall, scores for the industry are a bit below average, with Kaiser being a clear leader, especially in trust and loyalty Healthcare Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores

29   85%96%66% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 82% Measuring Usability LLC #uzwebinar SUPR-Q Retail Banking

30   Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Bank of America 66% 67% 70% 54% 52% Chase Bank 96% 97% 93% 94% 83% Wells Fargo 85% 88% 81% 84% 66% All Banking 82% 84% 81% 77% 67% Overall, scores for the industry are above average, with Chase Bank leading the pack, especially in usability and appearance Retail Banking Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores

31    Feb  2011   Oct  2011     NPS    73%    -­‐7%   Credibility    99.8%    62%   Usability    99.7%   73%   Appearance    99.7%    80%   Measuring Usability LLC #uzwebinar Netflix Case Study

32   Measuring  Usability  LLC  is  a  usability  and  quanAtaAve  research  firm  based  in   Denver,  Colorado  USA  focusing  on  the  staAsAcal  analysis  of  human  behavior   and  quanAfying  the  user  experience  for  Fortune  1000  companies.   www.MeasuringUsability.com Twitter @MeasuringU August 20-22nd 2014 Denver, CO Sponsored by UserZoom DenverUX.com Measuring Usability LLC #uzwebinar Measuring Usability LLC

33   Measuring Usability & UX: UX Score Tim Bosenick @GfK_UX

34   Development 2011-2014 GfK UX Score #uzwebinar© GfK 2014 | UX Score Development 34  

In 2011, GfK SirValUse and the GfK Verein conducted the basic research for the UX Score metric. Desk research Operationalisation & Design Draft model of User Experience Reduction to a core item set Eight different test objects 50 face-to-face interviews each Final examination of the model Four different test objects 400 face-to-face interviews Three qualities: task-oriented, self- oriented, aesthetic Standardised instrument for the measurement of UX Validation study I Validation study IIPreparation Model of UX #uzwebinar© GfK 2014 | UX Score Development 35  

Task-oriented qualities Learnability / Operability Self-oriented qualities Product Fit / Inspiration Aesthetic qualities Look & Feel Task-oriented qualities refer to the operation and learnability of the way in which the product works. User-oriented qualities refer to feelings of belonging and the stimulation of the user that the product engenders. Product-related aspects refer to the features of the product such as how it looks, feels and sounds. These are the three dimensions of UX that we found and validated with the first and the second validation study in 2011. #uzwebinar© GfK 2014 | UX Score Development 36  

In 2013, we conducted a third validation study in Germany, US and in China. Collection of all relevant metrics to put in our questionnaire, like: SUS, NPS, AttraktDiff, Price Challenger Creation of item substitutes for the 'difficult' items. The first part was the validation study in Germany. We conducted 100 face-to-face interviews of 60 minutes in our labs in Hamburg. After the analysis of the German data confirmed the UX Score metric, User Centric and GfK China conducted the identical study in their countries. After all the lab sessions had been conducted, we started with the online validation. The questionnaire was much shorter than the in the lab (see session structure). Validation in Germany Validation in the US and China Preparation Online Validation (2014) #uzwebinar© GfK 2014 | UX Score Development 37  

Test Stimuli with a standardized setup for all three countries iPad 4 Samsung Galaxy Note 10.1 #uzwebinar© GfK 2014 | UX Score Development 38  

•  Introduction •  Price Challenger pre-usage •  Device 1 (tablets were shown in rotated order) •  Tryout part for around 10 minutes •  Brand items (including NPS) •  UX Score •  SUS •  AttraktDiff •  Device 2 (repeat the tryout and the questions for the second device) •  Price Challenger post-usage Session Structure #uzwebinar© GfK 2014 | UX Score Development 39  

UX Score: Overall results The UX Score differentiates well between the devices in all three countries. UX Score Learnability Operability Product fit Stimulation Look and feel 5.4   4.1   4.4   4.4   5.0   4.6   USA (N=101) China (N=103)Germany (N=100) 4.6   4.0   4.3   3.9   4.4   4.2   5.3   4.0   4.3   4.5   5.2   4.6   5.2   4.4   4.7   4.6   5.0   4.8   5.2   4.8   4.8   4.9   5.1   5.0   4.5   4.2   4.1   4.3   4.8   4.4   iPad Samsung iPad Samsung iPad Samsung * * * * * * * * * * * * * * * * * * * * * * * * * Significant difference between the groups (significance level: 5%) #uzwebinar© GfK 2014 | UX Score Development 40  

UX Score: Factor analysis   DE USA CN   1 2 3 1 2 3 1 2 3 LE I  understand  the  terms  and  icons  used  by  my  tablet 0,819 0,145 0,183 0,864 0,023 0,202 0,845 0,251 0,175 LE I  can  learn  how  to  use  my  tablet  without  a  great  deal  of   thought 0,803 0,147 0,114 0,755 0,257 0,166 0,839 0,302 0,136 OP When  I  use  my  tablet,  it  always  does  what  I  want  it  to  do 0,773 0,312 0,217 0,651 0,523 0,065 0,426 0,761 0,232 OP Features  and  informaAon  are  always  where  I  expect  them   to  be 0,824 0,284 0,217 0,770 0,290 0,239 0,455 0,485 0,466 ST Using  this  tablet  is  inspiring   0,358 0,771 0,206 0,161 0,790 0,296 0,239 0,726 0,391 ST The  tablet  has  interesAng  new  features   0,065 0,847 0,169 0,075 0,844 0,072 0,225 0,829 0,189 PF The  tablet  is  right  for  me 0,459 0,568 0,363 0,410 0,703 0,298 0,420 0,672 0,428 PF The  tablet  has  exactly  the  features  I  need 0,450 0,586 0,281 0,379 0,733 0,166 0,354 0,720 0,377 LF I  like  the  look  of  the  tablet 0,215 0,304 0,830 0,120 0,439 0,751 0,150 0,332 0,887 LF The  tablet  looks  like  it  is  a  high-­‐quality  product 0,213 0,189 0,888 0,335 0,091 0,835 0,259 0,641 0,509 Total explained variance 74,5% 74,0% 78,6% Rotated component matrix. Extraction method: principle component analysis (Varimax with Kaiser normalization) •  The item set performs very well in all three countries. •  The explained variance of the UX Score metric is high. •  The factor structure can also be reproduced in all three countries. However, the factor loadings for three items are in China not as clear as they are in the other two countries. 41  

Brand metrics: NPS Promoters Passive Detractors 0   1   2   3   4   5   6   7   8   9   10   NPS (Net Promoter Score) = promoters - detractors 47%   19%   29%   36%   45%  45%   36%   48%   25%   26%   38%   26%   18%   33%   47%   39%   18%   30%   -15 -27 3 18 14 -31 iPad Samsun g Germany USA China iPad Samsun g iPad Samsung The NPS Score is different in all three countries. While the German and Chinese participants would rather recommend the iPad, the US participants are more into the Samsung Galaxy Note. * Significant difference Lab (level: 5%) ** Significant difference Online (level: 5%) © GfK 2014 | UX Score Development 42  

43© GfK 2014 | UX Score Development Explanation of the NPS by other global scales None of the other metrics is able to explain as much of the NPS as the UX Score. R² Dependent NPS DE USA CN UX Score 0,517 0,584 0,565 SUS 0,322 0,422 0,308 AttrakDiff 0,228 0,476 0,356 Dependent: NPS The diagram shows to which extent the Net Promotor Score (NPS) is explained by other global measures. An adjusted R² value of .50 and better can be interpreted as a relevant connection. #uzwebinar 43  

44© GfK 2014 | UX Score Development Explanation of other global scales by the UX Score Linear regression. Adjusted R² is diagrammed. The diagram shows to which extent the UX Score explains other global measures. An adjusted R² value of over .50 can be interpreted as a relevant connection. R² Predictor UX Score DE USA CN SUS 0,704 0,545 0,521 AttraktDiff 0,547 0,536 0,514 Intention to buy the brand 0,435 0,566 0,606 Intention to buy the product 0,457 0,572 0,636 Suitability of the brand 0,499 0,585 0,569 Positive feeling about the brand 0,469 0,460 0,568 Predictor: UX Score The UX Score proved to be a valueable predictor for many global measures, like NPS, SUS etc. In all three countries, the UX Score explaines the other global measures to a high extend. #uzwebinar 44  

45© GfK 2014 | UX Score Development % Market Share // Results of the Price Challenger This table shows the estimated market share from the Price Challenger tool. It shows the market share of both devices in relation to the height of the UX Score. The results clearly indicate that a good user experience is important for a high market share. The value of the UX Score can be used as a predictor for the market share. The higher the score, the higher the market share. Germany USA China iPad Samsung iPad Samsung iPad Samsung UX Score 0 – 3,9 points 4% 40% 2% 46% 41% 13% 4,0 – 4,4 points 10% 60% 17% 61% 57% 27% 4,5 – 4,9 points 25% 87% 21% 85% 59% 43% 5 – 6 points 51% 92% 39% 86% 72% 57% #uzwebinar 45  

www.linkedin.com/company/userzoom @UserZoom_UK www.userzoom.co.uk Contact us: Cheshire (UK) 50a Alderley Road Wilmslow, Cheshire SK9 1NT Phone: +44 (0) 1625 525 650 Any question?

#uzwebinar presentations

Add a comment

Related presentations

Related pages

UserZoom | Webinars on UX Research, Usability Testing, CX ...

... How UX Research & Usability Testing Can ... Join this webinar with Jeff Sauro ... Jon Innes of UX Innovation & Jeff Sauro of Measuring Usability ...
Read more

Ux & Usability | LinkedIn

UX & Usability resource gold mine, sharing is caring....founder of a ux agency based in Melbourne, I was asked recently asked to put together a workshop ...
Read more

Ux, Usability | LinkedIn

View 4241 Ux, Usability posts, presentations, experts, and more. Get the professional knowledge you need on LinkedIn.
Read more

Usability Marathon. Jeff Sauro. Measuring User Efficiency

Online free webinars in usability and user ... online webinars in usability and UX by leading international experts. Measuring User Efficiency Jeff Sauro ...
Read more

How to Master UX Testing in an Agile Design Process

... Jon Innes of UX Innovation & Jeff Sauro of Measuring Usability ... Free Webinar: Agile UX ... and Jeff Sauro of Measuring Usability uncover the ...
Read more

UserZoom | User Experience, UX Benchmarking & Usability ...

Measuring User Experience; UX ... Join this webinar with Jeff Sauro ... Best Practices for Benchmarking the Website User Experience with Jeff Sauro.
Read more

How to Measure Usability & UX - YouTube

In this webinar we will explain the ... recognized methods for measuring Usability & User ... How to Measure Usability & UX
Read more

Jeff Sauro It's Our Research interview - YouTube

An interview with Jeff Sauro about getting stakeholder buy-in for UX research. The interview ... Jeff is a Principal at Measuring Usability ...
Read more