Published on February 5, 2014
User research: a balanced approach A quick guide to the pros and cons of different research methods and how to combine them to elicit valid user requirements January 2014 www.thinkstrait.com ThinkStrait Ltd 145-157 St John Street London, EC1V 4P United Kingdom firstname.lastname@example.org +44 (0)20 8798 0696
User research: a balanced approach January 2014 The importance of different angles No product is successful if it doesn’t meet the needs of its users at a price they are willing to pay. That sounds simple enough. However, finding out what users want is anything but. Although there are a lot of ways to elicit user requirements, none gives you the full picture. Take a look at this cup of coffee. A familiar and welcome sight for most on Monday morning, surely. Now focus on the mug. Could you find an exact copy of it based on this picture alone? Probably not. However, if you were shown several “mug shots” –who could resist the pun?- taken at different angles then you just might. The same holds for understanding users: to get the full picture you have to combine different methods. The idea of combining different methods is not new in user research, but the implementation is often bungled. Modern Portfolio Theory (Markowitz, 1952) offers a useful lesson here. Prudent financial investors build a portfolio of investments to avoid the risk that comes with putting all their eggs in one basket. To make this work the performance of one investment needs to be largely independent from other investments in the portfolio. Without such diversification –for instance if all your investments depend on sunny weather to pay off- you might be taking on more risk than you think. It’s this rule that is often ignored in user requiremens gathering: a lot of pictures are taken but mostly from the same angle. So which methods should you combine to get a balanced portfolio? In short, methods with complementary characteristics. To simplify the selection process it’s helpful to use a classification structure like the 2X2 matrix on the right. The two dimensions of the matrix are 1) whether the method is focused on the problem or on features of the solution, and 2) whether the method uses an open ended or closed style in much the same way as exam questions can be open or closed. 2
User research: a balanced approach January 2014 Once you’ve arrived at a good mix of the methods from the different quadrants –watch out with quadrant 1 however for the reasons mentioned below- you can start adding methods from the same quadrant. This will generally protect you from putting all your eggs in one basket. Thus, inter-quadrant first, then intra-quadrant to finetune. The 4 quadrants will be discussed in more detail below. Quadrant 1: Solution focus & Open question Most companies and designers have learnt –some the hard way- that they can’t just rely on asking users what they need. This approach, which includes focus groups and interviews, suffers from at least four problems. Firstly, there are two kinds of user requirements: manifest needs and latent ones. In the case of latent requirements the user him/herself is not aware of the actual needs or at least not able to communicate them. As a result, you might fail to pick up on all the needs. Secondly, when you ask users what they need, more often than not you’ll hear what they want. Put this question to a group of 5-year-olds, and chances are that you’ll hear more answers related to candy than to vegetables. Wanting and needing are two different concepts, and not only for children. Thirdly, as Gustafsson et al. (2012) put it,“customers create solutions based on their previous experiences of usage of different products or services… Radical solutions can often be considered unthinkable in advance… but customers know a good idea when they see and use it.” That is, users are “anchored” to what they know and are not always able to break away from the available solutions, especially if there is no tangible design to guide them. As a result, you’ll get more incremental improvements than truly new insights. The development of Sony’s Walkman is often mentioned as an example of the limited use of asking customers when it comes to more radical innovations. Finally, whenever you ask questions to a user there’s a chance that the social desirability bias sneaks in, i.e. a tendency of respondents to answer questions in a way that makes them look good. Throw in a social setting, e.g. a focus group, and you have the possibility of confirmy bias raising its head. The bias can be quite strong as Asch (1956) showed in his classic experiment with different line sizes and can make user groups appear more homegenous than they are in reality. 3
User research: a balanced approach January 2014 Quadrant 2: Solution focus & C losed question One way of overcoming the issues associated with the first quadrant is by getting users to react to a design. This can take the form of presenting a list of design features or letting users interact with a prototype. The problem with using a list and asking “would you like this” is that it could lead to a lot of False Positives. That is, users might indicate that a feature is absolutely necessary but don’t go near it in the final product. Norman (1988) refers to the tendency to include more and more and as “creeping featurism”, and points out that more isn’t always better (1998). Conjoint analysis and similar approaches also fall into this quadrant but suffer less from the risk of producing “creeping featurism” as they actively gauge how important each feature is compared to others. Interacting with prototypes enables users to understand the system and form expectations about its behaviour. When these expectations are not met, the disconnect can point to a need, even if it had been latent till now. Caroll et al. (1991) point out that deploying several of these Task-artifact cycles may give the designer a clear picture of the user requirements. For most people, the word “prototype” is associated with sophisticated technology and polished design, the type you expect in concept cars. However, you can make prototypes as “lo-fi” as you want, using techniques like paper prototyping and “Wizard of Oz” simulations. In fact, there are good reasons to go this route. For starters, hi-fi prototypes can unintentionally convey that the design is as good as finished. This can make users hesitant to criticize it because they feel that they can no longer influence the design. In addition, a hi-fi prototype is more risky since it requires a large investment that may have to be discarded completely. So don’t throw away those post-it notes just yet. Two risks hoover over this quadrant. Firstly, a designer may not anticipate certain requirements and leave them out of the prototype or feature list. If the user doesn’t mention them, they won’t make it to the final design, even though they may turn out to be important requirements (i.e. risk of False Negatives). Although this is less of a concern when prototypes are used than with feature lists and conjoint analyses there is still a risk as users might be so absorbed by what the prototype offers that they fail to notice what it doesn’t. Secondly, when prototypes are used in an artifical setting, e.g. a usability lab, you might find that the findings don’t match up to what users do in real life. One way to balance this out is to go beyond the lab and conduct user tests with prototypes in the field although this usually requires a hi-fi prototype. 4
User research: a balanced approach January 2014 3. Problem focus & Open question To balance out the disadvantages of quadrant 2 you can choose to observe and/or interact with users in their natural environment. This doesn’t only give you insight into the problems they face, but also in the ways they go about solving them, and the context in which they do so. This approach, which includes design ethnography and contextual inquiry, has become increasingly popular over the last two decades. However, some caution is advised. First of all, interpreting the observations and translating them into design requirements might not be all that easy. As a result, the benefits of this approach depend to a large extent on the skills of the designer. A particular risk with this approach is stopping short of the actual root cause of the behaviour observed. Consider the following example about the redesign of an office floor. During your design ethnography studies you observe that your subject leaves her desk to get water between 2 and 5 times a day. The watercooler is on the other side of the building and when asked your subject says she would like to have the watercooler closerby. So, in the new design you put several watercoolers at different points in the office. Problem solved. Until you hear that nobody uses them. It’s only after diving deeper that you discover that the original watercooler was used as a central meeting point. To reduce the risk of falling into this trap you can start by using the “5 WHYs” that underpinned the Toyota Production System and earned a place in Kaizen, Lean and Six Sigma. The second disadvantage of this quadrant is that it can be resource-intensive. It’s likely to require traveling to the customer site and spending hours –a lot longer in most casescollecting data, only to spend a good amount on interpreting the results as well. Thirdly, because the method is so resource-intensive the number of subjects is usually quite small which means that the results from a contextual inquiry may be inadequate for conducting statistical inference. Although this doesn’t have to be the case you should keep in mind that your subject might not have been representative of the user population. 5
User research: a balanced approach January 2014 4. Problem focus & C losed question Surveys are a relatively resource-efficient way of gathering data and can add a quantitative component to the qualitative insights that quadrant 3 provides. This angle can protect you from being sidetracked by non-representative users (i.e. outliers). That said, there are some pitfalls to keep in mind when deciding on your survey design and how your surveys will be administered. Firstly, as was the case with the second quadrant, what is not included will be ignored. That is, if a question is not asked you won’t get an answer to it. Secondly, how participants answer a question depends to a large extent on how the question is phrased. Tversky and Kahneman (1981) clearly showed this framing bias by asking participants which type of treatment they would choose for a group of 600 people with a deadly disease. If treatment A was presented as “saves 200 lives” then 72% of participants preferred this option. This dropped to 22% if it was framed as “400 people will die”. The same effect of the treatment only worded differently rendered radically different answers. Thirdly, whenever a question touches on social norms and expectations you might bump into a social desirability bias (see quadrant 1). This effect can be exacerbated by the characteristics of the people involved especially if the results are not anonymous, e.g. a female researcher handing out surveys about gender equality to a group of men. When anonymity isn’t feasible you should consider using so called “lie scales”, which were introduced over 5 decades ago (Crowne & Marlowe, 1960). These scales measure replies to items with a fairly obvious socially desirable content, like “I have never stolen a thing in my life, not even a hairpin‟ allowing you to either control for this bias or at least interpret the rest of the answers with caution. Picture by Dan Piraro @ Bizarro.com 6
User research: a balanced approach January 2014 So, what now? The table below summarizes some of the pros and cons of the quadrants described above. If you’re about to start a new design project you could do a lot worse than using this table as a first step in building your balanced user research portfolio. However, before you dive in it’s worth noting that it’s not only about which methods you use but also when you use them. The most efficient strategy is probably to cast your net wide at first and then to gradually zoom in on the solution. Starting with quadrant 3 (e.g. observations), and working your way to quadarant 2 (prototypes) via quadrant 4 (surveys) makes more sense than the other way around. Missing an important requirements can make the difference between a successful design and one that fails in the market. So even if you’re already well underway it wouldn’t hurt to double check that you haven’t forgotten an essential angle. After all, who anticipated the mug at the beginning of this paper to look like this? 7
User research: a balanced approach January 2014 References Asch, S. E. (1956). Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs, 70. Carroll, J. M., Kellogg, W. A., and Rosson, M.B. (1991). The Task-Artifact Cycle. In: Carroll, John M. (ed.). Designing Interaction: Psychology at the Human-Computer Interface. Cambridge, UK: Cambridge University Press. Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354. Gustafsson, A., Kristensson, P., & Witell, L.. (2012). Customer co-creation in service innovation: a matter of communication? Journal of Service Management; 23 (3): 311-327 Markowitz, H. M. (1952). Portfolio Selection. In: Journal of Finance. 7, 1952, ISSN 00221082, S. 77–91. Norman, D.A. (1998). The invisible computer: why good products can fail, the personal computer is so complex, and information appliances are the solution. Cambridge, MA: The MIT Press. Norman, D.A. (1988). The psychology of everyday things. New York: Basic Books. Tversky, A., & Kahneman, D. (1981). The Framing of decisions and the psychology of choice". Science 211 (4481): p.453–458. Wixon, D., Holtzblatt, K., & Knox, S. (1990). Contextual Design: An Emergent View of System Design, in Proceedings of CHI ‘90: Conference of Human Factors in Computing Systems. Seattle, WA. 8
Canvas Prints at Affordable Prices make you smile.Visit http://www.shopcanvasprint...
30 Días en Bici en Gijón organiza un recorrido por los comercios históricos de la ...
Con el fin de conocer mejor el rol que juega internet en el proceso de compra en E...
With three established projects across the country and seven more in the pipeline,...
Retailing is not a rocket science, neither it's walk-in-the-park. In this presenta...
Jankowski, J., Balanced Approach to the Design of Conversion Oriented Websites with Limited Negative Impact on the Users, Computational Collective ...
A Balanced Approach to Capturing User Requirements in Business- to- Consumer Web Information Systems
Customer research: a balanced approach. by onno-romijn. on Dec 05, 2014. Report Category: Business
In a balanced approach to literacy instruction, teachers integrate instruction with authentic reading and writing and experiences so that
synthesise larger quantities of research approach needed ... Balanced Approach Needed For Students With Learning ... half of the balanced approaches ...
Evidence-Based User Experience Research, Training, and Consulting. ... When Is Image-Focused Design Appropriate? ... A Balanced Approach Works Best for ...
There are so many potential approaches that it’s hard to know which to use ... we’ve covered the need for a balanced set of user research tools ...
Balanced Approach Articles, experts, jobs, and more: get all the professional insights you need on LinkedIn
The table below summarizes these goals and lists typical research approaches and ... Here’s a short description of the user research methods shown ...