General Research Guides

Watching Them Work

What User Observation Can Tell You That Users Can't

We all work hard to understand what our campus users want from the technology and information services we provide. But how do you unearth problems that users don't recognize or can't fully describe? How do you zero in on the underlying cause of complaints like "this is hard to use" or "it's too confusing" or "I can't find what I need"?

An Information Services Forum

Presenters: Paula Lackie, Heather Tompkins, and Matt Ryan

We take a cue from the world of anthropology and watch our users at work in their natural habitat. You may be surprised how much you can learn from observation that you wouldn't have discovered any other way!

More Resources


So, you've got a website, application, device, tool, service, or process. Typically, most of the input into its design came from insiders -- the technologists who created it and the people who commissioned its purchase or development. However, none of these folks represent the typical user for the tool. They have a whole bunch of assumptions about how things should work and how information should be organized that they may not even be aware of. The technology might have made it easier to develop the tool in one way rather than another. The upshot, often, is that the tool reflects more the needs of the people who created or selected it than the needs of the people who will be using it.

This creates a potential problem -- the tool might be fantastically easy to use for the people who commissioned and created it, but have aspects that make it difficult to use or understand by the people for whom it is intended. How, as someone who is inside the process, can you find out what stumbling blocks and other issues real users have with a tool?

"Good systems cannot be built by design experts who proceed with only limited input from users. Even when designers and prospective users have unlimited time for conversation, there are many aspects of a work process--such as how a particular tool is held, or what it is for something to "look right"--that reside in the complex, often tacit, domain of context."

Profile 14. Participatory Design, Profile Authors: Sara Kuhn and Terry Winograd in Bringing Design to Software (c) Addison-Wesley, 1996

The answer is user research.


There are a variety of concepts/design principles that apply, but two common terms used to describe this sort of work are:

  • User-Centered Design
  • Participatory Design

The goal of this work is to understand your users as much as possible, the key principle is to respect and believe your users, and the key requirement to do this sort of research is empathy.


Types of user research include:

  • Usability Testing
  • Observation
  • Card Sorts
  • Focus Groups
  • Interviews
  • Photo Elicitation
  • Self Reports
  • Maps
  • Time Allocation Diaries

Usability Study SketchUsability Testing

  • Simple, but rigorous
  • User is presented with a mockup or real interface, and asked to complete a series of predefined tasks
  • Most problems are found with 5 users
  • Can be done using paper prototypes (many experts think it's best to start with this, because you can identify usability issues before you have spent a lot of time and money developing the site or tool)
  • Person who is giving the test is careful not to help user unless user gives up
  • Ideally done with separate administrator, observer, and (if using mockups) "machine"
  • Tests we've done at Carleton:
    • Off-Campus Studies
    • Student Gateway
    • Moodle
    • Sakai

Card Sort SketchCard Sorting

  • Method to discover how users organize information and what terms they use
  • Useful when you have a lot of information and not sure how to organize it
  • More art than science -- it doesn't replace smart thinking about information organization
  • Helps to ensure that the site organization maps well to how people think about the info
  • Card sorts we've done at Carleton:
    • Music department
    • Off-Campus Studies
    • Geology department
    • Library

Focus Group SketchFocus Groups

  • Useful method to get more general understanding of users or audience
  • Opportunity to discuss users' needs, desires, frustrations, etc.
  • Focus groups we've held at Carleton:
    • Student Gateway
    • Home Page -- Prospective Students

Interview SketchInterviews

  • Not the same as usability testing -- not in front of actual tool
  • Useful for when you have a specific idea of best organization, but want to test
  • Interviews we've done at Carleton
    • Bridge icons
    • Student Gateway
    • Moodle
    • Sakai

Readings/Additional Resources