|
Collecting
Data From Users
To design a site
that works for you and your intended audience, you have to know a lot
about those people. They may be "customers,""consumers,""researchers,"
"the public." Let's call them "users" because what you want them
to do is use your site.
What
to Consider
Verify or challenge
your assumptions about users. Thinking about users only gets you
so far in designing a successful site. Your thinking brings out your
assumptions about the users. To learn about users' reality, you need
to get out and meet them, work with them, involve them in helping you
to understand their:
- needs for information
- ways of thinking
about, grouping, and organizing information
- expectations
about your site
- levels of knowledge
about the subject matter
- levels of experience
with the Web and similar types of sites
By working with
users, you can gather many realistic scenarios and learn what makes
a Web site work or not work for them.
Let users help
you build the site for them. Many useful techniques have been developed
to get useful information from users and about users before you
design a site.
Understanding
and Comparing Techniques for Gathering Data From Users
The following is
an overview of data-gathering techniques, what they are, and how they
differ.
| Technique |
Characteristics |
| Early
usability tests |
- Users
usually come to you
- You
usually develop the scenarios
- Small
numbers: one or two users at a time
- Total
numbers: five to 12 users
- You
observe and listen to actual behaviors
- May
be formal or informal, quantitative
and/or qualitative results
- Tester
and user need not be at same location
|
| |
|
| Contextual
interviews |
- You go
to the user's home or work site
- Users
do their own work (different scenarios with different users)
- Small
numbers: one or two users at a time
- Total
numbers: five to 12 users
- You observe
and listen to actual behaviors
- You see
users' environments and the technology users have
- Usually
informal dialogue with user, qualitative results
- Interviewer
and user are physically at same location
|
| |
|
| Online
surveys |
- May have
large number of responses
- Get users'
self-report
- Good
for wish lists, attitudes, experiences; not for actual behaviors
- Usually
mostly closed questions (yes/no, multiple choice, short answer)
- May include
open-ended questions, but they require more analysis
- Users
may be located anywhere
- May be
single-survey or iterative series
|
| |
|
| Individual
interviews |
- Face
to face, by telephone, through instant messaging or other
computer-aided techniques
- Small
numbers: one user at a time
- Total
numbers: usually five to 15 users
- Rich
data — you can follow up on questions
- Can include
both closed and open-ended questions
- Self-report;
good for attitudes, experiences, wish lists
- Not good
for actual behaviors
|
| |
|
| Focus
groups |
- Small
group discussion
- Moderated
by trained facilitator
- Usually
everyone is in same location
- Self-report;
good for attitudes, experiences, wish lists
- Not usually
good for actual behaviors, but can combine with some aspects
of behavioral usability testing
- Discussion
influenced by group dynamics (for good or bad)
- Can be
done as an electronic meeting, which allows for anonymity
and reduces the effect of group dynamics
|
| |
|
| Card
sorting |
- Usually
used after gathering information with one or more of the other
techniques
- Each
card represents a possible topic on the site
- Need
a start on content topics — so have some cards to sort
- Usually
small numbers: one or two users at a time
- Typical
total numbers: five to 12 users
- You usually
observe and take notes as users talk about what they are doing
- Can be
done remotely with a Web-based tool — so can be large
numbers
|
Early
usability tests
Consider starting
your project with a usability test.
If you already have
a Web site, you can find out what works well for your users and what
does not. If you do not yet have a site, use a competitor's site or
one that has similar purposes.
You can learn a
great deal that will help you build a new site — what to keep,
what to expand on, what to change, how to avoid others' mistakes.
A usability test
can be done quickly and inexpensively. What a usability test reveals
about what users actually do is usually more valuable than what you
learn in interviews and focus groups where you ask users about themselves
and their work.
What users say
they do and what they actually do are often different — because
people aren't always aware of how they work. When talking about
our work, we all skip steps because we do them automatically. We often
cannot remember exactly how we do or did something. Watching and listening
as users work is the most informative way to see what people do —
and to get what you need to build a successful site.
For more on how
to plan, carry out, and analyze data from a usability test, see Conducting
and Using Usability Tests.
•
For
information on doing remote usability testing,
see http://research.cs.vt.edu/usability/projects/
remote%20evaln/remote%20evaln.htm
Contextual
interviews
Contextual interviews
are like usability tests because you watch and listen as users work.
They differ from usability tests in location . . . you go to the user.
That way, you see the user's environment and the actual technology the
user has to work with.
Seeing the user's
environment can be very useful. What is the social environment like?
Are there people around to help the user? What is the physical environment
like? Is the user on a slow modem? Does being online tie up a phone
line so the user wants to be on and off the Web quickly?
Contextual interviews
are more natural and realistic than usability testing. In a contextual
interview, you watch and listen as the user does his or her own work.
You don't usually impose tasks or scenarios on the user. In a usability
test, on the other hand, you usually have all users do the same scenarios,
which gives you comparative data from several people trying the same
thing.
You can, however,
combine aspects of both:
- Contextual interview:
Take scenarios along and combine watching the user do his own work
in his environment with asking the user to try a few of your tasks.
- Usability test:
Interview the user to find out the sorts of questions, issues, tasks
he or she would do with the site. Let the user do his or her own task.
Also have the user do some of your tasks to get data on tasks from
all the users.
A contextual interview
is usually informal. The observer listens to the user but may also ask
clarifying questions and probe to gain greater understanding of what
the user is doing and thinking. The results are usually qualitative
rather than quantitative.
Usability testing
in Web site development today is also often informal and is often conducted
much like a contextual interview. However, usability testing can range
from informal and qualitative to quite formal and quantitative.
Online
surveys
Use an online survey
to gather information from all of your Web audiences.
Some hints for effective
online surveys:
- Decide why you
are surveying your users.
• At this
early stage of Web site development, you probably want to learn
more about who they are, what experiences they have had with your
site or similar sites, and what they want — that's all!
- Decide where
you will find the people you want to survey.
• If you
already have a site, post it there.
• Also
consider other sites where users for your site might go and obtain
permission to announce your survey there.
• Consider
getting permission to send broadcast email about the survey to lists
of potential users through professional societies, listservs, discussion
groups, etc. Be sure to get permission from the list owners before
broadcasting email.
- Keep the survey
short. Under 10 items is preferable.
- Make the survey
easy for users to do. It should take no more than five or 10 minutes
to complete.
- Consider a mix
of closed questions (multiple choice) and open-ended questions (users
write down what they want to tell you).
• Closed
questions are easier to analyze.
• Open-ended
questions may give you richer data and offer a glimpse at the
terminology your users use.
- Consider a mix
of questions about demographics, users' prior experiences, and what
users want.
- Consider using
a series of surveys.
• Gather
information with a short survey that you make available where many
people might see it and respond.
• On the survey, ask if people are willing to answer more in-depth
questions — and ask those people to give you an email address.
•Send follow-up electronic questionnaires to willing respondents
who meet your criteria for the type of user you want to know more
about.
- Consider combining
an online survey with individual interviews.
• With
an online survey, you reach many people in many places, but you
don't get the depth of data that you get from individual interviews.
• If
you do individual interviews first, you get good ideas for survey
questions — and for the items on multiple-choice survey questions.
• If you
do individual interviews after gathering some survey data, you get
to follow up with some people on issues and ideas that come from
the survey answers.
Individual
interviews
What
do we mean by individual interviews?
Why conduct individual interviews?
When should you conduct individual interviews?
Individual
interviews and focus groups: What's the difference?
What makes an interview successful?
•
What do we mean
by individual interviews?
We are using "individual
interviews" to refer to talking with one user at a time (for 30
minutes to an hour) face to face or by telephone or with instant messaging
or other computer-aided means. These interviews do not involve watching
a user work. Thus, this is different from interviewing users in a usability
testing session or conducting contextual interviews.
•
Why conduct individual
interviews?
Individual interviews
can give you a deep understanding of people who come to your site. You
can probe their attitudes, beliefs, desires, experiences. You can ask
them to rate or rank choices for the Web site content.
•
When should you
conduct individual interviews?
Use individual interviews
to supplement online surveys. You can do interviews first to refine
questions for the survey. Or you can do interviews after a survey to
probe for details and reasons behind answers that users give on a survey.
•
Individual interviews
and focus groups: What's the difference?
Individual interviews
resemble focus groups because they involve talking with users. You do
not see them perform work/tasks as you do in usability tests and contextual
interviews. The obvious difference between an individual interview and
a focus group is that you are talking to one person at a time. In an
individual interview,
- you have more
time to discuss topics in detail
- you do not have
to worry about the group dynamics that inevitably occur in focus groups
- you can give
the interviewee your full attention and you can adjust your interviewing
style to draw out shy users and keep others on topic
•
What makes an interview
successful?
- Select participants
to represent the types of users you want to come to the Web site.
(This is true of all the data-gathering techniques.)
- Decide what you
want to learn. (This is also true for the other data-gathering techniques.)
- Write an "interview
protocol" for the interviewer to follow. (In focus groups, the comparable
document is called a "script." An interview protocol includes questions
and probes to use to follow up on questions.)
- Hire a skilled
interviewer who will make interviewees feel comfortable, ask questions
in a neutral manner, listen well, know when and how to probe for more
details, and keep track of time unobtrusively.
- Allow the interviewer
flexibility in using the protocol. (Although you want all the questions
answered, this is not a survey but can be an opportunity to get a
deep understanding of users.)
- Get permission
to tape the sessions and have one or more people take good notes.
(You are looking for answers to the questions and for insights about
users that will help you build a Web site that meets their needs.)
Focus
groups
What
is a focus group?
What
do you get from a focus group?
What do you not get from a typical focus group?
What makes a focus group work well?
•
What is a focus
group?
A focus group is
a moderated discussion among eight to 12 users or potential users of
your site. A typical focus group lasts about two hours and covers a
range of topics that you decide on beforehand.
Focus groups are
a traditional market research technique, so marketing departments are
often more familiar with focus groups than with usability testing or
contextual interviews. However, the techniques produce different kinds
of information. In a typical focus group, participants talk; you hear
them tell you about their work. In a typical usability test or contextual
interview, users act; you watch (and listen to) them doing their work.
•
What do you get
from a focus group?
- users' attitudes,
beliefs, desires
- users' reactions
to ideas or to prototypes
•
What do you not
get from a typical focus group?
- how users really
work with Web sites
- what problems
users really have with sites
•
What makes a focus
group work well?
- Select participants
to represent the types of users you want to come to the Web site.
(This is true of all the data-gathering techniques.)
- Decide what you
want to learn. (This is also true for the other data-gathering techniques.)
- Write a "script"
for the moderator to follow. (In usability testing, you write scenarios
— tasks — for users to perform. In contextual interviewing,
you let the context and the user's work shape the dialogue.)
- Hire a skilled
moderator to facilitate the discussion so that everyone participates
and the group stays on track. (In the other techniques, you need skilled
observers and listeners.)
- Allow the moderator
flexibility in using the script. The script usually gives the moderator
questions to ask and topics to cover. The moderator may change the
order of questions and topics to keep the discussion flowing smoothly.
The moderator has to be a good judge of time to decide when to encourage
more discussion on a topic and when to move on.
- Tape the sessions
and have one or more people take good notes. (This is also true of
other data gathering techniques. Good notes are critical to making
sense of what you see and hear in all these techniques.)
Card
sorting
What
is card sorting?
What
happens in a card sorting session?
Why use index cards with one topic per card?
How does card sorting work?
What about doing card sorting remotely with many
users?
•
What is card sorting?
This is a technique
that involves users in grouping information for a Web site. Card sorting
helps you build the site's structure and site map, decide what to put
on the home page, and label the home page categories.
•
What happens in
a card sorting session?
You give a user
(or two users working together) a set of index cards on which you have
put likely content topics for the site (one topic per card). The user
takes the first card and puts it on the table. The user takes the second
card and decides whether it belongs to the same group of information
as the first card or if it deserves its own category, etc. — and
so on through the set of cards. The user "thinks aloud" during
the session and describes his organizing strategy.
•
Why use index cards
with one topic per card?
- To allow users
to group — and regroup — the cards.
- To have users
build hierarchies that reflect the categories they want on the home
page and how they would group information in those categories on second-level
and lower-level pages.
•
How
Does Card Sorting Work?
Getting
the cards ready
Arranging for card sorting sessions
Conducting a card sorting session
Analyzing data
•
Getting the cards
ready
1. List the content
topics or types of information that you are likely to have on the
site. (These may come from your objectives for the site, from scenarios
you wrote, from what you learned from users with the other data gathering
techniques.)
2. Write each
content topic or information type on a separate index card. (Hint:
Use slef-adhesive labels and a word processor. These cards will be
neat, legible, and consistent. Also, you'll have the list of topics
in the computer for easy analysis later.)
3. Limit yourself
to less than 100 cards. (About 50 is a good number.)
4. Have blank
cards available for users to add topics and to name the groups they
make when they sort the cards. (Hint: Consider using a different colored
card for naming groups.)
5. Number the
cards in the bottom corner or on the back.
•
Arranging for card
sorting sessions
1. Select participants
to represent the range of users — pull from different user groups
with different levels of experience.
2. Plan about
one hour for each session — longer if you have many cards.
3. Arrange for
a space where the user has enough room to spread the cards out on
a table. A conference room works well.
4. Plan to have
someone take notes as the user works and thinks aloud.
5. As with other
techniques, arrange for payment or other incentives to thank the user
for spending the time and effort helping you.
•
Conducting a card-sorting
session
1. Show the user
the set of cards and explain what you want the user to do. Explain
that you are asking for help to find what categories of information
should be on the site's home page and what those catagories should
be called. Explain that you want to see what groupings of cards make
sense to the user and that when the user has grouped the cards, you
will ask that they name the groups.
2. Ask the user
to talk out loud while working. (This is the same technique employed
use in usability testing.) You want to understand the user's thoughts
and rationale.
3. Let the user
work. Also, let the user add cards — for example, to indicate
lateral hyperlinks or additional topics. Let the user put cards aside
to indicate topics the user would not want on the site. Mimimize interruptions,
but encourage the user to think aloud.
4. At the end,
if the user has too many groups for the home page, ask him or her
to create hierarchies of the groups.
5. Give the user
a different colored card for each group and ask the user to name the
group. What words would the user expect to see on the home page or
second-level page that would lead the user to that particular group
of cards?
6. At the end,
thank the user and give the user the payment or other gift as promised.
•
Analyzing data
1. Use the numbers
on the cards to quickly record what that user has done. Write down
the names that user gave to each grouping and the numbers of the cards
the user included under that name. Then you can reshuffle the cards
for the next session.
2. If you want
a complete picture of the detailed site map each user has created,
create a computer file for each session. Working from your original
list of topics, move topics around to recreate each user's groupings
and enter that user's names for the groupings.
3. For a less
detailed analysis, use your notes and recordings of the users' names
and card numbers under each person's name to find commonalities from
different sessions.
•
What about doing
card sorting remotely with many users?
The National Institute
of Standards and Technology (NIST) has developed a tool for card sorting.
You set up the cards, and you name the categories. Users drag and drop
cards into the categories.
For more information
on the NIST tool for card-sorting, see
http://zing.ncsl.nist.gov/WebTools/
Links
to Related Articles
Creating
Web Site Designs Based on User Expectations and Feedback, Jeannette
Fuccella and Jack Pizzolato, ITG Newsletter, http://internettg.org/newsletter/june98/web_design.html
|
