Getting Rapid and Representative User Input Using the Web
Karel Vredenburg, karel@ca.ibm.com
Paul McInerney, paulmci@ca.ibm.com
Scott Isensee, isensee@us.ibm.com
IBM Corporation
Abstract
A major challenge in doing User-Centered Design is getting
rapid and representative user input. A simple mechanism is
described that uses the Web in building a participant recruiting
database and carrying out Web user surveys. A recent study of
desktop user interface usage is used to illustrate the mechanism.
Overview
Despite the increase in focus on ease of use and usability
within the computer industry, many product teams still forego
involving users in the design and evaluation of their products
due to the difficulty in getting ready access to representative
users and the lack of rapid turn-around techniques to get user
input. In fact, even groups who have embraced User-Centered
Design in the past are finding it increasingly difficult to
involve users given ever decreasing development cycles, or the
new so-called "three-month web-year" phenomenon. In
addition, the increasing focus on designing for an international
audience drives a requirement to gather user input from a
world-wide sample.
Approach
The approach our team developed for addressing these
challenges involves two fundamental elements using the World-Wide
Web. The first is a participant recruiting database and
the second is a survey mechanism.
Participant Recruiting Database
This component of our approach is comprised of a demographics
and product usage survey on the internet, a central relational
database and web query interface on the intranet. Potential
participants complete the web-based form asking them information
such as name, address, e-mail address, characteristics of their
computer use, and information about the specific products they
use and their proficiency at using them. It also asks the type of
studies for which they would like to be considered - web survey,
telephone interview, on-location study, etc. Designers and other
practitioners within the company then use the intranet query
interface to the database to select random samples of particular
user characteristics for specific studies, of whatever type. If a
web-based survey will be conducted with the selected sample, the
e-mail addresses are used to invite potential participants. The
query interface also records the studies for which the names were
used to avoid participant study fatigue.
To ensure the database would be as representative of the population of users as possible, we identified the list of market segments, products, and user types that our company targets with our products, then found the usenet groups that the users in these categories used frequently and invited those visiting the usenet groups to help us design our products by registering themselves in our database. To increase the response rate, we offered a notebook computer to a randomly selected respondent who added him/herself to our database. Realizing that we still may have some bias in the sample, we also regularly check the demographics of people in our database with demographic information available on selected populations. We have recently also included our company product registration database records into our recruiting database to further increase representativeness. Designers and other practitioners can now identify and invite a sample of representative users within minutes using this database.
Note: Names in this screen image have been altered to protect privacy.
Web Survey Mechanism
The second component of our approach involves a web survey
mechanism that provides teams with survey templates for
conducting the most commonly requested surveys. This includes
customer satisfaction, requirements gathering, task analysis,
icon recognizability, terminology evaluation, design review, etc.
The most frequently used surveys, such as for customer
satisfaction and requirements gathering, are automated in that
practitioners can simply provide the product name and questions
they would like to ask in addition to a standard set we
provide, press OK and the survey is on the web. Participants are
selected using the recruiting database and invited via e-mail.
Within minutes the design team is receiving user input on the
intranet site where results are automatically analyzed and
displayed. In cases of more customized surveys, practitioners are
provided templates on the intranet website that they then modify
at will, submit to have posted to the web, and get files with the
aggregated results sent to them via e-mail.
Results
Our experience to date with these two components used in
combination shows that we can build surveys in minutes and
get 60-70 percent response rates with all information
typically collected within two to three days.
Example
The characteristics of the mechanism are best illustrated by
example. Recently, one of the teams within our company needed to
have information on the use of the various user interface
elements of the Microsoft® Windows 95™ desktop. However,
the team was on a very tight development schedule and decided
they didn’t have the time or resources to conduct the
required study. It was then suggested to the team to consider
using our new mechanism, which they did. The survey was
constructed, the participants selected and invited, the data
collected, analyzed, and presented back to the team all within
three days. The results yielded important insights for the team.
They learned that the majority of users do not change settings on
the interface from the defaults and little use is made of
advanced features. For example, the majority leave the task bar
at the bottom of the screen with auto-hide turned off. Results
also showed that despite having higher resolution monitors, a
large number of users maximize windows and switch between them
using the task bar. Results like these had a major input on the
subsequent design direction. Comments from the teams like this
one that have used this approach have been extremely positive.
They realize that the use of mechanisms like these for rapidly
collecting input from representative users not only increases the
knowledge they have of their users but also allows them to
collect this information when they don’t have the time to do
so using traditional methods. The input is also from an
international sample. Despite their speed and power, it should be
pointed out, however, that these methods should not be used as a
replacement to all of the typical User-Centered Design methods
and techniques. They should augment the traditional methods in
critical projects in cases where there is extremely time
pressure, where a larger sample is desired, and where a
world-wide audience is important. In certain circumstances where
a project would normally get no user input, these mechanisms can
be used to ensure that some user input is collected.
Additional Information
Further information is available on our website at www.ibm.com/easy.
© Internet Technical Group
Last update: September 8, 1998
URL: http://www.sandia.gov/itg/newsletter/sep98/isensee.html
hosted by Sandia National Labs
Disclaimer: Neither Sandia Corporation, the United States Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately-owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by Sandia Corporation, the United States Government, or any agency thereof. The views and opinions expressed herein do not necessarily state or reflect those of Sandia Corporation, the United States Government or any agency thereof.