Product Testing: For Usability and Market Research

Corrections or additions?

These articles by Michele Alperin and Barbara Fox were prepared for the July 11, 2001

edition of U.S. 1 Newspaper. All rights reserved.

In Software Ergonomics, Usability Is Key

The test laboratory is set up like any other Princeton

Forrestal Center office, with a desk and a computer. But instead of

the standard issue corporate art a large mirror covers half the wall.

Behind this mirror, really a one-way window, is an observation room

where two dozen people are clustered to watch a software


They stare at an enlarged computer screen identical to the one viewed

by the test subject. Another screen captures the test subject’s facial


Siemens Corporate Research hopes this new facility at 755 College

Road will help its engineers develop better products — everything

from software to washing machines. For a long time product development

has been the purview of "techies" who, if they thought at

all about end users, figured that consumer needs were pretty much

a clone of their own requirements. "There was a general assumption

that the customer would adapt to the product," says Heidi


head of corporate technology user interface design for Siemens



"User friendliness" is no longer simply an option that adds

value to a product, but rather it is integral to product quality on

par with the product’s technical functions. "The quality of a

product is defined by the quality of the usability of that product

by the target market," says Kroemker. The press of new


possibilities and the integration of computers into nearly all complex

products, she explains, is making software ergonomics, the


creation of software, ever more important.

Siemens Corporate Research opened its third User Interface Design

Lab this spring. Like its twins in Munich and Beijing, the Princeton

lab is responsible for designing user interfaces and evaluating how

real users respond to potential products. Although the lab will devote

about 80 percent of its time to evaluating Siemens products, its


eventually will also be available to outside companies.

The first Siemens usability lab, established in 1994, was the first

of its type in Germany. The lab was supposed to demonstrate how rapid

technological change and increasingly international markets are


sensitivity to product users.

But it takes education and exposure to usability concepts to convince

product developers. Even in the face of expert opinion from the labs’

interdisciplinary teams of computer scientists, industrial engineers,

physiologists, linguists, designers and psychologists, development

engineers are typically doubters. "Some of them couldn’t believe

their eyes when they saw how customers struggled with a product,"

says Kroemker. "We had to work hard to make development engineers

realize how important it was to consider the issue of user interface

design and usability engineering in the very early stages of product


An example: Klaus Obersteiner was product manager of the group in

Munich that developed a new fire alarm system for Siemens Building

Technology. He says the doubters in his product development group

worried that user testing would mean lost time and money. But after

the product was tested against consumer needs, he says, the market

demand for the product doubled over initial projections.

So how did he get his group to incorporate a user perspective? They

wanted an innovative product, and money was available, so he just

had to convince them to "trust that it will work."

Kroemker highlights a number of factors that influence the procedures

employed in the user interface labs:

Need for innovation : If you encourage the kind of customer

input that feeds product innovation, you need to develop special


for acquiring and evaluating new ideas.

Functional requirements : To discard seldom-used functions

in the product’s next generation, you need procedures that analyze

functions for their value and meaning.

Modularization : Products are often built in modules, which

are joined together to serve the needs of particular users. To


combine these modules, designers must know how a particular group

of users thinks and what tasks it must complete.

Internal communication : Software design, in particular,

demands systematic communication between representatives of marketing,

development, and sales to achieve a broad perspective that includes

all necessary knowledge and experience.

Cost reduction : Reducing costs and development time


more efficient methods of product development, including design.

Cultural differences : Because most products are sold


user interfaces must account for the specific cultural needs of


groups of users, and designers must be sensitive to different cultural


For example, because in China the color white tends to evoke

sadness or mourning, white washing machines would be a no-no. User

preferences on how to learn to use a mobile phone — whether to

ask friends, use trial and error, ask sales assistant for help,


or read documentation — also vary from culture to culture.

"The formation of a man-machine interface should be the start

of every design project," writes Kroemker. When the user interface

is developed early, "the development process can run smoothly

without expensive changes and time loss due to alterations."

So how does product design from a user perspective work? Kroemker


Determine necessary product functions. To do so, look systematically

at the tasks users must complete and the problems they must solve.

Information is gathered via focus groups, analysis of competitive

products, and expertise from marketing, development, and sales as

well as product development. Also, analyze how the user thinks and

operates. Given the increased number of possibilities for user


with software — through movements and speech — designers must

find the correct combination of communication elements, depending

on both the user and the task at hand.

With this research complete, the next step is to create a tentative

product design. Then, perform usability tests to gather input from

experts and newcomers as well as frequent and infrequent users.

Gavriel Salvendy of Purdue’s School of Industrial Engineering, the

keynote speaker at a demonstration for clients in the first week of

May, suggests that two levels of usability must be evaluated: A


"apparent" usability is the user’s first impression of the

interface; if it does not seem accessible, the user may not even want

to try it. The second level is the inherent usability of the product,

what it feels like to use once the user has tried it out. After


field tests, make adjustments to the product design based on user


Those who came to the lab opening on College Road were treated to

a live usability test of the user interface of the website


Standing in the observation room, they watched as a visitor, Ana


of Somerset-based OSS Nokalva, performed the following task: You are

looking for a new job, but you don’t want to move, and you hate


Find posted job offers within 25 miles of Princeton. As she performed

her task using the site, she was asked to think aloud

about what hindered and helped her. As she was guided through the

task by Kem-Laurin Lubin, a human factors specialist at Siemens, she

made a number of discoveries about the site’s usability:

The site was crowded with "a lot to read."

There was a checkbox with no apparent function.

There was no option for 25 miles; users had to choose 20 or


Search results lacked titles.

The results of keyword searches were hard to evaluate.

Job distance was the only criterion the user could specify.

During the test, video cameras recorded Greenspan’s face as

well as the screen she was seeing on her computer, and these were

displayed on the wall of the observation room.

In a real test, Lubin explained, there would be follow-up to try to

clarify and objectify the user’s subjective reactions. The user might,

for example, be shown her confused facial expression at a particular

point in the test and ask what she was thinking then. Or, she might

be asked to rate the user interface on a scale of 1 to 7 and explain

her reasons for the number she gave. In a period of burgeoning


problems in the interface with the user become apparent much sooner.

In the three months since it opened, the laboratory

has done work for various Siemens business units, including


a business to business website to make it more efficient. "We

had a lot of Siemens’ clients volunteering and wanting to help to

improve the usability of the website," says Arnold Rudorfer, who

is in charge of the Princeton lab. "The main motivation is to

make the working tool more efficient to use. The purpose of the


is to order equipment and to get service, so a better website helps

them increase customer satisfaction by providing timely service."

The lab has two full-time and two temporary staff members and expects

to hire more next year. Rudorfer, 34, studied at the Technical


of Graz, and has a master’s degree in software engineering and a minor

in business administration. He has worked in Spain and Sweden and

came to this job in April, 2000.

In the real world, the results from user testing are sometimes


For Obersteiner’s fire alarm system, the final product included a

relatively high-cost touch screen to simplify use, a comfortable


for frequent use, shortcuts for easy operation, and icons designed

for easy recognition.

Improvements in product usability mean increased user productivity

and satisfaction, reduced sources of error and confusion, and


training time. "If the competition is doing a better job,"

says Lubin, "they’re way ahead."

— Michele Alperin

Siemens Corporate Research Inc., 755 College Road,

Forrestal Center, Princeton 08540. Thomas Grandke, president and CEO.

609-734-6500; fax, 609-734-6565.

Top Of Page
Product Testing: For Usability and Market Research

Testing products for either usability or market research

purposes is a big business, and Princeton has more than its share

of the latter. Princeton has more than 50 market research firms,


such well-known ones as Gallup, Opinion Research, Mathematica, and

Roper Starch.

But Princeton does not lead the country in actually hosting usability

focus groups. Opinion Research Corporation had maintained a focus

group room when it was located on North Harrison Street, but now only

one company, Research 100, can rent out an actual focus group room

with a one-way mirror for observation. The next nearest rooms are

located in Woodbridge, Edison, and Marlton.

Siemens has just set up a room like this, but as explained in the

article above, that is mostly for company use. If you want to rent

the Siemens usability lab, you have to knock on the door — the

company is not actively marketing it, at least not right now. And,

as the Siemens people point out, their usability lab is


and meant for use by one person a time. It won’t hold a group of 10

people chatting with a moderator.

In contrast, Research 100 is eager to rent its room at Princeton


Center for both usability and market research studies. Founded in

1975, it offers custom market research to manufacturers, advertising

and service organizations and has 10 employees in 5,000 square feet.

A 1981 graduate of Ithaca College, Mark H. Sandler went right into

the family business, and is now president, while his 65-year-old


remains CEO.

For usability test purposes, Research 100 works with only one person

at a time, but for market research discussions it can hold 12


plus the moderator. The 22 by 18-foot focus group room was recently

refurbished with a new modular conference table, new seating and


and an upgraded camera. The viewing room, equipped with the requisite

one-way mirror, holds about 15 people and has an adjoining client

lounge for refreshments and discussion.

Any technology needed — computers, for instance — is added

to the room for a particular project. This contrasts with the


at Siemens, which brings in whatever equipment is needed for each


Just having a well-equipped room or lab facility is not enough, says

Sandler. The key to good research is who you find to test the


or give the opinions. And the jewel in Research 100’s crown is its

database of prospective participants. "We have a database of 7,000

people who have agreed to participate in focus groups. That allows

us to select people as appropriate for a particular project. We call

them on the phone to see if they qualify. If they use a certain brand

of shampoo. we invite them in," says Sandler.

For a standard two-hour group, consumers generally get paid $50.


of doctors might get $250 for 90 minutes. "You do those at 7 a.m.

and feed them breakfast and get them on their way," says Sandler.

Small business owners might get paid $100 to $125 depending on what

criteria they need to meet; if they are supposed to have a small


with a server, they would be paid more. "Virgin" participants,

those who have never participated in a market research project, are

particularly valuable.

Anyone who wants to participate in market research is invited to sign

up by calling Sandler, and by the end of the summer signups will be

offered at a website.

For clients to just rent the room costs $450 a session or $1,200 for

the day, including audiotaping. Add recruiting to that, for instance

— finding two groups of 12 women for a standard consumer product,

like toothpaste — would cost $2,500 for recruiting plus $900 for

the sessions. And to do an entire project — develop the


write the topic guide, recruit and pay the respondents, provide the

facility and the moderator, tape the sessions, and deliver the written

analysis — costs from $10,000 to $12,000.

Don’t think that Research 100 depends on rents for its livelihood.

A maximum of 10 or 15 percent of the company’s income is related to

focus groups at this location. Most of its qualitative studies are

done in cities close to the client, such as New York, Chicago, Dallas,

Houston, Tampa, Fort Lauderdale, and Atlanta. The company also does

quantitative studies such as in-store or telephone research.

What makes a good facility?

1. Great recruiting. "They find good qualified


who are what we asked for," says Sandler. "You know who the

ringers are when they open their mouth. They tell you what they think

you want to know about them."

2. A good quality sound system.

Everything else after that is peripheral, although good catering

helps, says Sandler. His favorite spot in Florida is located in a

high rise overlooking Tampa Bay, "but the reason we go to that

facility is that the recruiting is excellent."

And here’s an economy tip worthy of notice from Alan Greenspan.


the economy gets soft," says Sandler, "our qualitative work

increases rather than our quantitative." Telephone surveys are

expensive, so when the accountants are seeing red, they are hard to

justify. "But rather than make a business decision blind, our

clients ask for `a small study,’ which might be a focus group."

"We have been tracking the relationship between our work and the

economy for more than 20 years," says Sandler, "and our work

tends to lead the economy by about six months. When our business goes

up in terms of qualitative, in six months the economy will go down.

Right now, based on the business mix we are seeing, we are suggesting

that the economy will be soft through the end of the year."

— Barbara Fox

Research 100, 3490 Route 1, Building 16, Princeton

08540. Mark H. Sandler, president. 609-924-6100; fax, 609-452-0138.

Corrections or additions?

This page is published by

— the web site for U.S. 1 Newspaper in Princeton, New Jersey.

Facebook Comments