CSC 151-02, Fall 2006 : Usability Study Results

Usability Study Results

Some notes

• To protect confidentiality, never put the name of a participant on the card.  In a real usability study, each participant would be assigned a "subject code" (usually a number).
• Describe problems carefully enough so that another reader can understand them (or you later, after you forget :-)
• Designers would use observations like these to inform a new design or revise the existing design.  The facilitator/observer might demonstrate to the designer what the participant did or saw, to help the designer understand the problems.  The designer might also observe the study directly.
• Major problems are those that are observed by several participants (e.g., clock D: difficulty finding controls) or those that could result in an irrecoverable error (e.g., clock C: difficult to tell what time alarm is set for).  Fixing major problems should be a high priority! Problems that do not result in serious errors and are only experienced by one participant might be lower priority.
• I like that some observers noted positive comments made by the participant, as well as negative comments and problems.  Positive comments (expressing satisfaction!) can be helpful in choosing between alternative designs.
• Times have high variability.  We would need many participants (and a reliable stopwatch) to make statistically valid comparisons.  Which method for setting the alarm clock time do you hypothesize is the fastest/slowest?
• Based on these findings, which clock do you think is the most usable? the least usable?

Findings

Clock A: Curtis with time, hour, minute, al. set buttons

• Positive comment: "I want this clock, it's easy to use."  [It's on clearance for $4.99 at Paul's Ace Hardware!]
• 1/4 observed the snooze button is big (a nice feature?)
• 1/4 alarm set button clearly demonstrates alarm is on (a nice feature?)
• 1/4 slow response to button presses
• 2/4 not obvious need to hold down time button to set clock time
• 1/4 not sure whether time is AM or PM
• Task 4 times: 6 sec, 8 sec, 14 sec, 14 sec, 16 sec

Clock B: RCA with mode slider and fwd/rev buttons

• Negative comment: "I would not buy this clock."
• 2/5 expressed frustration
• 1/5 appreciated fast to set alarm time
• 2/5 difficult to find the mode slider to set the time
• 2/5 difficult to find the hour button
• 1/5 on/off button confusing
• 1/5 not obvious to hold down button to make it go faster
• 1/5 hesitated in figuring out how to set the alarm time
• 1/5 wasn't sure if alarm was set
• 1/5 overshot alarm time and could not go backwards
• Task 4 times: 8 sec, 23 sec., 24 sec, 37 sec, 40 sec

Clock C: Advance analog clock

• 2/4 reports setting time is tedious---have to turn the minute hand to set the hour
• 1/4 expresses frustration in setting alarm time
• 1/4 error in setting alarm time
• 1/4 difficult to tell what time the alarm is set for
• 2/4 difficult to set alarm time to the exact desired time
• 1/4 difficult to tell from the front of the clock whether the alarm is set
• Task 4 times: 7 sec, 7 sec, 22 sec

Clock D: Radio Shack travel clock with time, date, and day of week; set with mode slider and hr/min buttons

• 1/6 expresses frustration
• 1/6 expresses fondness for the clock
• 1/6 "cool snooze light"
• 3/6 difficulty finding controls
• 1/6 annoying to have to open the battery cover to change any settings
• 1/6 difficult to open the clock
• 1/6 too easy to lose the battery cover
• 1/6 time set buttons are too small
• 2/6 too much button pushing---too slow to set time
• 1/6 needed to read manual to set clock time
• 1/6 initial year is 1990---which is 16 years ago
• 3/6 had errors while setting the date---missed the correct year---not clear how to go back
• 1/6 difficult to push slider to alarm set
• 1/6 sets alarm for wrong day of week
• 2/6 alarm settings confusing---is the alarm set?
• Task 4 times: 11 sec, 16 sec, 31 sec, 69 sec

Clock E: Fancy transparent Timex with time and date; set with mode, adj, and set buttons

• 3/4 participants expressed frustration
• 1/4 clock is easy to drop
• 1/4 surprised by clock noises while inserting batteries
• 1/4 difficulty determining purpose of on/off button
• 3/4 needed to read the manual to determine the purpose of the mode, adj, set buttons
• 2/4 tedious to set minutes
• 1/4 accidentally pressed wrong buttons while setting time
• 1/4 suprised by display while setting date
• 2/4 had errors while setting the date---missed the correct year---not clear how to go back
• 1/4 not sure how to set the alarm
• 1/4 not sure how to use snooze button
• Task 4 times: 27 sec, 42 sec

Clock F: Simple travel clock

• 1/5 battery case "sticks a little"
• 2/5 difficult to tell whether time is AM or PM
• 1/5 error: pressed minute button instead of hour button
• 3/5 holding down a button to set the time is slower than repeatedly pushing it
• 1/5 error: missed correct hour
• 1/5 did not understand how to turn the alarm on
• 1/5 What does the Z mean?
• Task 4 times: 7 sec, 28 sec, 28 sec, 29 sec