Success Rates for Usability Objectives

Continuing our discussion about usability objectives from last Wednesday, a related question that comes up frequently is – What acceptance criteria should I use for my usability objectives? or Is it acceptable to have a usability objective of less than 100% success?

Ideally 100% of usability test participants would be able to complete a given task correctly or give the expected response. However, if this was the case we wouldn’t have to do usability testing. We are, after all, evaluating patterns of human behavior that often involve error.

When determining acceptance criteria, or a target success rate, looking back at where the usability objectives came from (safety concerns or business goals) usually helps. If a usability objective is safety related, 100% target success rate makes sense as the goal. No errors that could result in a safety hazard should go unnoticed. If the actual success rate observed from a safety related task during testing is less than 100%, this raises a red flag that user behavior possibly leading to a safety hazard needs to be better understood. Then the residual risk can be analyzed.

For objectives that are motivated by business goals, the target success rate should be the success rate that is satisfactory from a business standpoint to those who developed the goal. For example, if the objective is that users will be able to set up a new model of the product 3 minutes faster than an older model of the product, marketing or management representatives who set that goal may be satisfied with 80 or 90% of users actually meeting that objective in testing. There is much more flexibility in setting acceptance criteria when safety isn’t a concern.

If you’ve written usability objectives in the past, how do the methods I’ve outlined compare with your approach?

-KB