Textbook of Usability

Introduction

Usability testing (Chapter 15) observes real users performing real tasks. Expert review methods take a different approach: trained evaluators examine the interface and predict usability problems based on established principles and cognitive models. These methods are faster, cheaper, and less logistically demanding than user testing, making them valuable for early-stage evaluation, quick assessments, and situations where user recruitment is difficult. This chapter covers the two most widely used expert review methods: heuristic evaluation and cognitive walkthrough.

Heuristic Evaluation

Heuristic evaluation, developed by Jakob Nielsen and Rolf Molich Nielsen, 1990, is a systematic inspection method in which evaluators examine an interface and judge its compliance with recognised usability principles (heuristics). The heuristics serve as the evaluation criteria; the evaluator's expertise and judgment determine how they are applied Nielsen, 1994.

Procedure

1. Briefing: evaluators are given a description of the user population, the primary tasks, and the context of use.
2. Individual evaluation: each evaluator independently examines the interface, going through it at least twice — first to get an overall sense of the system, then to focus on specific elements.
3. Problem documentation: each evaluator records every usability problem found, noting which heuristic it violates, where it occurs, and its likely impact.
4. Consolidation: evaluators' findings are combined into a single list, with duplicates merged.
5. Severity rating: the consolidated list is rated for severity, either by the evaluators or by a separate group.

The Heuristics

The evaluation criteria are typically Nielsen's 10 heuristics (described in Chapter 8) Nielsen, 1994: visibility of system status, match between system and real world, user control, consistency, error prevention, recognition rather than recall, flexibility, aesthetic minimalism, error recovery, and help. Evaluators may supplement these with domain-specific heuristics — for instance, Gerhardt-Powals' cognitive engineering principles Gerhardt‐Powals, 1996 or medical device heuristics that include patient safety considerations.

Severity Rating

After consolidation, each problem is rated for severity. A common framework combines two dimensions. Impact: how serious is the problem when it occurs? (cosmetic, minor, major, critical) Frequency: how often will users encounter it? (rarely, sometimes, frequently) The product of impact and frequency yields a priority ranking that guides remediation efforts.

Strengths and Limitations

Strengths:

• Fast (a single evaluator can review a system in 1–2 hours)
• Inexpensive (no participant recruitment, no lab)
• Can be conducted early (on wireframes, prototypes, or specifications)
• Produces actionable, specific findings Limitations:
• Depends on evaluator expertise; novice evaluators miss problems and report false positives
• Does not reveal actual user behaviour, task times, or satisfaction
• Evaluators may disagree on severity ratings
• Cannot identify problems that arise from the user's mental model rather than from heuristic violations

Cognitive Walkthrough

The cognitive walkthrough, developed by Lewis, Polson, Wharton, and Rieman Lewis, 1990, focuses specifically on learnability. It traces the steps required to complete a specific task and, at each step, asks whether a new user would know what to do Wharton, 1994.

Procedure

1. Define the task: specify a realistic user task and the correct sequence of actions to complete it.
2. Define the user: describe the target user's goals, knowledge, and experience.
3. Walk through each step: for each action in the correct sequence, the evaluator answers four questions.

• Will the user try to achieve the right effect? (Does the user's goal match what the interface requires?)
• Will the user notice that the correct action is available? (Is the control visible and recognisable?)
• Will the user associate the correct action with the desired effect? (Does the label, icon, or affordance suggest the right action?)
• If the correct action is performed, will the user see that progress is being made? (Is the feedback adequate?)

4. Record failures: any step where the answer to one or more questions is "no" (or "probably not") represents a learnability problem.

Strengths and Limitations

Strengths:

• Focuses specifically on learnability — the first-use experience
• Forces the evaluator to adopt the user's perspective
• Identifies specific points of failure in specific tasks
• Can be conducted on paper prototypes or wireframes Limitations:
• Time-consuming (each task must be walked through step by step)
• Narrow focus (only evaluates the specific tasks analysed)
• Does not address efficiency, satisfaction, or error recovery
• Assumes a specific "correct" path, which may not match how users actually approach the task

Pluralistic Walkthrough

The pluralistic walkthrough combines elements of expert review and user testing. A group comprising users, developers, and usability experts walks through the interface together, discussing each step. The group format generates diverse perspectives: users reveal mental model mismatches, developers explain technical constraints, and usability experts identify principle violations. The pluralistic walkthrough is particularly effective for building shared understanding among team members about usability issues. Its limitation is the social dynamics of group settings — quieter participants may defer to more vocal ones, and the presence of developers may inhibit users from expressing confusion.

Comparing Expert Review with User Testing

Expert review and user testing are complementary, not competing methods. Research comparing the two approaches [Jeffries, 1991; Hertzum, 2003] consistently finds:

• Different problems: expert review methods and user testing identify overlapping but distinct sets of problems. Expert reviewers find problems that users work around (and therefore might not report), while users encounter problems that experts fail to predict.
• False positives: expert reviewers sometimes flag issues that do not cause problems for actual users (false positives). User testing avoids this because problems are identified by observing actual difficulty.
• Context sensitivity: user testing reveals problems that arise from the user's context, mental model, and task approach — factors that expert reviewers can only approximate.
• Efficiency: expert review is faster and cheaper; user testing is more expensive but produces more ecologically valid results.

When to Use Which Method

Use heuristic evaluation when:

• The design is early-stage (wireframes, prototypes)
• Budget or time constraints prevent user testing
• You need a quick assessment of a competitor's product
• You want to identify low-hanging fruit before user testing Use cognitive walkthrough when:
• Learnability is a primary concern (new users, infrequent use)
• The task flow is complex and sequential
• You want to evaluate whether a specific task can be completed without training Use user testing when:
• You need to validate that the design works for real users
• Quantitative metrics (task time, completion rate, satisfaction) are needed
• The design is at a stage where user feedback can still influence changes
• You want to discover problems that experts cannot predict Use multiple methods when:
• The stakes are high (safety-critical systems, high-traffic consumer products)
• The budget allows iterative evaluation
• You want the most comprehensive assessment possible