Textbook of Usability

Explainability in AI refers to the degree to which an AI system can communicate how and why it produced a particular output in terms a human user can understand. It is a usability requirement for any AI system whose outputs users must evaluate, trust, or override.

Many AI systems — particularly deep learning models — operate as black boxes: they produce outputs without transparent reasoning. A clinical decision support system that recommends a drug without explaining why provides no basis for the clinician to decide whether to trust, modify, or override the recommendation.

Different users need different levels of explanation. A clinical AI predicting patient deterioration might offer:

1. Summary level: "This patient has a high risk of deterioration in the next 12 hours."
2. Contributing factors: "Key factors: declining blood pressure trend, elevated heart rate, rising lactate."
3. Technical detail: "Prediction based on a gradient-boosted tree trained on 50,000 ICU admissions, with an AUROC of 0.87."

Most clinicians need level 1 (to act) and level 2 (to evaluate). Level 3 is relevant for informaticists validating the system.

The usability question is not "Can the system explain itself?" but "Does the explanation help the user make a better decision?" A technically correct explanation framed in machine-learning jargon may be useless to a clinician in an emergency; a simple list of contributing factors may be exactly what's needed.

Explainability is distinct from interpretability (which refers to the system's internal workings being understandable in principle). A system can be uninterpretable at the model level but explainable at the interface level if the designer builds meaningful explanations into the output.