StandardInternational professional-learning synthesis · ABET 2026–27 · IEA GAPC v4 · ASCE ethics and AI responsibility · NIST AI RMF 1.0
Source3 source files
Review stateTechnical and publication gates pending
After this chapter, you should be able to
- Construct a confusion matrix.
- Calculate precision, recall, specificity, F1 and accuracy.
- Explain class imbalance and calibration.
- Select thresholds using engineering consequence.
- Design human review and abstention states.
Engineering context and methodSource §Lesson 05 · Engineering context and method · GOV-08 · SW-01 · DATA-02
A classifier separates labels created by a definition and labelling process. Its threshold is an operational choice: reducing missed positives usually increases false alarms. Always state which class is positive, the unit of analysis, prevalence and the consequence of each cell. Anomaly scores indicate difference from a learned reference, not defect, damage cause or failure probability.
Verified worked exampleSource §Lesson 05 · Verified worked example · GOV-08 · SW-01 · DATA-02
Consequence-aware screening metrics
TP = 18, FP = 12, FN = 2 and TN = 68.
- Precision
18 / (18 + 12)
0.60 - Recall
18 / (18 + 2)
0.90 - Specificity
68 / (68 + 12)
0.85 - F1 and accuracy
2×0.60×0.90/(0.60+0.90); (18+68)/100
F1 = 0.72; accuracy = 0.86
Result. Two positives were missed and 12 false alarms were raised. The acceptable threshold depends on consequence and review capacity, not accuracy alone.
Practical lab · 5 h lesson effortSource §Lesson 05 · Practical lab · 5 h lesson effort · GOV-08 · SW-01 · DATA-02
- Define the positive class and create a confusion matrix at three thresholds.
- Plot precision, recall and review workload against threshold.
- Investigate every false negative and a sample of false positives.
- Write an abstention and competent-review route for low-confidence or out-of-domain cases.
Failure modes to investigateSource §Lesson 05 · Failure modes to investigate · GOV-08 · SW-01 · DATA-02
- Positive class not defined.
- Accuracy used on an imbalanced population.
- Threshold tuned on the final test set.
- Anomaly equated with damage.
- No capacity plan for generated reviews.
Knowledge checksSource §Lesson 05 · Knowledge checks · GOV-08 · SW-01 · DATA-02
| Question | Answer rationale |
|---|---|
| What does recall measure? | The fraction of actual positives detected under the labelled test set. |
| What does precision measure? | The fraction of predicted positives that match the reference labels. |
| Why is threshold contextual? | Error consequences, prevalence and review capacity differ by use. |
| Is an anomaly a defect? | No; it is a statistical deviation requiring contextual investigation. |
| What is abstention? | A controlled refusal to decide when evidence or operating conditions are inadequate. |
Key points
- Start from the accountable engineering decision and its consequence.
- Compare against a transparent non-AI baseline.
- Validate on a split that represents intended use and retain human authority.
Source references recorded by the supplied chapter
- NIST AI RMF 1.0, measurement and risk-characterisation guidance.
- scikit-learn classification metrics documentation.
- Course research register DATA-02 synthetic and public screening datasets.