Problem framing, baselines, leakage and validation design — AI for Civil Engineering

StandardInternational professional-learning synthesis · ABET 2026–27 · IEA GAPC v4 · ASCE ethics and AI responsibility · NIST AI RMF 1.0

Source3 source files

Review stateTechnical and publication gates pending

LEARNING OUTCOMES

After this chapter, you should be able to

Separate decision, prediction target and action.
Choose a transparent non-AI baseline.
Identify direct and indirect leakage.
Select grouped, spatial or temporal validation.
Define predeclared acceptance and abstention criteria.

Engineering context and methodSource §Lesson 03 · Engineering context and method · GOV-08 · CUR-01 · CUR-04 · SW-01

The correct split follows the intended deployment. Random rows are unsafe when records share assets, sites, batches, time windows, image campaigns or derivative features. Fit imputation, scaling, feature selection and model tuning only on training data. Keep a final independent test set untouched until choices are frozen. Report uncertainty and critical slices, not only a pooled score.

Illustrative grouped holdout. Actual grouping may be by site, section, batch, catchment, campaign or time according to the intended use.Original STRUCTURA review diagram · technical sign-off pending

Verified worked exampleSource §Lesson 03 · Verified worked example · GOV-08 · CUR-01 · CUR-04 · SW-01

WORKED EXAMPLE

Repeated bridge records

Four annual records exist for each of 80 bridges; the model will be used on bridges not seen during training.

Invalid split
random records → same bridge in train and test
asset identity leaks
Aligned split
GroupKFold(group = bridge_id)
entire bridges held out
Final test
freeze 16 bridges before tuning
64 development + 16 independent

Result. Report held-out-bridge performance and bridge-level uncertainty; random-row performance is not deployment evidence.

Practical lab · 5 h lesson effortSource §Lesson 03 · Practical lab · 5 h lesson effort · GOV-08 · CUR-01 · CUR-04 · SW-01

Write the decision, target, prediction time, available features and prohibited post-decision data.
Implement a simple rule or persistence baseline.
Compare random, grouped and time-ordered splits and explain the differences.
Freeze acceptance thresholds and critical slices before opening the final test result.

Failure modes to investigateSource §Lesson 03 · Failure modes to investigate · GOV-08 · CUR-01 · CUR-04 · SW-01

Same asset or specimen batch on both sides of a split.
Future measurements used to predict an earlier state.
Scaling or feature selection fitted before splitting.
Repeated tuning against the final test set.
A statistically good target that does not support the intended decision.

Knowledge checksSource §Lesson 03 · Knowledge checks · GOV-08 · CUR-01 · CUR-04 · SW-01

Five review questions and answer rationales
Question	Answer rationale
Who defines the correct split?	The intended use and data-generating process, not a universal recipe.
What is leakage?	Information unavailable at prediction time or shared structure that gives an unrealistically easy test.
Why retain a baseline?	It shows whether AI adds useful value over an auditable existing approach.
When is a test set independent?	When it remains outside preprocessing, model and threshold choices until those choices are frozen.
Why examine slices?	Pooled averages can hide failure in a consequential subgroup or condition.

Use these as formative checks. Technical and editorial review remain pending.

Key points

Start from the accountable engineering decision and its consequence.
Compare against a transparent non-AI baseline.
Validate on a split that represents intended use and retain human authority.

Source references recorded by the supplied chapter

scikit-learn, Cross-validation: evaluating estimator performance.
scikit-learn, Common pitfalls: inconsistent preprocessing and data leakage.
NIST AI RMF 1.0, Map and Measure functions.