Why population-aware evaluation matters

Bacterial isolates are not independent. Isolates from the same lineage share most of their genome and, often, their resistance phenotype. A predictor can look accurate simply because test isolates have close relatives in the training data. This vignette uses simulated data, where the truth is known, to show how AMRvalid exposes that.

library(AMRvalid)

A simulated collection

amr_simulate() creates isolates in lineages. Resistance depends on a causal marker and on a lineage effect that is inherited clonally but not explained by the marker. Two fixed predictors are then applied:

  • knn predicts from the 5 genomically nearest training isolates, so it memorises lineages;
  • marker predicts from the marker alone, so it learns the mechanism.

Each is evaluated under a random split and a lineage-held-out split, and on an external set of lineages never seen in training.

sim <- amr_simulate(n_lineages = 30, isolates_per_lineage = 15,
                    external_lineages = 30, seed = 2026)
x <- amr_import_predictions(sim$predictions)
x
#> <amr_data> 2160 predictions | 613 isolates | 1 species-drug strata | 4 model(s) | type: score
#>   isolate_id      species drug truth_sir prediction_type probability   model_id
#> 1      D0007 B_KLBSL_PNMN  MEM         S           score         0.6 knn/random
#> 2      D0016 B_KLBSL_PNMN  MEM         R           score         0.8 knn/random
#> 3      D0025 B_KLBSL_PNMN  MEM         R           score         0.8 knn/random
#> 4      D0031 B_KLBSL_PNMN  MEM         S           score         0.2 knn/random
#> 5      D0035 B_KLBSL_PNMN  MEM         S           score         0.2 knn/random
#> 6      D0043 B_KLBSL_PNMN  MEM         R           score         0.0 knn/random
#>   dataset_id split lineage
#> 1   internal  test   DL001
#> 2   internal  test   DL002
#> 3   internal  test   DL002
#> 4   internal  test   DL003
#> 5   internal  test   DL003
#> 6   internal  test   DL003
#> # ... with 2154 more rows

Step 1: check the split

Before trusting any number, check what the split actually separates.

checks <- amr_validate_splits(x, train_manifest = sim$train_manifest, group = "lineage")
checks$consistent_designs
#>         model_id isolate lineage temporal geographic
#> 1     knn/random    TRUE   FALSE       NA         NA
#> 2  marker/random    TRUE   FALSE       NA         NA
#> 3    knn/lineage    TRUE    TRUE       NA         NA
#> 4 marker/lineage    TRUE    TRUE       NA         NA

Both designs keep isolates apart, but only the lineage-held-out design keeps lineages apart. Claiming lineage-level generalisation for the random split fails:

random_only <- x[x$model_id == "knn/random" & x$dataset_id == "internal", ]
amr_validate_splits(random_only, train_manifest = sim$train_manifest,
                    group = "lineage", claimed_design = "group")
#> <amr_split_check> PROBLEMS FOUND | claimed design: group
#>  x Split is not consistent with: group
#>  v [knn/random] isolate: 0 of 90 test isolates also in training (0% of test isolates)
#>  ! [knn/random] lineage: 29 of 29 test lineage value(s) also in training (100% of test isolates)
#>  i AMRvalid checks supplied metadata only; it cannot certify that upstream feature selection
#>    or tuning used training data alone.

Step 2: internal versus external performance

The external lineages approximate what happens when the predictor meets new strains.

e <- suppressMessages(amr_evaluate(x, validate = FALSE))
auc <- as.data.frame(e)[e$metric == "roc_auc", c("model_id", "dataset_id", "estimate")]
auc_wide <- reshape(auc, idvar = "model_id", timevar = "dataset_id", direction = "wide")
names(auc_wide) <- c("model", "external", "internal")
auc_wide$optimism <- auc_wide$internal - auc_wide$external
auc_wide[, c("model", "internal", "external", "optimism")]
#>             model  internal  external    optimism
#> 4     knn/lineage 0.6221719 0.4414151  0.18075680
#> 16     knn/random 0.8345734 0.4501534  0.38442002
#> 28 marker/lineage 0.7684766 0.7932212 -0.02474456
#> 40  marker/random 0.7708333 0.7932212 -0.02238784

The memorising predictor looks good under a random split and collapses on new lineages. The lineage-held-out split is less optimistic, though with only six test lineages it is still noisy. The marker predictor is stable because it learned the mechanism.

Step 3: performance against distance to training

amr_distance_eval() makes the same point without needing an external dataset: it asks how performance changes as test isolates become less similar to anything in training.

knn_random <- x[x$model_id == "knn/random", ]
knn_random$dataset_id <- "all test isolates"
# Isolates within a lineage are less than about 20 units apart in this simulation
d <- amr_distance_eval(knn_random, distances = sim$distances,
                       train_manifest = sim$train_manifest, bands = c(0, 20, 100, Inf))
table(d$nearest$band)
#> 
#>    [0,20)  [20,100) [100,Inf] 
#>       106       273       161
d$trend[, c("model_id", "model", "effect", "lower", "upper", "n")]
#>     model_id                                            model     effect
#> 1 knn/random linear: squared error ~ distance (change per SD) 0.04067345
#>         lower      upper   n
#> 1 0.008731224 0.07261567 540

A positive slope for the squared error means predictions get worse with distance. Test isolates with a relative from the same lineage in training are easy; isolates from other lineages are close to a coin toss.

Step 4: intervals that respect clustering

Resampling single isolates treats related isolates as independent and gives intervals that are too narrow. Resampling whole lineages does not. Here both are applied to the marker predictor on the external set (30 lineages of 15 isolates).

ext <- x[x$model_id == "marker/lineage" & x$dataset_id == "external", ]
by_isolate <- as.data.frame(amr_bootstrap(ext, resample_by = "isolate_id", B = 300, seed = 1))
by_lineage <- as.data.frame(amr_bootstrap(ext, resample_by = "lineage", B = 300, seed = 1))
ci <- rbind(
  isolate = unlist(by_isolate[by_isolate$metric == "roc_auc", c("estimate", "lower", "upper")]),
  lineage = unlist(by_lineage[by_lineage$metric == "roc_auc", c("estimate", "lower", "upper")])
)
cbind(ci, width = ci[, "upper"] - ci[, "lower"])
#>          estimate     lower     upper      width
#> isolate 0.7932212 0.7490463 0.8293799 0.08033354
#> lineage 0.7932212 0.7199368 0.8560544 0.13611757

The lineage interval is substantially wider because 450 isolates carry the information of only 30 independent lineages. The isolate-level interval overstates precision.

Key points

  1. Check the split with amr_validate_splits() before reporting performance.
  2. Prefer lineage- or cluster-held-out designs, or external datasets, when the claim is about new strains.
  3. Use amr_distance_eval() to show how performance depends on similarity to the training data.
  4. Use amr_bootstrap() with resample_by = "lineage" (or genomic clusters) so the uncertainty reflects the number of independent lineages, not isolates.