This vignette evaluates
genome-based predictions on real, public data: Klebsiella
pneumoniae isolates from NCBI Pathogen Detection with laboratory
susceptibility results for meropenem and ciprofloxacin. The predictions
come from simple rules applied to AMRFinderPlus genotypes, two per
antimicrobial (see ?amr_example_data). The rules are
illustrative; the point is the evaluation.
library(AMRvalid)
d <- amr_example_data()
x <- amr_import_predictions(d)
x
#> <amr_data> 4754 predictions | 1255 isolates | 2 species-drug strata | 4 model(s) | type: binary
#> isolate_id species drug truth_sir prediction_type prediction
#> 1 SAMN03455989 B_KLBSL_PNMN CIP S binary S
#> 2 SAMN03455990 B_KLBSL_PNMN CIP R binary R
#> 3 SAMN03455995 B_KLBSL_PNMN CIP S binary S
#> 4 SAMN03892119 B_KLBSL_PNMN CIP R binary R
#> 5 SAMN03892120 B_KLBSL_PNMN CIP R binary R
#> 6 SAMN03892126 B_KLBSL_PNMN CIP R binary R
#> model_id dataset_id genomic_cluster
#> 1 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 <NA>
#> 2 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 PDS000006932.31
#> 3 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 <NA>
#> 4 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 PDS000045528.1
#> 5 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 PDS000011646.2
#> 6 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 PDS000045528.1
#> country collection_date bioproject collection_year amrfinder_version
#> 1 USA <NA> PRJNA279669 2014 4.2.7
#> 2 USA <NA> PRJNA430813 2014 4.2.7
#> 3 USA <NA> PRJNA279675 2015 4.2.7
#> 4 USA <NA> PRJNA278886 2015 4.2.7
#> 5 USA <NA> PRJNA278886 2012 4.2.7
#> 6 USA <NA> PRJNA278886 2015 4.2.7
#> # ... with 4748 more rowss <- amr_data_summary(x)
s[, c("model_id", "drug_name", "n_isolates", "n_S", "n_I_SDD", "n_R", "prevalence_R")]
#> model_id drug_name n_isolates n_S n_I_SDD n_R
#> 1 carbapenemase genes Meropenem 1183 514 49 620
#> 2 carbapenemase or porin + ESBL/AmpC Meropenem 1183 514 49 620
#> 3 QRDR mutations Ciprofloxacin 1194 285 52 857
#> 4 QRDR mutations or PMQR genes Ciprofloxacin 1194 285 52 857
#> prevalence_R
#> 1 0.5467372
#> 2 0.5467372
#> 3 0.7504378
#> 4 0.7504378Isolates come from 52 studies (BioProjects). Many are relatives: 867 isolates fall into 434 NCBI SNP clusters, and the largest cluster holds 30 isolates. That structure matters below.
amr_validate(x)
#> <amr_validation> OK | 0 error(s), 0 warning(s), 2 note(s)
#> i intermediate: Reference contains I or SDD results; binary metrics need an explicit policy (see `intermediate` in amr_metrics()). [202]
#> i breakpoint_standard: No 'breakpoint_standard' column; record the breakpoint authority and version for reproducibility.The reference contains intermediate (I) results. The rules only predict S or R, so I is handled explicitly: binary metrics exclude it (the default), and the AST agreement metrics count it as a minor error.
e <- amr_evaluate(x)
key <- c("sensitivity", "specificity", "categorical_agreement",
"very_major_error", "major_error", "minor_error")
tab <- as.data.frame(e)[e$metric %in% key, c("drug_name", "model_id", "metric", "estimate", "lower", "upper")]
tab[, c("estimate", "lower", "upper")] <- round(tab[, c("estimate", "lower", "upper")], 3)
rownames(tab) <- NULL
tab
#> drug_name model_id metric
#> 1 Meropenem carbapenemase genes sensitivity
#> 2 Meropenem carbapenemase genes specificity
#> 3 Meropenem carbapenemase genes categorical_agreement
#> 4 Meropenem carbapenemase genes very_major_error
#> 5 Meropenem carbapenemase genes major_error
#> 6 Meropenem carbapenemase genes minor_error
#> 7 Meropenem carbapenemase or porin + ESBL/AmpC sensitivity
#> 8 Meropenem carbapenemase or porin + ESBL/AmpC specificity
#> 9 Meropenem carbapenemase or porin + ESBL/AmpC categorical_agreement
#> 10 Meropenem carbapenemase or porin + ESBL/AmpC very_major_error
#> 11 Meropenem carbapenemase or porin + ESBL/AmpC major_error
#> 12 Meropenem carbapenemase or porin + ESBL/AmpC minor_error
#> 13 Ciprofloxacin QRDR mutations sensitivity
#> 14 Ciprofloxacin QRDR mutations specificity
#> 15 Ciprofloxacin QRDR mutations categorical_agreement
#> 16 Ciprofloxacin QRDR mutations very_major_error
#> 17 Ciprofloxacin QRDR mutations major_error
#> 18 Ciprofloxacin QRDR mutations minor_error
#> 19 Ciprofloxacin QRDR mutations or PMQR genes sensitivity
#> 20 Ciprofloxacin QRDR mutations or PMQR genes specificity
#> 21 Ciprofloxacin QRDR mutations or PMQR genes categorical_agreement
#> 22 Ciprofloxacin QRDR mutations or PMQR genes very_major_error
#> 23 Ciprofloxacin QRDR mutations or PMQR genes major_error
#> 24 Ciprofloxacin QRDR mutations or PMQR genes minor_error
#> estimate lower upper
#> 1 0.927 0.904 0.945
#> 2 0.903 0.874 0.925
#> 3 0.878 0.858 0.896
#> 4 0.073 0.055 0.096
#> 5 0.097 0.075 0.126
#> 6 0.041 0.031 0.054
#> 7 0.969 0.953 0.980
#> 8 0.852 0.819 0.880
#> 9 0.878 0.858 0.896
#> 10 0.031 0.020 0.047
#> 11 0.148 0.120 0.181
#> 12 0.041 0.031 0.054
#> 13 0.793 0.765 0.819
#> 14 0.972 0.946 0.986
#> 15 0.802 0.778 0.823
#> 16 0.207 0.181 0.235
#> 17 0.028 0.014 0.054
#> 18 0.044 0.033 0.057
#> 19 0.992 0.983 0.996
#> 20 0.698 0.643 0.749
#> 21 0.879 0.859 0.896
#> 22 0.008 0.004 0.017
#> 23 0.302 0.251 0.357
#> 24 0.044 0.033 0.057The broader rule for each antimicrobial trades very major errors (resistant isolates called susceptible) for major errors (susceptible isolates called resistant).
Public collections over-represent outbreaks. Weighting each SNP cluster equally (isolates outside a cluster count as their own group) shows how the rules perform across distinct strains rather than across isolates.
k <- c("sensitivity", "specificity", "very_major_error", "major_error")
iso <- as.data.frame(amr_evaluate(x))
clu <- as.data.frame(amr_evaluate(x, weight_by = "genomic_cluster"))
cmp <- merge(iso[iso$metric %in% k, c("model_id", "metric", "estimate")],
clu[clu$metric %in% k, c("model_id", "metric", "estimate")],
by = c("model_id", "metric"), suffixes = c("_isolates", "_clusters"))
cmp$change <- cmp$estimate_clusters - cmp$estimate_isolates
cmp[, 3:5] <- round(cmp[, 3:5], 3)
cmp <- cmp[order(cmp$metric, cmp$model_id), ]
rownames(cmp) <- NULL
cmp
#> model_id metric estimate_isolates
#> 1 carbapenemase genes major_error 0.097
#> 2 carbapenemase or porin + ESBL/AmpC major_error 0.148
#> 3 QRDR mutations major_error 0.028
#> 4 QRDR mutations or PMQR genes major_error 0.302
#> 5 carbapenemase genes sensitivity 0.927
#> 6 carbapenemase or porin + ESBL/AmpC sensitivity 0.969
#> 7 QRDR mutations sensitivity 0.793
#> 8 QRDR mutations or PMQR genes sensitivity 0.992
#> 9 carbapenemase genes specificity 0.903
#> 10 carbapenemase or porin + ESBL/AmpC specificity 0.852
#> 11 QRDR mutations specificity 0.972
#> 12 QRDR mutations or PMQR genes specificity 0.698
#> 13 carbapenemase genes very_major_error 0.073
#> 14 carbapenemase or porin + ESBL/AmpC very_major_error 0.031
#> 15 QRDR mutations very_major_error 0.207
#> 16 QRDR mutations or PMQR genes very_major_error 0.008
#> estimate_clusters change
#> 1 0.100 0.003
#> 2 0.152 0.004
#> 3 0.029 0.000
#> 4 0.284 -0.018
#> 5 0.902 -0.026
#> 6 0.958 -0.011
#> 7 0.723 -0.070
#> 8 0.989 -0.003
#> 9 0.898 -0.005
#> 10 0.846 -0.006
#> 11 0.972 0.000
#> 12 0.712 0.014
#> 13 0.099 0.026
#> 14 0.042 0.011
#> 15 0.273 0.067
#> 16 0.011 0.003The very major error rate is higher for all four rules once each cluster counts once, and QRDR-only sensitivity falls from 0.79 to 0.72: well-predicted outbreak clones make the isolate-level numbers look better than performance across distinct strains.
mero <- x[x$drug == "MEM", ]
b_iso <- amr_bootstrap(mero, resample_by = "isolate_id", B = 300, seed = 1)
b_clu <- amr_bootstrap(mero, resample_by = "genomic_cluster", B = 300, seed = 1)
vme <- function(b, unit) {
b <- as.data.frame(b)
r <- b[b$metric == "very_major_error", ]
data.frame(model = r$model_id, resampled = unit, groups = r$n_groups,
estimate = round(r$estimate, 3), lower = round(r$lower, 3), upper = round(r$upper, 3))
}
rbind(vme(b_iso, "isolates"), vme(b_clu, "SNP clusters"))
#> model resampled groups estimate lower upper
#> 1 carbapenemase genes isolates 1134 0.073 0.055 0.092
#> 2 carbapenemase or porin + ESBL/AmpC isolates 1134 0.031 0.018 0.045
#> 3 carbapenemase genes SNP clusters 739 0.073 0.050 0.096
#> 4 carbapenemase or porin + ESBL/AmpC SNP clusters 739 0.031 0.018 0.047Resampling clusters rather than isolates widens the intervals, because the 1134 isolates carry the information of only 739 independent groups.
amr_paired_compare(mero, models = c("carbapenemase genes", "carbapenemase or porin + ESBL/AmpC"),
resample_by = "genomic_cluster", B = 300, seed = 1)
#> <amr_paired_compare> carbapenemase or porin + ESBL/AmpC - carbapenemase genes | resampling unit: genomic_cluster | B = 300 | 95% percentile intervals
#> drug metric carbapenemase genes
#> Meropenem sensitivity 0.927
#> Meropenem specificity 0.903
#> Meropenem ppv 0.920
#> Meropenem npv 0.912
#> Meropenem accuracy 0.916
#> Meropenem balanced_accuracy 0.915
#> Meropenem f1 0.924
#> Meropenem mcc 0.831
#> Meropenem very_major_error 0.073
#> Meropenem major_error 0.097
#> carbapenemase or porin + ESBL/AmpC difference interval matched
#> 0.969 0.042 [0.024, 0.063] 1134
#> 0.852 -0.051 [-0.070, -0.031] 1134
#> 0.888 -0.032 [-0.047, -0.018] 1134
#> 0.958 0.047 [0.027, 0.067] 1134
#> 0.916 0.000 [-0.013, 0.013] 1134
#> 0.911 -0.004 [-0.016, 0.011] 1134
#> 0.927 0.003 [-0.008, 0.016] 1134
#> 0.834 0.003 [-0.022, 0.030] 1134
#> 0.031 -0.042 [-0.063, -0.024] 1134
#> 0.148 0.051 [0.031, 0.070] 1134
#> McNemar exact p (accuracy, assumes independent isolates): Meropenem 1The broader rule lowers the very major error rate and raises the major error rate by a similar amount, leaving overall accuracy unchanged. Which rule is preferable depends on which error matters more in practice.
x$period <- cut(d$collection_year, c(-Inf, 2016, 2020, Inf), labels = c("up to 2016", "2017-2020", "2021 on"))
by_period <- as.data.frame(amr_evaluate(x, by = "period"))
w <- by_period[by_period$metric == "very_major_error" & by_period$period != "(missing)",
c("model_id", "period", "denominator", "estimate", "lower", "upper")]
names(w)[3] <- "n_resistant"
w[, 4:6] <- round(w[, 4:6], 3)
rownames(w) <- NULL
w
#> model_id period n_resistant estimate lower
#> 1 carbapenemase genes up to 2016 215 0.065 0.039
#> 2 carbapenemase genes 2017-2020 132 0.076 0.042
#> 3 carbapenemase genes 2021 on 179 0.073 0.043
#> 4 carbapenemase or porin + ESBL/AmpC up to 2016 215 0.033 0.016
#> 5 carbapenemase or porin + ESBL/AmpC 2017-2020 132 0.015 0.004
#> 6 carbapenemase or porin + ESBL/AmpC 2021 on 179 0.050 0.027
#> 7 QRDR mutations up to 2016 281 0.110 0.079
#> 8 QRDR mutations 2017-2020 142 0.085 0.049
#> 9 QRDR mutations 2021 on 305 0.377 0.325
#> 10 QRDR mutations or PMQR genes up to 2016 281 0.000 0.000
#> 11 QRDR mutations or PMQR genes 2017-2020 142 0.007 0.001
#> 12 QRDR mutations or PMQR genes 2021 on 305 0.016 0.007
#> upper
#> 1 0.106
#> 2 0.134
#> 3 0.120
#> 4 0.066
#> 5 0.054
#> 6 0.093
#> 7 0.152
#> 8 0.142
#> 9 0.433
#> 10 0.013
#> 11 0.039
#> 12 0.038by = "period" evaluates each period separately within
each predictor; isolates without a collection year form a
"(missing)" group, left out of this table. The QRDR-only
rule misses far more resistant isolates collected from 2021 on, while
the rule that also counts plasmid-mediated quinolone resistance genes
stays accurate. Changes over time can reflect new resistance mechanisms,
changing breakpoints or different submitting studies; the table shows
that performance is not stable, not why.
dataset_info <- list(source = "NCBI Pathogen Detection PDG000000012.2568",
breakpoint_standard = "As submitted to BioSample (not recorded)")
model_info <- list(description = "Rule-based interpretation of AMRFinderPlus calls",
training_data = "None (rules, not trained models)",
threshold_selection = "Not applicable (binary rules)")
report <- amr_report(x, e, b_clu, dataset_info = dataset_info, model_info = model_info,
file = file.path(tempdir(), "ncbi-report.md"))
p <- amr_provenance(x, e, b_clu, dataset_info = dataset_info, model_info = model_info)
table(p$status)
#>
#> checked not assessable supplied
#> 16 9 5
p[p$status == "not assessable", c("tripod_item", "item")]
#> tripod_item item
#> 3 6a Setting
#> 4 6b Eligibility criteria
#> 9 8a Reference AST method
#> 14 12a Separation of training and test isolates
#> 17 12g How predictions were produced
#> 18 13 Class imbalance handling
#> 19 14 Fairness considerations
#> 27 18e Data availability
#> 28 18f Code availabilityThe report lists every TRIPOD+AI item it covers and flags the ones still missing. The list above is the author’s to-do list before submission.
amr_validate_splits() (see the
vignette on population-aware evaluation).