Evaluating rule-based predictors on public data

This vignette evaluates genome-based predictions on real, public data: Klebsiella pneumoniae isolates from NCBI Pathogen Detection with laboratory susceptibility results for meropenem and ciprofloxacin. The predictions come from simple rules applied to AMRFinderPlus genotypes, two per antimicrobial (see ?amr_example_data). The rules are illustrative; the point is the evaluation.

library(AMRvalid)
d <- amr_example_data()
x <- amr_import_predictions(d)
x
#> <amr_data> 4754 predictions | 1255 isolates | 2 species-drug strata | 4 model(s) | type: binary
#>     isolate_id      species drug truth_sir prediction_type prediction
#> 1 SAMN03455989 B_KLBSL_PNMN  CIP         S          binary          S
#> 2 SAMN03455990 B_KLBSL_PNMN  CIP         R          binary          R
#> 3 SAMN03455995 B_KLBSL_PNMN  CIP         S          binary          S
#> 4 SAMN03892119 B_KLBSL_PNMN  CIP         R          binary          R
#> 5 SAMN03892120 B_KLBSL_PNMN  CIP         R          binary          R
#> 6 SAMN03892126 B_KLBSL_PNMN  CIP         R          binary          R
#>         model_id                                dataset_id genomic_cluster
#> 1 QRDR mutations NCBI Pathogen Detection PDG000000012.2568            <NA>
#> 2 QRDR mutations NCBI Pathogen Detection PDG000000012.2568 PDS000006932.31
#> 3 QRDR mutations NCBI Pathogen Detection PDG000000012.2568            <NA>
#> 4 QRDR mutations NCBI Pathogen Detection PDG000000012.2568  PDS000045528.1
#> 5 QRDR mutations NCBI Pathogen Detection PDG000000012.2568  PDS000011646.2
#> 6 QRDR mutations NCBI Pathogen Detection PDG000000012.2568  PDS000045528.1
#>   country collection_date  bioproject collection_year amrfinder_version
#> 1     USA            <NA> PRJNA279669            2014             4.2.7
#> 2     USA            <NA> PRJNA430813            2014             4.2.7
#> 3     USA            <NA> PRJNA279675            2015             4.2.7
#> 4     USA            <NA> PRJNA278886            2015             4.2.7
#> 5     USA            <NA> PRJNA278886            2012             4.2.7
#> 6     USA            <NA> PRJNA278886            2015             4.2.7
#> # ... with 4748 more rows

Step 1: what is in the data?

s <- amr_data_summary(x)
s[, c("model_id", "drug_name", "n_isolates", "n_S", "n_I_SDD", "n_R", "prevalence_R")]
#>                             model_id     drug_name n_isolates n_S n_I_SDD n_R
#> 1                carbapenemase genes     Meropenem       1183 514      49 620
#> 2 carbapenemase or porin + ESBL/AmpC     Meropenem       1183 514      49 620
#> 3                     QRDR mutations Ciprofloxacin       1194 285      52 857
#> 4       QRDR mutations or PMQR genes Ciprofloxacin       1194 285      52 857
#>   prevalence_R
#> 1    0.5467372
#> 2    0.5467372
#> 3    0.7504378
#> 4    0.7504378

Isolates come from 52 studies (BioProjects). Many are relatives: 867 isolates fall into 434 NCBI SNP clusters, and the largest cluster holds 30 isolates. That structure matters below.

amr_validate(x)
#> <amr_validation> OK | 0 error(s), 0 warning(s), 2 note(s)
#>  i intermediate: Reference contains I or SDD results; binary metrics need an explicit policy (see `intermediate` in amr_metrics()). [202]
#>  i breakpoint_standard: No 'breakpoint_standard' column; record the breakpoint authority and version for reproducibility.

The reference contains intermediate (I) results. The rules only predict S or R, so I is handled explicitly: binary metrics exclude it (the default), and the AST agreement metrics count it as a minor error.

Step 2: performance per predictor

e <- amr_evaluate(x)
key <- c("sensitivity", "specificity", "categorical_agreement",
         "very_major_error", "major_error", "minor_error")
tab <- as.data.frame(e)[e$metric %in% key, c("drug_name", "model_id", "metric", "estimate", "lower", "upper")]
tab[, c("estimate", "lower", "upper")] <- round(tab[, c("estimate", "lower", "upper")], 3)
rownames(tab) <- NULL
tab
#>        drug_name                           model_id                metric
#> 1      Meropenem                carbapenemase genes           sensitivity
#> 2      Meropenem                carbapenemase genes           specificity
#> 3      Meropenem                carbapenemase genes categorical_agreement
#> 4      Meropenem                carbapenemase genes      very_major_error
#> 5      Meropenem                carbapenemase genes           major_error
#> 6      Meropenem                carbapenemase genes           minor_error
#> 7      Meropenem carbapenemase or porin + ESBL/AmpC           sensitivity
#> 8      Meropenem carbapenemase or porin + ESBL/AmpC           specificity
#> 9      Meropenem carbapenemase or porin + ESBL/AmpC categorical_agreement
#> 10     Meropenem carbapenemase or porin + ESBL/AmpC      very_major_error
#> 11     Meropenem carbapenemase or porin + ESBL/AmpC           major_error
#> 12     Meropenem carbapenemase or porin + ESBL/AmpC           minor_error
#> 13 Ciprofloxacin                     QRDR mutations           sensitivity
#> 14 Ciprofloxacin                     QRDR mutations           specificity
#> 15 Ciprofloxacin                     QRDR mutations categorical_agreement
#> 16 Ciprofloxacin                     QRDR mutations      very_major_error
#> 17 Ciprofloxacin                     QRDR mutations           major_error
#> 18 Ciprofloxacin                     QRDR mutations           minor_error
#> 19 Ciprofloxacin       QRDR mutations or PMQR genes           sensitivity
#> 20 Ciprofloxacin       QRDR mutations or PMQR genes           specificity
#> 21 Ciprofloxacin       QRDR mutations or PMQR genes categorical_agreement
#> 22 Ciprofloxacin       QRDR mutations or PMQR genes      very_major_error
#> 23 Ciprofloxacin       QRDR mutations or PMQR genes           major_error
#> 24 Ciprofloxacin       QRDR mutations or PMQR genes           minor_error
#>    estimate lower upper
#> 1     0.927 0.904 0.945
#> 2     0.903 0.874 0.925
#> 3     0.878 0.858 0.896
#> 4     0.073 0.055 0.096
#> 5     0.097 0.075 0.126
#> 6     0.041 0.031 0.054
#> 7     0.969 0.953 0.980
#> 8     0.852 0.819 0.880
#> 9     0.878 0.858 0.896
#> 10    0.031 0.020 0.047
#> 11    0.148 0.120 0.181
#> 12    0.041 0.031 0.054
#> 13    0.793 0.765 0.819
#> 14    0.972 0.946 0.986
#> 15    0.802 0.778 0.823
#> 16    0.207 0.181 0.235
#> 17    0.028 0.014 0.054
#> 18    0.044 0.033 0.057
#> 19    0.992 0.983 0.996
#> 20    0.698 0.643 0.749
#> 21    0.879 0.859 0.896
#> 22    0.008 0.004 0.017
#> 23    0.302 0.251 0.357
#> 24    0.044 0.033 0.057

The broader rule for each antimicrobial trades very major errors (resistant isolates called susceptible) for major errors (susceptible isolates called resistant).

Step 3: do outbreak clones flatter the results?

Public collections over-represent outbreaks. Weighting each SNP cluster equally (isolates outside a cluster count as their own group) shows how the rules perform across distinct strains rather than across isolates.

k <- c("sensitivity", "specificity", "very_major_error", "major_error")
iso <- as.data.frame(amr_evaluate(x))
clu <- as.data.frame(amr_evaluate(x, weight_by = "genomic_cluster"))
cmp <- merge(iso[iso$metric %in% k, c("model_id", "metric", "estimate")],
             clu[clu$metric %in% k, c("model_id", "metric", "estimate")],
             by = c("model_id", "metric"), suffixes = c("_isolates", "_clusters"))
cmp$change <- cmp$estimate_clusters - cmp$estimate_isolates
cmp[, 3:5] <- round(cmp[, 3:5], 3)
cmp <- cmp[order(cmp$metric, cmp$model_id), ]
rownames(cmp) <- NULL
cmp
#>                              model_id           metric estimate_isolates
#> 1                 carbapenemase genes      major_error             0.097
#> 2  carbapenemase or porin + ESBL/AmpC      major_error             0.148
#> 3                      QRDR mutations      major_error             0.028
#> 4        QRDR mutations or PMQR genes      major_error             0.302
#> 5                 carbapenemase genes      sensitivity             0.927
#> 6  carbapenemase or porin + ESBL/AmpC      sensitivity             0.969
#> 7                      QRDR mutations      sensitivity             0.793
#> 8        QRDR mutations or PMQR genes      sensitivity             0.992
#> 9                 carbapenemase genes      specificity             0.903
#> 10 carbapenemase or porin + ESBL/AmpC      specificity             0.852
#> 11                     QRDR mutations      specificity             0.972
#> 12       QRDR mutations or PMQR genes      specificity             0.698
#> 13                carbapenemase genes very_major_error             0.073
#> 14 carbapenemase or porin + ESBL/AmpC very_major_error             0.031
#> 15                     QRDR mutations very_major_error             0.207
#> 16       QRDR mutations or PMQR genes very_major_error             0.008
#>    estimate_clusters change
#> 1              0.100  0.003
#> 2              0.152  0.004
#> 3              0.029  0.000
#> 4              0.284 -0.018
#> 5              0.902 -0.026
#> 6              0.958 -0.011
#> 7              0.723 -0.070
#> 8              0.989 -0.003
#> 9              0.898 -0.005
#> 10             0.846 -0.006
#> 11             0.972  0.000
#> 12             0.712  0.014
#> 13             0.099  0.026
#> 14             0.042  0.011
#> 15             0.273  0.067
#> 16             0.011  0.003

The very major error rate is higher for all four rules once each cluster counts once, and QRDR-only sensitivity falls from 0.79 to 0.72: well-predicted outbreak clones make the isolate-level numbers look better than performance across distinct strains.

Step 4: intervals that respect clustering

mero <- x[x$drug == "MEM", ]
b_iso <- amr_bootstrap(mero, resample_by = "isolate_id", B = 300, seed = 1)
b_clu <- amr_bootstrap(mero, resample_by = "genomic_cluster", B = 300, seed = 1)
vme <- function(b, unit) {
  b <- as.data.frame(b)
  r <- b[b$metric == "very_major_error", ]
  data.frame(model = r$model_id, resampled = unit, groups = r$n_groups,
             estimate = round(r$estimate, 3), lower = round(r$lower, 3), upper = round(r$upper, 3))
}
rbind(vme(b_iso, "isolates"), vme(b_clu, "SNP clusters"))
#>                                model    resampled groups estimate lower upper
#> 1                carbapenemase genes     isolates   1134    0.073 0.055 0.092
#> 2 carbapenemase or porin + ESBL/AmpC     isolates   1134    0.031 0.018 0.045
#> 3                carbapenemase genes SNP clusters    739    0.073 0.050 0.096
#> 4 carbapenemase or porin + ESBL/AmpC SNP clusters    739    0.031 0.018 0.047

Resampling clusters rather than isolates widens the intervals, because the 1134 isolates carry the information of only 739 independent groups.

Step 5: comparing the two meropenem rules on the same isolates

amr_paired_compare(mero, models = c("carbapenemase genes", "carbapenemase or porin + ESBL/AmpC"),
                   resample_by = "genomic_cluster", B = 300, seed = 1)
#> <amr_paired_compare> carbapenemase or porin + ESBL/AmpC - carbapenemase genes | resampling unit: genomic_cluster | B = 300 | 95% percentile intervals
#>  drug      metric            carbapenemase genes
#>  Meropenem sensitivity       0.927              
#>  Meropenem specificity       0.903              
#>  Meropenem ppv               0.920              
#>  Meropenem npv               0.912              
#>  Meropenem accuracy          0.916              
#>  Meropenem balanced_accuracy 0.915              
#>  Meropenem f1                0.924              
#>  Meropenem mcc               0.831              
#>  Meropenem very_major_error  0.073              
#>  Meropenem major_error       0.097              
#>  carbapenemase or porin + ESBL/AmpC difference interval         matched
#>  0.969                              0.042      [0.024, 0.063]   1134   
#>  0.852                              -0.051     [-0.070, -0.031] 1134   
#>  0.888                              -0.032     [-0.047, -0.018] 1134   
#>  0.958                              0.047      [0.027, 0.067]   1134   
#>  0.916                              0.000      [-0.013, 0.013]  1134   
#>  0.911                              -0.004     [-0.016, 0.011]  1134   
#>  0.927                              0.003      [-0.008, 0.016]  1134   
#>  0.834                              0.003      [-0.022, 0.030]  1134   
#>  0.031                              -0.042     [-0.063, -0.024] 1134   
#>  0.148                              0.051      [0.031, 0.070]   1134   
#> McNemar exact p (accuracy, assumes independent isolates): Meropenem    1

The broader rule lowers the very major error rate and raises the major error rate by a similar amount, leaving overall accuracy unchanged. Which rule is preferable depends on which error matters more in practice.

Step 6: stability over time

x$period <- cut(d$collection_year, c(-Inf, 2016, 2020, Inf), labels = c("up to 2016", "2017-2020", "2021 on"))
by_period <- as.data.frame(amr_evaluate(x, by = "period"))
w <- by_period[by_period$metric == "very_major_error" & by_period$period != "(missing)",
               c("model_id", "period", "denominator", "estimate", "lower", "upper")]
names(w)[3] <- "n_resistant"
w[, 4:6] <- round(w[, 4:6], 3)
rownames(w) <- NULL
w
#>                              model_id     period n_resistant estimate lower
#> 1                 carbapenemase genes up to 2016         215    0.065 0.039
#> 2                 carbapenemase genes  2017-2020         132    0.076 0.042
#> 3                 carbapenemase genes    2021 on         179    0.073 0.043
#> 4  carbapenemase or porin + ESBL/AmpC up to 2016         215    0.033 0.016
#> 5  carbapenemase or porin + ESBL/AmpC  2017-2020         132    0.015 0.004
#> 6  carbapenemase or porin + ESBL/AmpC    2021 on         179    0.050 0.027
#> 7                      QRDR mutations up to 2016         281    0.110 0.079
#> 8                      QRDR mutations  2017-2020         142    0.085 0.049
#> 9                      QRDR mutations    2021 on         305    0.377 0.325
#> 10       QRDR mutations or PMQR genes up to 2016         281    0.000 0.000
#> 11       QRDR mutations or PMQR genes  2017-2020         142    0.007 0.001
#> 12       QRDR mutations or PMQR genes    2021 on         305    0.016 0.007
#>    upper
#> 1  0.106
#> 2  0.134
#> 3  0.120
#> 4  0.066
#> 5  0.054
#> 6  0.093
#> 7  0.152
#> 8  0.142
#> 9  0.433
#> 10 0.013
#> 11 0.039
#> 12 0.038

by = "period" evaluates each period separately within each predictor; isolates without a collection year form a "(missing)" group, left out of this table. The QRDR-only rule misses far more resistant isolates collected from 2021 on, while the rule that also counts plasmid-mediated quinolone resistance genes stays accurate. Changes over time can reflect new resistance mechanisms, changing breakpoints or different submitting studies; the table shows that performance is not stable, not why.

Step 7: a report for the paper

dataset_info <- list(source = "NCBI Pathogen Detection PDG000000012.2568",
                     breakpoint_standard = "As submitted to BioSample (not recorded)")
model_info <- list(description = "Rule-based interpretation of AMRFinderPlus calls",
                   training_data = "None (rules, not trained models)",
                   threshold_selection = "Not applicable (binary rules)")
report <- amr_report(x, e, b_clu, dataset_info = dataset_info, model_info = model_info,
                     file = file.path(tempdir(), "ncbi-report.md"))
p <- amr_provenance(x, e, b_clu, dataset_info = dataset_info, model_info = model_info)
table(p$status)
#> 
#>        checked not assessable       supplied 
#>             16              9              5
p[p$status == "not assessable", c("tripod_item", "item")]
#>    tripod_item                                     item
#> 3           6a                                  Setting
#> 4           6b                     Eligibility criteria
#> 9           8a                     Reference AST method
#> 14         12a Separation of training and test isolates
#> 17         12g            How predictions were produced
#> 18          13                 Class imbalance handling
#> 19          14                  Fairness considerations
#> 27         18e                        Data availability
#> 28         18f                        Code availability

The report lists every TRIPOD+AI item it covers and flags the ones still missing. The list above is the author’s to-do list before submission.

Caveats

  • Phenotypes were interpreted by the submitting laboratories under breakpoints that are not recorded, and may differ between studies.
  • The isolates are those with submitted antibiograms, not a random sample.
  • The rules have no training data, so no split checks apply here; for trained models, start with amr_validate_splits() (see the vignette on population-aware evaluation).