Javier García

Comment on Docket FDA-2026-N-7874
Sequential, harm-weighted benchmarking for GenAI-enabled devices used in critical care

Submitted by: Javier García, MD, intensivist (critical care physician), Buenos Aires, Argentina. Independent researcher in clinical AI evaluation. Comments submitted in a personal capacity.

I support the competency-based approach described in the discussion paper. My comments address premarket device benchmarking and postmarket monitoring for GenAI-enabled devices that support decisions for hospitalized and critically ill patients.

1. Benchmark how a device revises, not only what it answers.
In intensive care, the clinical picture changes over hours. A device that gives a correct initial assessment but fails to revise it when decisive new data arrive (for example, rising lactate, a positive blood culture, a new arrhythmia) may cause more harm than one that was initially wrong. Recent studies report that models often reach the correct diagnosis under sequential information delivery but fail to retain or revise it consistently. Single-snapshot vignettes cannot detect this failure mode. I recommend that benchmarking for devices intended for evolving clinical situations include staged scenarios in which information is revealed sequentially and scored at every stage.

2. Separate appropriate revision from instability using paired ("twin") scenarios.
A device that changes its recommendations frequently can appear responsive while being unstable or overly deferential to the user. I suggest paired scenarios that are identical up to a given stage: in one, decisive evidence warrants a change in management; in the other, non-decisive or irrelevant information is introduced. Reporting both the appropriate-revision rate and the inappropriate-revision rate, together with run-to-run variability across repeated runs, distinguishes genuine clinical updating from random variation. Final-answer accuracy alone does not capture this.

3. Weight errors by clinical harm.
Not all errors carry equal risk. For each decision point, clinicians from the intended-use specialty should predefine harm tiers (for example: suboptimal care, reversible harm, life-threatening harm). Benchmark results should report harm-weighted persistent-error rates, not only average scores.

4. Protect benchmark integrity against contamination.
Scenarios should be original and stage-locked (each stage frozen and time-stamped before the next is written), with a private holdout set that is never published, so that performance reflects clinical reasoning rather than memorization of public vignettes.

5. Use specialty-matched clinician adjudication.
Where the comparator is a clinician panel, adjudicators should be practicing clinicians from the device’s intended care setting (for example, intensivists for ICU devices), blinded to device identity, with inter-rater agreement reported. If an LLM is used as an adjudicator, it should first be validated against such a panel on the same task, with agreement thresholds stated in advance.

6. Re-benchmark whenever the underlying model changes.
GenAI-enabled devices often depend on foundation models updated by third parties. A fixed, version-controlled sequential test battery should be re-run whenever the underlying model, system prompt, or retrieval components change, with results compared against the prior version. This would make periodic benchmarking an operational postmarket control rather than a one-time premarket event.

7. Consider language and care-context generalizability.
Devices intended for Spanish-speaking patients, or for deployment outside the United States, should be benchmarked in those languages and care contexts, because clinical documentation, drug availability and practice patterns differ.

I would welcome the opportunity to contribute critical care scenarios and evaluation methodology to public or confidential benchmarking efforts.

Respectfully,
Javier García, MD
Intensivist, Buenos Aires, Argentina