Sarah Gebauer

Submitted by Sarah Gebauer, MD. I am an anesthesiologist and palliative care physician. I served on a hospital quality committee and helped build physician performance scorecards. I was a senior physician researcher at RAND on its AI evaluation team, led AI evaluation at MDCalc, and was the founder and CEO Validara Health, a company that evaluates clinical AI systems. These views are my own.

I support the discussion paper’s direction: evaluate generative AI devices on the elements of clinical judgment (triage, communication, knowing when to defer) and continue that evaluation after marketing through review of real interactions. I write to flag one problem that will decide whether the framework works. The paper proposes comparing devices to a "standard of care" or a "median clinician in practice." Neither comparator currently exists as measured data. If the agency does not address that, every evaluation will rest on an undefined reference point, and sponsors will define it for themselves.

1. The comparator column is empty

Medicine does not measure practicing physicians on most of the elements the paper lists. Board examinations test knowledge, pass or fail, and rarely after initial certification. Residency competency ratings are subjective, and programs have strong incentives to find residents competent. Hospital scorecards, which I helped build, largely track documentation and process measures that are easy to collect, not judgment or communication; the ones I built were easy to pass. Medicare’s physician scores have the same limitation, with a few exceptions such as cardiac surgery. The best evidence about what typical clinicians actually do on these elements comes from scattered research studies and, increasingly, from the very AI comparison studies the framework would govern. A comparator built from those sources is circular.

2. What "median clinician" should mean

I recommend that any claim that a device performs at the level of a standard of care or median clinician be supported by measuring real clinicians on the same cases and the same elements, under the same blinding, as the device. Specifically:

a. Standardized cases, not outcomes. Compare device and clinicians on a shared bank of standardized cases scored on the paper’s elements. Outcome-based comparisons on real patients penalize whoever takes the sickest patients and cannot be blinded.

b. A described reference population. "Median clinician" should state who was measured: specialty, setting, experience, and how many. A median of two academic attendings is not a median of practice.

c. Element-level reporting. Report device and clinician performance per element (triage, communication, deferral, uncertainty, recognizing a wrong premise), not as a single accuracy figure. A device can match clinicians on diagnosis and fail on telling a patient what it does not know.

d. Independent reviewers with stated agreement. Where physician reviewers score answers, report inter-reviewer agreement and the rubric. In my evaluation work, the hardest and least standardized step is deciding what "good enough" looks like for a given answer; agreement between reviewers is often lower than sponsors assume, and the disagreements are substantive (which guideline, how strong a warning).

3. Postmarket review of real interactions

I support random sampling of real interactions by independent clinicians. Two additions: reviewers should be independent of the sponsor and compensated by a mechanism the sponsor does not control, and the sampling frame should oversample the interactions where harm concentrates (uncertainty, deferral, patients who are not the typical case), not a uniform random draw.

4. Make the measurement useful to patients

Whatever reference data the agency assembles for devices will be the first systematic measurement of practicing clinicians on these elements. It should not stay inside premarket submissions. Reporting device performance against a described clinician reference, in plain terms, would give patients information they have never had about either. The same data would help answer the question medicine has avoided: what is good enough? The standard written for machines will become the standard physicians are held to. That is appropriate, and the agency should expect and welcome it.

Thank you for the opportunity to comment.

Sarah Gebauer, MD