The Upstream Evidence Project (Cassandra C. Chou, Benjamin Q. Huynh)

Please see the attached public comment evaluating the two-axis framework, benchmarking elements and pre-market evaluation, synthetic data generation, and foundation models (questions 1, 9, 13, 14, and 24).

Commissioner Kyle Diamantas October 17, 2026
Food and Drug Administration
10903 New Hampshihre Avenue
Silver Spring, MD 20993

Re: Considerations for the Regulation of Generative AI-Enabled Medical Devices
Dear Commissioner Diamantas,

We write on behalf of The Upstream Evidence Project, a collective of scientists and policy experts who
conduct research and advocacy on public policy. We appreciate the opportunity to provide public comment on the considerations for the regulation of generative AI-enabled medical devices. Our comments
and recommendations are as follows.

Q1: Improvements to the two-axis framework
We believe the proposed two-axis framework, which measures risk in terms of independence of activity vs
consequences of relying on an incorrect output, does not adequately capture potential risk. Instead, we
recommend a standard risk decomposition where the horizontal axis represents the probability a wrong
output is carried out and the vertical axis represents the harm arising from taking action based on a wrong
output.

First, independence is not a strong determinant of risk: informational or non-directive functions may still
pose high risk. For instance, plaintiffs in Raine vs. OpenAI allege that ChatGPT conversation contributed
to a teenager's suicide; a situation where GenAI had no directive or decision-making role. 1 Numerous
other wrongful death suits have been filed against major developers. It has also been shown that AI models
tend to exercise sycophancy, responding to match user beliefs rather than the truth. 2,3 This poses a risk
of models reinforcing a user's negative emotions or impulsive, harmful action. As a result, device activity,
or intended activity, is a weak proxy for influence and potential risk.

Section IV asks when and whether patient-facing functions, where users may lack the domain knowledge
to independently evaluate the output, should be considered higher risk. We agree that a user's lack of domain knowledge indicates higher risk—which applies to both patients using AI models for medical advice
and potentially general physicians using models for specialized medical information—but also recognize
that domain knowledge does not guarantee error detection. In a randomized study, systematically biased
predictions from a diagnostic AI model reduced clinician accuracy by 11.3 percentage points. 4 This auHigh
Higher
Specialized expected harm

Expected harm from following a wrong output
clinician tool

Fully autonomous
LLM medical coder

General−purpose chatbot

Low
Low High
Probability of carrying out a wrong output

Figure 1: Proposed two-axis framework.

tomation bias, the tendency to trust an automated system's output, affects trained and untrained users
alike 5,6 and should factor into risk placement regardless of the intended user. Moreover, clinician-facing
tasks may contain higher stakes than patient facing tasks, indicating that the domain knowledge of the
user may not be the most effective indicator for harm potential. Instead, we recommend CDRH should
base risk calculation on measured performance of the user and device together for both patient-facing and
HCP-facing functions.

As an alternative, we recommend a different set of axes for measuring potential risk, where the horizontal
axis represents the probability of carrying out a wrong output, and the vertical axis represents the expected
harm from following the wrong output (Figure 1). Formally, the horizontal axis decomposes into:

𝑃 (carrying out wrong output) = 𝑃 (wrong output) × 𝑃 (User doesn't catch error|wrong output)

The horizontal axis can therefore be measured by developers as "how likely is it for a user to take action
based on the wrong output," reportable in standard predictive performance metrics.

Points should have error bars along both axes that represent variation in user, predictive performance, and
potential harm of an action. We believe this better addresses the concerns that expertise is not a proxy for
independence from automation bias and informational-only AI models can still drive severe outcomes,
and better captures user performance.

Consider the following the three examples of how this framework might be used. A general-purpose chatbot has a large pool of users with varying amounts of domain knowledge and high variation in predictive
performance based on the task. As a result, it would have a large error bar in the horizontal direction.
Similarly, due to the high variation in possible tasks with variable harm, it would have a large error bar in
the vertical direction. A specialized clinician tool, on the other hand, would have a much more limited
scope in task, though the task is likely higher stakes, and less variation in domain knowledge. It would
still have error bars in both directions, but they would be considerably smaller than in the general-purpose
chatbot case. The third example is that of a fully autonomous LLM medical coder, which has been shown
to have poor performance in generating correct medical codes. 7

Q9: Robustness and Subgroup Performance should be mandatory
Section V.B.1 indicates that not all benchmarking elements will apply to all devices. We recommend that
the generalizability metrics (robustness/reliability/reproducibility and subgroup performance) should be
applied to all devices.

AI models can give different answers to the same question or questions with slightly variable wording
or formatting, 8 indicating potential reliability concerns for all GenAI-enabled medical devices. Element
R.1 contemplates testing reproducibility across repeated runs and paraphrases; we recommend that this
testing be mandatory, with the number of runs and variants prespecified, and that passing thresholds be
computed over all runs and variants rather than a single best run.

We further recommend that Element R.2 should be mandatory and require passing thresholds for each
relevant subgroups rather than an overall score. For instance, sponsors should be required to report underescalation and over-escalation rates by subgroup, as aggregate measures may not detect concentrations of
under- or over-escalation in certain populations. Prior work illustrates that LLMs can perform differently
across subgroups: Omar et al. found that "LGBTQIA+ subgroups were recommended mental health
assessments approximately six to seven times more often than clinically indicated." 9 All models may be
susceptible to this type of subgroup bias, so subgroup performance should be mandatory for all devices.

Q13: Synthetic data should not be generated from similar models to the device under
evaluation, and should be representative of the subgroup
Section V.C.1 states that synthetic data “modeled on real patient data could potentially supplement evaluation” for small or hard-to-sample subgroups, if it is representative of the deployment environment. As
written, this passage can be read in two different ways: is synthetic data supplementing real data from
the subgroup, or supplementing the overall evaluation such that the entire subgroup may entirely be represented by synthetic cases? CDRH should clarify that synthetic data may supplement, not replace, real
patient inputs from any subgroup.

Further, synthetic data must be representative of the subgroup itself, not only of the deployment environment. When the real data the synthetic cases are modeled on contains scarce subgroup data, the synthetic
data inherits and amplifies these gaps. Furthermore, creating synthetic data typically requires rich data
to begin with—the subgroups requiring additional data are therefore the groups for which synthetic data
generation would be least reliable. Shumailov et al. showed that model-generated data underrepresents
the tails of a distribution, 10 and small subgroups often fail in those tails. Additionally, sponsors must
ensure that synthetic data is not generated by the same base model as the one being tested, as this can
inflate performance. Sponsors should disclose which model generated the synthetic data and provide
justification for its independence from the testing model.

Benchmarking acceptance criteria should include uncertainty estimation to account for
AI output variability
CDRH should require a confidence interval for any accuracy metrics, 11 where the lower bound of the
confidence interval is required to meet a passing threshold. The FDA already applies this standard in 21
CFR §866.3328(b)(1): the special controls for influenza antigen detection tests require that the lower
bound of the 95% confidence interval for sensitivity and specificity meet a minimum value, not only the
point estimate. 12

Q14: Reference standards and comparative groups should be considered separately
Section V.D.1 describes that "performance might be compared to that of a panel of qualified clinicians
whose consensus reflects the applicable standard of care, or to that of a median clinician in practice."
We recommend separating the reference standard from the comparator when evaluating GenAI-enabled
devices, and using them for different purposes.

The reference standard should be used to define a "correct output" that reflects the standard of care. It
should be established by specialist consensus, which could be done through a panel of qualified clinicians.
Any device should be required to meet a threshold of performance using this reference standard.

Performance relative to the clinicians who would otherwise perform the task in the intended setting
(similar to a median clinician) should be reported separately to characterize clinical benefit, but should
not substitute the threshold determined by the standard of care. The median clinician standard is highly
susceptible to variation in the population this is determined by, such that a device that outperforms the
median clinician could still fall short of the standard of care. Additionally, clinicians vary in the ways
they make errors (displaying human variance), whereas AI models tend to make errors in similar ways
(displaying bias). As a result, a device approved solely on comparison to the median clinician may have
much worse performance in certain subpopulations and enable substandard care at scale.

Additionally, as presented in our response to Q1, human-AI team performance should always be evaluated
alongside device operation due to the potential of AI models to reduce clinician performance.

Q24: Require pinned model version and change notice
Changes in foundation models, especially underlying or silent changes, can shift model behavior and void
a premarket benchmark result, potentially harming businesses and patients. 13 For instance, the sycophantic ChatGPT model mentioned in section Q1 was the result of an in-place update: OpenAI updated
GPT-4o in April of 2025 under the same GPT-4o name, leading to the more sycophantic model, without
informing the public of the full extent of the model changes. 2 The CDRH should require device sponsors
to choose one foundation model that does not automatically update, known as a pinned model version.
Additionally, the CDRH should require model developer change notice agreements with the sponsor as
a condition of authorization, which would require developers to notify sponsors of what would otherwise
be a silent deployment in pinned models, such that devices using this model can be re-evaluated.

Conclusion
We support the CDRH's efforts to protect patients from potential harms from generative AI-enabled
medical devices, and appreciate the opportunity to provide public comment on this timely discussion.
Please feel free to contact us at bhuynh@upstreamevidence.org should you have any questions. Thank
you for your consideration.

Sincerely,

Cassandra C. Chou, ScM
Fellow, The Upstream Evidence Project
Doctoral Student in Biostatistics, Harvard T.H. Chan School of Public Health

Benjamin Q. Huynh, PhD
Executive Director, The Upstream Evidence Project
Assistant Professor of Environmental Health & Engineering, Johns Hopkins University

This comment is submitted on behalf of The Upstream Evidence Project. The views expressed herein are solely those of the Project.
Academic affiliations are for identification only and do not imply institutional endorsement.
References
[1] Raine v. OpenAI, Inc. Complaint, Superior Court of California, County of San Francisco,
2025. URL https://www.law.berkeley.edu/wp-content/uploads/2025/09/Raine-vOpenAI.pdf.

[2] Sycophancy in GPT-4o: What happened and what we’re doing about it. URL https://openai.
com/index/sycophancy-in-gpt-4o/.

[3] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. TOWARDS UNDERSTANDING SYCOPHANCY IN LANGUAGE MODELS. 2024.

[4] Sarah Jabbour, David Fouhey, Stephanie Shepard, Thomas S. Valley, Ella A. Kazerooni, Nikola
Banovic, Jenna Wiens, and Michael W. Sjoding. Measuring the Impact of AI in the Diagnosis of
Hospitalized Patients: A Randomized Clinical Vignette Survey Study. JAMA, 330(23):2275--2284,
December 2023. ISSN 0098-7484. doi: 10.1001/jama.2023.22295. URL https://doi.org/
10.1001/jama.2023.22295.

[5] Thomas Dratsch, Xue Chen, Mohammad Rezazade Mehrizi, Roman Kloeckner, Aline MähringerKunz, Michael Püsken, Bettina Baeßler, Stephanie Sauer, David Maintz, and Daniel Pinto dos
Santos. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology, 307(4):e222176, May 2023. ISSN 0033-8419. doi: 10.
1148/radiol.222176. URL https://pubs.rsna.org/doi/full/10.1148/radiol.222176.

[6] Raja Parasuraman and Dietrich H. Manzey. Complacency and bias in human use of automation:
an attentional integration. Human Factors, 52(3):381--410, June 2010. ISSN 0018-7208. doi:
10.1177/0018720810376055.

[7] Ali Soroush, Benjamin S Glicksberg, Eyal Zimlichman, Yiftach Barash, Robert Freeman, Alexander W Charney, Girish N Nadkarni, and Eyal Klang. Large language models are poor medical
coders—benchmarking of medical code querying. Nejm Ai, 1(5):AIdbp2300040, 2024.

[8] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. QUANTIFYING LANGUAGE MODELS’ SENSITIVITY TO SPURIOUS FEATURES IN PROMPT DESIGN. 2024.

[9] Mahmud Omar, Shelly Soffer, Reem Agbareia, Nicola Luigi Bragazzi, Donald U. Apakama, Carol R.
Horowitz, Alexander W. Charney, Robert Freeman, Benjamin Kummer, Benjamin S. Glicksberg,
Girish N. Nadkarni, and Eyal Klang. Sociodemographic biases in medical decision making by large
 language models. Nature Medicine, 31(6):1873--1881, June 2025. ISSN 1546-170X. doi: 10.1038/
s41591-025-03626-6. URL https://www.nature.com/articles/s41591-025-03626-6.

[10] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal.
AI models collapse when trained on recursively generated data. Nature, 631(8022):755--759, July
2024. ISSN 1476-4687. doi: 10.1038/s41586-024-07566-y. URL https://www.nature.com/
articles/s41586-024-07566-y.

[11] Evan Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations,
November 2024. URL http://arxiv.org/abs/2411.00640. arXiv:2411.00640 [stat.AP].

[12] U.S. Food and Drug Administration. 21 CFR 866.3328 -- Influenza virus antigen detection test system, 2026. URL https://www.ecfr.gov/current/title-21/part-866/section-866.
3328.

[13] Lingjiao Chen, Matei Zaharia, and James Zou. How Is ChatGPT’s Behavior Changing Over
Time? Harvard Data Science Review, 6(2), April 2024. ISSN 2644-2353,. doi: 10.1162/99608f92.
5317da47. URL https://hdsr.mitpress.mit.edu/pub/y95zitmz/release/2.