Guanbo Wang

See attached file(s)

Public Comment to the U.S. Food and Drug Administration

Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback
Docket No. FDA-2026-N-7874

Submitted by:
Guanbo Wang*, PhD
The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth
guanbo.wang@dartmouth.edu
Shu Yang, PhD
Department of Statistics, North Carolina State University
Tian Gu, PhD
Department of Biostatistics, Columbia University Mailman School of Public Health
James O’Malley
The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth

Executive Summary
We appreciate FDA's discussion paper and support a risk-proportionate, competency-based approach to generative
AI (GenAI)-enabled medical devices. Our comments focus on a practical regulatory question that cuts across
benchmarking, clinical confirmation, device modification, and postmarket monitoring: what evidence is needed for
the particular safety or effectiveness claim a sponsor wants to make?
Our principal recommendation is to align the evidence requirements with the claim rather than rely on a fixed
sequence of validation studies. A benchmark may provide strong evidence about a device competency without
showing that clinicians benefit from using the device. Shadow deployment can reveal behavior on live clinical inputs
without establishing how visible outputs change clinical decisions. Conversely, a randomized human-AI study may
answer a decision-making question directly even when its setting is less representative of routine care. The
appropriate design depends on the claim, risk, and residual uncertainty FDA needs to resolve.
For GenAI-enabled devices, FDA should also make cross-setting and cross-device generalizability an explicit part of
the evidence package. The same model may behave differently across imaging devices, acquisition protocols, EHR
systems, connected sensors, patient subgroups, institutions, clinical workflows, and user interfaces. Evidence
should therefore characterize heterogeneity, not only average performance.
Nine recommendations are especially important:
1. In response to Discussion questions 12, 14: Prespecify clinically meaningful performance measures, criticalerror criteria, acceptance thresholds, subgroup requirements, and the comparator appropriate to intended use.
Include patient-centered outcomes when the claim concerns care or health, and define the comparator as it
would be defined in a target trial.
2. In response to Discussion questions 1 and 8: Use FDA's two-axis risk framework as the starting point, while
allowing detectability, reversibility, meaningful human oversight, time to harm, patient vulnerability, and
autonomy to modify the strength and scope of evidence required.
3. In response to Discussion questions 7 and 11: Select clinical-confirmation designs according to the question
they answer. Retrospective evaluation, shadow deployment, human-GenAI studies, prospective deployment
studies, and target-trial emulations are complementary designs, not interchangeable rungs on a universal
ladder.
4. In response to Discussion questions 9, 10, and 11: Require explicit evidence for cross-device, cross-site, and
cross-workflow generalizability when intended use spans heterogeneous data sources or device configurations.
 Sponsors should report performance distributions, subgroup precision, and source-level heterogeneity rather
than only pooled averages.
5. In response to Discussion question 12: Do not combine evidence simply because it concerns the same device.
Evidence from another population, setting, synthetic distribution, or configuration raises an external-validity
problem; evidence about a different clinical quantity requires additional empirical support before it is used for a
downstream claim.
6. In response to Discussion questions 19 and 22: Make revalidation and postmarket monitoring responsive to
what a modification could change. Stable benchmark performance after an update does not necessarily imply
stable human-GenAI interaction, subgroup performance, or clinical effects.
7. In response to Discussion questions 11, 14 and 15: Specify the claim of what a product can do as a hypothesis
to be tested in a target trial, using the same protocol elements one would specify for a non-GenAI device:
eligibility, treatment strategies, assignment, follow-up, outcomes, and causal contrast. When the claim concerns
comparative performance, the statistical hypothesis and acceptance criteria should reflect the claim and clinical
consequences of performance differences.
8. In response to Discussion questions 9 and 10: Stress-test the configured device on unseen scenarios that
extend beyond the current range of the data or evidence pool, including novel presentations, unusual
conversational trajectories, plausible failure modes, and data-source combinations not represented in training or
in the primary test set.
9. In response to Discussion questions 11 and 15: When intended use includes efficiency, substitution for clinician
time, or reduced utilization, include resource-use and cost-effectiveness analysis as complementary evidence.
Such analyses should supplement as opposed to replace evidence of safety and effectiveness.

All nine recommendations draw on principles broadly applicable to intervention evaluation, but warrant particular
emphasis when evaluating GenAI-enabled medical devices. Recommendations 6 and 8 additionally address GenAIspecific considerations in revalidation, postmarket monitoring, and stress testing.

I. Start With the Regulatory Claim and the Target Trial
Before choosing a clinical-confirmation design, sponsors should state the material claim the evidence is intended to
support. At minimum, that statement should identify the outcome or performance quantity, intended patients and
users, workflow and setting, configured device version, data sources, and clinically relevant comparator.
This specification is most useful when written as a target trial: the protocol one would run if a randomized
experiment were feasible, whether or not the submitted evidence is experimental. GenAI-enabled devices should be
evaluated in the same terms as non-GenAI devices. The protocol should identify eligibility, the strategies being
compared, the assignment procedure, follow-up, the outcome, and the causal contrast. For replacement or
autonomous functions, the comparator should be an appropriate human-expert standard. For assistive functions, the
comparison is often clinician-plus-AI versus clinician without AI. In either case, the statistical hypothesis and
decision criteria should be prespecified and clinically justified, whether the analysis is frequentist or Bayesian.
FDA can then ask a simpler set of questions. What submitted evidence directly evaluates the claim? If evidence was
generated in another population, setting, synthetic distribution, data modality, or device configuration, what supports
its applicability? If evidence evaluates an upstream quantity, such as benchmark performance or response quality,
but the claim concerns clinician decisions or patient outcomes, what additional evidence supports that connection?
This approach need not create a blanket requirement for prospective trials. A low-consequence claim about device
behavior may be supported by rigorous benchmarking and clinically representative confirmation. A claim that device
use improves clinician decisions ordinarily requires evidence involving intended users. A claim of improved patient
outcomes may require direct downstream evidence or a well-supported evidentiary pathway to that outcome.
Observational or real-world analyses can support the same claim if they emulate the target trial and state which
protocol elements cannot be recovered from the data.
II. Risk Should Determine the Evidentiary Burden
We would retain FDA's two-axis framework—device activity and the consequence of relying on an incorrect output—
rather than add several new formal axes. Two devices in the same cell, however, may warrant different evidence if
an error in one is readily detected and reversible while an error in the other is difficult to recognize before irreversible
harm. Detectability, reversibility, meaningful human oversight, time to harm, patient vulnerability, and autonomy are
therefore useful modifiers of the evidence required.
These modifiers should affect both premarket evidence and the extent to which the FDA can rely on postmarket
monitoring. Postmarket detection is less reassuring when the first recognizable failure could itself cause serious or
irreversible harm. FDA should also distinguish under-escalation from over-escalation; their clinical consequences
are asymmetric and can be obscured by a single average accuracy measure.

III. Benchmark the Configured Device, but Be Explicit About What the Benchmark Establishes
We support evaluating the final configured device rather than a foundation model in isolation. For GenAI
applications, clinically relevant behavior may depend on prompts, retrieval, guardrails, tools, interface, conversation
history, human review, and other application-level components.
Benchmark plans should prespecify clinically meaningful performance and safety measures, critical-error categories,
subgroup analyses, and uncertainty. Where relevant, they should also assess calibration, abstention or deferral,
escalation, stochastic variability, and plausible distribution shifts. Benchmark provenance, test-set version,
application/model version, scoring rules, and adjudication procedures should be reproducible. They should also
stress-test unseen scenarios: inputs that are beyond the range of current inputs, novel presentations, unusual
conversational trajectories, and plausible failure modes that were not represented in training or in the primary test
set. Performance on familiar cases alone does not establish safe behavior in unfamiliar cases. Statistical analyses
of performance trends within the observed sample may inform predictions beyond the tested range, although such
extrapolation depends on assumptions that the observed data cannot fully verify. These predictions should therefore
be assessed alongside clinical and technical expert judgment and supplemented with targeted new data where
uncertainty remains material to safety or effectiveness.
The important limitation is interpretive. A benchmark can validly establish a device competency without establishing
that the competency improves clinician behavior or patient outcomes. If a sponsor uses benchmark results to
support such a downstream claim, FDA should ask for evidence supporting that connection. Sequestered test sets,
prespecified analyses, real-world holdouts, and independent adjudication can improve the credibility of the
benchmark itself, but they do not remove this distinction.

IV. Generalizability and Heterogeneity Should Be Treated as Evidence Questions

Many GenAI-enabled medical devices will be deployed across heterogeneous data sources and clinical
environments, including different imaging scanners and acquisition protocols, EHR systems, clinical note styles,
connected devices, laboratory platforms, patient populations, institutions, and workflows. A single pooled estimate
can be misleading if performance is strong in common settings but unreliable in a smaller subgroup, site, device
type, data modality, or workflow. Design-of-experiments approaches may help sponsors explore this heterogeneity
efficiently, both within individual studies and across coordinated studies, by systematically varying relevant factors
and assessing their interactions. The factors and combinations examined should reflect the device’s intended
deployment environments. Generalizability should therefore be evaluated as part of the evidentiary package rather
than inferred from average performance alone.

FDA should ask sponsors to define the intended generalization domain and assess whether the submitted evidence
adequately covers that domain. Useful approaches include external validation across independent sites or device
types, leave-one-source-out evaluation, hierarchical or random-effects summaries of source-level performance,
prespecified subgroup analyses, calibration assessment, and sensitivity analyses for missing, noisy, or discordant
modalities. Sponsors should report clinically important heterogeneity and, where evidence is sparse in particular
settings or subgroups, explicitly describe the resulting uncertainty and limits of applicability. Pooled estimates should
not obscure these differences or gaps in the evidence.
For high-consequence devices, FDA may also consider requiring a transportability analysis that states the
assumptions under which evidence from one setting supports use in another. This is especially important when the
claim spans both technical performance and human decision making, because heterogeneity may arise from the
data source, model architecture, adaptation procedure, user interface, clinician behavior, or clinical workflow. Such
analyses would help FDA distinguish evidence that is truly generalizable from evidence that is valid only in the
development or testing context.

V. Clinical Confirmation Should Be Selected by the Question
FDA’s proposed options for clinical confirmation are useful, but they answer different questions. Shadow
deployment illustrates the point. Running a device silently on live clinical inputs can provide compelling evidence
about its behavior under real workflow conditions and permit comparison of its outputs with clinicians’ initial
decisions. Such comparisons may identify potential opportunities for improvement, but they cannot establish
whether clinicians would benefit from seeing the outputs, either through immediate decision support or through
learning over time. A simulated randomized human-AI study may be less representative of routine practice yet more
directly assess whether access to AI improves clinicians’ decisions.
Approach Question it can address well Important limitation
Retrospective clinical evaluation How does the configured device Does not reproduce prospective
perform on real clinical cases interaction or workflow effects.
with an available reference?
Shadow deployment How does the device behave on Hidden outputs do not establish
live clinical inputs and workflow effects on clinician decisions or
conditions? patient outcomes.
Human-AI or simulated How does access to AI change Effects may not transport to
interaction study decisions, reliance, routine practice or to other
communication, or workflow? users.
Prospective comparative What happens when the AI- More resource intensive;
deployment enabled workflow is deployed, endpoints should be justified by
including downstream the claim and risk.
outcomes?
Cross-site or cross-device Does performance generalize Requires enough sources to
validation across sites, device types, data estimate heterogeneity; pooled
systems, acquisition protocols, averages can obscure sourceor workflows? specific failures.
Target-trial emulation using What would have been Requires a complete protocol
real-world data observed in the specified target and explicit assumptions;
trial, under stated assumptions? cannot non-parametrically
recover elements the data do
not contain.
The least-burdensome approach is therefore not the least intensive design in the abstract. It is the least
burdensome design, or combination of designs, that resolves the uncertainty relevant to the claim and risk at issue.
Where confirmation uses retrospective or real-world data, we recommend designing the analysis as an emulation of
the target trial specified for the claim. The sponsor should map each protocol element from the ideal trial to the
observed data and identify elements that cannot be emulated—for example, a well-defined assignment procedure or
contemporaneous expert care. This keeps confirmation from collapsing into a convenience sample of benchmarklike cases and aligns GenAI evaluation with methods already used for non-GenAI devices and other medical
interventions.
VI. Synthetic, Multimodal, and Real-World Evidence Should Be Combined Selectively and in a
Manner Consistent with Statistical Design Principles
Synthetic data are useful for rare-event stress testing, controlled perturbations, standardized scenarios, and
targeted evaluation of underrepresented cases. They are less persuasive when the phenomenon of interest
depends on complex clinical heterogeneity, workflow, or correlations that the generator may not reproduce. They
are particularly useful for stress-testing unseen scenarios that are rare, ethically difficult to observe, or outside the
support of available clinical data.
Many GenAI devices will rely on multimodal data integration, such as EHR variables, clinical notes, images,
waveforms, laboratory results, wearable or connected-device data, and patient-reported information. Sponsors
should describe modality provenance, timing, missingness, measurement error, harmonization, and how the model
resolves conflicts across modalities. FDA should be cautious when evidence is drawn from deeply phenotyped
research cohorts but the intended use is routine clinical care, or vice versa, unless the sponsor provides a clear
data-integration and transportability rationale.
We recommend reporting synthetic, retrospective, prospective, and real-world results separately before combining
them. Pooling or borrowing should be justified by the relationship between the source distributions, prespecified
integration methods, and evidence that aggregation does not conceal clinically important differences. If synthetic
cases are generated by the same or a closely related model family, correlated blind spots deserve particular
scrutiny.
Borrowing strength from historical or auxiliary data may reduce the need for new data when the relevance of the
earlier evidence is adequately justified. FDA’s guidance on Bayesian medical device trials recognizes that effects of
devices with physical, local mechanisms may sometimes be predictable from earlier device generations when
modifications are minor. For GenAI-enabled devices, sponsors should explicitly assess whether changes in the
model, application configuration, data sources, or workflow preserve the relevance of historical evidence. Statistical
analyses of observed performance patterns, together with clinical and technical expert assessment, can inform the
extent of borrowing and identify uncertainties requiring new data.
More broadly, two sources should not be combined simply because they arose from evaluating the same device. If
they address the same clinical quantity in different populations, settings, or device configurations, generalizability or
transportability methods may be appropriate. If one evaluates device competency and another evaluates human-AI
performance, simple pooling has no clear interpretation. The sponsor should instead explain what evidence
connects those quantities.
Reproducibility: Because AI algorithms and data used for training are constantly evolving, the results obtained from
using them may change. This problem is more extensive than failing to fix a random number seed as the reason for
the variability may be systematic as opposed to random.

VII. Comparators and Acceptance Criteria Should Follow Intended Use
The comparator is a vital part of almost all clinical questions and should be specified before the evidence is
interpreted. Depending on intended use, the relevant comparison may be device performance against a reference
standard, clinician plus AI versus clinician without AI, or AI-enabled care versus usual care.
The comparator should also reflect the realistic alternative. A generalist-assistance tool should not automatically be
judged only against a subspecialist, while an autonomous specialty function should not be justified by comparison
with an inadequately trained user. For some patient-facing informational functions, the realistic alternative may be
the usual information environment rather than immediate specialist consultation.
When a device is intended to perform a task that a qualified clinician would otherwise perform, its performance
should be evaluated against an appropriate human-expert comparator matched to the intended user and task. The
statistical hypothesis should reflect the claim and clinical consequences of performance differences. Non-inferiority
may be appropriate when a limited decrement in the specified outcome can be clinically justified, but should not be
adopted by default or interpreted as evidence of equal performance. Any margin should be prespecified and justified
by the consequences of the errors it permits. Separate acceptance criteria should address critical omissions and
unsafe recommendations so that favorable average performance does not obscure unacceptable safety failures.
Results should report the estimated performance difference and its uncertainty, alongside whether the prespecified
criteria were met.
FDA’s decision remains one of safety and effectiveness. Where appropriate, joint statistical analysis of safety and
effectiveness outcomes may improve statistical efficiency, while preserving distinct conclusions and acceptance
criteria for each.

When intended use includes efficiency, reduced utilization, or substitution for clinician or specialist time, resourceuse and cost-effectiveness analyses can complement the evidentiary package. They can clarify whether an
apparent performance gain is offset by added review time, over-escalation, or downstream testing. These analyses
do not replace evidence that the device is safe and effective for its intended use.
For open-ended outputs, a single accuracy threshold may be inadequate. Safety-critical omissions, factual
correctness, clinical appropriateness, calibration, escalation behavior, consistency, communication quality, subgroup
performance, and cross-source performance may need separate criteria. Independent benchmark custody or clinical
adjudication can improve objectivity without transferring accountability away from the sponsor.

VIII. Revalidation Should Follow the Consequences of the Change
Sponsors should document changes to training data, model versions, prompts, retrieval, guardrails, tools, interfaces,
intended use, source data, and subgroup performance well enough to determine what could plausibly change
clinically.
Re-benchmarking against the original premarket baseline is an important first step, but it is not automatically
sufficient. FDA’s guidance on Bayesian medical device trials (2010 version, section 2.2) recognizes that effects of
devices with physical, local mechanisms may sometimes be predictable from earlier device generations when
modifications are minor. For GenAI-enabled devices, sponsors should justify whether a comparable expectation of
stable behavior is warranted. If an update is narrowly scoped to a defined device competency and evidence
supports stability of other clinically relevant behavior, targeted regression testing may be sufficient. If it plausibly
changes how users interpret, trust, override, or act on outputs, renewed human-AI evaluation may be needed.
Changes to autonomy, workflow, intended population, target device environment, or mechanisms closely tied to
downstream outcomes may justify broader clinical confirmation.
This distinction is especially important for third-party foundation models, where a manufacturer may not control or
fully observe upstream changes. Sponsors need reliable version identification, change notification or detection
where feasible, regression gates (prespecified performance and safety checks that an updated device must pass
before deployment or continued use), and a predefined response when the clinical significance of a change cannot
be characterized. Predetermined change control plans can support this process when the classes of change,
evaluation methods, acceptance criteria, target-adaptation rules, and rollback or escalation conditions are bounded
in advance.

IX. Postmarket Monitoring Should Reflect Clinically Relevant Failure Modes
Postmarket monitoring should follow the claims and failure modes that mattered for authorization rather than rely
only on aggregate device performance. For some devices, technical errors, calibration, abstention, escalation, and
subgroup performance will be central. Where user interaction is safety-relevant, overrides, disagreements, reliance
patterns, failed handoffs, and near misses may be equally informative. When downstream clinical outcomes are
material to intended effect, those outcomes should be monitored when feasible.
Monitoring should combine prespecified periodic review with event-triggered reassessment. For higherconsequence devices, event-level provenance should permit investigators to identify the relevant model or
application version, data source, device configuration, modality availability, and workflow when a safety signal
occurs.
Internet-connected GenAI-enabled devices may provide substantial opportunities to generate postmarket evidence.
Automated learning and modification, however, may mean that evidence accumulates for an evolving intervention
rather than a fixed device configuration. Sponsors should therefore link postmarket observations to detailed records
of changes, allowing findings to be interpreted alongside studies of subsequent device generations. Such records
may also facilitate rollback when earlier validated configurations can be restored, although this capability should be
demonstrated rather than assumed.
Greater reliance on postmarket evidence is most defensible when degradation can be detected before unacceptable
harm accumulates and when intervention or rollback is feasible. Monitoring should return timely feedback to the
manufacturer and, where appropriate, to clinicians and patients, so that errors can be recognized and corrected
promptly. When patient-centered outcomes were material to the authorization claim, they should be among the
monitored endpoints, rather than relying solely on aggregate accuracy.

X. A Short Evidentiary Checklist
For each material safety or effectiveness claim, FDA could ask sponsors to summarize the following in a concise
evidentiary table:
Item Question
Claim What precisely is being asserted?
Intended use context Which patients, users, workflows, settings, data
sources, modalities, and configured versions are
covered?
Comparator/reference What is the clinically relevant alternative or
reference standard?
Direct evidence Which evidence directly evaluates the claimed
quantity?
Applicability If evidence comes from another population,
setting, synthetic distribution, data source, or
configuration, what supports its use here?
Heterogeneity How do results vary across sites, subgroups,
device types, workflows, and modality-availability
patterns?
Additional support If evidence evaluates an upstream or
intermediate quantity, what supports extension to
the stated claim?
Uncertainty How are subgroup precision, source
incompatibility, model stochasticity, and missing
modalities reflected?
Lifecycle plan Which changes or signals trigger reassessment,
additional confirmation, rollback, or user
notification?
Target-trial protocol Which eligibility, strategies, assignment, followup, outcomes, and causal contrast define the
claim?
Unseen-scenario stress tests Which beyond the observed data range, sourceshifted, or previously unobserved cases were
evaluated, and what were the results?
Patient-centered outcomes and timely feedback Which patient-centered endpoints and feedback
latencies are claimed or monitored?
 Resource use and cost-effectiveness If efficiency or substitution is part of intended use,
what resource-use or cost-effectiveness
evidence is offered?

Conclusion
FDA's discussion paper provides a useful foundation for risk-proportionate oversight of GenAI-enabled medical
devices. The central improvement we recommend is straightforward: choose the evidence package or requirements
according to the claim FDA needs to evaluate and the uncertainty that remains.
Benchmarks, synthetic evaluations, retrospective clinical data, shadow deployment, human-AI studies, cross-device
validation, transfer-learning studies, prospective comparative studies, and postmarket evidence each have
legitimate roles, but they are not interchangeable. Evidence generated in another context requires an applicability
argument made by content-area experts and supported by statistical analysis of empirical data when it becomes
available. Evidence about an upstream competency should not automatically substitute for evidence about clinician
behavior or downstream care.
A claim-aligned approach embodying pre-specified hypotheses and analyses can support both rigor and leastburdensome regulation: require the evidence necessary for the clinical question and risk at issue, rather than a fixed
sequence of studies for every GenAI-enabled device. The same target-trial elements used for non-GenAI devices—
population, strategies, follow-up, patient-centered outcomes, and a justified contrast—should organize that package.
Stress tests of unseen scenarios, explicit heterogeneity analysis, target-context validation for transfer learning,
timely feedback after deployment, and, when efficiency is part of intended use, cost-effectiveness or resource-use
analysis help ensure that the evidence matches the claim.
Thank you for the opportunity to comment.

Selected References
1. U.S. Food and Drug Administration, Center for Devices and Radiological Health. Considerations for the
Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback. August 2026.
2. U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change
Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final Guidance. August 2025.
3. Vasey B, Nagendran M, Campbell B, et al. Reporting Guideline for the Early-Stage Clinical Evaluation of Decision
Support Systems Driven by Artificial Intelligence: DECIDE-AI. Nature Medicine. 2022;28:924-933.
4. Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK. Reporting Guidelines for Clinical Trial Reports for
Interventions Involving Artificial Intelligence: CONSORT-AI Extension. Nature Medicine. 2020.
5. Rivera SC, Liu X, Chan AW, Denniston AK, Calvert MJ. Guidelines for Clinical Trial Protocols for Interventions
Involving Artificial Intelligence: SPIRIT-AI Extension. Nature Medicine. 2020.
6. Patel B, Blumenthal D. A Novel Approach to Overseeing the Clinical Application of Generative AI. JAMA Health
Forum. 2026;7(3):e256947.
7. Garcia V, Sidulova M, Badano A. Performance Assessment Strategies for Language Model Applications in
Healthcare. Artificial Intelligence in the Life Sciences. 2026;9.
8. Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available.
American Journal of Epidemiology. 2016;183(8):758-764.
9. U.S. Food and Drug Administration. Non-Inferiority Clinical Trials to Establish Effectiveness. Guidance for
Industry. November 2016.
10. U.S. Food and Drug Administration. Principles for Selecting, Developing, Modifying, and Adapting PatientReported Outcome Instruments for Use in Medical Device Evaluation. Guidance. January 2022.