Daniel O. Olofin

Please find attached my comment on Questions 12 and 13, with reference to Questions 9 and 10, regarding the FDA discussion paper on generative AI-enabled medical devices (Docket No. FDA-2026-N-7874).

This comment reflects my individual perspective and does not represent the views of my employer or institution.

Thank you for considering my submission.

Respectfully,
Daniel O. Olofin

Docket: FDA-2026-N-7874
Submitted by: Daniel O. Olofin
Professional background: Researcher in biostatistics, machine learning, and computational statistics
Capacity: Individual comment; views expressed are my own and do not represent my employer or institution
Date: October 19, 2026
Questions addressed: 12 and 13, with reference to Questions 9 and 10

To the Food and Drug Administration:

Thank you for the opportunity to comment on the Center for Devices and Radiological Health discussion paper
regarding generative artificial intelligence-enabled medical devices. My research involves statistical modeling,
machine learning, and simulation-based analysis in medicine and biology. I submit this comment in my
individual capacity; the views expressed are my own and do not represent my employer or institution.

My perspective comes from designing a validation plan for a simulation-trained neural network that counts and
sizes particles in fluorescence microscopy images. Although this project is neither a medical device nor
generative AI, it presents a statistical challenge relevant to Questions 12 and 13: abundant synthetic data with
known simulation labels, alongside limited real data and few independent sampling units.

Summary of Comment

Synthetic data can broaden test coverage and reveal device failure modes, but additional generated cases do not
constitute additional independent real-world observations. I recommend that sponsors define the unit of
inference, account for clustering and repeated outputs, and report the independent real units supporting
performance estimates. Synthetic and real-data results should be reported separately unless their combination is
justified through a prespecified statistical approach with sensitivity analyses. Synthetic data are particularly
useful for controlled stress testing but require validation before supporting claims about real-world or subgroup
performance. Safeguards should address shared model assumptions, data leakage, benchmark validity, and
uncertainty in reference standards. Claims about real-world performance should remain anchored in
independent real-data evaluation.

1. Distinguish independent real-world evidence from generated test cases (Q12).

Synthetic inputs can expand test coverage without adding independent patients, sites, clinicians, or acquisition
settings. Treating related synthetic cases, repeated generations, or prompt variants as independent observations
can understate uncertainty about real-world performance.

I recommend that sponsors define the target population, performance endpoint, and unit of inference; report the
number of independent real units supporting each estimate; and account for relevant clustering within patients,
encounters, sites, or sessions. For stochastic GenAI outputs, sponsors should distinguish variability across cases
from variability across repeated outputs for the same case.

Synthetic cases should not be counted as additional independent real-world observations. Any effective sample
size attributed to model-based borrowing from synthetic data should be reported separately, with its
assumptions and sensitivity analyses. When independent real units are few, that limitation should remain
explicit.

2. Report evidence separately unless combination is justified (Q12).

Benchmarking and clinical confirmation are evaluation stages, whereas synthetic and real data are evidence
sources. Sponsors should identify both distinctions clearly.
A single combined performance estimate should target a clearly defined population and endpoint, use
compatible scoring rules, and evaluate the same deployed device configuration. It should also include a
prespecified justification for relating the evidence sources and accounting for differences between their
distributions. Overlapping cases and other dependencies should be addressed.

Without such justification, sponsors should report estimates separately and anchor claims about real-world
performance in independent real-data evaluation. If synthetic evidence is used to borrow strength, a prespecified
hierarchical or discounted-borrowing analysis may be appropriate. However, the model should account for
potential source differences and uncertainty in the generator. Sensitivity analyses should include no borrowing
and plausible synthetic-to-real discrepancies.

Sponsors should also report a synthetic-to-real performance gap using prespecified metrics under comparable
conditions, with uncertainty and stratification by relevant factors such as input quality, acquisition setting, and
subgroup. Agreement on measured factors should not be treated as proof that all relevant distributional
differences have been addressed.

3. Use synthetic data for questions it can credibly answer (Q13).

Synthetic data are particularly useful for controlled stress testing, where a known factor can be varied
systematically for example, noise, blur, overlapping structures, missing information, contradictory inputs, or
unit errors. Blank or null inputs can also help characterize false-positive behavior where applicable. These uses
can reveal failure modes that would be difficult to study efficiently using real data alone.

Synthetic data are less persuasive for estimating prevalence-weighted real-world performance when the
frequency and combination of clinical conditions are not established. They are also insufficient, by themselves,
to establish subgroup performance when the generator has not been validated for the relevant subgroup.
Generating more examples from an inadequately represented subgroup does not establish that its clinically
important features have been captured.

4. Address circularity, leakage, and benchmark validity (Q13, with reference to Q9 and Q10).

I recommend the following safeguards:

• Disclose shared dependencies. Sponsors should disclose shared training data, model lineage, and
development dependencies between the generator and the device under evaluation. A different generator
does not necessarily provide an independent test if both systems share the same assumptions or blind
spots.
• Validate the generator for its intended evaluation role. Sponsors should compare synthetic and real
inputs on features that plausibly drive device performance, including subgroup-specific features, and
document known limitations. Visual or linguistic realism alone is insufficient.
• Protect final testing from development decisions. Real data used to design, tune, or select the
generator, benchmark, scoring rule, or device should be treated as development data. Final evaluation
should use separate real units, with related patient or encounter inputs kept within one partition. Sitelevel separation should be used when needed to support claims about performance at new sites.
• Prespecify comparative claims. Comparators, metrics, acceptance criteria, and any superiority,
noninferiority, or equivalence analyses should be fixed before final testing.
• Demonstrate sensitivity to meaningful failures. A benchmark should be shown to detect clinically
important errors, including errors arising from assumptions shared by the generator and evaluated
device. Where independent real-data testing is unavailable, claims about real-world generalization
should be identified as preliminary.
5. Use appropriate real-data reference standards for measurement functions.

Section IV notes that measurement and signal-processing functions may produce outputs whose correctness
users cannot readily assess. For these functions, synthetic labels provide a reference standard within the
simulator’s definition of the target; they do not establish that the definition or measurement process corresponds
to reality.

Final measurement claims should therefore be supported by independent real-data evaluation against an
appropriate reference standard. Where expert annotation is needed, sponsors should use a written annotation
protocol, qualified independent reviewers, and a defined process for resolving disagreements. Referencestandard uncertainty should also be reported.

I support the responsible use of synthetic data to broaden evaluation coverage and investigate failure modes. Its
contribution should be clearly distinguished from independent evidence about performance in the intended realworld population.

Thank you for considering these comments.

Respectfully,
Daniel O. Olofin
MS Mathematics – Statistics
Submitted in an individual capacity

Reference

U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical
Devices: Discussion Paper and Request for Feedback. August 2026. FDA discussion paper.