GFLOPS Co., Ltd. (Ryosuke Suzuki)

GFLOPS Co., Ltd. (Tokyo, Japan) submits the attached comment on the Discussion Paper "Considerations for the Regulation of Generative AI-Enabled Medical Devices." The comment reports measurements of whether large language models give exactly the same output when the same request is repeated, and makes recommendations on benchmarking element R.1 and Questions 1, 9, 12, 19, 20, 22, 24 and 25. Please see the attached PDF for the full comment.

Comment on Docket No. FDA-2026-N-7874
Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback

Submitted by: GFLOPS Co., Ltd., Tokyo, Japan
Contact: Ryosuke Suzuki, Co-founder and CTO, contact@gflops-ai.com
Date: October 19, 2026

Summary
We are submitting the results of a study we conducted for a different purpose: to test whether large
language models (LLMs), when assessing whether a medical device's documents meet regulatory
requirements, give exactly the same assessment when the same request is repeated. Its findings bear
directly on two proposals in the Discussion Paper: treating reproducibility across repeated runs as part
of benchmarking element R.1, and using a device's premarket assessment as a baseline for reevaluating later changes, including upstream changes to foundation models. Both proposals can be
interpreted only if it is known how much the outputs vary when nothing has changed. The methods and
results are reported in the preprint (R. Suzuki, "Making every stage of medical device document
assessment with large language models deterministic," arXiv:2610.21300v1, October 15, 2026,
https://arxiv.org/abs/2610.21300); we summarize here only what bears on the Discussion Paper.
For each requirement, the model returned four things: an assessment label (Satisfied, Not satisfied or
Cannot determine), a short reason, quoted evidence (passages copied word for word from the
documents to support the label), and the documents from which those passages were quoted.

Key findings
1. Settings meant to make outputs more deterministic did not make commercial services
reproducible. We sent each of 70 different requests five times on each of two days to 14
commercial deployments (7 models, each reached through two services), asking for temperature 0,
a fixed random seed and a fixed output format wherever the service accepted them. All 14 returned
a different response to at least one request, and in 12 the assessment label changed.
2. The quoted evidence changed more often than the label. In 12 of the 14 deployments, more
requests changed the passages quoted as evidence than changed the label, by up to three times.
3. Behavior depended on more than the model name. One model's quoted evidence and full
response varied significantly more through one service than through the other. Another gave the
same label in all five runs on each day, yet a different label on the second day than on the first,
under the same model name.

1 of 3
 4. Run-to-run variation was eliminated by design in the system we tested. A complete assessment
system we built for reproducibility, from text extraction through search to the model's output, gave
byte-for-byte identical results at every step for 337 documents and 776 assessments. This held
across reruns, interruptions, reordering, parallel processing, other machines of the same GPU
model, and other days. When one serving setting was turned off, the same model on the same
hardware changed its full output on 58 of 70 requests processed in parallel, an effect that one-at-atime testing would miss. Reproducibility came from how requests were executed, not from
running the model ourselves.
5. The configuration that matters includes the GPU model. On a different GPU model, with
identical software and settings, 34 of 776 labels changed.

Recommendations
R.1, Q1, Q9 and Q12: Specify how reproducibility across repeated runs is measured. R.1 already
lists reproducibility across repeated runs. Our results suggest three specifications. First, it should be
measured on the configuration that will actually be deployed, including the service through which the
model is reached, on at least two different days. Stated settings such as temperature 0 should not count
as evidence. Second, R.1 treats variation in phrasing as acceptable and variation in safety-critical
behaviors, such as escalation, refusal or diagnostic conclusions, as failure. Between these lie the parts
of an output that support a conclusion, such as cited sources and stated reasons. In our study these
changed more often than the conclusion itself. Where users rely on them, and especially where
traceability to primary source materials is part of the risk assessment (Q1), they should also be
compared across runs. Third, submissions should report the number of repetitions, the observed
variation, and what the test can rule out under stated assumptions. For example, if each run is
compared with a fixed reference output, five runs without a change rule out, at the one-sided 95%
level, only a per-run probability of change above 45%. A lack of observed variation in a few runs does
not show that outputs are reproducible.
R.1, Q22 and Q24: Include how the model is run in the device configuration. The Discussion Paper
lists the underlying model, prompts, retrieval strategies, guardrails and orchestration logic as features
that can change. In our study, outputs also changed with the service through which the same model was
reached, with the GPU model, and with a serving setting whose effect appeared only when several
requests were processed together. Records of the deployed configuration should include these, to the
extent the sponsor controls them. A change to any of them should not be assumed to leave outputs
unchanged; it should be documented and its effect assessed, with the extent of re-evaluation
proportionate to the risk and the observed effect (Q22). The requested model name and any version
reported by the service should be recorded with every output. Because behavior can change while the
model name stays the same, as we observed, these records do not by themselves detect upstream
changes; probe testing, as described next, and provider change notifications are also needed (Q24).
Q19, Q22 and Q24: Base change detection on measured run-to-run variation. Using the premarket
assessment as a baseline, or a fixed set of probe inputs to detect upstream changes, requires
distinguishing ordinary run-to-run variation from changes in device behavior. In 12 of the 14
deployments we measured, the assessment label itself changed on repetition. Unless this variation is
measured, a difference observed after a change cannot be attributed to the change, and the absence of a
difference does not show that nothing changed. The baseline should therefore include the device's runto-run variation, measured over several days, and re-benchmarking and probe testing should state the
size of change they can detect.

2 of 3
Where a sponsor shows that both the original and the modified configurations reproduce their outputs
exactly, a single run of each on the prespecified inputs identifies which outputs the change altered. This
does not show that the altered outputs are acceptable, or that outputs for untested inputs are unchanged.
We suggest recognizing exact reproducibility as one acceptable way to address R.1 and to simplify
change detection, not as a requirement; it is not by itself sufficient for validation. It has been suggested
that exact reproduction of generative AI output is infeasible, or unnecessary. For the system we tested,
it was feasible; the setting that kept outputs identical under parallel processing lengthened processing
time by 18% in our test.
Q20: Evaluate the repeatability of LLMs used as adjudicators or supervisory agents. The
Discussion Paper notes that adjudicator considerations still apply when the adjudicator is itself an
LLM, and Q20 asks about the reliability of supervisory agents. Such LLMs are subject to the same runto-run variation. Their repeatability should be measured and their configuration recorded, in proportion
to their role and risk.
Q25: Include serving behavior in Foundation Model Device Master Files. In addition to model
updates, Foundation Model MAFs could state whether identical requests return identical outputs under
stated settings, whether behavior differs between the services that offer the same model, and whether
serving can change under an unchanged model name. Sponsors would still measure the behavior of
their own deployed configuration.

Scope
These comments concern reproducibility and configuration control only. Reproducibility is not
accuracy: a system can reproduce an incorrect answer exactly. We do not suggest that FDA require any
particular way of running models, such as self-hosting or access to model weights. The study's
limitations are stated in the preprint; among them, it covered document assessment rather than a
clinical function, and its results have not yet been reproduced outside GFLOPS. Its relevance to the
Discussion Paper is methodological; the clinical significance and acceptable size of output variation
must be established for each intended use. We would welcome independent verification.

Disclosure
GFLOPS develops and sells software that uses large language models to prepare preliminary
assessments of whether a medical device's documents and records meet regulatory requirements, for
qualified experts to check. The reproducible system described in this comment is that software. The
study was funded by GFLOPS. Its author, Ryosuke Suzuki, is GFLOPS's co-founder and CTO and the
inventor on a provisional patent application covering that system. The study has not been peer
reviewed.
Respectfully submitted,
Ryosuke Suzuki
Co-founder and CTO
GFLOPS Co., Ltd.

3 of 3