Visium SA (Tomislav Ilicic)

Please find attached the comments of Visium on the discussion paper “Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback” (Docket No. FDA-2026-N-7874). Visium is a Switzerland-based data science and AI consultancy. Our submission responds in full to Discussion Questions 7, 9, 10, 16, 18, 19, 20 and 24, briefly to twelve further questions, and summarizes eight principal recommendations. Contact: Dr Tomislav Ilicic Tomislav.ilicic@visium.com

19 October 2026
Dockets Management Staff (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Room 1061
Rockville, MD 20852
Submitted electronically via www.regulations.gov

Re: Docket No. FDA-2026-N-7874 — Considerations for the Regulation of Generative AI-Enabled Medical
Devices: Discussion Paper and Request for Feedback
Dear Dockets Management Staff:
Visium welcomes the opportunity to comment on the Center for Devices and Radiological Health’s discussion
paper on the regulation of generative AI-enabled medical devices.
Visium is an independent, Switzerland-based data science and AI consultancy. We do not manufacture medical
devices and we do not develop foundation models; our work consists of designing, building, validating, and
operating AI systems — increasingly generative AI systems — on behalf of organizations in the life sciences
sectors. Much of that work takes place in GxP-regulated environments, where the questions this discussion
paper raises are not hypothetical: which exact system configuration was validated, how an open-ended output
is scored defensibly, how performance is monitored once ground truth lags, and what happens when an
upstream model provider changes a model the system depends on. Our teams combine machine-learning
engineering with experience gained in roles at pharmaceutical, medical-technology, and AI organizations, and
we have drawn on that combined experience in preparing these comments.
The practice areas from which we comment map directly onto the paper’s structure. We design and run
evaluation harnesses for generative AI applications — task-specific test sets, scoring rubrics with human and
LLM adjudication, and the validation of those adjudicators against human judgment — which is the subject
matter of the benchmarking elements in Section V.B and Appendix A. We build the infrastructure that makes
such evaluations reproducible: pinned model and configuration versions, controlled inference stacks, and
regression suites that are re-executed whenever a component changes, which is the practical substance of
Sections V.B.2 and VI.C. We generate and validate synthetic test data where real data are scarce or cannot be
exposed, and we design post-deployment monitoring — proxy metrics, drift detection, and human review
sampling — for systems whose outputs cannot be fully verified at runtime, the concern of Section VI.A.
Because the systems we build are predominantly assembled on third-party foundation models, change
management for upstream model updates (Sections VI.C and VII.A) is a routine engineering problem for us
rather than a policy abstraction. Our comments are confined to these areas; where the paper raises clinical
questions outside our competence, we have deliberately said less.
We support the paper’s overall direction: a risk-based, total product life cycle approach that accepts the openended nature of GenAI-enabled devices rather than forcing them into evaluation paradigms built for bounded
software, and that looks for least burdensome, scientifically sound ways to give patients timely access to safe
and effective devices. Our comments are intended to help make that approach operable.
We have chosen depth over coverage. Section 2 responds in full to the eight questions where our evaluation
and assurance experience is most directly relevant (Questions 7, 9, 10, 16, 18, 19, 20, and 24). Section 3 offers
brief responses to twelve further questions. Section 1 summarizes our principal recommendations.
Throughout, we reference the paper’s own sections and benchmarking elements, and where possible we build
on sources the paper itself cites.
1. Summary of principal recommendations
1. Bind competency assessment to a versioned configuration manifest, with reproducible evaluation
procedures and explicit tolerances for clinically relevant behavior; use changes to the manifest to trigger
proportionate impact assessment (Question 7).
2. Make the reliability of the evaluation harness an explicit evidence requirement: judge–human agreement,
inter-adjudicator reliability, and rubric stability for every non-deterministically scored element; design the
premarket suite for re-execution (Question 9).
3. Distinguish screening from gating uses of benchmarks. Public benchmarks screen; gating requires
demonstrated construct validity and sequestered, post-cutoff, rotating held-out sets under independent
custody, with contamination audits as supporting evidence (Question 10).
4. Use independent third parties for the functions sponsors cannot perform credibly for themselves —
sequestered data custody, independent adjudication, reproducible execution, and test-method standards
— sequenced as standards, then accreditation, then conformity assessment on existing ASCA and MDDT
rails, under structural independence criteria and capability-based accreditation (Question 16).
5. Accept greater premarket uncertainty only against monitoring programs that are powered and binding —
prespecified thresholds, demonstrated power and time-to-detect, trigger-bound playbooks, rollback
capability, and reporting commitments — and not for autonomous severe-consequence functions
(Question 18).
6. Make reassessment event-driven, with prespecified triggers; stratify clinician-review samples by risk;
monitor validated proxy metrics per deployment context against baselines frozen at authorization
(Question 19).
7. Permit machine-based supervisory agents as a screening layer only, validated against human-adjudicated
reference sets, independent of the supervised device’s model family, and with uncertainty-aware routing
to human review (Question 20).
8. Treat adoption of a new version of a named third-party foundation model as a prespecifiable PCCP change
category, supported by version pinning, automated regression batteries, and minimum contractual notice
and deprecation terms (Question 24).

2. Responses to selected discussion questions
Question 7 — Competency-based approach
Is the competency-based approach described above, i.e., device benchmarking followed by clinical
confirmation, a useful and appropriate framework for evaluating GenAI-enabled devices?

Yes. We support the competency-based approach. Scalable non-clinical benchmarking followed by riskproportionate clinical confirmation mirrors how mature AI evaluation practice separates capability
measurement from contextual validation, and it can be operated at the pace at which GenAI-enabled devices
evolve.
Its value depends on one operationalization. The clinician analogy holds for how competence is assessed but
breaks on what is being credentialed. A clinician is a continuous entity; a GenAI-enabled device is a
configuration of components — the foundation model version, system and task prompts, retrieval assets,
guardrails, orchestration logic, and user interface. Section V.A already states that the final user-facing device,
as configured for deployment, is the object of evaluation. We recommend that CDRH make “as configured”
auditable by expecting sponsors to document a versioned configuration manifest identifying every behaviordetermining component of the assessed system. The manifest answers a question that reviewers will
otherwise be unable to answer later — which exact system was assessed — and it gives Section VI a clean
trigger: any change to a manifest element invokes change control, with the premarket benchmark serving as
the regression baseline anticipated in Section V.B.2.
The empirical case for version binding is strong. When the March and June 2023 versions of the same
commercial LLM service were evaluated on identical tasks, accuracy on prime-number classification fell from
84% to 51%, partly because the later version was less willing to follow chain-of-thought instructions [1]. This
demonstrates that model updates can materially change behavior; it does not establish an unannounced
change within a pinned version. A competency assessment should therefore remain tied to the configuration
that was evaluated.
In our work engineering AI evaluation platforms for life-sciences clients, including our internal Visium Bench,
we have found that even within a fixed model version, outputs can diverge across runs because of floatingpoint round-off and parallel execution in the inference stack. To achieve the reproducibility expected in GxPregulated environments — identical outputs for identical inputs across runs — we bind evaluations not only to
a model version but to an explicit infrastructure configuration. We frequently use open-weight models for this
reason: when model weights, hardware, and token serving are under the sponsor’s control, reproducibility can
be demonstrated rather than assumed, whereas API-only models limit a sponsor’s control over weights,
versioning, and inference conditions and thereby constrain deterministic reproducibility. Therefore we
recommend, to also require to document not only the model, but also the infrastructure configuration.
Finally, as Freyer et al. note in the paper’s footnote 16, the analogy also breaks on accountability: clinicians
face professional and legal consequences for substandard care. Competency assessment should therefore be
understood as complementing, not substituting for, clear sponsor accountability across the total product life
cycle.

Question 9 — Benchmarking elements
Would a benchmarking structure such as the one described above be likely to provide adequate evidence of
clinical knowledge, analytic capabilities, safety behavior, communication, and generalizability to support a
reasonable assurance of safety and effectiveness? Are there elements that are missing, redundant, or
inappropriately categorized? Are there externally developed standards that could be leveraged?

The proposed structure — Safety, Clinical Proficiency, Generalizability, and Agentic AI Capabilities — is
coherent and would, in our assessment, capture the right competencies. The consistent two-directional error
framing (under- and over-escalation in S.1, under- and over-refusal in S.2, deferral without over-deferral in S.3
and E.2) reflects how these systems actually fail in use, and we endorse it.
One gap we identified is the reliability of the evaluation harness itself, and should be an explicit cross-cutting
consideration. Wherever device outputs are scored against rubrics or by adjudicators — human or LLM — the
measurement instrument is, in our experience, frequently the largest uncontrolled source of variance in a
GenAI evaluation. GPT-4-class judges agree with human preference in over 80% of cases, but carry
documented position, verbosity, and self-enhancement biases and limited ability to grade reasoning [2]; on
challenging response pairs with objectively correct answers, many strong judge models perform only slightly
better than random guessing [3]; and large-scale comparisons recommend validating any LLM judge against
human annotators for each use case rather than assuming transfer [4]. We recommend that sponsors report
judge–human agreement, inter-adjudicator reliability, and rubric stability, together with sensitivity to clinically
important errors, false-clearance rates, subgroup performance, and uncertainty in these estimates. Overall
agreement alone can conceal failure on rare but consequential cases. Section V.B.2 already addresses
adjudicator qualifications and independence; these measures would make the evidentiary expectation more
explicit. The paper’s general principle that these considerations apply when the adjudicator is itself an LLM
(Section V.B.2) points in this direction; we suggest making the evidence requirement explicit.
A second point is less a missing element than one to make explicit: design for re-execution. Element R.1 covers
run-to-run reproducibility within a version, and Section VI anticipates re-benchmarking after modification. The
benchmarking chapter should state that the premarket suite is expected to be versioned, sequestered where
feasible, and automatable, so that re-benchmarking against “the same capabilities” (Section VI.C) is technically
achievable rather than aspirational.
On categorization, adversarial robustness and prompt injection currently appear under S.2 and A.1. We
recommend clarifying that resistance to injection through user inputs, retrieved content, and tool outputs
applies to every conversational GenAI-enabled device that uses retrieval or tools, not only to agentic ones, and
consolidating this with R.1’s adversarial-input language.
We also recommend making clinically consequential omissions and information provenance explicit within
E.1–E.3 and R.1. For functions that extract, summarize, or write clinical information back to a record, testing
should follow material facts through the complete information pathway, including subsequent reuse. Relevant
failures include loss of patient identity, temporality, negation, units, and evidentiary status, as well as treating
a previously generated summary as independent corroboration of the original finding. Sponsors should retain
sufficient provenance to distinguish source observations from generated inferences and to trace clinically
material statements to their supporting information. The depth of traceability should follow the intended use
and risk; it need not entail disclosure of model reasoning or indefinite retention of all prompts. Medical
summarization research distinguishes omissions from unsupported additions and evaluates their clinical
severity [21]. Testing these pathways would also build on FDA’s existing interoperability considerations [22].
On standards: existing rails cover risk and management processes — AAMI TIR34971:2023 (successor to the
FDA-recognized AAMI CR34971:2022) for applying ISO 14971 to machine learning [5], ISO/IEC 23894 for AI risk
management, ISO/IEC 42001 for AI management systems, and ISO/IEC TS 4213 for classification-performance
assessment. What is missing are GenAI-specific test-method standards for the elements in Appendix A. That
gap is a concrete area where qualified third parties could contribute (see our response to Question 16).

Question 10 — Benchmark validity, contamination, and sponsor-developed benchmarks
CDRH recognizes that publicly available benchmarking assets may be subject to data contamination,
saturation, and limited real-world representativeness. How should a sponsor establish that performance on
a given benchmark predicts safe and effective real-world behavior for the device’s intended use? What
evidence should support the construct validity of a benchmark used to gate device evaluation, and what
role should sponsor-developed benchmarks play given potential concerns around independence and
optimization to the test?

We recommend that CDRH draw a clear line between screening and gating uses of benchmarks. Publicly
available assets should be classed as screening evidence; gating decisions should rest on assets with
demonstrated construct validity and controlled exposure.
Saturation is documented. Knowledge-based medical benchmarks show saturation, with leading models
reaching 84–90% on USMLE-style examinations [6], and Med-Gemini reported 91.1% on MedQA [7]. A
saturated benchmark no longer discriminates between adequate and excellent performance, and high scores
may reflect memorization rather than capability.
Construct validity must be demonstrated, not asserted. The sharpest published warning we are aware of
matched MedQA items to real electronic-health-record decisions requiring the same medical knowledge: GPT4 scored 0.71 on the benchmark but 0.28 on the matched real-world decisions [8]. Benchmark-to-deployment
correspondence evidence — task fidelity, population and setting match, and input-distribution realism —
should be a submission expectation for any benchmark used to gate.
Contamination audits are necessary but insufficient. Overlap-based detection presumes access to training data
that sponsors building on third-party foundation models do not have, and it is bypassed by paraphrased or
translated test items: models fine-tuned on rephrased benchmark samples reach GPT-4-level scores while
evading n-gram decontamination [9], and cross-lingual contamination evades current detection methods
entirely [10]. The load-bearing control is therefore sequestered, post-training-cutoff, rotating held-out test sets
under independent custody (see Question 16), with contamination screening and saturation reporting as
supporting evidence.
Sponsor-developed benchmarks are legitimate and often necessary for intended-use specificity, provided they
operate under controls: acceptance criteria prespecified before test execution (consistent with Section V.B.2);
independent custody of at least a held-out partition; documented item provenance and clinical validation; and
disclosure of any optimization loop between development and testing. This extends the concern about trainto-the-test overfitting raised by the FDA authors cited in the paper’s footnote 18 [11] into a workable control
set. Finally, no single evidence source should gate: benchmark evidence should be triangulated with
deployment-representative testing and a clinical-confirmation tier matched to the device’s position in the twoaxis risk framework.

Question 16 — Independent third parties
Should independent third parties be involved in some or all aspects of a competency-based approach,
including device benchmarking and clinical confirmation? If so, in what ways might qualified, independent
third-party participation contribute to this approach, and what qualifications and independence criteria
should apply? Are there aspects of the assessment for which third-party involvement would be impractical
or inadvisable? What safeguards or program design features would be critical to prevent third-party
participation from limiting competition or preventing innovation?

Yes, for the functions that sponsors and foundation-model developers cannot credibly perform for themselves.
In our view these are: custody and rotation of sequestered evaluation datasets; independent adjudication by
qualified human panels and by validated automated judges; reproducible execution of benchmark suites at
initial review and at re-benchmarking after modifications; and the development of FDA-recognizable testmethod standards for GenAI evaluation.
A working federal precedent exists. NIST’s Face Recognition Technology Evaluation (formerly FRVT) has run
sequestered, black-box testing since 2017: developers submit algorithms, never see the test data, and
resubmit as they update — 1,385 algorithms from 426 developers to date [12]. One program design
simultaneously mitigates contamination and train-to-the-test (Question 10) and provides a natural mechanism
for post-modification re-benchmarking (Questions 22 and 24).
The rails exist; the standards do not. ASCA has been a permanent program since September 2023 under
MDUFA V, and MDDT can qualify evaluation tools; but ASCA’s present scope is bench-testing standards for
biocompatibility and basic safety and essential performance [13]. Extending these rails to GenAI presupposes
FDA-recognized GenAI test methods. We therefore recommend sequencing: standards development first, then
accreditation, then conformity assessment.
Independence criteria should bind any assessor. We note for transparency that Visium may in future offer
evaluation or assurance services of the kind discussed here, and we propose criteria that would constrain us
equally: structural and financial separation from both the device sponsor and the foundation-model developer;
no contingent or success-based fees; a prohibition on assessors authoring the acceptance criteria they later
assess against; cooling-off periods between consulting for and assessing the same device; and full conflict-ofinterest disclosure.
To prevent third-party participation from limiting competition, accreditation should be capability-based rather
than size-based, with transparent published criteria; multiple bodies should be accredited; FDA should retain a
direct-review path so that third-party routes remain voluntary and least burdensome; and assessors should be
periodically re-accredited with their own performance published. Two aspects are inadvisable for third parties:
the benefit-risk determination itself, which must remain FDA’s; and any arrangement in which the same party
both builds a sponsor’s evaluation infrastructure and certifies its results.

Question 18 — Accepting greater premarket uncertainty
CDRH is considering whether it may be appropriate to accept greater premarket uncertainty regarding a
GenAI-enabled device’s benefit-risk profile through greater reliance on postmarket monitoring. Under what
conditions might such an approach be appropriate, and what characteristics of a monitoring program
would need to be in place to justify reduced premarket evidence? Are there device types or risk profiles for
which this approach would not be appropriate?

We support this approach where the residual uncertainty is specific, the expected benefits justify it, and the
monitoring and containment arrangements can address it. For GenAI-enabled devices the premarket input
space cannot be enumerated, so shifting some evidentiary weight to the postmarket phase is the realistic path;
the alternative is premarket testing that creates an appearance of coverage without the substance.
The condition is that the monitoring program is a clear requirement, rather than aspirational. At minimum it
should include: prespecified metrics with degradation thresholds; demonstrated statistical power to detect a
defined degradation within a defined time window; defined trigger events with a predefined response
playbook; technical containment capability (version pinning, staged rollout, rollback); and binding
commitments on reporting cadence and mandatory action on threshold breach. The uncertainty being
accepted should also be characterized rather than generic: sponsors should identify which subgroups,
conditions, or interaction patterns are under-evidenced at the premarket stage and design monitoring to close
exactly those gaps (element R.2).
External-validation experience with deployed clinical prediction models shows why these conditions matter.
We offer this as an analogy from predictive machine learning rather than as GenAI evidence, but the transfer
mechanism — site, workflow, and population shift — applies with greater force to open-ended systems. A
widely implemented proprietary sepsis model showed 33% sensitivity, a positive predictive value of 12%, and
an area under the curve of 0.63 in external validation across 38,455 hospitalizations [14]; an emergencydepartment validation found 14.7% sensitivity and 7.6% positive predictive value [15]; and even the updated
version, while achieving AUROC between 0.82 and 0.92 across four health systems, showed high institutional
variability, low positive predictive value, and high alert burden [16]. They illustrate the need to define
operating thresholds, prediction horizons, deployment contexts, and alert burden in a monitoring program.
Restricting a device to a read-only advisory role can limit direct changes to clinical systems, but cannot
establish that its outputs are harmless. In our work on regulated review workflows, we have used read-only
access, deterministic rules, and extraction of source evidence to constrain GenAI components. Incorrect or
incomplete evidence can nevertheless influence a human decision. Where human review is relied upon as a
safeguard, sponsors should demonstrate that intended users detect and correct clinically important errors
under representative workload, time pressure, expertise, and interface conditions. Simulated use with
deliberately seeded errors can test this interception without exposing patients to experimental
misinformation.
This approach should not be available for fully autonomous action-taking functions at the severe end of the
consequences axis, nor where errors are irreversible or harm materializes faster than monitoring can detect it.
The two-axis framework of Section IV is the appropriate instrument for drawing that boundary.

Question 19 — Postmarket performance evaluation approaches, cadence, and triggers
Please comment on the potential approaches to postmarket performance evaluation, including periodic rebenchmarking, sample-based clinician review, and performance degradation monitoring. What additional
approaches should CDRH consider, and how should the cadence and triggering events for reassessment be
determined?

The three approaches detect different failure classes and should be treated as complements rather than
alternatives: re-benchmarking detects capability regression; sample-based clinician review detects contextual
quality and appropriateness failures; degradation monitoring detects distribution shift and silent drift. A
credible program includes all three, weighted by the device’s risk position.
Cadence should be event-driven first and calendar-based as a backstop. We recommend prespecified trigger
events: any change to a configuration-manifest element (Question 7); an upstream foundation-model version
change or deprecation notice (Question 24); input-distribution shift beyond prespecified bounds; a proxymetric threshold breach; and complaint or adverse-event signals. Calendar frequency should scale with the
device’s position in the two-axis framework.
Within clinician review, sampling frames should be stratified by risk rather than drawn at random. Safetyrelevant interactions — escalation decisions, dosing and contraindication queries, long conversations, lowprevalence subgroups — are rare by construction, and random samples systematically under-power them. This
is a methodological recommendation from evaluation practice; it should be reflected in how sponsors justify
sample design.
Degradation monitoring must confront delayed or absent clinical ground truth at runtime. The practical
approach is to monitor validated proxy metrics — escalation-rate bands, in-scope and out-of-scope refusal
rates, output-property distributions, retrieval quality — against baselines frozen at authorization, and to
evaluate them per deployment context as well as in aggregate; the documented inter-institutional variability of
deployed clinical models [16] argues against relying on global averages alone. One additional approach worth
naming is structured in-workflow clinician flagging that feeds the review sample: it is low-cost and catches
failure modes that metrics miss.

Question 20 — Machine-based supervisory agents
Please comment on whether the proposed approaches to postmarket monitoring can be facilitated by
machine-based supervisory agents. What considerations, including the evaluation and reliability of the
supervisory agent itself, should CDRH take into account for such an approach?

Machine-based supervision is the only economical way to observe GenAI-enabled devices at production
volume, and we support its use — architected as a screening layer whose flags route to human adjudication,
never as the sole authority on device performance.
The supervisory agent is itself a GenAI system with the same failure modes as the device it supervises, and it
should be validated like the measurement instrument it is: agreement measured against human-adjudicated
reference sets before deployment, agreement audited on a defined cadence thereafter, and drift monitoring
applied to the judge itself.
Two structural rules follow from the evidence. First, the supervisor should not share a model family with the
supervised device. LLM evaluators score their own outputs higher than those of other models while human
annotators rate them as equal in quality, and the strength of this self-preference correlates linearly with the
model’s ability to recognize its own text [17]; self-enhancement bias is independently documented [2]. Shared
lineage means correlated blind spots, which defeats the purpose of supervision. Second, cases the supervisor is
least certain about must be routed to humans by design: on hard, correctness-grounded discriminations,
strong judge models perform only slightly better than chance [3], and those are precisely the clinically
consequential cases.
For governance, the supervisor’s version should be pinned in the configuration manifest (Question 7), its scope
prespecified like any other component, and changes to it treated as trigger events for reassessment (Question
19). This directly addresses the question of the supervisory agent’s own evaluation and reliability.

Question 24 — Changes initiated by third-party foundation model developers
For devices built on third-party foundation models, changes to the underlying model may be initiated by the
third-party model developer rather than the device manufacturer. How can a manufacturer detect,
evaluate, and respond to such changes in a timely manner? What mechanisms—for example, contractual,
technical, or through a PCCP—could provide reasonable assurance that third-party developer-initiated
changes do not compromise the safety or effectiveness of the device?
The safe engineering assumption is that upstream models change silently. Materially different behavior behind
an unchanged service name within three months is documented — 84% to 51% on identical tasks between
successive versions of the same commercial model [1] — which makes notification-dependent designs
untenable on their own; our own experience with API-only models (see our response to Question 7) points the
same way. Detection should rest on continuous behavioral fingerprinting: a compact, automated regression
battery drawn from the premarket benchmark (Section V.B.2), run on a schedule and on any provider
announcement, with alerting on deviation.
Technically, manufacturers should deploy against pinned, explicitly identified model versions rather than a
floating “latest”; adopt new versions only behind re-benchmarking gates; and retain the ability to fall back to
the last validated configuration. Contractually, minimum terms should include advance change notification,
deprecation windows long enough to re-validate, documented version identifiers, and access to change
descriptions. We note candidly that negotiating power varies with manufacturer size; where such terms cannot
be secured, the residual risk should surface explicitly in the device’s risk assessment and monitoring intensity
rather than disappear into an assumption — and this is exactly what a voluntary Foundation Model Master File
(Section VII.A) could help standardize.
Within the PCCP framework, “adoption of a new version of the named underlying foundation model” is a
prespecifiable change category. A PCCP covering it would specify the re-benchmarking protocol (the same
suite and acceptance criteria as the premarket baseline), the rollback criteria, and the reporting obligation,
operationalizing Section VI.C through the final PCCP guidance cited in the paper’s footnote 22 [18].
For awareness rather than as a recommendation to adopt: under the EU AI Act, providers of general-purpose
AI models have been obliged since 2 August 2025 to draw up, keep up to date, and make available to
downstream providers documentation covering the model’s capabilities, limitations, and integration
requirements (Article 53(1)(b) and Annex XII of Regulation (EU) 2024/1689) [19]. The practical relevance for
CDRH is that manufacturers active in both markets already operate these documentation and update channels
with their model providers, so US expectations along the lines above could draw on existing machinery at low
marginal burden — consistent with the Innovation and Global Leadership pillar’s interest in regulatory
frameworks that keep the US approach the global reference point.
3. Brief responses to further discussion questions
Question 1 — Dimensions of the risk framework
We support the two-axis framework as the primary heuristic. The additional dimensions named in the question
are real but operate better as modifiers of position within the grid than as new axes: reversibility and the
availability of downstream safeguards shift a function along the consequences axis, since an irreversible or
unsupervised action is in effect more severe; time pressure and output traceability affect the user’s ability to
detect and avoid relying on an incorrect output, which the paper already recognizes for measurement and
signal-processing functions. We recommend that sponsors be asked to state reversibility and errordetectability explicitly as named modifiers in the risk assessment.
Question 2 — The directiveness continuum
We support assessing directiveness functionally — by the substance, specificity, and personalization of the
output in context — rather than by trigger words, and we agree that disclaimers do not reduce directiveness.
For predictability, sponsors could characterize directiveness empirically rather than by assertion: a sample of
representative outputs scored by independent reviewers against a prespecified rubric (general information;
contextualized; endorsing; instructing) would make a device’s position on the continuum evidence-based and
comparable across submissions.
Question 5 — Multi-turn conversational trajectories
Risk for conversational devices should be assessed at the trajectory level, not only per turn. Simulated multiturn encounters with scripted user behaviors — minimization of symptoms, pressure for a recommendation,
gradual drift out of scope — are the practical instrument, consistent with the testing the paper envisages for
elements S.1 and S.2, and acceptance criteria should be set on conversation-level outcomes: whether the
device escalated, refused, or deferred appropriately by the end of the exchange, and how early. The intended
use of such a device can be characterized by the set of conversational trajectories it is designed and tested to
handle, with out-of-envelope trajectories governed by S.2 boundary behavior.
Question 8 — Linking the risk framework to the level of evidence
The two-axis position should determine three things: which benchmarking elements are required, the depth
and adversarial intensity of test conditions (for example the extent of R.1 adversarial and long-conversation
testing), and the clinical-confirmation tier selected from the Section V.C ladder. A simple mapping — lower-left
functions relying primarily on benchmarking plus retrospective evaluation, upper-right functions requiring
shadow deployment or prospective study — would give sponsors predictability without a rigid matrix. See also
our responses to Questions 7 and 18.
Question 12 — Statistically meaningful performance measurement
For GenAI outputs, variance is a first-class result rather than noise. Performance should be reported with
confidence intervals across repeated runs and across semantically equivalent paraphrases, because a point
estimate from a single run overstates precision for a stochastic system. Where synthetic inputs supplement
real data, performance on real and synthetic strata should be reported separately before any pooled estimate,
with pooling weights justified by the anticipated real-world input distribution. We would be cautious about
combining benchmarking and clinical-confirmation evidence into a single performance estimate: they measure
different constructs and are better presented as a coherent chain of evidence than as one number.
Question 13 — Synthetic data
Synthetic data is well suited to stress-testing rare presentations, safety-critical edge cases, and adversarial
inputs, and poorly suited as the sole basis for subgroup performance claims. The principal safeguard is
generator independence: synthetic inputs produced by the same model family as the device will tend to
reproduce that family’s blind spots, and contamination research has found benchmark material propagating
into synthetic datasets generated by frontier models [9].
In our own evaluation work we generate synthetic test inputs with a generator independent of the system
under test and validate their fidelity against held-out real examples before use. Sponsors should document the
generator, demonstrate fidelity against held-out real data for the subgroup in question, and treat synthetic
evidence as supplementary wherever real data exist.
Question 14 — Comparators and acceptance criteria for open-ended outputs
The comparator should follow the intended workflow. Where the device is used by a clinician, human–AI team
performance against unaided clinicians is the relevant evaluation; where the device acts alone, device-alone
performance against a qualified reference panel is. A consensus panel reflecting the standard of care is in most
cases the more defensible comparator than a median clinician, because it is explicitly defined and auditable.
Where LLM adjudicators are used in either setting, the considerations in our responses to Questions 9 and 20
apply.
Question 21 — Roles of other stakeholders in postmarket monitoring
Clinicians, institutions, and professional societies are best placed to contribute structured in-workflow
feedback, local performance data, and clinical reference standards; standards bodies are best placed to codify
test methods. These contributions should feed the manufacturer’s monitoring program rather than replace it.
The manufacturer should remain the single accountable party for monitoring, reporting, and corrective action,
and any third-party role should be subject to the independence criteria described in our response to Question
16.
Question 22 — Scaling re-evaluation to the modification
The extent of re-benchmarking should follow a change taxonomy tied to the configuration manifest (Question
7). Changes to the foundation model or to safety-critical prompts and guardrails warrant full re-execution of
the premarket suite; changes to non-safety-critical presentation or orchestration may warrant a targeted
subset; documentation-only changes may be managed within the quality management system. Categories of
change that are predictable in kind but not in timing — including upstream model updates — are well suited to
a PCCP, as discussed in our response to Question 24.
Question 23 — PCCP concepts when modifications cannot be fully prespecified
When the nature of future modifications cannot be fully prespecified, the PCCP can still prespecify the process:
the trigger events that invoke re-evaluation, the benchmark suite and acceptance criteria to be re-executed,
the rollback criteria, and the reporting obligations. A process-level PCCP is, in our experience, more robust for
GenAI-enabled devices than attempts to enumerate every modification in advance.
Question 25 — Foundation Model Master Files
We support voluntary Foundation Model Master Files. Given limited incentives for disclosure, their value will
depend on a minimum content set that is directly usable in review: version identifiers and change-notification
commitments, deprecation policy, documented safety-relevant behaviors and guardrails, and healthcarerelevant evaluation results. A master file should inform the review of a device built on the model and never
substitute for device-level evidence, as the paper correctly states. Our response to Question 24 notes, for
awareness only, that comparable documentation duties already exist for general-purpose model providers in
the EU.
Question 26 — Agentic GenAI-enabled devices
We endorse element A.1 and recommend two additions. Compliance with human-oversight checkpoints
before irreversible or high-consequence actions should be a tested behavior rather than a documented
intention, with acceptance criteria set on action-level outcomes across complete task sequences. Resistance to
injection through retrieved content and tool outputs should be tested for every GenAI-enabled device that
uses retrieval or tools, not only those classified as agentic, as noted in our response to Question 9.

4. Closing
The discussion paper poses the right questions and, in the competency-based approach and the two-axis
framework, offers structures that practitioners can work with. Our recommendations are intended to make
those structures verifiable in practice: by binding assessment to a defined system configuration, by holding
evaluation instruments to the same evidentiary standard as the devices they measure, by controlling
benchmark exposure through independent custody, and by making postmarket reliance conditional on
monitoring with real detection power. We would welcome the opportunity to discuss these comments with
CDRH staff and to contribute to the development of test methods and standards for the evaluation of GenAIenabled devices.

Respectfully submitted,
Dr Tomislav Ilicic
Partner Lifesciences AI

Visium
Löwenstrasse 49, 8001 Zürich, CH
E-Mail: anton.vanpamel@visium.com
References
[1] Chen L, Zaharia M, Zou J. How Is ChatGPT’s Behavior Changing over Time? arXiv:2307.09009 (2023);
Harvard Data Science Review (2024).
[2] Zheng L, Chiang W-L, Sheng Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances
in Neural Information Processing Systems 36 (2023); arXiv:2306.05685.
[3] Tan S, Zhuang S, Montgomery K, Tang WY, Cuadron A, Wang C, Popa R, Stoica I. JudgeBench: A Benchmark
for Evaluating LLM-based Judges. International Conference on Learning Representations (2025);
arXiv:2410.12784.
[4] Bavaresco A, et al. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation
Tasks. arXiv:2406.18403 (2024).
[5] AAMI TIR34971:2023, Application of ISO 14971 to machine learning in artificial intelligence — Guide (also
published as BS/AAMI 34971:2023); successor to AAMI CR34971:2022, recognized by FDA as a consensus
standard in January 2023.
[6] Systematic review of clinical large language model benchmarks. Journal of Medical Internet Research
2025;27:e84120.
[7] Saab K, et al. Capabilities of Gemini Models in Medicine. arXiv:2404.18416 (2024).
[8] Alaa A, Hartvigsen T, Golchini N, Dutta S, Dean F, Raji ID, Zack T. Position: Medical Large Language Model
Benchmarks Should Prioritize Construct Validity. Proceedings of the 42nd International Conference on Machine
Learning (2025); arXiv:2503.10694.
[9] Yang S, Chiang W-L, Zheng L, Gonzalez JE, Stoica I. Rethinking Benchmark and Contamination for Language
Models with Rephrased Samples. arXiv:2311.04850 (2023).
[10] Yao F, Zhuang Y, Sun Z, Xu S, Kumar A, Shang J. Data Contamination Can Cross Language Barriers.
arXiv:2406.13236 (2024).
[11] Garcia V, Sidulova M, Badano A. Performance Assessment Strategies for Language Model Applications in
Healthcare. Artificial Intelligence in the Life Sciences 2026;9:100162 (cited in the discussion paper, footnote
18).
[12] National Institute of Standards and Technology. Face Recognition Technology Evaluation (FRTE), 1:1
Verification — participation statistics as of March 2026. https://pages.nist.gov/frvt/
[13] U.S. Food and Drug Administration. Accreditation Scheme for Conformity Assessment (ASCA) Program;
conversion from pilot to permanent program effective September 19, 2023, as authorized by MDUFA V.
[14] Wong A, Otles E, Donnelly JP, et al. External Validation of a Widely Implemented Proprietary Sepsis
Prediction Model in Hospitalized Patients. JAMA Internal Medicine 2021;181(8):1065–1070.
[15] Ostermayer DG, Braunheim B, Mehta AM, Ward J, Andrabi S, Sirajuddin AM. External validation of the Epic
sepsis predictive model in 2 county emergency departments. JAMIA Open 2024;7(4):ooae133.
[16] Wong A, et al. Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model.
JAMA Network Open 2026;9:e260181.
[17] Panickssery A, Bowman SR, Feng S. LLM Evaluators Recognize and Favor Their Own Generations. Advances
in Neural Information Processing Systems 37 (2024).
[18] U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change
Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final guidance, December 2024
(cited in the discussion paper, footnote 22).
[19] Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 53(1)(b) and Annex XII; obligations for
providers of general-purpose AI models applicable from 2 August 2025.