AI Nova Strategy (Kevin J. Hacker)

Please see the attached joint comment from Kevin J. Hacker, Ph.D. (AI Nova Strategy) and Andriy Zezekalo. It responds to all 26 discussion questions in Appendix B of the discussion paper.

Our central point is that evidence about a GenAI-enabled device can be interpreted only when it can be tied to what produced it. We recommend that:

1. Evidence be tied to a defined configuration of model or hosted service version, software, prompts, retrieval sources, and safety controls, including provider-controlled components that can change without a new version identifier (Q22, Q25).

2. Device inputs and benchmark, synthetic, and clinical evidence be traceable to their origin and transformations, starting with the instrument run for sensor data and documented authorship for clinical narrative and EHR inputs (Q1, Q10, Q13, Q17).

3. Postmarket observations and modifications be linked to the authorized baseline, preserved as a record because a retired model version may no longer run, and governed through existing QMSR, 21 CFR Part 803, and PCCP processes (Q18, Q19, Q21 to Q24).

4. Recorded changes in input sources, interfaces, and components trigger reassessment before performance degrades, alongside statistical drift monitoring (Q19).

5. Runtime safeguards, supervisory agents, and agentic action controls be independently validated and scaled to risk (Q5, Q20, Q26).

The views expressed are our own and do not represent any current or former employer.

Kevin J. Hacker, Ph.D.
Andriy Zezekalo

FDA-2026-N-7874: Considerations for GenAIEnabled Medical Devices
Document Type: Consolidated Joint Public Comment Submission
Agency Docket Reference: CDRH Discussion Paper: Considerations for the Regulation of
Generative AI-Enabled Medical Devices
Submission Date: October 19, 2026 (Submitted to the CDRH Digital Health Center of Excellence)
Authors: Kevin J. Hacker, Ph.D. (AI Nova Strategy, kevin@ainovastrategy.ai) and Andriy Zezekalo
(formerly Senior Quality Engineering / QA Leader, Medtronic Diabetes, azezekalo@gmail.com)
Affiliation Disclaimer: The views and technical positions expressed in this submission represent
the independent personal and professional opinions of the authors and do not represent the official
positions of any current or former employer.

Executive Summary
Our comments center on a common principle: evidence about a GenAI-enabled medical device can
be interpreted only when it can be tied to what produced it. Question numbers in parentheses refer
to the discussion questions in Appendix B of the FDA paper [1]. Three themes recur throughout this
joint submission:

1. Configuration Identity: Evidence should be associated with a defined, reproducible configuration
of model or hosted service version, software build, system prompts, retrieval corpora, orchestration
logic, and runtime safety controls, including provider-controlled components that can change without
a new version identifier (Q22, Q25).

2. Evidence Lineage: Device inputs and benchmark, synthetic, clinical, and reference standard
evidence should be traceable to their origin, transformations, and custody history. Traceability is only
as reliable as its first link: the instrument run for sensor data, and documented authorship and
transformations for clinical narrative and EHR inputs (Q1, Q10). Multimodal provenance should be
recorded for each modality before fusion (Q17), and synthetic evidence requires model, data, and
reference answer lineage (Q12, Q13). An integrity check shows that a record did not change. It does
not show that the record is accurate or representative (Q10).

3. Lifecycle Continuity & Quality Governance: Postmarket observations, field reports, drift
monitoring, and post-deployment modifications should be linked to the configuration that produced
them and bound to the authorized premarket baseline (Q18, Q21, Q22). That baseline should be
preserved as a record, because a retired model version may no longer run (Q23). These records
should be governed through existing quality system regulations (QMSR / ISO 13485:2016) [2, 3],
medical device malfunction reporting (21 CFR Part 803) [4], and Predetermined Change Control
Plans (FD&C Act Section 515C) [5] rather than separate documentation silos (Q16, Q19, Q24).

Provenance also serves as an early warning. When a device records the source, interface, units,
and component versions of its inputs, a discrete change can trigger reassessment before
performance degrades. The same records let a sponsor tell a change in source data or configuration
apart from a change in model behavior (Q18, Q19). Provenance complements runtime safety
controls and statistical drift monitoring. It does not replace them (Q5, Q19, Q20).

Table of Contents & Response Roadmap

· Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices:
Questions 1–6 (Two-axis framework, directiveness spectrum, patient vs. HCP delivery,
generalist/specialist risk, multi-turn trajectory risk, asymmetric care escalation).

· Section V: A Competency-Based Approach for Premarket Evaluation: Questions 7–17
(Competency framework, risk integration, benchmarking structure, benchmark
contamination/saturation, real-world input distributions, synthetic data statistical rigor, synthetic
data clinical domains & 3-tier lineage, open-ended comparators, counterfactual comparators,
third-party benchmark custody & ASCA, multimodal vision-language models and world models).

· Section VI: Postmarket Monitoring for GenAI-Enabled Devices: Questions 18–24
(Premarket uncertainty vs. postmarket evidence & Rule of Three sample sizing, reassessment
triggers & cadence, machine-based supervisory agents, stakeholder roles & triage handoff
controls, premarket baseline & 3-tier change control matrix, adapting PCCPs for unprespecified
changes & bounded envelopes, third-party foundation model changes & purchasing controls).

· Section VII: Other Topics: Questions 25–26 (Foundation Model Device Master Files,
agentic GenAI devices & propose-then-authorize protocol).

Section IV: Considerations for the Assessment of Risk for GenAIEnabled Devices

Question 1: Two-Axis Risk Framework & Additional Dimensions

FDA Prompt: Does the two-axis risk framework, organized around device activity and the
consequence of relying on an incorrect output, appropriately capture the dimensions most relevant to
the risk of a GenAI-enabled software function? If there are additional dimensions—such as the
reversibility of a resulting action, the availability of downstream safeguards, the time pressure of the
deployment setting, or the traceability of the output (i.e., to primary source materials)—that should be
represented in a risk framework, please describe and provide examples of how they should be
represented.

The two-axis risk framework (device activity vs. consequence of erroneous output) is necessary but
incomplete for generative AI. Three additional dimensions should be formally represented:

· a. Reversibility of Resulting Action and Deployment Time Pressure: If an output triggers an
immediate or irreversible clinical intervention without sufficient clinician time window to detect
 and intercept an error (e.g., an acute emergency department triage recommendation where a
clinician has seconds to evaluate an alert), the function should sit in the highest-risk tier of the
framework. Reversibility and available downstream latency are primary determinants of safety
criticality.

· b. Traceability to Primary Sources and Ingestion Attribution: Traceability is only as reliable
as its first link. For sensor-driven devices, this requires verifiable linkage to the physical
instrument run; for clinical narratives and EHR inputs, it requires tamper-evident audit logs
documenting authorship and transformations. This provenance enables reviewers to determine
whether an incorrect output stemmed from corrupt source data, faulty preprocessing, or model
error. Section IV of the paper [1] notes that measurement and signal processing functions may
be of higher risk because the user typically cannot independently evaluate the basis for the
output. For those functions, the provenance record becomes the main way to check how an
output was produced. It does not by itself show that the output is correct.

· c. Stateful Trajectory Risk: Generative devices rarely operate as single, isolated
transactions. Risk accumulates and migrates across multi-turn interactions, requiring evaluation
across full dialogue trajectories rather than isolated turns.

Question 2: Informational Functions Spectrum (Non-Directive vs. ActionDirecting)

FDA Prompt: CDRH seeks input on how to account for the spectrum of GenAI-enabled informational
functions that vary in the degree to which they direct a user to a particular action (i.e., between “nondirective” and “action-directing”). What characteristics of an output—such as its wording, specificity,
personalization, or context—could be considered as modifiers of the risk associated with an
informational function, after accounting for the device’s overall functionality and intended use? What
additional information would provide manufacturers with sufficient clarity and predictability around risk
assessment for informational functions while recognizing that directiveness may exist along a
continuum rather than as a binary distinction?

Directiveness is a continuous spectrum governed by substance, specificity, personalization, and
clinical actionability, not solely by linguistic phrasing (such as "recommend" vs. "consider"). For
example, an output stating "some patients in your situation take 20 mg" is substantively actiondirecting even if linguistically phrased as informational. Modifiers of directiveness should include: (1)
Specificity of treatment or dosage; (2) Personalization directly tied to user-specific clinical
biomarkers; and (3) Explicitness of actionable next steps. Output substance should be bounded by
deterministic post-generation validation to prevent prescriptive directives unless cleared under an
action-directing intended use.

Question 3: Patient-Facing vs. HCP-Facing Delivery of Clinical Information
 FDA Prompt: CDRH seeks input on whether and when a GenAI-enabled function that results in
delivery of clinical information to patients, as opposed to HCPs, could present different or higher risks,
while also recognizing the potential benefits associated with improved patient empowerment,
engagement, and access to clinical information. What device characteristics, output features, or
safeguards might mitigate risks that could arise when a user lacks the domain knowledge to
independently evaluate an output, without unnecessarily underestimating patient capability?

Patient-facing outputs present elevated clinical risks due to asymmetry in medical domain
knowledge, but safeguards should balance protection against unnecessary disempowerment
(avoiding over-refusal, which is itself a failure mode under Element S.2). Safeguards should include:
(1) Validated plain-language formatting (for example, a target of Flesch-Kincaid [6] grade 6 to 8,
validated via human-factors comprehension testing under IEC 62366-1 [7] rather than readability
formulas alone); (2) Prohibition of raw, uncontextualized diagnostic probabilities without clear
explanatory guidance; (3) Explicit fail-safe routing to clinical care when red-flag clinical symptoms
are identified.

Question 4: Generalist vs. Specialist Physicians Risk Differentiation

FDA Prompt: CDRH seeks input on whether and how the distinction between generalist and
specialist physicians might inform the assessment of risk for HCP-facing GenAI-enabled functions.
Under what circumstances, if any, might risk be affected when an HCP who lacks the relevant clinical
specialist knowledge receives information that falls within a specialist area of practice? What device
characteristics or safeguards might mitigate such risks?

Risk escalates significantly when specialist recommendations are delivered to generalist clinicians
who lack the subspecialty depth to detect subtle hallucinations. Because generative models can
fabricate citations, safeguards should include: (1) Source citations verified programmatically against
the retrieved clinical guideline prior to display; (2) Measured, calibrated uncertainty metrics anchored
in Appendix A Element S.3 rather than raw self-reported model confidence; (3) Explicit clinical
contraindication warnings calibrated to the user credential level.

Question 5: Multi-Turn Conversational Devices & Trajectory Risk Migration

FDA Prompt: For multi-turn conversational GenAI-enabled devices that may migrate from providing
“non-directive” information to “action-directing” information over the course of an exchange, how
should risk be assessed across realistic conversational trajectories? How could the intended use of
such a device be characterized when its behavior is emergent across a conversation?

Single-turn static evaluation is structurally incapable of assessing conversational risk. Risk should be
evaluated across realistic temporal trajectories using a Conversation State Machine (CSM):
 · 1. Deterministic State Tracking: The GenAI model should operate under an external statecontroller that continuously classifies dialogue trajectory vectors into predefined clinical boundary
zones (Zone 1: Educational/Fact-Retrieval, Zone 2: Contextual Interpretation, Zone 3: Directive
Action/Prescription).

· 2. Characterizing Emergent Intended Use: The intended use should explicitly define the
permitted conversational zones and prespecify the device’s deterministic response when a
conversation attempts to leave those boundaries (e.g., terminating generation, suppressing the
token stream, and serving a standardized escalation prompt).

· 3. Trade-Off Quantification: While state tracking introduces latency overhead, False Refusal
Rate (FRR) where benign queries are halted should be treated as a formal acceptance criterion
under Element S.2 to prevent degraded clinical utility.

Question 6: Care Escalation Functions & Asymmetric Loss (Under- vs. OverEscalation)

FDA Prompt: For GenAI-enabled care escalation functions, CDRH is considering how the evaluation
may account for both under-escalation and over-escalation. How could manufacturers characterize
and weigh these two directions of error, given that they may not be commensurable and that
acceptable trade-offs may vary by clinical context?

In safety-critical clinical environments, under-escalation (false negative) and over-escalation (false
positive) are incommensurable and should not be collapsed into a single scalar loss function:

· 1. Independent Prespecified Acceptance Criteria: Sponsors should evaluate and report
under-escalation and over-escalation as separate, independent performance thresholds. Underescalation represents acute safety failure (deterioration missed), requiring strict sensitivity
bounds (e.g., illustrative target Sensitivity >= 98%). Over-escalation represents operational and
safety risk via alert fatigue, unnecessary emergent care utilization, and downstream clinician
desensitization; it should be quantified through Positive Predictive Value (PPV) or alerts per
patient-day tailored to deployment workflow.

· 2. Calibrated Indeterminate Default: Rather than relying on uncalibrated token-level entropy,
uncertainty should be derived from empirically calibrated confidence estimators (aligned with
Element S.3). When a case falls within an indeterminate confidence band, the system should
default to safe human escalation, provided the indeterminate band is prespecified and clinically
validated.

Section V: A Competency-Based Approach for Premarket Evaluation
of GenAI-Enabled Devices
Question 7: Competency-Based Approach Appropriateness

FDA Prompt: Is the competency-based approach described above, i.e., device benchmarking
followed by clinical confirmation, a useful and appropriate framework for evaluating GenAI-enabled
devices?

Yes. The competency-based model provides an appropriate framework for non-deterministic GenAI.
Benchmarking across the Appendix A elements (clinical proficiency, safety behavior, generalizability,
and agentic competencies) establishes core technical capability, while Clinical Confirmation
evaluates real-world workflow integration and human-AI dyad performance.

Question 8: Integrating the Two-Axis Risk Framework into Competency
Evaluation

FDA Prompt: How might the two-axis risk framework described in Section IV be considered within
the competency-based approach described above to help determine the level of evidence needed for
a premarket submission?

The risk matrix should directly scale premarket evidence requirements: low-risk informational
functions can rely primarily on benchmarking (including synthetic assets with documented
provenance) paired with clinician adjudication of retrospective cases (Section V.C.1). Functions with
action-directing output, irreversible clinical consequences, or autonomous execution warrant the
most rigorous confirmation approaches, up to prospective clinical studies, perturbation stress testing,
and out-of-band supervisory runtime controls.

Question 9: Benchmarking Structure Adequacy & External Standards

FDA Prompt: Would a benchmarking structure such as the one described above be likely to provide
adequate evidence of clinical knowledge, analytic capabilities, safety behavior, communication, and
generalizability to support a reasonable assurance of safety and effectiveness? Are there elements
that are missing, redundant, or inappropriately categorized? Are there externally developed standards
that could be leveraged?

The benchmarking structure across Appendix A is comprehensive, but requires specific
methodological refinements:

· 1. Simulation-Based Perturbation Testing: For pure SaMD and conversational devices,
sponsors should execute simulation-based perturbation harnesses testing output stability against
reordering of clinical information (e.g., medication list presented before symptoms),
paraphrasing, and adversarial clinical framing. Hardware-in-the-Loop (HIL) automation should be
 reserved specifically for devices integrated with physical hardware or closed-loop delivery
actuators.

· 2. Missing Element — Output Source Traceability: Traceability of generated outputs to
underlying source material and clinical guidelines should be formally established as an explicit
Appendix A benchmarking element.

· 3. Applicable Medical Device Standards: Benchmarking protocols should be grounded in
medical device quality and risk standards: AAMI TIR34971:2023 [8] (application of ISO 14971 [9]
to machine learning), IEC 62304 [10] / IEC 82304-1 [11] (health software lifecycle), and IEC
62366-1 [7] (usability engineering).

Question 10: Benchmark Contamination, Saturation & Construct Validity

FDA Prompt: CDRH recognizes that publicly available benchmarking assets may be subject to data
contamination, saturation, and limited real-world representativeness. How should a sponsor establish
that performance on a given benchmark predicts safe and effective real-world behavior for the
device’s intended use? What evidence should support the construct validity of a benchmark used to
gate device evaluation, and what role should sponsor-developed benchmarks play given potential
concerns around independence and optimization to the test?

A benchmark score is credible only when the sponsor can show where each benchmark item came
from, how it changed, and that the benchmark stayed fixed. Verifiable origin, versioning, and
cryptographic integrity are necessary for that showing. They are not sufficient.

· 1. For instrument data, each item should trace to the original instrument record. For clinical
narrative and EHR inputs, no instrument record exists, so the sponsor should document who or
what authored each item and every transformation applied to it. An integrity check shows that a
note did not change. It does not show that the note is accurate or representative.

· 2. Sponsors should address contamination with dated exposure records compared against
the training cutoff of the underlying model, and with checks against their own fine-tuning data,
retrieval sources, and prompt examples.

· 3. Sponsor-developed benchmarks should be version-frozen and checksummed before
finalizing device configuration, with an append-only log of every evaluation run and a
sequestered portion held by an independent party, refreshing exposed items through protected
anchors or statistically equated forms.

· 4. Construct validity then rests on two comparisons: benchmark composition against real
deployment inputs, and benchmark scores against clinical confirmation results. A crossreference to Question 25 applies: a Foundation Model Device Master File (MAF) disclosing
 contamination screening methods and findings would support pre-evaluation contamination
screening.

Question 11: Real-World Input Distribution & Clinical Confirmation Rigor

FDA Prompt: CDRH is considering that clinical confirmation for a GenAI-enabled device might not
require a prospective clinical study in every case, and has described above a range of approaches of
increasing rigor and patient exposure. How might a sponsor select and justify a confirmation
approach tailored to a device’s intended use and proportionate to the device’s risk profile? Are there
device types or risk profiles for which one or more of these approaches would be insufficient or
inappropriate? How might the anticipated distribution of real-world inputs be taken into consideration?
Are there other methods of clinical confirmation that might help inform the evaluation of GenAIenabled devices?

Clinical confirmation should be risk-graduated using shadow deployments and retrospective clinical
datasets to capture the real-world input distribution across multiple clinical sites, EHR vendors, and
patient demographics. Coverage gaps between benchmark datasets and real-world inputs should
directly dictate confirmation rigor. Prospective clinical studies remain essential for functions with
irreversible treatment implications, while lower-risk informational and decision-support functions can
leverage shadow clinician adjudication (recognizing that live-patient shadow deployment may still
require IRB review under 21 CFR Part 56 [12]).

Question 12: Statistically Meaningful Performance & Synthetic vs. Real Data

FDA Prompt: How should sponsors achieve statistically meaningful performance measurement for
GenAI-enabled devices? Where synthetically generated inputs supplement real patient data, how
should sponsors account for differences between the synthetic and real-world distributions when
estimating performance, and under what conditions, if any, is it appropriate to combine benchmarking
evidence and clinical confirmation evidence to support a single performance estimate?

To achieve statistically meaningful measurement for stochastic outputs, each evaluation item should
undergo repeated runs to report run-to-run variance, with sample size determined by precision
bounds on safety-critical endpoints. Sponsors should record every evaluation item as real, synthetic,
or mixed when created, carrying that record through every downstream transformation with
documented generator version and source cohort derivation. Combining synthetic benchmarking
scores with real-world clinical confirmation to create a unified performance estimate should not be
the default; it may be acceptable only when synthetic and real cohorts are demonstrated to be
comparable on relevant subgroups, using prespecified Bayesian discounting frameworks (e.g.,
power priors [13]) with formal sensitivity analyses.

Question 13: Clinical Domains for Synthetic Data & Model-Family Bias
Safeguards
 FDA Prompt: For which clinical domains, device functions, or subpopulations is synthetic data
particularly well-suited, or particularly inadequate, as a supplement to real-world evidence? What
safeguards would mitigate the risk that synthetic data generated by models of the same class as the
device under evaluation reproduces the very performance gaps the evaluation is intended to detect,
particularly for underrepresented subgroups?

Synthetic data are most appropriate as supplemental evidence when both the validity of the case
and its reference answer can be established independently of the generative process. Examples
include physiological, imaging, or genomic scenarios whose defining parameters are known, and
dosing or unit-conversion cases whose correct answer can be computed (Appendix A, E.3). They
are least reliable where validity depends on reproducing complex clinical interactions, social
determinants of health, nuanced demographic variation, or subgroup characteristics for which little
independent real-world evidence exists. If the source data underrepresents a clinically relevant
subgroup or feature, synthetic generation may reproduce that gap rather than reveal it.
Safeguards should let FDA reconstruct the chain from source data to generator, synthetic case,
reference answer, and device result, and determine whether any shared dependency could hide the
failure the test is meant to detect. That requires three kinds of lineage:

· 1. Model Lineage: Sponsors should document dependencies across the synthetic data
generator, the device under evaluation, and any model-based evaluator, including shared
foundation models or weights and distillation relationships. Different commercial labels or model
families do not establish independence.

· 2. Data Lineage: Lineage should also cover training and fine-tuning datasets and the source
cohorts used to build the generator. Models with different architectures do not provide
independent evidence when they inherit the same data blind spots. Sponsors should confirm that
the generator's source data do not overlap with the real test set or the device's training data.
Consistent with Question 12, each synthetic item should retain its real, synthetic, or mixed
status, generator and version, source cohort version, generation settings, and later
transformations. Where an item is derived from specific real records, that parentage should be
traceable under appropriate privacy controls. Otherwise, cohort-level lineage is appropriate.
Sponsors should also assess whether generated cases reproduce identifiable source patients.

· 3. Truth Lineage: The reference answer for a synthetic case should not come from the
same generative process that created the case. It should be established by construction, by an
independent clinical reference standard, or by independent expert adjudication.
Subgroup Limits: Sponsors should compare synthetic and real performance across clinically
relevant subgroups and report the amount and uncertainty of the real evidence behind each
comparison. When real subgroup data are too sparse to establish that the generator is faithful for
that subgroup, synthetic results should carry less weight and serve mainly as challenge or stresstesting evidence. A failure on a validated synthetic challenge case can identify a credible
vulnerability. Passing synthetic cases alone should not establish that the vulnerability is absent in
real use, or how often it occurs.
Question 14: Comparator Selection for Open-Ended Outputs & Human-AI Teams

FDA Prompt: For open-ended device outputs, how should performance comparators and acceptance
criteria be selected? When a panel of qualified clinicians serves as the comparator, how should the
applicable standard (for example, the standard of care versus the performance of a median clinician
in practice) be defined and justified? Should generalist or specialist physicians be used as a
performance standard? When should human-AI team performance, rather than the device operating
alone, serve as the basis for evaluation?

For open-ended outputs, comparator selection should reflect the intended clinical context and
intended user:

· 1. Consensus-Adjudicated Standard: When clinician panels serve as the comparator,
sponsors should prespecify an achievable agreement threshold (referencing the 0.61–0.80
"substantial agreement" band per Landis & Koch, 1977 [14], or task-appropriate Krippendorff’s
alpha [15]) backed by a formal protocol for resolving discordant ratings (e.g., senior specialist
adjudication). Very high consensus thresholds (for example, kappa of 0.85 or above) may be
unrealistic for complex clinical judgment.

· 2. Comparator Tiering (Generalist vs. Specialist): The comparator standard should match
the intended user credential level. If a device is labeled for generalist use to interpret specialist
findings, performance should be evaluated against board-certified specialists as the gold
standard, measuring whether the generalist-AI team achieves specialist-level care.

· 3. Human-AI Team Evaluation: For devices used with a clinician in the loop, evaluation
should measure the clinician-device dyad rather than the model in isolation, specifically
quantifying clinician susceptibility to automation bias (agreeing with incorrect AI advice) and alert
fatigue.

Question 15: Comparators Relative to Course of Care in Device Absence

FDA Prompt: Are there ways in which performance might be assessed relative to the care,
technology, or course of action likely to occur in the absence of the device, rather than to the
comparators described in this section? What approaches might be used to identify and justify a
comparator such as unaided clinical judgment, delayed specialist review, or no intervention?

Shadow deployment protocols provide a natural counterfactual comparator by recording standard-ofcare decisions made in the absence of the device, establishing empirical baselines for time-totreatment and specialist referral delays. Because shadow data reflect clinical decisions rather than
downstream patient outcomes, comparisons should be supplemented with historical cohort controls
or stepped-wedge rollout designs, with counterfactual comparators (e.g., delayed specialist
consultation in rural settings) justified directly from actual site-level care pathways.
Question 16: Independent Third-Party Participation in Benchmarking

FDA Prompt: Should independent third parties be involved in some or all aspects of a competencybased approach, including device benchmarking and clinical confirmation? If so, in what ways might
qualified, independent third-party participation contribute to this approach, and what qualifications and
independence criteria should apply? Are there aspects of the assessment for which third-party
involvement would be impractical or inadvisable? What safeguards or program design features would
be critical to prevent third-party participation from limiting competition or preventing innovation?

Independent third parties are most valuable where independence protects the validity of the
evidence: custody of sequestered benchmark assets, execution of blinded regulatory evaluations,
and independent adjudication. Their involvement should be proportionate to device risk rather than
universally required.
Independence alone does not protect a sequestered benchmark. The custodian should keep a
traceable record that links each official evaluation attempt to the device configuration, the underlying
model and version, the benchmark version, the scoring method, the adjudication, and the reported
result. All official attempts should be recorded and disclosed, not only the one submitted. The main
risk is not the number of runs. It is feedback. When scores from repeated runs guide changes to the
device, the hidden benchmark becomes an undeclared development loop (Dwork et al., Science,
2015) [16]. Custodians should separate exploratory testing from blinded regulatory evaluation and
limit the feedback that regulatory evaluation returns. NIST's Artificial Intelligence Technology
Evaluation program, launched in July 2026 [17], shows that a sequestered, blind-data testbed is
operationally feasible.
FDA proposes evaluating the final device as deployed. For many devices, evaluation inputs pass
through the sponsor's infrastructure and an external foundation model service. The evaluation record
should identify that data path and the retention, logging, training-use, and access controls that apply
to it. Transmission outside the controlled benchmark environment should be treated as a potential
exposure event. Continued use of affected items should depend on documented evidence that
confidentiality was preserved. Benchmark refresh should preserve both confidentiality and
comparability, for example through protected anchor items or statistically equated parallel forms, so
that re-benchmarking after a modification tests the same capabilities without reusing exposed items
indefinitely.
ASCA [18] and MDDT [19] provide useful precedents for different parts of such a framework. ASCA
addresses the competence and impartiality of testing organizations, including assessment under
ISO/IEC 17025 [20]. MDDT addresses whether a tool is fit for a defined context of use. Neither
program currently covers GenAI benchmark custody, adaptive reuse, or benchmark exposure, so
additional program controls would be needed. Independence criteria should follow the paper's
criteria for expert adjudicators. The custodian should be structurally independent from the sponsor
and from the developer of any foundation model the device uses.
Third-party involvement should not become a mandatory step for every assessment or modification.
Clinical site activities and frequent low-risk changes may remain sponsor or site operated, with
independent adjudication or periodic audit where warranted. Third parties can contribute to
consensus standards and reference methods, but they should not set device-specific acceptance
criteria after seeing results, and FDA should retain authority over regulatory decisions. Competition
safeguards should include multiple qualified providers, transparent and nondiscriminatory access
and pricing, reasonable turnaround times, conflict of interest controls that bar a custodian from
evaluating a competitor's device, published benchmark methods without disclosure of protected test
content, and a path for technical appeal.
Quality System Governance (ISO 13485 and CAPA Integration): Manufacturers should qualify
and control external benchmark testing providers within their QMS (ISO 13485:2016 clause 7.4.1)
[3], commensurate with the risk and criticality of the service. A benchmark execution anomaly or
data leakage should trigger a formal Nonconformance Report (NCR) promptly, for example within 48
hours. Benchmark modifications should undergo Change Control Board (CCB) review, and a
systemic integrity breach should trigger a formal CAPA with requalification of the provider before retesting.

Question 17: Multimodal Vision-Language Models & World Models

FDA Prompt: CDRH is exploring whether a competency-based approach could be applied to devices
that incorporate a variety of underlying model architectures, such as multimodal vision-language
models and generative or predictive world models. Are there aspects of the described approach that
might be ineffective or inapplicable to these devices? What additional or distinct considerations might
need to be addressed?

Sponsors should record provenance separately for each modality, such as images, waveforms, and
text, at acquisition and before fusion, and carry it through every transformation. Provenance should
be captured when the data are generated. Lineage reconstructed later is least complete for the
transformations that matter most, those applied inside acquisition devices and vendor software
before the data reach the model.
Whole-device evaluation should remain the primary test. Sponsors should add tests that show what
each modality contributes, using methods that fit the architecture, such as removing or altering one
input. Where the architecture does not allow this, the sponsor should say so and explain how it
detects a masked failure.
Benchmark element R.1 already covers missing and contradictory inputs. For multimodal devices, it
should also cover conflicts between modalities: inputs that disagree, a missing or degraded modality,
and inputs from different times, such as a current waveform paired with an old medication list.
For predictive world models, sponsors should record what data the model had seen at the time of
each prediction, so that reviewers can score each prediction against what later happened.
Evaluation should cover how errors grow over longer prediction horizons and whether the model
updates when new data contradict an earlier prediction. A device that claims to predict the effect of
an intervention should validate that claim separately.

Section VI: Postmarket Monitoring for GenAI-Enabled Devices

Question 18: Premarket Uncertainty Reliance on Postmarket Monitoring
 FDA Prompt: CDRH is considering whether it may be appropriate to accept greater premarket
uncertainty regarding a GenAI-enabled device’s benefit-risk profile through greater reliance on
postmarket monitoring. Under what conditions might such an approach be appropriate, and what
characteristics of a monitoring program would need to be in place to justify reduced premarket
evidence? Are there device types or risk profiles for which this approach would not be appropriate?

Greater reliance on postmarket monitoring is appropriate only when the uncertainty accepted
premarket can be resolved by postmarket evidence. FDA should be able to trace each accepted
uncertainty to the evidence intended to resolve it, and trace that evidence to the device configuration
that produced it. FDA's 2019 guidance on uncertainty in benefit-risk determinations [21], cited in
footnote 21 of the paper [1], provides a useful precedent. Its factors include probable benefits and
risks, public health need, the benefit of earlier access, the feasibility of further premarket evidence,
and the likelihood that complete and timely postmarket data can address the remaining uncertainty.

· 1. Uncertainty-to-evidence traceability: Each material uncertainty accepted at authorization
should be identified and linked to its intended postmarket evidence: data source, endpoint,
population, device configuration, evaluation criteria, timeframe, and action threshold. Monitoring
should answer the specific question left open premarket, not serve as a general commitment to
watch the device.

· 2. Outcome observability: Reduced premarket evidence is defensible only when clinically
meaningful failures can be captured with adequate completeness, timeliness, and attribution. A
patient-facing triage function that wrongly advises a patient to stay home may log a normal
interaction while the harm appears in another facility's records. Where appropriate, monitoring
should link the input and context, device configuration, output, resulting clinical action, and
reference outcome, using EHR linkage, registries, or clinician follow-up where device logs
cannot.

· 3. Configuration-specific evidence: Postmarket evidence applies to the configuration that
generated it. Monitoring records should identify the software build, foundation model or service
version, retrieval and safety-control configuration, and other components that affect behavior.
These records also allow sponsors to tell a model change apart from a site-specific input shift. A
material configuration change should trigger an impact assessment of whether accumulated
evidence still applies (see Questions 19, 21, 24, and 25).

· 4. Prespecified response: The monitoring plan should define metrics, precision
requirements, reassessment triggers, and actions, including targeted re-evaluation, rollback or a
safe degraded mode where a provider has retired the prior model version, labeling or use
restrictions, and other measures available under the applicable pathway. Where consistent with
the authorized intended use, limited initial deployment can help manage residual uncertainty
while evidence accrues.
Non-Suitability Criteria & Rare Failure Bounds: Greater reliance on postmarket evidence is not
appropriate where failure may cause immediate or irreversible serious harm before intervention is
possible; where relevant outcomes cannot be captured reliably and promptly; where rare serious
failures would require impractically long exposure to characterize, since even under ideal
assumptions, ruling out a 1-in-1,000 failure rate with 95 percent confidence takes about 3,000 fully
followed uses with no failures observed (the statistical rule of three) [22]; or where safety-relevant
configuration changes cannot be reliably identified, detected, or bounded. Greater caution is also
warranted where users cannot detect incorrect outputs and no timely independent reference
outcome exists.

Question 19: Approaches to Postmarket Performance Evaluation & Cadence

FDA Prompt: Please comment on the potential approaches to postmarket performance evaluation,
including periodic re-benchmarking, sample-based clinician review, and performance degradation
monitoring. What additional approaches should CDRH consider, and how should the cadence and
triggering events for reassessment be determined?

The three approaches CDRH describes are complementary. Periodic re-benchmarking shows
whether the deployed device still meets its defined competencies, but the benchmark may not reflect
changes in the actual deployment population or environment. Sample-based clinician review
examines real inputs and outputs against prospectively defined criteria, but it is limited by sample
size and may detect problems only after exposure. Performance degradation monitoring can detect
changes in inputs, outputs, and device behavior between formal reassessments, but clinically
meaningful reference outcomes vary in availability and timeliness across devices.
CDRH should also consider provenance-based change detection as an upstream layer of
postmarket monitoring. When a device records the source, site, interface, schema, units, and
component versions of its inputs and compares them against its authorized deployment
configuration, a discrete change becomes an explicit reassessment trigger rather than a problem
discovered after performance degrades. For example, a new laboratory interface that reports a
measurement in different units can be identified at onboarding, before it affects quantitative
interpretation, which Appendix A addresses under E.3. Discrete changes are most often missed at
handoffs between vendors and inside proprietary preprocessing, such as a firmware or application
update that changes how an instrument or wearable signal is processed before the device receives
it. Recording the version of each upstream processing step lets the sponsor detect such a change
when it occurs.
Provenance does not replace statistical drift monitoring. Patient mix, clinician documentation,
terminology, and clinical practice can change gradually without any change in recorded source
information. A postmarket program should therefore monitor both discrete configuration changes and
statistical distribution changes, and route both to the same reassessment decision.
Two early warning approaches can supplement these. A version-controlled sentinel suite, with stable
anchor cases and rotating probes, can run on a defined schedule against the same external model
endpoints the deployed device uses, in an environment isolated from patient care. Safety-critical
cases may need repeated runs to account for output variability. Label-free indicators, such as
refusal, escalation, deferral, retrieval failure, and clinician override rates, can prompt investigation,
but they should not by themselves be read as evidence of degraded clinical performance.
Reassessment should combine a prespecified minimum cadence with event-driven triggers. The
minimum cadence should be proportionate to risk and should also reflect use volume, dependence
on externally controlled components, frequency of component change, how readily routine
monitoring detects deterioration, and the time between deterioration and potential harm. Triggers
should include material changes to the model, prompts, retrieval sources, guardrails, orchestration
logic, deployment architecture, data interfaces, or user populations; foundation model changes made
by the provider, whether announced or detected; input or behavioral drift that crosses a prespecified
threshold; and relevant trends in complaints and adverse event reports.
A trigger need not require full re-benchmarking. Each trigger should start an impact assessment that
identifies the competencies the change could affect, followed by targeted or broader reassessment
in proportion to the scope, uncertainty, and safety relevance of the change.
Quality System Integration (QMSR and 21 CFR Part 803):

· 1. Complaint Trending (ISO 13485 Clauses 8.2.2 and 8.4): Manufacturers should apply
statistical trending methods to complaint intake that are appropriate to the data structure,
exposure volume, and risk. When complaints about a specific clinical sub-competency exceed a
prespecified threshold, a reassessment trigger opens.

· 2. 21 CFR Part 803 and GenAI Malfunctions: An erroneous or unexpected GenAI output
should first be evaluated to determine whether it is a malfunction, meaning a failure to meet the
device's performance specifications or otherwise perform as intended. If it is a malfunction,
manufacturer reporting under 21 CFR 803.50(a)(2) [4] applies when the device or a similar
device would be likely to cause or contribute to death or serious injury if the malfunction
recurred. An event that may have caused or contributed to a death or serious injury is evaluated
separately under 21 CFR 803.50(a)(1). FDA clarification would be valuable on how the
malfunction definition applies to stochastic devices when an individual erroneous output falls
within a validated error distribution.

· 3. CAPA Closed-Loop Verification: Targeted re-benchmarking against a sequestered
challenge set serves as the formal Verification of Effectiveness (VoE) under ISO 13485 Clause
8.5.2 before closing the CAPA investigation.

Question 20: Machine-Based Supervisory Agents

FDA Prompt: Please comment on whether the proposed approaches to postmarket monitoring can
be facilitated by machine-based supervisory agents. What considerations, including the evaluation
and reliability of the supervisory agent itself, should CDRH take into account for such an approach?

Machine-based supervisory agents provide critical continuous assurance, but their design and
evaluation should follow these principles:
 · 1. Separation of Supervisory Roles: CDRH should distinguish between (a) Postmarket
Monitoring Agents that detect, log, and trend drift out-of-band; and (b) Runtime Safety Guards
that actively intercept and block outputs. Runtime guards are active device software components
requiring full premarket validation and change control.

· 2. Structural Heterogeneity & Self-Preference Avoidance: Supervising a generative model
with an LLM from the same architectural family should be avoided due to common-mode failure,
circular reasoning, and demonstrated self-preference bias (Panickssery et al., 2024) [23]. The
supervisor should be architecturally distinct (e.g., rule-based clinical boundary engines,
deterministic physiological bounds checkers, or narrow specialized classifiers).

· 3. Out-of-Band Deployment: The supervisor should monitor patient data available to the
device (including physiological telemetry where present) out-of-band, without relying on the
primary model’s internal attention states.

· 4. Enforceable Safety Mechanism: Where warranted by risk, runtime safety guards should
incorporate an independently enforceable mechanism that intercepts outputs violating
prespecified physiological boundaries and forces a safe default state.

· 5. Supervisor Evaluation and Audit: The supervisory agent itself should undergo
independent validation reporting measured sensitivity and specificity against labeled clinical
hazard sets, change control under QMS, and immutable audit logging of every triggered veto.

Question 21: Stakeholder Roles in Postmarket Monitoring

FDA Prompt: What roles might clinicians, healthcare institutions, professional societies, standardssetting bodies, and other stakeholders appropriately play in postmarket monitoring, and how can
these roles be structured without diffusing manufacturer accountability?

Postmarket monitoring can draw on distributed sources of evidence without distributing
accountability for the device. Clinicians and patients can identify unsafe, incorrect, or unexpected
behavior that automated monitoring may miss. Healthcare institutions can contribute local
deployment context, interface and workflow changes, real-world performance data, and cases for
clinician review. Professional societies can define clinically meaningful reference standards and
adjudication frameworks, and they can support registries that aggregate signals across institutions.
Standards development organizations can develop consensus methods for logging, provenance,
drift characterization, and safety event terminology. Foundation model providers can supply change
notifications and safety-relevant model information (see also Question 24).
These contributions supplement manufacturer monitoring. They do not replace it. CDRH states that
manufacturers are responsible for postmarket monitoring of their devices while recognizing roles for
the broader ecosystem (Section VI.B). Part 803 follows the same pattern. Device user facilities
report qualifying deaths and serious injuries under section 803.30, and those reports do not transfer
the manufacturer's own obligations. Under section 803.50, the manufacturer must seek reasonably
available information, investigate each event, and evaluate its cause. Under section 803.17, it must
maintain written procedures for identifying and evaluating potentially reportable events.
Accountability stays clear when the manufacturer's monitoring and complaint handling procedures
specify how stakeholder signals are received, triaged, investigated, trended, and acted on.
Stakeholders supply observations and evidence. The manufacturer decides what that evidence
means for its device and takes the actions required of it.
For GenAI devices, a report is useful only if it can be traced to the device configuration that
produced the behavior. Site staff should not be expected to know the model version or software
build. The person reporting should describe what happened, when, and in what clinical context, with
a transaction or encounter identifier where available. The device should retain enough identifiers for
the manufacturer to link that report to the software build, model or external service identifier,
configuration, retrieval source version, and relevant input and output, to the extent the information is
technically available. This matters most when a third-party foundation model can change outside the
manufacturer's control. Clinical context linked to a report should be limited to what the investigation
needs and protected by appropriate privacy and security controls.
For devices used outside a Part 803 device user facility, including physician offices and home use,
the mandatory user facility reporting pathway may not apply. Manufacturer surveillance, complaint
intake, and voluntary reporting by patients and clinicians through MedWatch [24] become especially
important sources of safety signals for those devices.
Signal Handoff and Quality Governance Controls:

· 1. Signal Handoff Targets: Manufacturers should define internal service-level targets for
stakeholder signal triage, with standardized context capture. For example, high-severity adverse
signals could be transmitted and triaged within 24 to 48 hours. These are internal triage targets
and do not change the reporting timelines in 21 CFR Part 803.

· 2. Two-Way Closure Protocols: Manufacturers should establish closed-loop communication
confirming signal intake, investigation outcome, and corrective actions back to reporting
institutions.

Question 22: Premarket Baseline for Re-Evaluating Modifications & PCCP Scope

FDA Prompt: CDRH envisions that a manufacturer’s premarket competency-based assessment
might serve as a baseline against which post-deployment modifications could be re-evaluated. How
might the extent of re-benchmarking or other evidence be scaled to the nature and expected impact
of a given modification? For example, are there categories of change that might not significantly affect
safety or effectiveness of a GenAI-enabled device, or might be appropriately managed within a
sponsor’s quality management system versus requiring FDA premarket review and authorization
before implementation? Are there categories of changes that might be appropriate for inclusion in a
PCCP?

The premarket competency-based assessment provides an essential versioned authorization
baseline against which post-deployment modifications can be quantitatively re-evaluated.
Baseline identity and change lineage. Re-evaluation is meaningful only if the authorization baseline
is a reproducible, versioned record. The baseline should identify the device configuration evaluated,
including the software build, foundation model or hosted service identifier and version, prompts,
retrieval sources, safety controls, orchestration logic, assessment version, scoring method, and
expected run-to-run variability for stochastic outputs. Each later modification should be linked both to
that authorization baseline and to the immediately preceding configuration. A foundation model
update should be treated as a configuration change even when it stays within the same model
family. Whether it can be managed under the QMS or an authorized PCCP should depend on its
expected impact and the authorized scope of the PCCP, not on model family membership alone.
Changes initiated by a foundation model provider should be classified the same way, using the
detection and response mechanisms described in Question 24.
The extent of re-benchmarking should be risk-scaled, drawing on FDA’s 2017 guidance Deciding
When to Submit a 510(k) for a Software Change to an Existing Device [25] for devices subject to
510(k) change requirements, 21 CFR 814.39 [26] for PMA devices, FDA’s final AI PCCP guidance
[27], and ISO 13485:2016 design change control (clause 7.3.9). It should be applied across defined
device surfaces: foundation model weights, system prompts, retrieval corpora, safety guardrails,
orchestration logic, and user interfaces. CDRH should consider a clear three-tier change-control
matrix:

· 1. Tier 1: Changes Potentially Manageable Under the QMS Without a New Submission,
Following Documented Impact Assessment (QMSR / ISO 13485:2016 Clause 7.3.9): Changes
that do not alter the clinical algorithm, intended use, or risk profile. Examples include: (a)
Hyperparameter adjustments (temperature, top-p) strictly within validated, authorized stability
bounds; (b) Non-substantive UI styling, layout, or accessibility adaptations that do not alter how
clinical information or warnings are displayed; (c) Bug fixes and security patches in non-clinical
orchestration wrappers that preserve validated behavior and performance within prespecified
acceptance criteria. Verification requires standard software regression testing against versioned
verification test suites under internal design controls.

· 2. Tier 2: Authorized Predetermined Change Control Plan (PCCP under Section 515C):
Bounded, prespecified modifications that improve model accuracy or update knowledge bases
within the authorized intended use. Examples include: (a) Retraining or fine-tuning foundation
weights on newly acquired clinical data within predefined clinical domains and demographic
bounds; (b) Updating retrieval-augmented generation (RAG) corpora with newly published peerreviewed guidelines from recognized professional bodies; (c) Migrating to an updated foundation
model version within the authorized scope of the PCCP, supported by full re-benchmarking,
because membership in the same model family does not establish behavioral continuity.
Implementation requires formal re-benchmarking against the versioned authorization baseline
(Appendix A), meeting prespecified acceptance criteria, and logging versioned release manifests
with automated rollback capabilities.

· 3. Tier 3: Premarket Review Required (New 510(k) or PMA Supplement): Modifications that
alter the safety-risk profile or expand clinical functionality beyond the cleared baseline. Examples
include: (a) Changes to intended use or indications for use (e.g., expanding from adult to
pediatric populations, or migrating from informational support to autonomous action-directing
 guidance); (b) Relaxing safety guardrails or clinical boundary vetoes; (c) Architectural overhauls
introducing new tool-calling agentic capabilities. Prompt and guardrail modifications can be
safety-critical for GenAI devices and should be classified by documented impact assessment
rather than by category. Relaxing a guardrail should not be treated as QMS-only or covered
under a basic PCCP. Tightening a guardrail can also change behavior, because over-refusal is a
failure mode under Element S.2, so it should also undergo impact assessment.

Question 23: Adapting PCCP Concepts for Unprespecified Modifications

FDA Prompt: How might PCCP concepts or other change-control approaches be adapted for GenAIenabled devices when the nature or scope of future modifications cannot be fully prespecified?

Under Section 515C of the FD&C Act and FDA’s final AI PCCP guidance, a PCCP cannot serve as
an open-ended "blank check" for arbitrary postmarket modification. When the exact nature of future
weights or training runs cannot be fully prespecified at authorization, PCCP concepts can be
adapted by prespecifying bounded classes of change paired with authorized, prespecified
verification and acceptance methods:

· 1. Prespecified Modification Envelope: Rather than attempting to prespecify future model
weights, the PCCP establishes the allowable parameter space: (a) Permitted foundation model
versions or families, with lineage disclosure for each new version, and API parameter bounds;
(b) Approved data ingestion classes (e.g., vetted EHR registries and prespecified authoritative
guideline sources, with source, version, date, and update provenance, excluding unvalidated
web corpora); (c) The fixed operational envelope (input modalities, supported languages, and
clinical task boundaries). Any modification exceeding these defined boundaries falls outside the
PCCP.

· 2. Prespecified Modification Protocol (Verification and Acceptance Methods): The PCCP
should prespecify the verification and acceptance methods that gate deployment: (a) Execution
against versioned, sequestered re-benchmarking suites with protected anchors and refreshed
items (Appendix A; see Question 16); (b) Quantitative acceptance thresholds enforcing noninferiority margins on core clinical competencies, with prespecified, clinically justified subgroup
acceptance criteria and appropriate uncertainty estimates; (c) Automated rollback triggers and
safe degraded modes (e.g., rule-based fallback) if sentinel queries or post-deployment
monitoring detect performance degradation; (d) Formal traceability manifests recording
checksums of weights, prompts, retrieval snapshots, and validation test reports under ISO
13485:2016 Clause 7.3.9 prior to commercial release.
Change chain and cumulative impact. FDA's final PCCP guidance for AI-enabled device software
functions [27] already recommends verifying a modified device against both the original device and
the last modified version, and assessing the cumulative impact of all modifications. GenAI devices
add a practical problem: the original authorized configuration may no longer be executable once a
provider retires the underlying model version. The authorization baseline should therefore be
preserved as a record, including the assessment items, outputs, scores, and run-to-run variability, so
that comparison with the original device remains possible without re-running it. Release records
should link each implemented modification to the original authorized device and to every
intermediate version, identifying the model or service version, device configuration, data and
retrieval source versions, verification evidence, and impact decision for each release. The PCCP's
Impact Assessment should address how each class of change, alone and in combination with
others, could affect benefits and risks. Where a hosted foundation model does not expose its
weights, the provider's model or service identifier and snapshot version should be recorded in place
of a weights checksum.

Question 24: Third-Party Foundation Model Changes & QMSR Change Control

FDA Prompt: For devices built on third-party foundation models, changes to the underlying model
may be initiated by the third-party model developer rather than the device manufacturer. How can a
manufacturer detect, evaluate, and respond to such changes in a timely manner? What
mechanisms—for example, contractual, technical, or through a PCCP—could provide reasonable
assurance that third-party developer-initiated changes do not compromise the safety or effectiveness
of the device?

Third-party foundation models should be governed under QMSR / ISO 13485:2016 Design Change
Control (Clause 7.3.9) and Purchasing Controls (Clauses 7.4.1–7.4.3):

· 1. Contractual Supplier Controls (ISO 13485 Clause 7.4.2): Manufacturers should establish
supplier agreements requiring formal advance notification of model deprecation, API shifts, and
zero-data-retention (ZDR) terms to prevent benchmark test item contamination.

· 2. Automated Sentinel Regression Verification (ISO 13485 Clause 7.4.3): Manufacturers
should deploy automated sentinel regression suites executing on a defined schedule (for
example, daily) against live vendor endpoints using protected anchor cases and rotating probes.
Acting as incoming product verification, unexpected behavioral drift triggers nonconformance
investigations and safe maintenance tripwires.

· 3. Snapshot Pinning and Migration Planning: Devices should bind to immutable snapshot
endpoints, maintaining pre-validated migration paths and re-benchmarking against frozen test
suites before vendor snapshot retirement dates. Where pinning is unavailable, continuous
sentinel monitoring and bounded runtime guardrails become primary safeguards.

Section VII: Other Topics

Question 25: Foundation Model Device Master Files (MAFs)
 FDA Prompt: Would voluntary Foundation Model MAFs be practical for foundation model developers
to provide and to keep current, and what content might they include? Given that participation would
be voluntary and that model developers may have limited incentive to disclose safety-relevant
information, what would make such a program sufficiently useful for premarket review? Are there
alternative mechanisms CDRH should consider for obtaining information about underlying foundation
models?

Voluntary Foundation Model Device Master Files [28] can be practical and useful if they serve as
versioned evidence records rather than static model descriptions. The unit FDA needs to reference
is not a commercial model name. It is a dated configuration that identifies the model snapshot and
the provider-controlled components that can change device behavior, such as safety or refusal
controls, system-level instructions, and serving configuration. The MAF should state which
components that identifier covers and which can change without a new identifier, since changes
outside the identifier are a main source of the unannounced behavior changes described in Question
24. A developer could hold one MAF per model family, with a dated amendment for each
configuration offered for use in regulated devices and for each change relevant to safety or
effectiveness. The MAF should also include update notification commitments and planned retirement
dates.
For training lineage, MAF content should include training data provenance by category; data
windows or cutoff dates for pretraining and for each material post-training stage, such as fine-tuning;
exclusion, filtering, and deduplication methods; and the benchmarks screened for contamination, the
screening methods used, and known findings (see Question 10). A cutoff date alone does not
establish freedom from contamination, because exposure can occur in later training stages.
Consistent with the content the paper suggests in footnote 24 [1], a MAF could also describe known
limitations and failure modes, healthcare-relevant evaluation results including clinically meaningful
subgroups, and the audit logging available to device manufacturers.
The main incentive for developers is less repeated disclosure. FDA can review confidential
information once, and many device sponsors can reference it with the holder's authorization. Each
sponsor remains responsible for showing the safety and effectiveness of its own device.
Where no MAF is available, a supplier agreement alone does not close the information gap.
Manufacturers should apply risk-based supplier controls, including written change-notification
requirements where applicable under ISO 13485 clause 7.4.2. They should also obtain available
model documentation and verify the deployed configuration through their own testing (see Questions
19 and 24). Large model providers often offer standard service terms rather than negotiated supplier
agreements, so the manufacturer's own verification will often carry most of the assurance.
Manufacturers that host open-weight models can pin the model version themselves, which removes
provider-initiated changes, although training lineage remains relevant. Qualified independent thirdparty assessments could add assurance if FDA establishes qualification and independence criteria
(see Question 16).
The configuration identifier recorded with each device output should map to the corresponding MAF
amendment (see Questions 21 and 24). Postmarket observations and provider-initiated changes can
then be linked to the exact evidence FDA reviewed.
Question 26: Agentic GenAI Devices, Tool Calling & Autonomous Action
Boundaries

FDA Prompt: Are there additional considerations that inform the premarket and postmarket
evaluation of agentic GenAI-enabled devices, beyond those applicable to non-agentic GenAI-enabled
devices? How should the elevated risk associated with autonomous multi-step action, tool use, and
reduced opportunity for human review be reflected in acceptance criteria and oversight?

Agentic devices executing autonomous tool use introduce elevated clinical risk requiring rigorous
architectural sandboxing and verification boundaries:

· 1. Strict Tool Sandboxing & Context Provenance: All tool-calling execution environments
(database writes, EHR updates, device actuator commands) should be isolated with zero
arbitrary execution privileges. The agent should log the source, version, and trust level of every
context item and tool output to prevent prompt-injection attacks through untrusted retrieved data.

· 2. Transactional Propose-Then-Authorize Protocol: For any irreversible clinical action,
Phase 1 logs the proposed action and rationale; Phase 2 requires an authenticated clinician
digital signature before network or actuator execution. Expired authorization gates should
deterministically abort the transaction.

· 3. Tighter Acceptance Criteria for Agentic Execution: Acceptance criteria for autonomous
multi-step execution should enforce substantially tighter statistical error thresholds than
informational functions, reporting task completion rates, hallucinated parameter rates, and tool
invocation accuracy separately.

References
[1] U.S. Food and Drug Administration, Center for Devices and Radiological Health. Considerations for
the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for
Feedback. August 2026. Docket No. FDA-2026-N-7874.
https://www.fda.gov/media/194242/download
[2] U.S. Food and Drug Administration. Medical Devices; Quality System Regulation Amendments.
Final rule. 89 Fed. Reg. 7496 (February 2, 2024), effective February 2, 2026 (21 CFR Part 820).
[3] ISO 13485:2016. Medical devices: Quality management systems: Requirements for regulatory
purposes. Geneva: International Organization for Standardization; 2016.
[4] 21 CFR Part 803, Medical Device Reporting (including 803.3, 803.17, 803.30, and 803.50).
[5] Federal Food, Drug, and Cosmetic Act section 515C, 21 U.S.C. 360e-4, Predetermined change
control plans for devices.
[6] Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of New Readability Formulas
(Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted
Personnel. Research Branch Report 8-75. Millington, TN: Naval Technical Training Command;
1975.
[7] IEC 62366-1:2015+AMD1:2020. Medical devices, Part 1: Application of usability engineering to
medical devices. Geneva: International Electrotechnical Commission.
[8] AAMI TIR34971:2023. Application of ISO 14971 to machine learning in artificial intelligence: Guide.
Arlington, VA: Association for the Advancement of Medical Instrumentation; 2023.
[9] ISO 14971:2019. Medical devices: Application of risk management to medical devices. Geneva:
International Organization for Standardization; 2019.
[10] IEC 62304:2006+AMD1:2015. Medical device software: Software life cycle processes. Geneva:
International Electrotechnical Commission.
[11] IEC 82304-1:2016. Health software, Part 1: General requirements for product safety. Geneva:
International Electrotechnical Commission; 2016.
[12] 21 CFR Part 56, Institutional Review Boards.
[13] Ibrahim JG, Chen MH. Power prior distributions for regression models. Statistical Science.
2000;15(1):46-60.
[14] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics.
1977;33(1):159-174.
[15] Krippendorff K. Content Analysis: An Introduction to Its Methodology. 4th ed. Thousand Oaks, CA:
SAGE; 2018.
[16] Dwork C, Feldman V, Hardt M, Pitassi T, Reingold O, Roth A. The reusable holdout: Preserving
validity in adaptive data analysis. Science. 2015;349(6248):636-638.
[17] National Institute of Standards and Technology. Announcing NIST's Artificial Intelligence
Technology Evaluation (AITE). July 27, 2026. https://www.nist.gov/newsevents/news/2026/07/announcing-nists-artificial-intelligence-technology-evaluation-aite
[18] U.S. Food and Drug Administration. Accreditation Scheme for Conformity Assessment (ASCA)
Program. Center for Devices and Radiological Health.
[19] U.S. Food and Drug Administration. Medical Device Development Tools (MDDT) Program. Center
for Devices and Radiological Health.
[20] ISO/IEC 17025:2017. General requirements for the competence of testing and calibration
laboratories. Geneva: International Organization for Standardization; 2017.
[21] U.S. Food and Drug Administration. Consideration of Uncertainty in Making Benefit-Risk
Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian
Device Exemptions. Guidance for Industry and Food and Drug Administration Staff. August 30,
2019. https://www.fda.gov/media/115672/download
[22] Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero
numerators. JAMA. 1983;249(13):1743-1745.
[23] Panickssery A, Bowman SR, Feng S. LLM evaluators recognize and favor their own generations.
Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2404.13076.
[24] U.S. Food and Drug Administration. MedWatch: The FDA Safety Information and Adverse Event
Reporting Program.
[25] U.S. Food and Drug Administration. Deciding When to Submit a 510(k) for a Software Change to an
Existing Device. Guidance for Industry and Food and Drug Administration Staff. October 25, 2017.
https://www.fda.gov/media/99785/download
[26] 21 CFR 814.39, PMA supplements.
[27] U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined
Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Guidance for
Industry and Food and Drug Administration Staff. August 18, 2025 (originally issued December 4,
2024). https://www.fda.gov/media/166704/download
[28] U.S. Food and Drug Administration. Device Master Files. https://www.fda.gov/medicaldevices/premarket-submissions-selecting-and-preparing-correct-submission/device-master-files