Sia Partners (David Zar, Yiannis Antoniou)

See Attached

FDA-2026-N-7874 1 / 33

Sia Partners Comments on
FDA-2026-N-7874
Sia Partners welcomes CDRH’s discussion of generative AI-enabled medical devices as
these systems enter clinical workflows. Variable outputs, multi-turn interactions, and
changing models create evaluation challenges beyond conventional software. Evidence
should address the deployed device, intended users, clinical setting, and foreseeable
consequences of failure.

Sia is a global management consulting group with more than 3,000 consultants across 48
offices in 19 countries. We employ an AI-first consulting model and employ more than 500
FDEs deploying and operating AI at scale in 6 AI Hubs across North America, EMEIA and
APAC, supported by broader expertise across healthcare, financial services, strategy, digital
transformation, IT, supply chain, procurement, operations, pricing, customer experience,
and organizational effectiveness.

Sia Partners advises medical device manufacturers on regulatory strategy, quality systems,
and AI. Our selected responses propose practical, risk-proportionate approaches to
premarket evidence, clinical confirmation, change management, and postmarket oversight.

Principal recommendations

/ Assess realistic clinical use. Test conversational trajectories, clinical scope, escalation,
human factors and clinician–AI team performance.
/ Align evidence with risk. Link three presumptive evidence tiers to clinical function,
potential harm, confirmation and monitoring, allowing justified alternatives.
/ Make performance claims statistically defensible. Prespecify populations, failure
definitions, sampling units, and uncertainty bounds. Base clinical claims on
representative real cases; use synthetic cases mainly for stress testing.
/ Monitor the deployed configuration. Control models, prompts, retrieval, guardrails and
tools; pair risk-based re-benchmarking with automated screening, clinical adjudication
and production checks.
/ Clarify regulatory responsibilities and burden. Clarify supplier evidence, cybersecurity,
reporting and rollbacks while keeping implementation expectations proportionate.
This response is guided by the principle that proposed methods require applicationspecific validation. Note that illustrative risk ratings, where used, are qualitative premitigation judgments—not measured probabilities. We try to address feasibility based on
 FDA-2026-N-7874 2 / 33

implementation burden, separate from patient-risk acceptability. Evidence tiers and
monitoring schedules are proposed starting points.

Prepared by David Zar (david.zar@sia-partners.com), a Principal Consultant with more than
25 years of healthcare industry experience, and by Yiannis Antoniou (yiannis.antoniou@siapartners.com), an Engagement Director with deep expertise in AI and Data for more than 20
years.

Finally, we focus on questions where we feel we can offer informed experience rather than
simply agreeing. We hope this encourages additional exploration and welcome continued
discussion.

/ Question 5: Conversational trajectories and human factors

FDA question: For multi-turn conversational GenAI-enabled devices that may migrate from
providing “non-directive” information to “action-directing” information over the course of
an exchange, how should risk be assessed across realistic conversational trajectories? How
could the intended use of such a device be characterized when its behavior is emergent
across a conversation?

Summary: Assess risk across realistic conversations as well as individual turns. State the
clinical scope, intended users and setting, and the highest level of advice or action the
device is permitted to provide. Risk placement should reflect the most consequential
reasonably foreseeable trajectory within that scope, including relevant misuse scenarios. 1

Usability testing and validation

IEC 62366-1 provides a usability engineering process that can be applied to generated
interfaces. A sponsor should demonstrate the absence of critical use errors or design a
summative test to show they are sufficiently mitigated; for example, whether a user
correctly follows an instruction to seek emergency care. Validation should involve
representative intended users, including people with low health literacy where relevant, in
realistic conversations. Include exchanges that move from general information to advice
about the user’s own situation. 2

The benchmark scenario library can supply cases for usability testing. Simulated users can
broaden coverage, but do not replace human participants in summative usability validation.
Test incomplete or conflicting histories, repeated reassurance-seeking, changing
symptoms, user pressure, escalation and handoff failures. Assess comprehension, overreliance, and recovery from errors. The use-related risk analysis should address
benchmarking element E.4 and automation bias; conversation-level clinical validation is also
needed for claims that depend on interaction. 2
 FDA-2026-N-7874 3 / 33

Intended use and device specifications should set enforceable limits on clinical scope and
directiveness. Scope maintenance testing (S.2) should assess whether the device stays
within those limits over long conversations. An output that exceeds a specified limit fails
the device’s specification. An opening prompt or disclaimer is insufficient evidence that the
device stays within scope.

For users without clinical expertise, fixed escalation rules for red-flag symptoms and design
controls that enforce the permitted level of advice can reduce reliance on user knowledge.
ISO 14971 clause 7.1 prioritizes inherent safety by design, then protective measures, then
information for safety and training. Where HCP-facing devices require training, evaluate its
effectiveness and the foreseeable limits of human oversight. 3

/ Question 8: Mapping the risk framework to required evidence
FDA question: How might the two-axis risk framework described in Section IV be
considered within the competency-based approach described above to help determine the
level of evidence needed for a premarket submission?

Summary: The two-axis framework should set a presumptive evidence tier for each device
function, published by CDRH as a non-binding matrix, so the evidentiary bar does not rest
on sponsor justification alone. The IMDRF SaMD clinical evaluation framework provides a
useful precedent for relating evidence to risk but does not validate this proposed matrix.
FDA’s computational modeling guidance similarly offers a risk-based analogy within its
distinct scope. Aligning terms where appropriate would help sponsors and reviewers. 4,5

A presumptive mapping from risk position to evidence
We propose three evidence tiers. Each tier sets the extent of clinical confirmation and
scales the benchmarking: the difficulty and breadth of test conditions, subgroup coverage,
the stringency of acceptance bounds, and the intensity of postmarket monitoring.

/ Tier 1: all applicable benchmarking elements, always including safety elements, using
inputs representative of real use, plus proportionate postmarket monitoring. This is a
proposed evidentiary starting point, not an exemption from otherwise applicable device
evidence requirements.
/ Tier 2 adds limited clinical confirmation: retrospective evaluation on real inputs or
clinician adjudication of real cases, with shadow deployment where feasible.
/ Tier 3 requires clinical confirmation capable of assessing consequential workflow
effects: shadow evaluation where informative, followed by prospective evaluation as
needed, including human-AI team performance for supervised functions.
Randomization should be considered where feasible and ethical, especially for
autonomous actions with severe consequences.
 FDA-2026-N-7874 4 / 33

Consequence Informational: non- Informational: action- Action-taking: HCP- Action-taking: fully
directive directing supervised autonomous
Severe Tier 2: Quantitative Tier 3: Patient-facing Tier 3: Drafting Tier 3: Initiating a timeimaging or lab-derived advice on whether anticoagulant dosing for critical treatment
measures the user cannot symptoms need clinician sign-off protocol
verify emergency care
Moderate Tier 1: Summarizing a Tier 2: Advising primary Tier 2: Drafting a chronic- Tier 3: Refilling a stable
patient's chart for a versus urgent care for a disease medication maintenance medication
clinician non-emergent symptom titration for approval within protocol
Limited Tier 1: General education Tier 1: Self-care advice for Tier 1: Pre-filling a routine Tier 2: Routing nonon a medication's a minor sprain screening order for sign- urgent patient messages
common side effects off to the right care team
Activity increases left to right; consequence of relying on an incorrect output increases bottom to top.

Examples are illustrative. Placement applies to the device function as deployed, including its
built-in safeguards.

Refining placement within the framework
While we support the two-axis framework as the organizing structure, four refinements
would make placement more consistent and less dependent on how a sponsor describes its
device:
/
Keep severity and detectability separate. For this proposed mapping, document
severity, probability of harm, and the user’s ability to detect and correct an error
separately, consistent with ISO 14971. Treat detectability as an explicit modifier of
residual risk rather than equating it with severity. This would make placement traceable
without implying that FDA’s conceptual axes are a formal classification.
/ Place by realistic trajectory, not nominal use. A conversational device can drift from
informational to action-directing within a single exchange. Its placement should reflect
the highest cell it reaches in realistic multi-turn testing. Demonstrated scope adherence
is what justifies a lower placement. 1
/ Assess both directions of escalation error. Under- and over-escalation carry different
harms. Evidence should address the severity and frequency of both, with the more
consequential failure modes driving the required assurance.
/ Let verified safeguards inform placement. Downward departures should require
safeguards enforced by design, evidence that they reduce the relevant risk, and
documented justification. No automatic tier reduction follows from a safeguard.
Sponsors whose functions sit on a boundary can discuss placement through a PreSubmission.

Implementation burden

Clinical confirmation can substantially drive submission costs, particularly for higher-risk
functions. Tier 1 should remain proportionate to risk under least-burdensome principles
while providing adequate evidence. The matrix should avoid incentives to describe
functions as less severe than they are and should be calibrated with realistic examples.
 FDA-2026-N-7874 5 / 33

Benchmarking 2,000 scenarios with three independent reviews of six minutes each requires
600 clinician hours before disagreement resolution, case preparation, or analysis. Costs
depend on specialty, contracting, and review arrangements. Shared scenario libraries could
reduce case-development work; validated model-assisted labeling may reduce adjudication
work where its assumptions hold, as discussed in Question 12. CDRH could publish a burden
estimate before setting expectations for repeated benchmarking.

/ Question 11: Selecting a clinical confirmation approach
FDA question: CDRH is considering that clinical confirmation for a GenAI-enabled device
might not require a prospective clinical study in every case, and has described above a
range of approaches of increasing rigor and patient exposure. How might a sponsor select
and justify a confirmation approach tailored to a device's intended use and proportionate to
the device's risk profile? Are there device types or risk profiles for which one or more of
these approaches would be insufficient or inappropriate? How might the anticipated
distribution of real-world inputs be taken into consideration? Are there other methods of
clinical confirmation that might help inform the evaluation of GenAI-enabled devices?

Summary: The evidence tier in Question 8 should set a presumptive confirmation approach,
with the sponsor selecting a design that observes the critical failure modes and justifying
departures. Real-world and non-US data should be assessed for fitness and transportability
to the intended US use. Reweighting is appropriate only when its assumptions and data
support it.

Tying the confirmation ladder to risk
Evidence tier Presumptive minimum confirmation Move up a rung when
Tier 3 Prospective confirmation appropriate to Strengthen design or coverage
consequential workflow effects; shadow components when material uncertainty remains;
where informative; randomization where feasible and Tier 3 is already the highest.
ethical
Tier 2 Retrospective evaluation on real inputs or clinician The device's output changes
adjudication of real cases; shadow deployment where subsequent care or user behavior
feasible; standardized patients for conversational
devices
Tier 1 Representative real-input benchmarking may suffice if Benchmark inputs are not
justified; no automatic exemption from applicable representative of real use.
evidence.

Within a tier, select an approach that can observe critical failure modes. Each approach has
limits:

/ Retrospective evaluation can support conversational and human-AI functions but
cannot, by itself, establish device-induced changes in behavior or care. Reconstruct
 FDA-2026-N-7874 6 / 33

only information available at the decision time, since later results leaking into inputs can
inflate performance.
/ Shadow evaluation can replay observed conversational histories or assess
noninteractive components but cannot reproduce the full interaction when device
outputs would change the next user response. Patient-facing conversational claims
therefore need suitable interactive evidence.
/ Standardized patients can test defined interactions, but may not represent all ages,
cognitive abilities, literacy levels, languages, or real distress. Validate their suitability
and include representative intended users in human-factors evaluation where needed.
/ Clinician adjudication is insufficient where correctness depends on later outcomes,
such as a diagnosis confirmed by follow-up.

Accounting for the real-world input distribution
Define the expected US distribution of performance-relevant factors, such as acuity, age,
language, health literacy, setting, and practice patterns. Report raw results and, where
justified by measured covariates, overlap, and transportability assumptions, reweighted
estimates. Strata with sparse or no supporting data remain evidence gaps.

Transportability of real-world and non-US data
Assess real-world and non-US data for relevance to the intended US use and explain the
assumptions needed to apply the results. A transportability assessment should:

/ Characterize differences from US settings in population, care pathways, standards of
care, and language.
/ Assess coverage of relevant US subgroups, settings, care pathways, and clinically
important effect modifiers; identify gaps rather than assuming that demographic
overlap alone establishes transportability.
/ Adjust by reweighting where adequate overlap and measured covariates support the
required assumptions. Dahabreh et al. describe methods for extending trial results to a
target population; however, these methods do not automatically establish
transportability for every device study. 6
/ Obtain appropriate US bridging evidence where material transportability gaps remain.
Translated inputs alone do not establish performance for native language interactions.
Apply 21 CFR 812.28 where relevant and FDA’s December 2025 device real-world evidence
guidance. ICH E5 offers a pharmaceutical bridging analogy, not a device-specific
acceptance rule. 7,8,9
 FDA-2026-N-7874 7 / 33

Other confirmation methods to consider
/ Summative usability testing under IEC 62366-1 for patient-facing comprehension and
use errors. 2
/ Staged or stepped-wedge studies across sites, where scientifically justified and
consistent with applicable investigation, ethics, and authorization requirements.
/ Target-trial emulation in real-world data when the question, recorded variables, and
causal assumptions support it. Address confounding, selection, time zero, and missing
outcomes; the design does not remove these limitations.
A minimum by tier gives predictability, but a rigid rule could require an infeasible study
design. We suggest a presumptive minimum with alternatives agreed through a PreSubmission.

/ Question 12: Statistically meaningful performance measurement
FDA question: How should sponsors achieve statistically meaningful performance
measurement for GenAI-enabled devices? Where synthetically generated inputs
supplement real patient data, how should sponsors account for differences between the
synthetic and real-world distributions when estimating performance, and under what
conditions, if any, is it appropriate to combine benchmarking evidence and clinical
confirmation evidence to support a single performance estimate?

Summary: Measure performance at the clinical-case level and report it as an interval, using
the lower bound for desirable performance and the upper bound for harmful failure rates.
Report benchmarking and clinical confirmation separately by default. We propose excluding
synthetic benchmark borrowing from safety-critical acceptance decisions and permitting it
for other elements only under strict, prespecified conditions.

Making performance estimates statistically meaningful
A GenAI-enabled device can produce different outputs for the same input, so evaluation
should sample both cases and repeat runs in proportion to the expected variability and risk.
More independent cases improve precision across the intended population. Repeat runs
characterize instability and can improve precision for average stochastic performance, but
do not create independent patients. We recommend that sponsors:

/ Define, before testing, what each benchmark element measures: the input population,
the case or patient as the unit of analysis, the locked rubric, and how refusals and
deferrals are counted. The ICH E9(R1) estimand framework offers a methodological
analogy; it is pharmaceutical guidance, not a device-specific requirement. 10
 FDA-2026-N-7874 8 / 33

/ Compute confidence intervals that account for dependence within patients, cases, and
conversational trajectories. Set acceptance criteria for the lower bound of sensitivity or
pass rates and the upper bound of harmful failure rates.
/ Treat automated or LLM graders as measurement instruments. Prediction-powered
inference combines model-generated labels on real inputs with a human-labeled
sample. The sample must be independently adjudicated and drawn either at random or
with known selection probabilities, which one uses to weight the estimate and correct
average prediction error. Valid inference requires representative sampling, a suitable
estimand, and a fixed judge or valid sample splitting. It does not automatically correct
bias from synthetic-input distributions or provide adequate coverage for rare harms. 11

Controlling multiplicity across benchmark elements
The adjustment should follow the decision. Requiring every safety element to pass an
individually valid test is an intersection-union decision. It does not, by itself, require an
alpha adjustment, though each element must be adequately powered. Marginal confidence
intervals should be labeled as such; simultaneous intervals are needed when making a joint
coverage claim. Comparative and subgroup claims should follow a prespecified multiplicity
strategy, safety first. FDA’s Multiple Endpoints in Clinical Trials guidance offers a useful
methodological analogy, not a device-specific requirement. The larger practical risk is
hidden multiplicity: repeatedly tuning prompts and rubrics against the same test set.
Elements, rubrics, and thresholds should be locked before final evaluation. 12

When benchmarking and clinical confirmation evidence can be combined
Benchmarking shows capability across constructed, often enriched inputs; clinical
confirmation shows performance in the intended population and workflow. A naive pooled
estimate may describe neither. We support separate reporting by default, with confirmation
evidence governing acceptance, and a combined estimate only when:

/ Both measure the same estimand with the same rubric and device configuration, and the
sponsor justifies transportability to the intended population and workflow.
/ Representative real cases provide coverage of clinically relevant populations, settings,
and failure modes. Synthetic cases may add information but should not substitute for
missing real-data coverage.
/ The sponsor prespecifies clinically justified equivalence margins and evaluates source
agreement with adequate precision. Failure to detect a difference is insufficient;
equivalence alone also does not establish transportability.
/ Borrowing uses a prespecified, justified model, discounts incompatible evidence, and is
capped so inexpensive benchmark data cannot outweigh confirmation data. FDA’s
 FDA-2026-N-7874 9 / 33

Bayesian device guidance provides principles for assessing exchangeability and
planning borrowing; sources should not be naively pooled. 13
/ As a conservative policy recommendation, no synthetic benchmark borrowing is used to
establish safety-critical acceptance. For other elements, the confirmation-only result is
shown alongside the combined result. This restriction does not prohibit validated
grading or inference methods applied to representative real cases.

Accounting for synthetic versus real-world differences

There are two distinct gaps: case-mix and fidelity. Sponsors may address a case-mix gap by
reweighting to the intended-use distribution when relevant variables are measured,
adequate overlap exists, and conditional transportability is justified. A fidelity gap arises
when synthetic cases do not reproduce clinically important features of real cases. Sponsors
should assess it on independent, adjudicated real cases, report residual uncertainty, and
avoid unsupported corrections. Reweighting or prediction-powered inference alone does
not repair this gap.

Reporting uncertainty in performance estimates
/ Pass rates should be reported with a 95% confidence interval and the bound used for
acceptance, using methods appropriate to the sample size, clustering, and event rarity.
Avoid unverified normal approximations for small evaluation sets. Bowyer et al. discuss
this problem and Bayesian alternatives; credible intervals should be identified as such. 14
/ Report zero observed failures with the upper bound they support, not as evidence of a
zero underlying failure rate. For n independent, representative Bernoulli trials, the exact
one-sided 95% upper bound is 1 − 0.05^(1/n), approximately 3/n. Clustering or selective
sampling changes the calculation. 15
/ Stratify results by element, risk level, subgroup, and real versus synthetic source. Report
run-to-run variability separately. Paired comparisons and equivalence margins should
reflect clinical importance; observed stochastic noise is not an acceptance threshold.
These recommendations can use existing frameworks. Acceptance criteria belong in the
ISO 14971 risk management plan; assess unrepresentative data and grader bias using AAMI
TIR34971:2023; place evaluation settings under IEC 62304 configuration control; use IEEE
2801-2022 for dataset quality management; and use IEEE 2802-2022 for performance and
safety evaluation terminology. CDRH should determine how/where these standards apply
and identify gaps for further consensus work. 3,16,17,18,19
 FDA-2026-N-7874 10 / 33

/ Question 13: Synthetic data suitability and safeguards
FDA question: For which clinical domains, device functions, or subpopulations is synthetic
data particularly well-suited, or particularly inadequate, as a supplement to real-world
evidence? What safeguards would mitigate the risk that synthetic data generated by models
of the same class as the device under evaluation reproduces the very performance gaps the
evaluation is intended to detect, particularly for underrepresented subgroups?

Summary: Synthetic data are best used to find failures, with representative real data needed
to substantiate performance claims. Synthetic data are well suited to stress-testing defined
hazards and to controlled variations of real cases. They are less dependable for estimating
real-population performance, especially in underrepresented subgroups. When used,
employ four safeguards: measured shared-error risk; a validated synthetic-to-real
comparison; adequate power for subgroup claims; and real-data confirmation of claimsupporting results.

Where synthetic data fits, and where it does not
/ Well suited: scope and boundary testing; adversarial and prompt-injection testing;
possibly rare but well-specified hazards such as dosing calculations, unit errors,
contraindications, and escalation triggers; and controlled perturbations of real cases,
changing one attribute (age, dialect, literacy, comorbidity) to test whether the output
changes when it should and holds steady when it should not.
/ Poorly suited without strong validation: estimating real-world performance or case
mix; representing patient communication and behavior; and filling evidence gaps in
subgroups whose presentations are thinly documented, including some pediatric,
pregnant, rare-disease, and low-resource language populations. Suitability depends on
the generator, task, and available validation evidence.
The subgroups where real data are scarcest may also be those a generator represents least
well. Shumailov et al. show loss of distributional tails under recursive training on generated
data; that finding does not establish that every synthetic-data generator loses rare
presentations. For underrepresented subgroups, synthetic data can reveal a gap but cannot,
on its own, establish its absence. 20

Safeguards against reproducing the gaps under evaluation
/ Measured shared-error risk. As a default safeguard, use a generator from a different
model family where feasible and include a non-LLM source, such as clinician-authored
cases, rule-based simulations, or perturbed real cases. A different family does not prove
statistical independence: errors can be correlated across providers and architectures.
Test shared blind spots on an independent real reference set and prespecify when error
overlap makes the generator unsuitable for a particular claim or subgroup. 21
 FDA-2026-N-7874 11 / 33

/ A validated synthetic-to-real comparison. One option is a classifier two-sample test,
with separate training and held-out real and synthetic test cases. An AUC near 0.5
indicates failure to distinguish the tested distributions at the available power, not proof
of clinical equivalence. Report uncertainty and subgroup results alongside clinician
assessment of clinically important features. Independently validate the metric,
reference labels, and test data to avoid circular evaluation. Qualification through FDA’s
voluntary Medical Device Development Tool (MDDT) program may be useful for a
reusable tool; it should not be a requirement for every high-risk device. 22
/ Minimum detectable difference for subgroup claims. State the smallest clinically
important gap the evaluation is designed to detect and the power and precision
supported by real cases in each subgroup. If the evaluation cannot resolve the
prespecified margin, report insufficient evidence rather than comparable performance.
/ Real-data confirmation. Confirm passing results that support a claim on real cases.
Investigate and fix failures found with synthetic data.

These safeguards fit existing risk-management and dataset-quality frameworks. FDA’s
computational modeling credibility guidance directly covers first-principles modeling and
relevant first-principles components of hybrid models, not standalone ML or AI generators.
Its context-of-use and credibility principles are useful here by analogy. AAMI TIR34971:2023
and IEEE 2801-2022 provide more directly relevant risk and dataset frameworks. 4,16,18

/ Question 14: Comparators and acceptance criteria for openended outputs
FDA question: For open-ended device outputs, how should performance comparators and
acceptance criteria be selected? When a panel of qualified clinicians serves as the
comparator, how should the applicable standard (for example, the standard of care versus
the performance of a median clinician in practice) be defined and justified? Should
generalist or specialist physicians be used as a performance standard? When should
human-AI team performance, rather than the device operating alone, serve as the basis for
evaluation?

Summary: A clinician panel is a measurement instrument and should be built and
characterized like one: sized to a target reliability, with agreement reported for every
benchmark element. The comparator standard should follow the device's intended use and
evidence tier, with one fixed rule: safety elements are always judged against the standard of
care, never against typical clinician error rates. Human-AI team performance should be the
 FDA-2026-N-7874 12 / 33

primary basis whenever the intended use requires clinician review, with device-alone
performance always reported alongside.

Constructing the reference panel
/ Use separate adjudicator and clinician comparator groups. Adjudicators who define
correct answers should be specialists in the clinical domain, independent of the sponsor
and model developer, and blinded to whether an output came from the device or a
clinician. The clinicians the device is compared against are a separate group, matched to
the device's intended user. 23
/ Size the panel to a reliability target. Under the Spearman-Brown assumption, an
individual-rater reliability of 0.5 yields a four-rater average reliability of 0.8. Raw
pairwise agreement of 0.5 is not the same quantity, and correlated or systematic errors
can invalidate this calculation. We suggest three adjudicators per case as a practical
starting point, with a prespecified adjudication rule and enough raters to meet the
validated reliability target; an odd number does not guarantee consensus for every
rubric. 24
/ Resolve rubric problems before combining raters’ scores. Low agreement may indicate
an ambiguous rubric, heterogeneous cases, or legitimate clinical uncertainty.
Investigate the cause, refine and retest the rubric where appropriate, and retain
unresolved uncertainty in the reported results.

Reporting inter-rater reliability
Report agreement for every element before adjudication resolves disagreements, using a
standard agreement statistic suited to the number of raters and the type of rubric. When
one answer dominates, as with rare escalation cases, overall agreement can be misleading,
so agreement should also be reported separately for positive and negative cases.

Standard of care or median clinician
Make a median-clinician claim measurable. Compare device performance against the
prespecified median performance of a separate, representative clinician comparator group
on the same cases, using the independent adjudicated reference. Define a clinically justified
non-inferiority or equivalence margin and confidence interval. Falling anywhere within a
clinician range does not establish median-clinician performance. The choice of standard
then follows the intended use:

/ Standard of care applies to safety elements and claims of guideline-concordant or
specialist-level performance. Define risk-justified safety bounds; typical clinician errors
alone do not justify accepting a hazardous device error. Device failures may be
systematic across many users, so shared failure modes require explicit testing.
 FDA-2026-N-7874 13 / 33

/ Median clinician is appropriate for clinical proficiency elements when the device
performs a task routinely done by practicing clinicians and claims parity with them.
/ The counterfactual alternative applies when the device extends access where no
clinician is available; our response to Question 15 addresses this.
On generalists versus specialists: the comparator group should match whoever performs
the task without the device. Compare a primary care tool with primary care clinicians and
use suitably qualified independent adjudicators. Tu et al. illustrate evaluating conversational
diagnostic AI against clinicians using separate specialist assessments; the design must still
match the proposed intended use. 23

When human-AI team performance should be the basis
/ Clinician-supervised functions: team performance against clinicians working alone
should be primary when the intended use depends on review. A paired or randomized
reader study may be appropriate; FDA’s CADe guidance is a design precedent within
radiology, not a universal requirement. Include enough device-error cases to assess
correction and over-reliance, while preserving a representative primary analysis or
justified weighting. 25
/ Autonomous and patient-facing functions: the device alone is the basis, with user
comprehension testing for patient-facing outputs.
/ Report device-alone performance alongside team performance to characterize intrinsic
failures. It supplements workflow evaluation; it does not by itself predict the clinical
consequences of supervision lapses.
Acceptance criteria then follow from the chosen standard: non-inferiority margins against
the comparator for proficiency elements, and absolute bounds for safety elements, as
described in our response to Question 12.

/ Question 15: Counterfactual comparators
FDA question: Are there ways in which performance might be assessed relative to the care,
technology, or course of action likely to occur in the absence of the device, rather than to
the comparators described in this section? What approaches might be used to identify and
justify a comparator such as unaided clinical judgment, delayed specialist review, or no
intervention?

Summary: For devices that extend care where clinicians are scarce, assess whether patients
are better off than with the care expected without the device. Document that comparator
from evidence about the intended setting and reflect it in the claim and labeling where
appropriate. Both comparative benefits and harms matter, while risk-justified safety criteria
 FDA-2026-N-7874 14 / 33

remain essential. FDA already considers available alternatives in benefit-risk
determinations. 26

Defining the causal comparison
Compare two care pathways in the same population: care with the device and care as it
would otherwise likely unfold. Specify eligibility, strategies, outcomes, and follow-up as for
a target trial. Where randomization is impractical, observational emulation may inform the
comparison if its identification assumptions are justified and sensitivity analyses address
residual bias.

Criteria for choosing and justifying the comparator
/ It reflects what actually happens, not what is assumed. The sponsor documents the
usual pathway in the intended setting with data: time to specialist review, the share of
patients who receive no care, and what patients do instead with typical outcomes.
/ It is realistic. For many patient-facing functions, the true alternative is not "no
intervention" but an Internet search, a general-purpose chatbot, or a nurse line.
Comparing against nothing when patients would have done something overstates the
device's benefit.
/ Measure comparable patients and outcomes. For delayed specialist review, use
adjudication and clinical follow-up to establish the reference; the specialist’s eventual
decision is not automatically ground truth. Time to correct action may matter alongside
accuracy.
/ Retain explicit safety criteria. A device can improve access and still have unacceptable
hazards. Evaluate harms within both pathways, and justify absolute safety bounds by
intended use, severity, and available alternatives. Scarcity alone does not justify
avoidable harmful performance.

Tying the comparator to the intended use

The comparator is valid only in the setting it describes. Define any access or delay condition
in the claim and labeling using evidence, rather than an arbitrary time threshold. Two
safeguards follow:

/ Labeling and design that hold the device to that setting, so it is not presented as a
substitute where proper clinicians are available.
/ Monitoring for substitution, meaning use that displaces better care that was in fact
available, such as patients skipping an available appointment. If substitution is
detected, the counterfactual no longer holds, and the benefit claim must be reassessed.
 FDA-2026-N-7874 15 / 33

Access and equity
Counterfactual comparators can support fair evaluation of access-extending tools. Avoid a
two-tier standard in which scarcity alone excuses preventable harm: assess comparative
benefits and harms, preserve justified safety criteria, and monitor substitution. ICH E10
offers methodological principles for choosing controls in drug trials; this is an analogy
rather than a device requirement. 27

/ Question 17: Other model architectures
FDA question: CDRH is exploring whether a competency-based approach could be applied
to devices that incorporate a variety of underlying model architectures, such as multimodal
vision-language models and generative or predictive world models. Are there aspects of the
described approach that might be ineffective or inapplicable to these devices? What
additional or distinct considerations might need to be addressed?

Summary: The competency-based structure carries over to multimodal and world-model
devices, but the benchmark content must match their functions. Multimodal devices need
tests of clinically relevant grounding and cross-modal consistency. World models need
evaluation of prediction, calibration, forecast horizon, and any claimed intervention effects.
Computational modeling credibility principles may help by analogy; FDA’s guidance does
not directly cover standalone learned AI models. Strengthen clinical confirmation wherever
adequate benchmarks are unavailable. 4

Multimodal vision-language device considerations:
/ Grounding and cross-modal consistency. A device can reach a correct answer with a
flawed explanation or image interpretation, as Jin et al. demonstrate. Use clinically
meaningful ablations and counterfactual edits with prespecified expected behavior.
Removing an input without changing an answer does not prove that the input was
unused: other inputs may be redundant. Test missing, conflicting, and misleading
modalities as well as ordinary cases. 28
/ Patient, encounter, and temporal association. Outputs must link the right image to the
right patient, side of the body, and time point, and correctly compare against prior
studies. These errors are invisible to text-only benchmarks.
/ Acquisition variability. Report performance by device, protocol, and capture condition,
including patient-captured photos, as imaging AI already expects.
/ Error propagation. A misreading in one modality flows into the reasoning that follows.
Where the architecture allows, sponsors should evaluate each stage separately and
score clinical claims individually so they can trace errors to their source.
 FDA-2026-N-7874 16 / 33

Generative and predictive world models
For world models, assess accuracy, calibration, error growth over the forecast horizon, and
validity of any simulated intervention effects. Predictive accuracy alone does not establish
causal validity. Intervention claims need evidence and justified causal identification,
potentially using trials, established effects, or appropriate observational designs with
sensitivity analyses. FDA’s computational modeling guidance applies directly to firstprinciples modeling and relevant hybrid components; its credibility principles are an
analogy for learned world models. Prospectively registered forecasts scored as outcomes
can confirm predictions, but they do not alone validate intervention effects. Apply the
synthetic-data safeguards in Question 13 when these models generate test inputs. 4

Gaps in available benchmarks
Adequate multimodal and predictive benchmarks may be limited for a particular intended
use. Assess their outcome linkage, representativeness, and possible training-data
contamination rather than assuming that all public benchmarks are compromised. Where
gaps remain, we recommend two steps:

/ Strengthen clinical confirmation. Fill the specific evidentiary gap with an appropriate
study, including prospective evidence where needed. Tier 3 is already the highest, so the
response is a stronger design or broader coverage rather than an automatic move up
one rung.
/ Build shared, sequestered test sets. MIDRC illustrates a data commons with held-back
imaging data for evaluation. CDRH could encourage comparable resources for
multimodal and predictive tasks, with voluntary MDDT qualification where useful for a
defined context of use. 22,29

/ Question 18: Trading premarket evidence for postmarket
monitoring
FDA question: CDRH is considering whether it may be appropriate to accept greater
premarket uncertainty regarding a GenAI-enabled device's benefit-risk profile through
greater reliance on postmarket monitoring. Under what conditions might such an approach
be appropriate, and what characteristics of a monitoring program would need to be in place
to justify reduced premarket evidence? Are there device types or risk profiles for which this
approach would not be appropriate?

Summary: Greater premarket uncertainty may be justified when a demonstrated detection
and response capability is in place, with evidence that it will work. Define the residual
uncertainty, ensure the initial benefit-risk case is acceptable, and monitor for relevant
 FDA-2026-N-7874 17 / 33

failures before unacceptable harm accumulates. FDA’s 2019 uncertainty guidance addresses
this principle, but it does not create a general evidence trade-off for every pathway. 30

Conditions for accepting greater premarket uncertainty
/ Define the residual uncertainty. Generalization, subgroup precision, and effect-size
uncertainty can all affect safety. Meet essential premarket safety criteria and justify any
remaining uncertainty claim by claim.
/ Errors are observable after deployment. Ground truth must be obtainable through
outcomes or clinician adjudication with appropriate records and lawful access. Offline
operation is not automatically disqualifying, but a potentially harmful function without
adequate logs, follow-up, or reliable sampling cannot claim monitoring credit for
unobservable failures.
/ Detection is faster than harm. Demonstrate that detection and action can occur before
the specified degradation causes unacceptable harm.
/ Response is credible and enforceable where applicable. Demonstrate suspension,
rollback, or another effective intervention, with named ownership and response times.
Specify which commitments are legally enforceable under the applicable pathway or
surveillance authority and which are sponsor commitments. Preserve logs and actions
to complete all applicable reporting obligations.

Validating monitoring coverage
An absence of reported events is not evidence of safety. Many GenAI failures, such as
missed escalation or reinforced avoidance of care, may generate no complaint. Validate
monitoring using known-failure cases and controlled injected degradations, and show
detection within the stated window. This establishes coverage of the tested failure modes,
not unknown failures. Assess clinically important claims and subgroups, and include
clinician-review samples drawn independently of automated flags.

A detection specification
State each monitored claim in a standard, reviewable form that also informs the evaluation
frequency discussed in Question 19. For example:
Field Example
Monitored claim Escalation sensitivity, adult chest-pain presentations
Degradation to detect Sensitivity decreases from a known 95% baseline to 90% (illustrative fixedwindow design)
Detection window One fixed 30-day window, with adjudication completed in time to act
Sampling plan 300 independent cases requiring escalation, sampled representatively;
additional subgroup evidence as needed
Confidence Alarm at 22 or more misses: 4.86% one-sided false-alarm probability and
95.42% power for this single fixed-window test
Response and owner Immediate safety triage; investigation and action within justified deadlines;
rollback or suspension when indicated; FDA reporting when applicable
 FDA-2026-N-7874 18 / 33

This example assumes a known 95% baseline, independent labeled escalation-positive
cases, and one fixed-window test. Of note is that if the required cases and labels cannot
accrue within the detection window, reduce the monitoring credit or change the plan.

Where the trade should not apply

/ Autonomous functions with severe consequences, where no human is positioned to
catch an error before it reaches the patient.
/ Irreversible or time-critical actions, such as initiating therapy or under-escalating an
emergency, where detection necessarily comes after harm.
/ Rare catastrophic failure modes and small populations. With no failures, bounding a
rate below 1 in 1,000 at one-sided 95% confidence requires about 3,000 independent,
representative cases. Monitoring may therefore be too slow for rare severe harms, and
small pediatric or rare-disease populations may not support the required detection
window. Feasibility must be assessed for the specific population. 15

The pharmacovigilance precedent, as an analogy
Pharmacovigilance is a useful analogy (e.g., ICH E2E pharmacovigilance plans) for specific,
timed follow-up and the limits of spontaneous reporting, but device sponsors should
determine the specific obligations. Section 522 surveillance and post-approval study
conditions apply in defined circumstances; ISO 14971 and ISO/TR 20416 support lifecycle
surveillance planning. A detection specification makes the proposed monitoring
testable. 3,31,32

/ Question 19: Postmarket performance evaluation approaches
and cadence
FDA question: Please comment on the potential approaches to postmarket performance
evaluation, including periodic re-benchmarking, sample-based clinician review, and
performance degradation monitoring. What additional approaches should CDRH consider,
and how should the cadence and triggering events for reassessment be determined?

Summary: The three approaches are complementary because each catches failures the
others miss. Set the sampling rate to detect the specified degradation within the window
proposed in our response to Question 18. Scheduled full re-benchmarking should also set a
 FDA-2026-N-7874 19 / 33

maximum interval between evaluations. Specified events should also trigger prompt
reassessment.

Combining the monitoring approaches
Approach What it catches What it misses
Periodic re-benchmarking Regressions on known Failure modes the benchmark never
competencies; effects of model anticipated; shifts in real-world inputs
changes
Sample-based clinician Real-world errors visible to an Rare, high-severity events at feasible
review expert, including new failure sample sizes; harms that appear only
modes downstream
Degradation monitoring Gradual drift in inputs or outputs, Problems that do not change measured
continuously and at low cost signals; it lags until thresholds are
crossed

Additional approaches CDRH should consider
/ Production canaries. Run a controlled set of cases against the live device at a riskappropriate frequency. Compare prespecified behavior metrics with the expected
stochastic baseline. An alarm can signal a model or administrator and should trigger
investigation; it does not prove an undeclared update or guarantee detection of every
change.
/ User-behavior signals. Track clinician overrides, edits, and rejections, as well as patient
abandonment or repeat contact. These can signal possible degradation but also depend
on case mix, staffing, and workflow; investigate them alongside adjudicated
performance and outcomes.
/ Outcome-linked surveillance. Linking device outputs to downstream outcomes in realworld data, such as delayed diagnoses or unplanned emergency visits, captures harms
that transcript review cannot see.
/ Knowledge-currency checks. When a relevant clinical guideline changes, the device
should be tested against it. A device can become wrong without any change to the
device itself.

Setting cadence on a statistical basis
Each monitored claim in the detection specification defines a degradation to detect, a
confidence level, and a detection window. From these, the sponsor calculates the number of
adjudicated cases needed, and the device's case volume determines how quickly that
sample accumulates. Cadence is then a trade-off between detection latency and review
burden, which the sponsor makes explicit:

/ Sequential statistical methods, such as CUSUM charts, may reduce detection delay for
specified shifts. Select and validate the design for the expected event rate, shift size,
 FDA-2026-N-7874 20 / 33

label delay, and false alarm burden, including repeated monitoring and multiple claims.
Feng et al. describe continual monitoring methods; no method is uniformly faster under
all conditions. 33
/ If the achievable latency exceeds the window, the sponsor must increase sampling,
narrow the claim, or accept that the claim is not monitorable, which bears directly on the
trade-off in Question 18.
/ A scheduled full re-benchmark against the locked suite sets a maximum interval,
because a monitor that never alarms is consistent with both a stable device and a faulty
monitor.

Triggering events for unscheduled reassessment
The following should trigger prompt impact assessment, independent of the schedule, with
testing scope set by the risk-based matrix in Question 22. Not every trigger requires the full
benchmark; reassessment should match the change's scope and risk. Some examples of
triggering events:

/ A device or component change, including third-party model updates, or a material
canary alarm. Changes to the probes themselves require controlled revalidation of the
monitoring process.
/ A shift in the input population beyond a prespecified bound, or onboarding a new site,
population, or language.
/ A control-chart alarm, a confirmed serious incident, or a cluster of complaints or
overrides.
/ A material change in clinical guidelines within the device's scope.
/ Deployment milestones, which add statistical power and expose new conditions of use.

/ Question 20: Machine-based supervisory agents
FDA question: Please comment on whether the proposed approaches to postmarket
monitoring can be facilitated by machine-based supervisory agents. What considerations,
including the evaluation and reliability of the supervisory agent itself, should CDRH take into
account for such an approach?

Summary: Supervisory agents may screen more outputs than clinician review alone,
potentially every output where technically feasible. Screening volume is not evidence of
 FDA-2026-N-7874 21 / 33

sensitivity. Monitoring credit should depend on measured detection of the specified harms,
shared-error risk, ongoing human audit, and accountable ownership of the resulting actions.

Independence from the supervised device
/ Diverse model and checking approaches. Use a different model family where feasible
and combine it with independently designed checks. A different family safeguards
against some shared errors, but it does not prove independence or provide a universally
sufficient minimum.
/ Measure shared misses. Model errors can be correlated across providers and
architectures. Measure the supervisor’s probability of missing a harmful device output
on an independent, clinician-adjudicated reference set, with clinically important
subgroups and uncertainty intervals. Monitoring credit should reflect this joint failure
risk. 21
/ Diverse checks where risk is high. Pairing an AI supervisor with non-AI checks, such as
rule-based dose limits or verification of cited sources, follows the long-standing safetyengineering practice of using diverse redundancy to avoid common-cause failure.
/ Resistance to manipulation. Device outputs pass through the supervisor, so text that
steers or instructs the supervisor is an attack path. Supervisors should be tested against
it.

Qualifying the supervisor
The supervisor should meet the same kind of competency standard as the device, for the
specific harms it is meant to detect:

/ Sensitivity for each targeted harm, such as missed escalation or an unsafe dose,
measured against clinician adjudication, along with the false-alarm rate. Overall
agreement is not enough, because the majority of correct outputs dominates it.
/ Audit flagged and passed outputs. A random sample of passed outputs estimates the
rate of harmful outputs among those passed, not sensitivity by itself. Estimate
sensitivity and false-alarm rates from adjudicated samples of both flagged and passed
outputs, with sampling weights where needed and sufficient harmful cases.
/ Change control. A change to the supervisor, including an update to its underlying
model, changes the monitoring program and is re-qualified using the change-impact
approach in our response to Question 22.

Role determines the level of control
Role determines the controls. Software used solely for production or quality monitoring
may fall within FDA’s February 2026 Computer Software Assurance guidance. A supervisor
that blocks, alters, or releases clinical outputs may perform a device function and requires
 FDA-2026-N-7874 22 / 33

the corresponding device lifecycle, risk, and verification controls. Classify the actual
function rather than its “supervisor” label. 17,34

Accountability stays with people
Assign responsibility for the monitoring process to a qualified person. Under the QMSR,
management responsibility remains with the manufacturer’s people. Assign a qualified
owner to the monitoring process, define severity-based triage and response times, and
retain records of reviews and actions. Clinically significant findings require appropriate
human escalation; a supervisor’s silence is not evidence of safety. 35,36

/ Question 21: Stakeholder roles and manufacturer accountability
FDA question: What roles might clinicians, healthcare institutions, professional societies,
standards-setting bodies, and other stakeholders appropriately play in postmarket
monitoring, and how can these roles be structured without diffusing manufacturer
accountability?

Summary: Allocate operational tasks explicitly while keeping manufacturer duties intact.
Define local configuration limits, secure lawful access to monitoring records, and assign
reporting and corrective-action responsibilities before deployment.

/ Deployer configuration. Manufacturers should define the locally configurable items,
permitted limits, and installation acceptance tests in labeling and applicable change
controls. These should cover local retrieval sources, protocols and formularies,
prompts, tools and guardrails. Local retrieval and prompt changes can affect safety
even within a nominal envelope. PCCP coverage must be specifically authorized;
deployer changes outside the validated configuration require assessment under
applicable FDA modification policy. 37,38 As a comparator, the EU AI Act Article 25
similarly addresses circumstances in which a deployer or another party becomes a
provider following substantial modification. 39

/ Access to logs. Manufacturer-deployer agreements should establish lawful access to
records needed for monitoring, security responsibilities, and data retention. HIPAA
business associate terms are appropriate where the relationship requires them; deidentification and other lawful arrangements may also be relevant. Agreements
should specify the records, permitted uses, retention period, access process, and
deadlines for providing material safety information. They do not, by themselves,
authorize disclosure of protected health information. 40 Again, as a comparator, EU AI
Act Article 26(6) specifies retention of automatically generated logs under the
deployer’s control for at least six months, unless applicable law provides otherwise.
 FDA-2026-N-7874 23 / 33

Application of that obligation depends on the system’s classification and the relevant
commencement date. 39

/ Other parties and accountability. Clinicians and healthcare institutions can
adjudicate cases, provide clinical follow-up, and provide evidence on workflow
performance. Professional societies are well placed to develop the clinical rubrics
judges and reviewers use, based on their guidelines. Standard-setting bodies can
support common evaluation methods, logging formats, and incident definitions. User
facilities already have reporting obligations under 21 CFR part 803. Agreements
should identify who collects each signal, routes reports, investigates failures, and
initiates action, with escalation deadlines. Complaint handling, the manufacturer’s
reporting obligations, and device corrective action remain the manufacturer’s
responsibility. 36,41

Illustrative risks and controls. The qualitative ratings below are priorities for this proposed
framework, not measured probabilities or device-specific risk determinations. L =
likelihood; I = impact; H = high; M = medium. Sponsors should reassess the ratings for the
intended use.

Risk Type L I Rating Mitigation and framework
Deployer changes take Safety H M High Validated local limits; installation tests; specific
the device outside its authorized PCCP coverage where applicable.
validated configuration
Manufacturers cannot Feasibility H M High Lawful log-access terms; business associate
obtain logs needed for agreements where required; validated demonitoring identification
Responsibility diffused Safety M M Medium Roles defined in agreements; manufacturer
across parties. retains complaint handling and reporting

/ Question 22: Scaling re-benchmarking to modification impact
FDA question: CDRH envisions that a manufacturer's premarket competency-based
assessment might serve as a baseline against which post-deployment modifications could
be re-evaluated. How might the extent of re-benchmarking or other evidence be scaled to
the nature and expected impact of a given modification? For example, are there categories
of change that might not significantly affect safety or effectiveness of a GenAI-enabled
device, or might be appropriately managed within a sponsor's quality management system
versus requiring FDA premarket review and authorization before implementation? Are there
categories of changes that might be appropriate for inclusion in a PCCP?

Summary: Re-benchmarking should scale with what a change can plausibly affect,
documented in a change-impact matrix. Preserve an invariant full safety core for higher-risk,
broadly influential, or uncertain changes. Allow targeted testing for narrow, lower-risk
 FDA-2026-N-7874 24 / 33

changes when an impact assessment justifies the exclusions and essential safety regression
checks still pass. The regulatory route depends on the authorized PCCP and applicable
modification policy, not on a test result alone.

A change-impact matrix
The matrix below supports a documented impact assessment. Testing scope scales with the
device’s evidence tier and the change's severity, breadth, and uncertainty. The risk-based
rules below apply across all rows. An authorized PCCP’s required tests remain binding on
implementation under that plan.
Change type Examples Minimum re-evaluation scope Route
Non-output Logging, security patches, Document impact assessment; QMS if
changes or nonclinical UI changes verify affected functions and applicable
shown not to affect essential safety regression modification
inference, safety controls, checks policy
latency, availability, or supports it;
clinical use. otherwise
authorized
PCCP or FDA
review
Changes Quantization, distillation, Risk-based scope under the rules Authorized
intended to serving-stack, hardware, below; paired equivalence where PCCP if
preserve or hosting changes justified; verify affected controls specifically
behavior and new hazards covered;
otherwise
apply the
relevant
modification
policy
Knowledge and Guideline refresh, Affected knowledge, clinical Authorized
retrieval updates approved-source proficiency, calibration, and PCCP if
additions subgroups; full safety core or covered;
justified targeted scope under otherwise
the rules below assess need
for FDA review
Prompt, System prompt or refusal- Affected elements, trajectories, Authorized
guardrail, policy revisions and subgroups; full safety core or PCCP if
orchestration justified targeted scope under covered;
the rules below otherwise
assess need
for FDA review
Model weights, Fine-tuning on like data; Normally full benchmarking and Authorized
same family foundation-model version affected subgroup analysis; PCCP if
upgrade clinical confirmation specifically
proportionate to risk and impact covered;
otherwise
assess need
for FDA review
 FDA-2026-N-7874 25 / 33

Change type Examples Minimum re-evaluation scope Route
New capability or New tools or agentic Full impact assessment, affected No automatic
model family actions, new modality, benchmarking, and appropriate route: assess
different model family clinical confirmation authorized
PCCP limits
and
applicable
FDA
modification
requirements
Intended use or New users or population, Re-established evidence at the FDA review
risk position more directive or new tier where
autonomous function required by
the applicable
pathway;
outside PCCP
if beyond its
authorized
limits

Three rules apply across the matrix:

/ Preserve the full safety core where risk warrants it. Re-run safety-critical recognition
and escalation, scope adherence, and calibration and deferral in full for Tier 3 functions,
changes that may affect high-severity hazards or increase autonomy, broad model or
serving changes, and changes whose impact is uncertain. For a narrow lower-risk
change, permit targeted testing only with a documented impact analysis explaining why
excluded elements are unaffected, traceability to hazards and dependencies, and a
minimum set of essential safety regression checks. Unexpected failures or sharedcomponent effects trigger expansion to the full core.
/ Test elements whose exposure to the change remains uncertain. If the impact
assessment cannot rule out an effect on an element, re-run it. Any reduced scope must
comply with the authorized PCCP or other applicable change controls. Document why
the change has limited effects.
/ Compare to the authorized baseline. Judge results against the originally authorized
performance as well as the prior version. Schedule periodic full re-benchmarking at a
justified interval and after accumulated changes warrant it, so repeated small exclusions
cannot conceal cumulative drift.

Fit with ISO 14971 and ISO 13485
The matrix can support existing change-control processes under the QMSR:

/ ISO 13485:2016, clause 7.3.9 (control of design and development changes) requires
sponsors to determine the significance of a change and to review, verify, and validate it
before implementation. The matrix provides a consistent, documented basis for that
significance determination and for the verification scope. 35
 FDA-2026-N-7874 26 / 33

/ ISO 14971 requires lifecycle risk management, including assessment of changes and
production and post-production information. New tools, data sources, populations, and
serving conditions may introduce hazards. Every change needs an impact assessment;
even a change intended to preserve behavior requires assessing new hazards and
verifying affected risk controls. Successful equivalence testing alone does not complete
that assessment. 3
/ FDA’s 2017 software-change guidance provides a decision framework for changes to
devices cleared through 510(k). Other pathways have their own requirements. Use test
evidence with the applicable policy and authorized PCCP; a passing test alone does not
establish that FDA review is unnecessary. 37,38

/ Question 23: PCCPs for unforeseeable modifications
FDA question: How might PCCP concepts or other change-control approaches be adapted
for GenAI-enabled devices when the nature or scope of future modifications cannot be fully
prespecified?

Summary: When exact future versions cannot be prespecified, define bounded
modifications and the evaluation gates together. A gate-based PCCP should retain FDA’s
three components: a sufficiently specific description of modifications, a modification
protocol, and an impact assessment. Passing a gate does not authorize a modification
outside the authorized plan.37

A gate-based PCCP

/ Describe the permitted modifications. The sponsor describes limited, sufficiently
specific modifications, potentially including prompt and guardrail revisions, retrieval
changes within approved sources, or defined model and serving changes. Broad classes
alone are insufficient. Changes outside the authorized PCCP, including changes beyond
its intended-use limits, require assessment under the applicable modification policy. 37
/ Locked gates. Prespecify the controlled benchmark suite, acceptance thresholds, and
verification protocol. Apply the risk-based testing rules in our response to Question 22:
a full safety core for higher-risk or uncertain changes, and justified targeted testing for
narrow lower-risk changes. Changes to the authorized evaluation protocol itself require
assessment under the applicable FDA policy; the sponsor cannot redefine a failing gate
to make a change pass.
/ Decision rules. Pass: implement only within the authorized PCCP and document the
result. Staged or shadow rollout is permissible only when the authorized protocol allows
it and the applicable stage’s criteria are met. Fail: do not implement the modification
under that PCCP. A marginal or failed result does not permit exposing patients while
seeking evidence. 37
 FDA-2026-N-7874 27 / 33

/ Rollback triggers. Prespecified postmarket signals trigger suspension or reversion to an
appropriate controlled version within a demonstrated time. Hosted third-party models
require contractual and technical support for version control, rollback, or a safe fallback;
a sponsor should not promise pinning the supplier cannot provide.
/ Cumulative drift guard. Each change is compared against the originally authorized
baseline, not only the previous version, so a series of small, individually acceptable
changes cannot add up to an unacceptable one. 37

Compression and serving changes
Compression and serving changes cut across the paper’s categories of sponsor-initiated
changes, model evolution, and third-party updates. Although these changes are intended to
preserve behavior, quantization, pruning, distillation, hardware, and serving changes can
affect safety and performance. Hong et al. provide evidence that compression can change
trustworthiness. Classify both who initiates a change and what it may affect. For changes
intended to preserve behavior, use clinically justified paired equivalence testing where
appropriate, together with checks for new hazards such as latency, availability, and
cybersecurity. Apply the testing scope in Question 22; intent to preserve behavior is not
proof of low risk. 42

Integration with IEC 62304 and the quality system

Integrate the gate into existing lifecycle controls. IEC 62304 addresses problem and
modification analysis (clause 6.2), risk management of software changes (7.4), and software
configuration management (clause 8). Applicable SOUP controls should cover third-party
software, with model-specific controls added where necessary. ISO 13485 design change
control under the QMSR provides the review, verification, and validation process; the
authorized PCCP adds the agreed modification limits and acceptance criteria. Each control
addresses a different part of the release decision. 17,35,36

Who owns the gate
Quality should own the documented release process, with engineering executing tests and
qualified clinical oversight proportionate to safety impact. FDA’s guidance permits
automatic implementation when it is specifically covered by an authorized PCCP and
validated controls. Automation should execute authorized criteria and preserve
accountable ownership; it should not release marginal results that fail those criteria. 37
 FDA-2026-N-7874 28 / 33

/ Question 25: Foundation-model master files and supplier
information
FDA question: Would voluntary Foundation Model MAFs be practical for foundation model
developers to provide and to keep current, and what content might they include? Given that
participation would be voluntary and that model developers may have limited incentive to
disclose safety-relevant information, what would make such a program sufficiently useful
for premarket review? Are there alternative mechanisms CDRH should consider for
obtaining information about underlying foundation models?

Summary: A voluntary foundation-model master file could reduce duplicated review if its
content is tied to an identifiable model snapshot and kept current. Its usefulness depends
on access to relevant evidence, effective rights of reference and confidentiality protections,
and the work required to file and maintain it. 43

Documentation prepared for downstream providers under EU AI Act Article 53 and Annex
XII could supply part of the file. Reusing that material could reduce filing burden, although
FDA-specific evidence, confidential information, version updates and rights of reference
would still need to be addressed. We ask FDA to consider Annex XII as a starting structure,
supplemented with evidence specific to medical-device use. That documentation alone
should not be presumed sufficient for FDA review. 39

Useful content would include architecture and training-data summaries; intended and
unsupported uses; limitations and known healthcare failure modes; evaluation methods and
results; change history; version and deprecation policies; and commitments on change
notification. Each should identify the model snapshot to which it applies.

A master file could support supplier evaluation under ISO 13485 clause 7.4. The
manufacturer would still need to evaluate the deployed device and its integration. FDA’s
existing Device Master Files process provides an administrative basis for authorizing
reference to confidential information; it does not establish acceptance of a foundation
model. We recommend snapshot-specific rights of reference, maintenance arrangements,
and notice of safety-relevant changes. 35,43

If a developer does not file supplier documentation, contractual disclosures, independent
evaluation and version-specific testing may provide alternative evidence. Supported pinned
versions or self-hosting may help control the configuration. None automatically replaces
information essential to the safety assessment. Where that information is unavailable,
identify the residual uncertainty and obtain adequate evidence or restrict the use. Canary
probes can detect some changes, but cannot replace adequate supplier controls.

Illustrative risks and controls. The table gives illustrative priorities before mitigation, not
measured probabilities or device-specific risk determinations. L = likelihood; I = impact; H =
 FDA-2026-N-7874 29 / 33

high; M = medium. Feasibility ratings describe implementation burden, separately from
acceptable patient risk.

Risk Type L I Rating Mitigation and framework
MAF out of date as the Safety H M High MAF content tied to snapshot identifiers;
model is updated commitment to notify updates; manufacturer
canary testing
Developers decline to file. Feasibility M M Medium Reuse Annex XII content where useful;
demonstrate adequate device evidence without
presuming equivalence.

/ Other considerations: Third-party software, reporting, and
cybersecurity
Foundation models as SOUP. Where a third-party model is treated as software of unknown
provenance under IEC 62304, identify its configuration and assess available published
anomalies. We ask CDRH to clarify how snapshot identifiers, model or system cards, supplier
information, and manufacturer anomaly records can support clauses 8.1.2 and 7.1.3 when
developers do not publish conventional anomaly lists. System-level benchmarking under
clause 5.7 does not create a blanket exemption from software-unit verification under clause
5.5. Integration code, interfaces, and safety controls need their applicable lifecycle
evidence. 17

Supplier controls. Agreements should address change notices, deprecation, version
pinning, and access to evaluation information. A stable API name is insufficient for
configuration identification. If necessary controls are unavailable, evaluate alternatives such
as supported pinned versions or self-hosting. Canary testing can complement supplier
controls but cannot make an uncontrolled arrangement equivalent to a controlled one. 17,35

Adverse event reporting. Under 21 CFR part 803, manufacturers assess whether an event
may have caused or contributed to death or serious injury, or involved a malfunction likely to
cause or contribute to death or serious injury if it recurred. A benchmark label of “critical
failure” does not replace those criteria. We ask FDA for examples covering stochastic
outputs and clarification of eligibility and exclusions under the Voluntary Malfunction
Summary Reporting Program. Deaths and serious injuries remain individually reportable;
summary reporting applies only to eligible malfunctions under the program’s conditions. 41,44

Corrections and removals. A model rollback or prompt change can be a correction under 21
CFR part 806. Reportability depends on the action’s purpose and circumstances, including
section 806.10 and applicable exemptions. An authorized PCCP does not waive these
duties. We ask FDA for examples distinguishing routine updates from actions taken to
reduce a risk to health or remedy a violation that may present a health risk. 37,45
 FDA-2026-N-7874 30 / 33

Cybersecurity. For products meeting the statutory definition of a cyber device, assess
prompt injection and jailbreak pathways in threat modeling and vulnerability management,
using FDA’s current cybersecurity guidance as applicable. Include data exfiltration, tool
misuse, and compromised retrieval where relevant. Identify models and guardrail
components by provider and version in the component inventory and document external
API dependencies. We ask FDA to clarify how those dependencies should be represented
within, or alongside, the required software bill of materials. Classify security threats by their
mechanisms and consequences; an undesirable output is not necessarily a cybersecurity
vulnerability. 31,46

Illustrative risks and controls. The table gives illustrative priorities before mitigation, not
measured probabilities or device-specific risk determinations. L = likelihood; I = impact; H =
high; M = medium. Feasibility ratings describe implementation burden, separately from
acceptable patient risk.

Risk Type L I Rating Mitigation and framework
SOUP identification Feasibility H H High Traceable snapshot identifiers; supplier notice;
cannot be met for API canary complements identification
models that change
behind a stable name
Supplier terms do not Feasibility H M High Evaluate controlled hosting or supported versions;
provide change notice or canary is not equivalent to supplier control.
snapshot pinning
Inconsistent or excessive Feasibility H M High Apply Part 803 legal criteria; examples for
malfunction reporting for stochastic outputs; summary reporting only when
stochastic outputs eligible.
Jailbreak vulnerabilities Safety M H High Threat modeling for applicable cyber devices;
handled outside component and dependency inventory;
cybersecurity processes. appropriate SBOM

/ References
1. US Food and Drug Administration. Considerations for the Regulation of Generative AIEnabled Medical Devices: Discussion Paper and Request for Feedback. Published August
2026. Accessed October 15, 2026. https://www.fda.gov/media/194242/download
2. International Electrotechnical Commission. Medical devices—Part 1: Application of
usability engineering to medical devices. IEC 62366-1:2015+A1:2020. IEC; 2020.
3. International Organization for Standardization. Medical devices—Application of risk
management to medical devices. ISO 14971:2019. ISO; 2019.
4. US Food and Drug Administration. Assessing the Credibility of Computational Modeling
and Simulation in Medical Device Submissions. Published November 2023. Accessed
October 15, 2026. https://www.fda.gov/media/154985/download
5. International Medical Device Regulators Forum. Software as a Medical Device (SaMD):
Clinical Evaluation. IMDRF/SaMD WG/N41FINAL:2017. Published 2017.
https://www.imdrf.org/documents/software-medical-device-samd-clinical-evaluation
 FDA-2026-N-7874 31 / 33

6. Dahabreh IJ, Robertson SE, Steingrimsson JA, Stuart EA, Hernán MA. Extending
inferences from a randomized trial to a new target population. Stat Med. 2020;39:19992014. https://doi.org/10.1002/sim.8426
7. Acceptance of Data From Clinical Investigations Conducted Outside the United States, 21
CFR § 812.28. Accessed October 15, 2026. https://www.ecfr.gov/current/title21/chapter-I/subchapter-H/part-812/subpart-A/section-812.28
8. US Food and Drug Administration. Use of Real-World Evidence to Support Regulatory
Decision-Making for Medical Devices. Published December 2025. Docket FDA-2023-D4395.
9. US Food and Drug Administration. E5 Ethnic Factors in the Acceptability of Foreign
Clinical Data. Published June 1998. https://www.fda.gov/media/71293/download
10. US Food and Drug Administration. E9(R1) Statistical Principles for Clinical Trials:
Addendum: Estimands and Sensitivity Analysis in Clinical Trials. Published May 2021.
https://www.fda.gov/media/148473/download
11. Angelopoulos AN, Bates S, Fannjiang C, Jordan MI, Zrnic T. Prediction-powered
inference. Science. 2023;382(6671):669-674. https://doi.org/10.1126/science.adi6000
https://pubmed.ncbi.nlm.nih.gov/37943906/
12. US Food and Drug Administration. Multiple Endpoints in Clinical Trials. Published
October 2022. https://www.fda.gov/media/162416/download
13. US Food and Drug Administration. Guidance for the Use of Bayesian Statistics in Medical
Device Clinical Trials. Published February 2010. https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/guidance-use-bayesian-statisticsmedical-device-clinical-trials
14. Bowyer S, Aitchison L, Ivanova DR. Position: Don’t use the CLT in LLM evals with fewer
than a few hundred datapoints. In: Proceedings of the 42nd International Conference on
Machine Learning. PMLR; 2025;267.
https://proceedings.mlr.press/v267/bowyer25a.html
15. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting
zero numerators. JAMA. 1983;249(13):1743-1745.
https://doi.org/10.1001/jama.1983.03330370053031
https://pubmed.ncbi.nlm.nih.gov/6827763/
16. Association for the Advancement of Medical Instrumentation. Application of ISO 14971 to
machine learning in artificial intelligence—Guide. AAMI TIR34971:2023. AAMI; 2023.
17. International Electrotechnical Commission. Medical device software—Software life cycle
processes. IEC 62304:2006+A1:2015. Edition 1.1. IEC; 2015.
18. Institute of Electrical and Electronics Engineers. IEEE Standard for Recommended
Practice for the Quality Management of Datasets for Medical Artificial Intelligence. IEEE
2801-2022. IEEE; 2022.
19. Institute of Electrical and Electronics Engineers. IEEE Standard for the Performance and
Safety Evaluation of Artificial Intelligence Based Medical Device: Terminology. IEEE 28022022. IEEE; 2022.
 FDA-2026-N-7874 32 / 33

20. Shumailov I, Shumaylov Z, Zhao Y, Papernot N, Anderson R, Gal Y. AI models collapse
when trained on recursively generated data. Nature. 2024;631:755-759.
https://doi.org/10.1038/s41586-024-07566-y
21. Kim EM, Garg A, Peng K, Garg N. Correlated errors in large language models. In:
Proceedings of the 42nd International Conference on Machine Learning. PMLR;
2025;267:30038-30066. https://proceedings.mlr.press/v267/kim25e.html
22. US Food and Drug Administration. Qualification of Medical Device Development Tools.
Published July 2023. https://www.fda.gov/regulatory-information/search-fdaguidance-documents/qualification-medical-device-development-tools
23. Tu T, Schaekermann M, Palepu A, et al. Towards conversational diagnostic artificial
intelligence. Nature. 2025;642:442-450. https://doi.org/10.1038/s41586-025-08866-7
24. Spearman C. Correlation calculated from faulty data. Br J Psychol. 1910;3(3):271-295.
https://doi.org/10.1111/j.2044-8295.1910.tb00206.x
25. US Food and Drug Administration. Clinical Performance Assessment: Considerations for
Computer-Assisted Detection Devices Applied to Radiology Images and Radiology
Device Data in Premarket Notification (510(k)) Submissions. Published September 2022.
Accessed October 15, 2026. https://www.fda.gov/media/77642/download
26. US Food and Drug Administration. Factors to Consider When Making Benefit-Risk
Determinations in Medical Device Premarket Approval and De Novo Classifications.
Published August 2019. Accessed October 15, 2026.
https://www.fda.gov/media/99769/download
27. US Food and Drug Administration. E10 Choice of Control Group and Related Issues in
Clinical Trials. Published May 2001. https://www.fda.gov/media/71349/download
28. Jin Q, Chen F, Zhou Y, et al. Hidden flaws behind expert-level accuracy of multimodal
GPT-4 vision in medicine. npj Digit Med. 2024;7:190. https://doi.org/10.1038/s41746024-01185-7
29. Baughan N, Whitney HM, Drukker K, et al. Sequestration of imaging studies in MIDRC:
stratified sampling to balance demographic characteristics of patients in a multiinstitutional data commons. J Med Imaging (Bellingham). 2023;10(6):064501.
https://doi.org/10.1117/1.JMI.10.6.064501
30. US Food and Drug Administration. Consideration of Uncertainty in Making Benefit-Risk
Determinations in Medical Device Premarket Approvals, De Novo Classifications, and
Humanitarian Device Exemptions. Published August 2019.
https://www.fda.gov/media/115672/download
31. Federal Food, Drug, and Cosmetic Act, 21 USC §§ 360e-4, 360l, 360n-2. Accessed
October 15, 2026. https://uscode.house.gov/
32. International Organization for Standardization. Medical devices—Post-market
surveillance for manufacturers. ISO/TR 20416:2020. ISO; 2020.
33. Feng J, Phillips RV, Malenica I, et al. Clinical artificial intelligence quality improvement:
towards continual monitoring and updating of AI algorithms in healthcare. npj Digit Med.
2022;5:66. https://doi.org/10.1038/s41746-022-00611-y
 FDA-2026-N-7874 33 / 33

34. US Food and Drug Administration. Computer Software Assurance for Production and
Quality Management System Software. Published February 2026.
https://www.fda.gov/media/188844/download
35. International Organization for Standardization. Medical devices—Quality management
systems—Requirements for regulatory purposes. ISO 13485:2016. ISO; 2016.
36. Quality Management System Regulation, 21 CFR pt 820. Accessed October 15, 2026.
https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part-820
37. US Food and Drug Administration. Marketing Submission Recommendations for a
Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software
Functions. Published August 2025. Accessed October 15, 2026.
https://www.fda.gov/media/166704/download
38. US Food and Drug Administration. Deciding When to Submit a 510(k) for a Software
Change to an Existing Device. Published October 2017.
39. European Parliament and Council of the European Union. Regulation (EU) 2024/1689
laying down harmonised rules on artificial intelligence. Published July 12, 2024. Accessed
October 15, 2026. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
40. US Department of Health and Human Services. Business Associate Contracts. Published
January 25, 2013. Accessed October 15, 2026. https://www.hhs.gov/hipaa/forprofessionals/covered-entities/sample-business-associate-agreementprovisions/index.html
41. Medical Device Reporting, 21 CFR pt 803. Accessed October 15, 2026.
https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part-803
42. Hong J, Duan J, Zhang C, et al. Decoding compressed trust: scrutinizing the
trustworthiness of efficient LLMs under compression. In: Proceedings of the 41st
International Conference on Machine Learning. PMLR; 2024;235:18611-18633.
https://proceedings.mlr.press/v235/hong24a.html
43. US Food and Drug Administration. Device master files. Accessed October 15, 2026.
https://www.fda.gov/medical-devices/premarket-submissions-selecting-andpreparing-correct-submission/device-master-files
44. US Food and Drug Administration. Voluntary Malfunction Summary Reporting Program.
Updated March 27, 2025. Accessed October 15, 2026. https://www.fda.gov/medicaldevices/medical-device-reporting-mdr-how-report-medical-deviceproblems/voluntary-malfunction-summary-reporting-program
45. Medical Devices; Reports of Corrections and Removals, 21 CFR pt 806. Accessed
October 15, 2026. https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part806
46. US Food and Drug Administration. Cybersecurity in Medical Devices: Quality
Management System Considerations and Content of Premarket Submissions. Published
February 2026. Docket FDA-2021-D-1158.