Circuit Breaker Labs

Please see the attached file for responses and comment on all questions posed in "Considerations for the Regulation of Generative AI-Enabled Medical Devices" from the perspective of Circuit Breaker Labs (Industry) whose work focuses on operationalizing Clinical AI Safety at speed, sophistication, and scale. Further queries can be directed to shirali@circuitbreakerlabs.ai.

Considerations for the
Regulation of Generative
AI-Enabled Medical Devices

Circuit Breaker Labs | FDA Public Comment

October 19, 2026
Question 1: Does the two-axis risk framework, organized around device activity and the
consequence of relying on an incorrect output, appropriately capture the dimensions most
relevant to the risk of a GenAI-enabled software function? If there are additional
dimensions—such as the reversibility of a resulting action, the availability of downstream
safeguards, the time pressure of the deployment setting, or the traceability of the output (i.e.,
to primary source materials)—that should be represented in a risk framework, please describe
and provide examples of how they should be represented.

The proposed two-axis framework provides a useful foundation, particularly because it recognizes risk
as a continuum. However, device activity and the consequences of an incorrect output do not fully
characterize the risks of generative AI-enabled medical devices. We recommend retaining the two axes
while introducing additional axes.

First, risk depends on the characteristics of the intended user. A patient, a practicing clinician, a
specialist, and a clinician in training may interact with the same system differently and have different
abilities to recognize an incorrect response. Even within these populations, age, health literacy, clinical
complexity, and patterns of use can materially affect risk. For instance, adolescent users may use
language that differs substantially from the system's expected inputs. Risk assessment should consider
foreseeable use behaviors rather than assuming all users interact with the device as intended.

Second, informational functions warrant greater differentiation. A resource navigator that directs users
to established clinical information differs meaningfully from a system that synthesizes medication
information in the context of an individual's treatment history. Even without an explicit
recommendation, personalized information may influence clinical decisions. The extent to which a
system incorporates patient-specific information should therefore modify its risk classification.

Third, the presence of human supervision should not automatically be treated as equivalent to effective
oversight. A clinician who reviews every output before it reaches a patient provides a different
safeguard from one who retrospectively reviews a small sample of interactions. Risk assessment should
consider the timing, frequency, coverage, clinical competence, and intervention authority associated
with human oversight. Retrospective sampling may detect systematic problems but cannot prevent
harm from an individual output already delivered.

Fourth, traceability and robustness should be considered together. A system may correctly handle a
clinically significant concept when expressed using formal terminology but respond differently when
the same information is expressed colloquially. For example, requests involving “acetaminophen” and
“Tylenol” should not produce materially different safety-critical behavior merely because of the
terminology used, but we have observed materially different behavior in our evaluations at Circuit
Breaker Labs. Traceability to appropriate clinical sources is valuable, but it does not itself establish that
the system will apply those sources consistently.

Rather than adding numerous independent axes, CDRH could establish a small set of risk modifiers
and associated testing expectations. This would preserve the flexibility of the proposed framework
while giving manufacturers more predictable, application-specific requirements.

Question 2: CDRH seeks input on how to account for the spectrum of GenAI-enabled
informational functions that vary in the degree to which they direct a user to a particular
action (i.e., between “non-directive” and “action-directing”). What characteristics of an
output—such as its wording, specificity, personalization, or context—could be considered as
modifiers of the risk associated with an informational function, after accounting for the
device’s overall functionality and intended use? What additional information would provide
manufacturers with sufficient clarity and predictability around risk assessment for
informational functions while recognizing that directiveness may exist along a continuum
rather than as a binary distinction?

The degree to which an informational function directs behavior cannot be assessed solely through
explicit instructions or recommendations. Risk also depends on how the information is
communicated, the user's circumstances, and the degree of authority or interpersonal trust the system
may implicitly convey.

A particularly important consideration is anthropomorphism, the attribution of human characteristics
or understanding to an AI system. A system that speaks in a highly personal, emotionally affirming, or
companion-like manner may encourage users to place greater confidence in its outputs, even when
those outputs are nominally non-directive. Generative systems may also mirror or reinforce a user's
beliefs and emotions, a behavior that can be inappropriate in certain clinical contexts.

We recommend a principle of minimum necessary anthropomorphism: manufacturers should justify
personifying design features according to their intended clinical benefit and assess whether those
features introduce undue trust, emotional dependence, or inappropriate behavioral influence. This
need not prohibit first-person language or empathetic communication. Indeed, some therapeutic
applications may benefit from carefully designed relational interactions. The important distinction is
between communication that supports appropriate engagement and communication that encourages
users to overestimate the system's understanding, competence, or authority.

CDRH could provide greater predictability through a behavioral assessment framework considering
the use of patient-specific information, implied authority, emotional influence, and persistence of
advice across repeated interactions.
We also recommend that manufacturers evaluate these properties across relevant patient populations,
clinical conditions, and conversational contexts. Attachment-related risks may emerge gradually,
particularly during extended interactions, and may not be apparent from isolated responses.

Rather than requiring manufacturers to eliminate all personalization, CDRH should encourage
empirical evaluation of how personalization affects comprehension, trust calibration, and downstream
decision-making.

Question 3: CDRH seeks input on whether and when a GenAI-enabled function that results
in delivery of clinical information to patients, as opposed to HCPs, could present different or
higher risks, while also recognizing the potential benefits associated with improved patient
empowerment, engagement, and access to clinical information. What device characteristics,
output features, or safeguards might mitigate risks that could arise when a user lacks the
domain knowledge to independently evaluate an output, without unnecessarily
underestimating patient capability?

Patient-facing generative AI applications offer substantial potential benefits, including more timely
access to health information, improved patient understanding, and support when a clinician is not
immediately available. These benefits should not be discounted merely because patients lack formal
clinical training.

However, patient-facing systems may require stronger safeguards because the recipient may not be able
to identify an incorrect output or recognize when an apparently routine question involves clinically
significant risk.

We recommend an approach in which multiple complementary safeguards address different failure
modes. These should include clinically-grounded constraints, appropriate uncertainty
communication, reliable escalation pathways, and testing against foreseeable real-world inputs.

An important consideration is the reliability of the safeguards themselves. Some systems use generative
language models to evaluate or moderate the outputs of other generative models (‘LLM-as-a-judge’).
Such approaches can introduce additional model-dependent error or variability. For safety-critical
judgments, manufacturers should characterize the evaluator's sensitivity, specificity, reproducibility,
variance, and susceptibility to adversarial or out-of-distribution inputs.

Patient-facing systems should also be evaluated using realistic input variation, including misspellings,
colloquialisms, incomplete medical histories, contradictory information, differences in health literacy,
and population-specific language. The goal should be reliable performance across the intended
population.
Clinical claims should be supported by appropriate, current evidence, with uncertainty clearly
communicated where evidence is limited. In clinically ambiguous or high-consequence circumstances,
systems should be able to defer or escalate appropriately without over-refusing routine informational
requests.

Importantly, these safeguards should be assessed through observable behavior, not solely through
documentation that a safety policy or guardrail has been implemented.

Question 4: CDRH seeks input on whether and how the distinction between generalist and
specialist physicians might inform the assessment of risk for HCP-facing GenAI-enabled
functions. Under what circumstances, if any, might risk be affected when an HCP who lacks
the relevant clinical specialist knowledge receives information that falls within a specialist
area of practice? What device characteristics or safeguards might mitigate such risks?

The distinction between generalist and specialist clinicians is important, but risk assessment should
reflect the actual clinical workflow. We recommend involving both generalist and specialist clinicians in
the development and evaluation of relevant GenAI-enabled functions. Generalists can help assess how
a system is likely to be used in routine practice, while specialists can identify subtle errors,
contraindications, and clinically significant failure modes that might otherwise go undetected.

This is particularly important for systems that provide specialized information to users without
corresponding specialist training. A physician may appropriately rely on an AI system for support
outside their primary field but may be less equipped to recognize when its recommendations are subtly
incorrect.

Behavioral health provides a useful example. An AI system that repeatedly reassures or validates a user's
concerns may appear an empathetic sounding board. However, in the context of obsessive-compulsive
disorder, repetitive reassurance can reinforce maladaptive cycles. Similarly, uncritical affirmation of
delusional beliefs may be clinically inappropriate. Evaluations developed without relevant specialist
input may fail to distinguish appropriate emotional support from harmful reinforcement.

We recommend that clinical specialists contribute not only to reviewing outputs, but also to defining
failure categories, evaluation criteria, and clinically appropriate safeguards. Generalist clinicians should
participate in realistic workflow testing to determine whether the system communicates these
distinctions effectively to its actual users.

For applications whose intended use population could reasonably span multiple clinical domains, the
evaluation should also include relevant adjacent specialties and common comorbid conditions. A
device intended for one specialty may create risks outside that specialty through medication
interactions, overlapping symptoms, or treatment decisions.

Question 5: For multi-turn conversational GenAI-enabled devices that may migrate from
providing “non-directive” information to “action-directing” information over the course of
an exchange, how should risk be assessed across realistic conversational trajectories? How
could the intended use of such a device be characterized when its behavior is emergent across
a conversation?

Multi-turn conversational devices introduce risks that cannot be adequately characterized through
isolated prompts. Safety-critical behavior may change over the course of a conversation as information
accumulates, earlier details become less salient, and the user's presentation evolves.

One particularly important concern is context pollution. Clinically consequential information
introduced early, e.g., during patient history, may no longer influence later outputs appropriately
because of context-window limitations, summarization errors, retrieval failures, or the model's
inconsistent use of available information.

For example, a patient may disclose a serious medication allergy early in an interaction, then return to
medication-related questions much later. The continued absence of an allergic reaction does not make
that contraindication irrelevant. Similarly, a system instructed to provide non-directive information
may gradually transition toward individualized treatment recommendations as conversation context
accumulates.

We recommend that CDRH encourage evaluation of complete conversational trajectories, including
scenarios in which risks emerge gradually, relevant information is disclosed indirectly, users resist
escalation, and important details are separated by substantial conversational distance.

Testing should examine whether safety-critical constraints persist despite changes in topic, user
framing, and conversation length. Manufacturers should characterize the reliability of their memory
mechanisms, context-management strategies, and safety controls under realistic and adversarial
conditions.

Where appropriate, safety-critical information should be preserved in structured, persistent
representations rather than relying exclusively on the generative model to recall and correctly prioritize
it. Such representations should themselves be tested for accuracy, appropriate updates, and protection
against conflicting or misleading information.

We further recommend distinguishing turn-level safety from trajectory-level safety. A system may
produce individually plausible responses while its cumulative behavior becomes unsafe. The unit of
evaluation must therefore include the conversation and its clinical consequences, not merely each
response in isolation.

Question 6: For GenAI-enabled care escalation functions, CDRH is considering how the
evaluation may account for both under-escalation and over-escalation. How could
manufacturers characterize and weigh these two directions of error, given that they may not
be commensurable and that acceptable trade-offs may vary by clinical context?

Under-escalation and over-escalation should be characterized separately because their consequences
differ across clinical contexts and cannot always be reduced to a common error measure.

-​ Under-escalation may result in delayed treatment or preventable harm.
-​ Over-escalation can create unnecessary anxiety, increase healthcare utilization, impose financial
burdens, and undermine user confidence.

Repeated unnecessary escalation may also discourage patients from engaging with the system or
seeking appropriate care in the future.

We recommend defining escalation thresholds according to the severity of potential harm, time
sensitivity, uncertainty, availability of clinical supervision, and reversibility of an incorrect decision.

For instance, a clinician-supervised informational function may appropriately provide differentiated
risk information without reflexively escalating every ambiguous presentation. Conversely, an
autonomous function operating without timely human review may require a lower threshold for
escalation in high-consequence situations.

Clinical prevalence and patient heterogeneity should also inform evaluation. In rare or highly variable
conditions, the absence of a familiar symptom pattern may provide limited reassurance. However,
rarity alone should not automatically mandate escalation. The relevant consideration is whether
uncertainty and potential harm justify additional review.

CDRH should encourage manufacturers to report under-escalation and over-escalation as distinct
outcomes, with clinically justified thresholds for each. Evaluations should include true emergencies,
borderline presentations, and clearly non-urgent cases.

Particular attention should be given to escalation timing. A system that ultimately recommends
emergency care may still fail if it first provides several rounds of inappropriate reassurance or delays the
recommendation beyond a clinically meaningful point.
Question 7: Is the competency-based approach described above, i.e., device benchmarking
followed by clinical confirmation, a useful and appropriate framework for evaluating
GenAI-enabled devices?

We support the proposed competency-based approach, particularly the distinction between scalable
non-clinical benchmarking and clinical confirmation under representative conditions.

That said, representative, high-volume testing of open-ended generative systems is not impractical. As a
point of evidence: Circuit Breaker Labs has designed an evaluation system for monitoring of clinical AI
that can run hundreds of thousands of dynamic, multi-turn tests each day, building meaningful
clinical evidence at rapid speed.

Importantly, competency should not be understood as a static property established at the time of
initial evaluation. Generative AI-enabled devices may change because of modifications to underlying
models, prompts, guardrails, retrieval systems, user interfaces, or other components. Even changes not
explicitly intended to affect clinical or safety performance may alter behavior in a clinically significant
way.

We therefore recommend treating competency-based evaluation as the foundation of a continuous
assurance process. Premarket benchmarking would establish a characterized baseline, clinical
confirmation would provide evidence of performance in representative use, and postmarket
reassessment would determine whether that competency is maintained as the system evolves.

The core competencies and acceptance criteria remain stable to support meaningful longitudinal
comparisons. At the same time, individual test cases should be refreshed and expanded to account for
emerging failure modes, changing clinical practices, and previously unidentified populations or
conditions.

This would preserve standardization without allowing initial benchmark performance to become a
substitute for continued evidence of safety and effectiveness.

Question 8: How might the two-axis risk framework described in Section IV be considered
within the competency-based approach described above to help determine the level of
evidence needed for a premarket submission?

The two-axis framework could provide the foundation for determining the rigor and scope of
premarket evidence, but the additional risk modifiers discussed in Question 1 should also inform what
manufacturers are expected to demonstrate.
In particular, CDRH should consider the intended user, degree of personalization, clinical supervision
model, patient population, potential for extended interactions, and clinical dependencies extending
beyond the device's primary specialty.

A device providing general cardiovascular education, for example, should not necessarily face the same
evidentiary expectations as one generating individualized cardiovascular treatment recommendations.
Likewise, a cardiology-focused device may need to demonstrate appropriate handling of pulmonary
disease, renal impairment, and other relevant comorbidities when they influence its intended function.

We recommend that manufacturers establish a risk-to-evidence mapping before testing. This would
identify which clinically significant behaviors must be evaluated, which populations and deployment
conditions must be represented, and what acceptance thresholds are justified for each category.

This mapping would also provide a reproducible starting point for postmarket evaluation. When
device components change, manufacturers could determine which risk categories may have been
affected and conduct targeted reassessment, while retaining broader regression testing for unexpected
effects.

A standardized approach to mapping risk categories to evidence requirements could improve
regulatory predictability without imposing identical testing obligations on devices with materially
different risk profiles.

Question 9: Would a benchmarking structure such as the one described above be likely to
provide adequate evidence of clinical knowledge, analytic capabilities, safety behavior,
communication, and generalizability to support a reasonable assurance of safety and
effectiveness? Are there elements that are missing, redundant, or inappropriately categorized?
Are there externally developed standards that could be leveraged?

The proposed benchmarking elements provide a strong foundation. We particularly support the
inclusion of safety-critical escalation, scope adherence, uncertainty communication, robustness,
subgroup performance, and agentic capabilities.

However, we recommend some refinements.

First, CDRH should distinguish reusable evaluation criteria from reusable test instances. Standardized
clinical competencies, scoring definitions, and acceptance criteria are essential for comparability.
Repeated reliance on a fixed, publicly available set of prompts or data is less reliable, particularly when
models or applications are often optimized against those exact examples. The evaluation framework
should therefore support standardized evaluation metrics applied to both reference cases and
independently generated, sequestered, or refreshed scenarios.
Second, clinically significant context retention should be an explicit competency. Devices should
demonstrate that they maintain and appropriately apply relevant patient information across extended
interactions, including contraindications, allergies, comorbidities, and changes in clinical status. This
competency spans clinical reasoning, safety, robustness, and agentic behavior, but its importance
warrants explicit consideration.

Third, anthropomorphism, inappropriate agreeableness, and automation bias should receive greater
attention. A system may provide factually accurate information while encouraging excessive reliance or
reinforcing harmful beliefs. These risks can emerge through communication style, emotional
affirmation, and repeated interactions, particularly in behavioral health applications.

Fourth, clinical provenance should be assessed. Where a device presents information as clinically
established, users should be able to distinguish established guidance from emerging evidence, inference,
or uncertain claims. Unsupported assertions of authority should be considered separately from
ordinary factual errors.

Finally, the proposed elements overlap in clinically meaningful ways. For example, an agent that
reorders a medication without adequately considering a relevant contraindication may demonstrate
failures of clinical proficiency, context handling, and agentic execution simultaneously.

We do not recommend treating these categories as mutually exclusive. Instead, CDRH could retain the
organizing structure while allowing individual failures to map to multiple competencies, with careful
safeguards against double-counting in aggregate scores.

Question 10: CDRH recognizes that publicly available benchmarking assets may be subject to
data contamination, saturation, and limited real-world representativeness. How should a
sponsor establish that performance on a given benchmark predicts safe and effective
real-world behavior for the device’s intended use? What evidence should support the
construct validity of a benchmark used to gate device evaluation, and what role should
sponsor-developed benchmarks play given potential concerns around independence and
optimization to the test?

CDRH identifies a central challenge in generative AI evaluation: performance on a known benchmark
does not necessarily predict performance under novel, real-world conditions.

We recommend distinguishing the clinical construct being measured from the particular examples used
to measure it. Clinical risk categories, evaluation criteria, and acceptance thresholds should be clearly
specified and sufficiently stable to support reproducibility. However, the scenarios used to assess those
should be capable of evolving as new failure modes and patterns of use emerge.
A robust approach could combine common reference benchmarks, independent sequestered cases, and
dynamically generated scenarios tailored to a device's intended use. This would allow manufacturers to
assess established competencies while reducing overreliance on a finite set of recognizable inputs.

The construct validity of such benchmarks should be supported through qualified clinical review,
explicit links to clinically meaningful failure mechanisms, testing across realistic input variation, and
comparison with independent clinical adjudication where appropriate.

We also recommend evaluating measurement reliability separately from model performance. When a
scoring method produces inconsistent judgments across equivalent inputs or repeated assessments, an
apparent change in device performance may reflect measurement variation rather than an actual
change in safety. Manufacturers should characterize the repeatability, agreement with qualified
adjudicators, and limitations of their evaluation methods.

Independent evaluation organizations can help identify blind spots arising from sponsor-developed
test suites, particularly when they collaborate with clinical specialists. However, independence should
not preclude device-specific customization. A useful structure could include a common set of clinically
defined risks supplemented by independently developed scenarios addressing the manufacturer's
intended use, population, and known vulnerabilities.

Manufacturers should document what risks their systems may encounter, why those risks matter, and
independent evaluators can assess handling of these risks and provide detail on how the scenarios
represent intended use, and what important uncertainties remain.

Finally, benchmark maintenance should be treated as a continuing scientific activity. As underlying
models and clinical applications evolve, evaluation methods must be updated without sacrificing
reproducibility or longitudinal comparability.

Question 11: CDRH is considering that clinical confirmation for a GenAI-enabled device
might not require a prospective clinical study in every case, and has described above a range of
approaches of increasing rigor and patient exposure. How might a sponsor select and justify a
confirmation approach tailored to a device’s intended use and proportionate to the device’s
risk profile? Are there device types or risk profiles for which one or more of these approaches
would be insufficient or inappropriate? How might the anticipated distribution of real-world
inputs be taken into consideration? Are there other methods of clinical confirmation that
might help inform the evaluation of GenAI-enabled devices?

We support a risk-proportionate approach to clinical confirmation that allows manufacturers to select
methods appropriate to their device's intended use, degree of autonomy, and potential consequences of
failure. However, we recommend that the selection process begin with a structured characterization of
clinical risks rather than a determination of which study design is most convenient.

Manufacturers could first describe their device's risk profile using the proposed two-axis framework,
supplemented by the additional considerations discussed in our earlier responses. This assessment
should identify both the situations the device is intended to address and foreseeable circumstances in
which its outputs could cause harm.

For example, a cardiology-focused system may not explicitly address pulmonary disease, yet many
patients presenting with cardiovascular concerns also have respiratory conditions or overlapping
symptoms. A risk assessment that excludes these clinical intersections could overlook consequential
failure modes. CDRH could request additional evidence in areas where the intended function creates
foreseeable dependencies on adjacent clinical specialties.

We recommend that manufacturers document these anticipated risks and provide corresponding
performance evidence, including findings from qualified independent evaluators where appropriate.
Independent evaluators can help identify blind spots that may be overlooked when manufacturers
develop their own evaluation criteria, while clinical specialists and generalist physicians can provide
complementary perspectives on clinical complexity and actual patterns of use.

The anticipated distribution of real-world inputs should inform both test coverage and the amount of
testing performed. A device expected to support large volumes of patient interactions should be
evaluated across a sufficiently broad range of scenarios to characterize uncommon but clinically
significant failures.

High-throughput automated testing makes it feasible to examine many more scenarios than traditional
manual evaluation alone.

For perspective, our work at Circuit Breaker Labs has focused on scalable, clinically-informed testing
methods capable of evaluating large numbers of interactions efficiently. Such methods can supplement
clinical confirmation by systematically examining variation in patient characteristics, language,
comorbidities, and conversational trajectories.

It is also important to distinguish reproducibility of the evaluation method from variability in the
device being evaluated. Reproducible scoring methods can make longitudinal comparisons more
reliable, but their clinical validity must still be established through appropriate expert agreement and
other evidence.

Finally, we recommend a staged approach in which scalable benchmarking identifies areas of
uncertainty that then inform more resource-intensive confirmation methods. For instance, failures
involving complex comorbidities discovered through synthetic testing could guide targeted
retrospective analysis, standardized patient interactions, or shadow deployment.

This would allow manufacturers to use clinical evidence more efficiently while directing greater
scrutiny toward the conditions most likely to produce significant harm.

Question 12: How should sponsors achieve statistically meaningful performance
measurement for GenAI-enabled devices? Where synthetically generated inputs supplement
real patient data, how should sponsors account for differences between the synthetic and
real-world distributions when estimating performance, and under what conditions, if any, is
it appropriate to combine benchmarking evidence and clinical confirmation evidence to
support a single performance estimate?

Statistically meaningful performance measurement is particularly challenging for generative
AI-enabled devices because outputs may vary across repeated interactions, clinically significant failures
may be uncommon, and the real-world distribution of patient inputs may be incompletely
characterized.

We recommend that CDRH encourage large-scale, risk-stratified evaluation supported by
clinically-informed synthetic data, while clearly distinguishing performance under synthetic testing
conditions from estimated real-world performance.

Real-world patient data is essential for understanding how individuals actually interact with medical
AI, including the conditions they present with, their language patterns, and the frequency of different
clinical scenarios. However, particularly for emerging conversational and patient-facing applications,
available data may not adequately represent the full range of potential interactions.

Privacy-preserving analysis of real-world data can help characterize these patterns and inform the
construction of synthetic test scenarios. Clinical experts can then assess whether the generated cases are
medically plausible, reflect meaningful patterns of presentation, and cover relevant populations and
conditions.

Importantly, synthetic data should not merely reproduce the observed distribution of existing patient
interactions. Real-world datasets may systematically underrepresent rare conditions, underserved
populations, atypical symptoms, or clinically important scenarios that have not yet been encountered.
Synthetic testing creates an opportunity to intentionally examine such situations before they result in
real-world harm.

Advances in automated evaluation have made testing at the scale of hundreds of thousands or millions
of interactions technically feasible, as evidenced by the in-market tool developed by Circuit Breaker
Labs. We believe this capability should be leveraged to characterize performance across a broad range of
clinically meaningful scenarios, including variations in patient language, age, comorbidities, and
conversational histories.

We recommend separating three components of performance evidence:

-​ Real-world representative performance, estimated using data that reflects the anticipated
distribution of actual patient interactions.
-​ Targeted safety performance, assessed through deliberately enriched datasets emphasizing rare,
high-consequence, or otherwise difficult clinical scenarios.
-​ Measurement reliability, establishing that the evaluation methods themselves provide
zero-variance and clinically meaningful results.

We also recommend reporting uncertainty intervals, clinically relevant subgroup performance, and
failure severity rather than relying exclusively on a single aggregate score.

The objective should be to make statistically rigorous evaluation more accessible and scalable.

Question 13: For which clinical domains, device functions, or subpopulations is synthetic
data particularly well-suited, or particularly inadequate, as a supplement to realworld
evidence? What safeguards would mitigate the risk that synthetic data generated by models
of the same class as the device under evaluation reproduces the very performance gaps the
evaluation is intended to detect, particularly for underrepresented subgroups?

We believe synthetic data can make substantial contributions across clinical domains, device functions,
and patient populations, particularly where real-world data is insufficient to characterize the full range
of foreseeable inputs.

Conversational and behavioral health applications illustrate both the need for synthetic data and the
importance of developing it carefully. Unlike many structured clinical datasets, patient-facing
conversations can include indirect disclosures, evolving symptoms, incomplete histories, informal
language, and substantial variation in how individuals describe similar experiences. Two patients
presenting with the same underlying clinical concern may use entirely different language, and relatively
small wording differences may influence whether an AI system recognizes a safety-critical condition or
responds appropriately.

For example, a patient describing significant psychological distress may not use formal diagnostic
terminology or explicitly identify the condition they are experiencing. A system evaluated primarily on
textbook descriptions may perform well on a benchmark while failing to recognize clinically
meaningful variations encountered in practice.
Synthetic testing offers a way to systematically explore these differences. It can generate variations in
wording, age-related language, clinical context, comorbidities, symptom progression, and conversation
length while preserving a clinically specified underlying scenario.

However, the clinical risks being tested should not be done solely by the same class of generative
models being evaluated (‘LLM-as-a-judge approaches’). Otherwise judging may reproduce overlapping
assumptions or blind spots.

Clinical experts should establish the relevant risk categories and behavioral expectations. Synthetic
scenarios can then be generated through complementary techniques, including structured simulation
and steering. Finally, outputs should demonstrate reliability and clinical validity.

At Circuit Breaker Labs, our approach emphasizes clinically-informed risk criteria and reproducible
evaluation methods rather than solely on one generative model to independently produce scenarios
and another to judge the results. We believe this separation is an important design principle for
developing trustworthy evaluation infrastructure.

The involvement of clinicians experienced with underrepresented populations is especially important.
Available patient datasets may not fully reflect how individuals across age groups, dialects, health
literacy levels, cultural contexts, and co-occurring conditions describe their concerns. Specialists and
generalists can identify clinically meaningful variations that would otherwise be absent from
test-generation processes.

We also recommend that synthetic datasets be continuously expanded and validated as new evidence
emerges from clinical practice, postmarket monitoring, patient feedback, and independent evaluations.

The appropriate goal is therefore not to replace real-world evidence, but to extend the range of
conditions under which safety can be examined before patients encounter those conditions themselves.

Question 14: For open-ended device outputs, how should performance comparators and
acceptance criteria be selected? When a panel of qualified clinicians serves as the comparator,
how should the applicable standard (for example, the standard of care versus the performance
of a median clinician in practice) be defined and justified? Should generalist or specialist
physicians be used as a performance standard? When should human-AI team performance,
rather than the device operating alone, serve as the basis for evaluation?

We recommend that performance comparators and acceptance criteria be determined according to a
device's intended use, level of autonomy, patient population, and clinical consequences of failure.

The proposed axis framework could provide a starting point for establishing baseline expectations.
Manufacturers should define the functions within the device's intended scope, the behaviors that
would constitute unacceptable deviations from that scope, and the clinical competencies required for
safe operation.

Minimum requirements could include recognition of clinically significant contraindications,
appropriate escalation in emergencies, adherence to the device's intended scope, and avoidance of
clearly harmful recommendations. These should be grounded in applicable clinical standards and
established with input from qualified specialists.

Comparative performance could then be assessed against clinicians or human-AI teams appropriate to
the intended workflow.

The relative contributions of generalist and specialist physicians deserve particular consideration.

-​ Specialists are especially valuable in defining performance standards for complex or
high-consequence clinical cases that genAI models have not trained on. Their experience with
unusual presentations, treatment complications, and clinically significant exceptions can reveal
important failure modes that might not be apparent from routine cases alone.
-​ Generalist clinicians provide a complementary perspective. In many healthcare settings, they
are the first point of contact for patients and must make decisions before the clinical picture is
complete. Their experience can help characterize how AI will be used during early information
gathering, initial assessment, referral, and routine patient interactions.

We therefore recommend involving both groups in evaluation design while assigning responsibility for
defining particular clinical acceptance criteria to professionals with qualifications appropriate to the
clinical question.

The degree of human involvement in analogous real-world operation should also determine which
comparator is most relevant. If a system is intended to support a clinician who reviews and acts on its
recommendations, the combined performance of the clinician and AI may be a central endpoint. If the
system makes decisions or takes actions autonomously, its independent performance may be assessed.

Patient-facing systems present an additional consideration. The absence of professional review at the
point of interaction may necessitate more conservative safety requirements, stronger escalation
pathways, and ongoing monitoring. However, retrospective human review should not automatically be
treated as equivalent to clinician supervision before a consequential response reaches a patient.

Finally, because generative systems can produce many different clinically acceptable responses, scoring
should focus on whether the response satisfies clinically meaningful criteria rather than whether it
matches a single reference answer.
Question 15: Are there ways in which performance might be assessed relative to the care,
technology, or course of action likely to occur in the absence of the device, rather than to the
comparators described in this section? What approaches might be used to identify and justify
a comparator such as unaided clinical judgment, delayed specialist review, or no intervention?

We strongly support consideration of the care or intervention that would realistically occur in the
absence of a generative AI-enabled device. This is particularly important for applications intended to
improve access-to-care.

In behavioral health, for example, the relevant alternative to an AI-enabled support tool may not
always be an appointment with a licensed therapist. For some individuals, the alternative may be
delayed care, limited access to clinical information, reliance on informal support, or no intervention at
all.

Evaluating a device exclusively against an idealized healthcare scenario may therefore underestimate its
potential benefits, particularly for populations facing substantial barriers to access.

However, we recommend distinguishing benefit relative to the available alternative from the device's
absolute safety. A device may offer meaningful benefits where no timely care would otherwise be
available while still creating risks that require independent assessment.

One particularly important question is whether an AI interaction introduces a clinically significant
harm that would not otherwise have occurred.

For example, in behavioral health, a user experiencing ordinary sadness may seek conversational
support. An AI system that amplifies emotional distress, or encourages inappropriate dependence
could potentially worsen the user's condition, even if it never provides an explicitly dangerous
instruction.

Conversely, a well-designed system may provide helpful information, encourage appropriate
care-seeking, or offer beneficial support during periods when professional care is unavailable.

We recommend that clinical confirmation consider whether the device produces a net benefit, neutral
effect, or additional harm relative to a realistic counterfactual, while separately measuring serious
adverse outcomes that should not be offset by average benefit.

Potential comparators could include usual care, delayed specialist review, unaided clinical judgment,
available digital resources, or no intervention, depending on the intended use and actual deployment
setting.
For devices performing clinical reasoning or decision-making, comparison with qualified clinicians
remains important. Specialist judgment may be particularly relevant for assessing complex cases, while
generalist judgment can help characterize the decisions likely to occur earlier in a patient's care
pathway.

When feasible, studies could examine not only whether the device's decision agrees with clinical
experts, but also whether exposure to the device changes patient understanding, decision-making,
care-seeking behavior, or clinical outcomes.

Such assessments should account for baseline patient characteristics and other factors that may
influence outcomes independently of the AI system.

We believe this approach would allow FDA to recognize the substantial potential of generative AI to
expand access to healthcare without overlooking the possibility that some applications create new
harms rather than simply performing existing clinical functions less effectively.

Question 16: Should independent third parties be involved in some or all aspects of a
competency-based approach, including device benchmarking and clinical confirmation? If so,
in what ways might qualified, independent third-party participation contribute to this
approach, and what qualifications and independence criteria should apply? Are there aspects
of the assessment for which third-party involvement would be impractical or inadvisable?
What safeguards or program design features would be critical to prevent third-party
participation from limiting competition or preventing innovation?

We strongly support the involvement of qualified independent third parties throughout the
competency-based evaluation process. Blind spots are natural, and independent scrutiny is particularly
valuable for identifying failure modes that may not be apparent to the organizations developing a
device.

Independent evaluators can contribute during premarket development by identifying vulnerabilities,
assessing clinically significant behaviors, and providing evidence that informs design decisions and
safeguards. This need not be limited to a final assessment immediately before submission. Early and
iterative evaluation can help manufacturers address risks while products are still being developed,
rather than discovering them after deployment.

The resulting evidence can also establish a baseline for postmarket monitoring. By evaluating devices
against consistent clinical factors over time, independent organizations can help manufacturers
identify emerging vulnerabilities, quantify regressions, and determine whether modifications have
affected previously demonstrated competencies.
We recommend that qualifications reflect both clinical and technical expertise. Evaluators should
demonstrate the ability to define clinically meaningful failure categories, incorporate perspectives from
relevant specialists and patient populations, conduct sufficiently broad and high-volume testing, and
produce reproducible, auditable findings. Technical testing capacity alone should not be sufficient;
evaluation methods must also demonstrate clinical validity.

Importantly, independent evaluation need not require access to a manufacturer's proprietary models,
training data, system prompts, or internal architecture. For example, at Circuit Breaker Labs, we
conduct external evaluations through API endpoint-based access to deployed systems, allowing
manufacturers to retain control of their underlying intellectual property. This approach can reduce
barriers to participation while enabling independent assessment of observable device behavior.

Finally, we suggest that CDRH avoid creating an evaluation framework that inadvertently concentrates
market access among a small number of organizations. Qualified for-profit companies, nonprofit
laboratories, academic groups, and other independent evaluators should be able to participate under
transparent, technology-neutral qualification criteria.

Independence should be defined by the integrity of the evaluation process, not the legal or commercial
structure of the organization conducting it. Such an approach would encourage competition in
evaluation methodologies, encourage innovation in the evaluation process itself, protect manufacturer
intellectual property, and support the development of safety infrastructure capable of keeping pace
with innovation.

Question 17: CDRH is exploring whether a competency-based approach could be applied to
devices that incorporate a variety of underlying model architectures, such as multimodal
vision-language models and generative or predictive world models. Are there aspects of the
described approach that might be ineffective or inapplicable to these devices? What
additional or distinct considerations might need to be addressed?

We believe the competency-based approach can be applied across different model architectures,
provided that the competencies are defined according to the device's intended clinical function rather
than the particular architecture of its underlying model.

The central regulatory question should remain whether the final deployed device performs its intended
function safely and effectively under foreseeable conditions. However, multimodal devices introduce
additional failure modes that may not be adequately captured through evaluations designed primarily
for text-based systems.
For example, a voice-enabled device may respond appropriately to an accurate transcript while failing
to recognize clinically significant information when it is spoken with an unfamiliar accent, interrupted,
or affected by background noise. We have observed this with our voice red-teaming offering at Circuit
Breaker Labs. Similarly, an image-based system may accurately interpret a clear clinical image but
behave differently when the image is incomplete, poorly illuminated, or inconsistent with
accompanying patient information.

We therefore recommend evaluating both individual components and the complete end-to-end clinical
interaction. Component-level testing can help identify whether a failure originated in speech
recognition, image interpretation, retrieval, reasoning, or another process. However, successful
performance by individual components does not necessarily establish that the complete device will
behave safely when those components interact.

An especially important competency is the handling of conflicting information across modalities. For
example, a patient's verbal description of a condition may conflict with an image, a physiological
measurement, or a previously documented clinical history. Evaluation should establish whether the
device appropriately recognizes uncertainty, gathers additional information, or defers when necessary
rather than arbitrarily prioritizing one input source.

Multimodal evaluation should also consider patient-specific communication patterns, including
accents, dialects, age-related language differences, interrupted speech, and changes in how users present
information across the course of an interaction.

Multi-participant red-teaming, an offering Circuit Breaker Labs, is also critical to consider as some
AI-enabled medical devices may involve one or more patients and clinicians interfacing with an AI
product, where risk is between human-AI and human-human.

For generative models, additional considerations may include the accuracy of modeled state transitions,
the system's response to uncertainty, and the consequences of incorrect predictions over multiple steps.
A plausible simulated future state does not necessarily establish that the predicted outcome is clinically
reliable.

These systems should be assessed through realistic scenarios in which initial information may be
incomplete, assumptions may be incorrect, and errors can accumulate over time.

Question 18: CDRH is considering whether it may be appropriate to accept greater
premarket uncertainty regarding a GenAI-enabled device’s benefit-risk profile through
greater reliance on postmarket monitoring. Under what conditions might such an approach
be appropriate, and what characteristics of a monitoring program would need to be in place
to justify reduced premarket evidence? Are there device types or risk profiles for which this
approach would not be appropriate?

We recognize the rationale for considering greater reliance on postmarket monitoring, particularly
because generative AI-enabled medical devices may operate in settings where real-world usage patterns
and the full distribution of clinical risks are not yet well characterized.

However, postmarket monitoring should complement a meaningful premarket evidence baseline. This
is particularly important for emerging patient-facing applications. In behavioral health, for example,
available real-world data may not adequately represent the range of ways patients discuss sensitive
conditions. Privacy constraints, limited data availability, and the evolving nature of human-AI
interactions can make it difficult to construct comprehensive evaluation datasets from historical
conversations alone.

We therefore recommend that manufacturers use clinically-informed synthetic scenarios to supplement
available real-world evidence during premarket evaluation. These scenarios can systematically explore
clinically plausible variations in patient characteristics, language, symptom presentation,
comorbidities, and conversational trajectories.

High-volume testing is especially valuable in this context because it enables manufacturers to examine a
broad range of foreseeable failures before users encounter them. Clinical experts should be involved in
defining the risks being tested and evaluating whether synthetic scenarios reflect meaningful clinical
situations.

Postmarket monitoring can then assess whether performance remains consistent with that baseline,
whether changes introduce new vulnerabilities, and whether previously unidentified failure modes
emerge as real-world evidence accumulates.

Greater reliance on postmarket evidence may be reasonable when the device has demonstrated
adequate initial safety, the remaining uncertainties can be monitored meaningfully, and mechanisms
exist to identify and remediate clinically significant problems in a timely manner.

For patient-facing conversational systems, human psychology introduces further considerations. A
device's outputs may influence trust, emotional reliance, care-seeking behavior, and subsequent
disclosures. These effects may develop across extended interactions and should not be assumed to be
captured by conventional single-turn benchmarks.

We therefore recommend an approach in which robust premarket synthetic and clinical evaluation
establishes a safety baseline, while continuous monitoring expands the evidence base and detects
deviations after deployment.
This would support innovation without treating patients as the primary means of discovering risks
that could reasonably have been identified before market entry. Humans should not be Guinea pigs.

Question 19: Please comment on the potential approaches to postmarket performance
evaluation, including periodic re-benchmarking, sample-based clinician review, and
performance degradation monitoring. What additional approaches should CDRH consider,
and how should the cadence and triggering events for reassessment be determined?

We support periodic re-benchmarking, sample-based clinician review, and performance degradation
monitoring. However, we recommend that CDRH explicitly distinguish event-triggered reassessment
from time-based reassessment, and consider both essential components of postmarket oversight.

Event-triggered evaluations are valuable when manufacturers make known changes that may affect
device performance. Examples include modifications to the underlying foundation model, prompts,
retrieval architecture, safety controls, guardrails, or clinical functionality.

However, manufacturers may not always know when their systems have changed in clinically
significant ways. Third-party model providers may modify hosted services, previously unrecognized
vulnerabilities may emerge, and changes in deployment conditions may affect behavior without a
clearly identifiable software modification.

For this reason, postmarket monitoring should not depend exclusively on manufacturers recognizing
that a change has occurred.

We recommend recurring re-benchmarking against established clinical safety criteria, even when no
known modification has taken place. Such evaluations could identify clinically meaningful regressions,
newly emerging vulnerabilities, or changes in the consistency of safety-critical behavior.

The appropriate cadence should be proportionate to the device's risk profile, anticipated usage, clinical
setting, and rate of technical change. Monthly or quarterly evaluation may be appropriate for some
applications, while higher-risk functions may require more frequent monitoring.

Clinical expertise should remain central throughout this process. Clinicians should contribute to
defining evaluation criteria, identifying emerging clinical risks, and adjudicating cases where automated
methods cannot reliably determine the clinical significance of an output.

We further recommend that monitoring include refreshed or independently developed challenge cases
to help detect vulnerabilities that may not appear in previously used benchmarks.

The objective should be to create a monitoring system that identifies both regressions in known
competencies and failures that were not anticipated during initial evaluation.
Question 20: Please comment on whether the proposed approaches to postmarket monitoring
can be facilitated by machine-based supervisory agents. What considerations, including the
evaluation and reliability of the supervisory agent itself, should CDRH take into account for
such an approach?

Machine-based supervisory systems can absolutely play an important role in scalable postmarket
monitoring, particularly where devices produce large volumes of open-ended outputs and tests are
necessary for generative AI systems, but cannot feasibly be reviewed individually by clinicians.

However, we caution against assuming that one generative model can reliably validate another.
LLM-as-a-judge approaches exhibit several documented limitations, including:

-​ Verbosity bias, where longer or more elaborate responses may receive preferential judgments
even when they are not more clinically appropriate.
-​ Position bias, where preference between responses can change depending on the order in
which they are presented.
-​ Self-enhancement bias, where AI-generated outputs, irrespective of model family, are favored.
-​ Judgment variability, where repeated evaluations of the same output may yield different scores,
a fundamental technical vulnerability in generative AI evaluators like LLM-as-a-judge.

These concerns are especially significant when supervisory systems are used to determine whether a
safety-critical behavior has improved, degraded, or remained unchanged.

A broader concern is correlated error. When the device under evaluation and the supervisory model
share similar assumptions, training influences, or failure modes, agreement between the two may create
a false impression of independent validation.

We therefore recommend that CDRH encourage the use of complementary and independently
validated evaluation mechanisms, including non-generative methods, specialized scoring systems, and
qualified clinician review where appropriate.

Non-generative evaluation techniques can be designed to produce reproducible numerical scores while
assessing clinically relevant contextual features. Circuit Breaker Labs, for example, develops
clinically-informed machine-based evaluation methods that do not rely on a general-purpose LLM to
assign final judgments.

We recommend that manufacturers characterize the reliability of any supervisory evaluation system by
requesting repeated assessments of identical outputs.

CDRH should also consider evaluation throughput and operational scalability. A central advantage of
machine-based supervision is the ability to evaluate substantially larger volumes of interactions than
manual review alone. However, this advantage is meaningful only when the evaluation method is
sufficiently accurate, reliable, and clinically appropriate.

Ultimately, the supervisory system should itself be treated as a measurement instrument whose validity,
reliability, and limitations must be demonstrated. A changing device cannot be meaningfully
monitored using an evaluation method whose own behavior is insufficiently characterized.

Question 21: What roles might clinicians, healthcare institutions, professional societies,
standards-setting bodies, and other stakeholders appropriately play in postmarket
monitoring, and how can these roles be structured without diffusing manufacturer
accountability?

All these stakeholders have valuable roles in postmarket monitoring, but CDRH should also recognize
the important contributions of independent, for-profit evaluation organizations. Given the pace at
which generative AI is advancing, a sustainable safety ecosystem must be capable of innovating at a
comparable pace to for-profit manufacturers. Commercial evaluation organizations can have incentives
to invest in scalable testing infrastructure, develop new methodologies, and rapidly adapt to emerging
risks and new capabilities. Their involvement should be encouraged alongside academic institutions,
professional societies, clinicians, and other stakeholders.

Shared participation in monitoring need not mean shared accountability for remediation.
Manufacturers can remain responsible for defining their device's risk profile and demonstrating that
appropriate safeguards are in place. Independent evaluators and other stakeholders can supplement
this work by identifying previously unrecognized vulnerabilities, assessing performance against
established clinical criteria, and detecting emerging risks through postmarket monitoring and reporting
those back to manufacturers.

When monitoring identifies a clinically significant issue, the manufacturer should be responsible for
investigating and addressing it within a timeframe proportionate to its severity. Where remediation
cannot be completed within that timeframe, manufacturers should document a corrective action plan,
interim risk mitigations, and an expected resolution timeline, with reporting to FDA where applicable.
Particularly serious risks may require immediate intervention rather than awaiting completion of a
remediation plan.

This approach allows different stakeholders to contribute their expertise without obscuring
responsibility. Independent organizations can identify and characterize failures, clinicians and
professional societies can inform their clinical significance, and manufacturers remain accountable for
maintaining the safety and effectiveness of their devices.
Question 22: CDRH envisions that a manufacturer’s premarket competency-based assessment
might serve as a baseline against which post-deployment modifications could be re-evaluated.
How might the extent of re-benchmarking or other evidence be scaled to the nature and
expected impact of a given modification? For example, are there categories of change that
might not significantly affect safety or effectiveness of a GenAI-enabled device, or might be
appropriately managed within a sponsor’s quality management system versus requiring FDA
premarket review and authorization before implementation? Are there categories of changes
that might be appropriate for inclusion in a PCCP?

We recommend that post-deployment reassessment be proportional to the potential clinical
consequences of a modification rather than solely its technical complexity or the manufacturer's
intended purpose for the change.

A relatively minor software modification may have outsized implications for device safety. For example,
changes to a system prompt, knowledge retrieval mechanism, context management strategy, or
underlying foundation model may alter how a device interprets contraindications, recognizes clinical
emergencies, or responds to patients outside its intended scope.

We propose a risk-based re-benchmarking framework built on the premarket competency assessment.

Manufacturers should maintain a mapping between their device's components, intended functions,
clinically-significant failure categories, and established acceptance criteria. When a modification occurs,
this mapping can help identify the competencies potentially affected and the evidence needed to
demonstrate continued performance.

For example, a change to the user interface that does not materially affect clinical information or user
decision-making may warrant limited reassessment. A change to the underlying language model, safety
architecture, or conversational memory system may warrant broader testing, including multi-turn
scenarios and relevant clinical subpopulations.

However, even targeted reassessment should be supplemented by cadenced safety-critical regression
tests. Generative systems can exhibit unexpected interactions between components, and manufacturers
should not assume that a modification affects only the functions they intended to change.

We recommend distinguishing regression testing from discovery-oriented testing. Regression testing
determines whether established competencies have been maintained. Discovery-oriented testing uses
refreshed, independently developed, or adversarial scenarios to identify new vulnerabilities that may
emerge unexpectedly.
Changes that fall within a prospectively defined and appropriately validated modification framework
may be suitable for management through a PCCP, consistent with applicable regulatory requirements.
Changes that introduce new intended functions, substantially alter risk exposure, or exceed the
boundaries of an established PCCP may require additional evidence or FDA review.

Importantly, the amount of evidence should be informed by the severity and detectability of potential
harm, the degree of clinical autonomy, and the availability of effective downstream safeguards.

This approach would make re-benchmarking more efficient and predictable while recognizing that the
safety impact of a modification cannot always be inferred from the size or apparent significance of the
underlying engineering change.

Question 23: How might PCCP concepts or other change-control approaches be adapted for
GenAI-enabled devices when the nature or scope of future modifications cannot be fully
prespecified?

PCCPs are valuable for modifications manufacturers can reasonably anticipate, including changes to
underlying models, prompts, retrieval systems, or other components that may affect device
performance. However, generative AI-enabled devices present an additional challenge: clinically
meaningful behavioral changes may occur even when the manufacturer has not knowingly
implemented a modification.

For example, a third-party foundation model provider may update a hosted model without providing
sufficient visibility into the change. Even when the model identifier remains unchanged, differences in
backend configuration or other dependencies may potentially affect performance.

We therefore recommend complementing modification-triggered PCCPs with proactive, recurring
evaluation. Rather than testing only when a manufacturer anticipates that something has changed, a
device could be periodically reassessed against its established clinical competencies and safety
requirements, regardless of whether a modification has been identified.

This would create two complementary safeguards: predefined change-control procedures for
anticipated modifications, and ongoing monitoring to detect unexpected changes, previously
unidentified vulnerabilities, or performance degradation.

Importantly, the monitoring framework should define which safety-critical behaviors must remain
consistent, how changes will be measured, what constitutes a clinically significant regression, and what
corrective actions would follow. Evaluation methods should produce numerical, sufficiently
reproducible results that can be tracked to distinguish changes in device performance from variability
in the measurement process itself.
This approach would allow manufacturers to prepare for the reality that not every future change can
be predicted, without requiring them to prespecify every possible modification. The central principle is
that manufacturers may not know what will change, but they can define what safety-critical behaviors
must continue to hold.

Question 24: For devices built on third-party foundation models, changes to the underlying
model may be initiated by the third-party model developer rather than the device
manufacturer. How can a manufacturer detect, evaluate, and respond to such changes in a
timely manner? What mechanisms—for example, contractual, technical, or through a
PCCP—could provide reasonable assurance that third-party developer-initiated changes do
not compromise the safety or effectiveness of the device?

Third-party foundation model dependencies create a significant challenge because manufacturers may
not control, anticipate, or even be notified of changes that affect their device's clinical performance.

We recommend that manufacturers implement ongoing behavioral monitoring and comparative
evaluations of their deployed systems. Independent evaluation organizations, including companies
such as Circuit Breaker Labs, can support this process through recurring safety assessments and
controlled comparisons, or bakeoffs, between existing and prospective model configurations. For
example, when a manufacturer considers replacing an underlying foundation model, the current and
proposed versions of the complete device can be evaluated against common clinical safety criteria,
including relevant failure modes and population-specific requirements. Because Circuit Breaker Labs
produces numerical, explainable results, manufacturers can identify potential improvements and
regressions before implementing the change.

The same approach can help detect unexpected changes introduced by third-party model providers. By
periodically evaluating the deployed device against an established baseline, manufacturers can identify
behavioral regressions even when they have not received notification of an underlying modification.

We believe recurring evaluation should be a reasonable expectation for GenAI-enabled medical devices,
with frequency proportionate to clinical risk and the pace of system change. Monthly or quarterly
assessments may be appropriate for some applications, while high-consequence systems or those
dependent on frequently updated foundation models may warrant more frequent or event-triggered
testing.

Importantly, periodic testing should complement, rather than replace, evaluations conducted before
known model changes are deployed. Manufacturers should also consider technical and contractual
mechanisms such as model-version pinning, advance update notifications, access to previous versions,
and rollback capabilities where available.
Advances in automated, clinically-informed evaluation make broad regression testing substantially
more scalable than exclusively manual approaches. This creates an opportunity to maintain rigorous
oversight without imposing the same cost and operational burden as repeated full-scale clinical studies.

Ultimately, the appropriate regulatory expectation should not be that manufacturers can prevent every
third-party model change. It should be that they maintain reliable mechanisms to detect clinically
significant changes, assess their consequences, and take timely corrective action, regardless of whether
the change originated within or outside their organization.

Question 25: Would voluntary Foundation Model MAFs be practical for foundation model
developers to provide and to keep current, and what content might they include? Given that
participation would be voluntary and that model developers may have limited incentive to
disclose safety-relevant information, what would make such a program sufficiently useful for
premarket review? Are there alternative mechanisms CDRH should consider for obtaining
information about underlying foundation models?

We appreciate the potential value of voluntary Foundation MAFs, particularly as a mechanism for
improving transparency about underlying models used in medical devices. However, we are concerned
that voluntary disclosure alone may not provide sufficiently current or application-specific evidence of
safety.

Foundation model developers may be slow to continually update regulatory documentation,
particularly when they are not themselves developing the downstream medical device and may not be
attuned to the specific risks.

A more fundamental limitation is that foundation model safety is not equivalent to device safety. A
foundation model's performance depends substantially on how it is incorporated into a specific
application. System prompts, retrieval-augmented generation, conversational memory, and guardrails
can all introduce new vulnerabilities or alter existing ones.

Similarly, the clinical safety requirements for a behavioral health application may differ substantially
from those for a medication management system, even when both use the same underlying foundation
model.

For these reasons, we would not recommend relying on a voluntary MAF as the primary mechanism
for establishing or maintaining the safety of downstream devices.

Instead, CDRH could consider a complementary framework involving ongoing independent
foundation model evaluations and application-specific testing.
For example, qualified independent evaluation organizations could periodically compare major
foundation models against common healthcare-relevant competencies, including clinical knowledge,
safety-critical recognition, uncertainty communication, and consistency of behavior.

Such comparative evaluations, “bakeoffs” as Circuit Breaker Labs refers to them, could provide useful
evidence about the relative strengths, limitations, and changing performance of different underlying
models.

CDRH could consider maintaining or referencing a body of independently-generated evaluation
evidence, with clear documentation of model versions, assessment methods, intended contexts, and
known limitations. This could provide reviewers with additional information when assessing devices
built on commonly used foundation models.

At Circuit Breaker Labs, comparative evaluation of different model configurations is one approach we
use to help characterize clinically relevant performance differences. We believe a similar methodology
could contribute to a broader evidence ecosystem without requiring foundation model developers to
disclose proprietary training information or pending non-public updates.

However, these comparisons should not be interpreted as general authorization or certification of an
underlying model for medical use. A model that performs well on common healthcare benchmarks
may still behave unsafely when integrated into a particular device or altered via RAG, etc.

If this approach is taken, we recommend treating foundation model evidence as a useful reference that
can inform, but not replace, application-specific evaluation.

Voluntary MAFs could still contribute important information, including version identification,
known limitations, major safety-relevant changes, and update-notification practices. These disclosures
would be most useful when combined with proactive, recurring behavioral testing rather than relied
upon as a static representation of model safety.

The objective should be to establish a regulatory evidence framework that remains informative even
when model providers make unanticipated changes, use cases evolve, or the performance characteristics
of a previously evaluated model shift over time.

Question 26: Are there additional considerations that inform the premarket and postmarket
evaluation of agentic GenAI-enabled devices, beyond those applicable to nonagentic
GenAI-enabled devices? How should the elevated risk associated with autonomous multi-step
action, tool use, and reduced opportunity for human review be reflected in acceptance criteria
and oversight?
Agentic generative AI-enabled devices warrant additional scrutiny because their risks may emerge
across sequences of decisions and actions rather than from any individual output.

This is particularly important for systems capable of using external tools, maintaining conversational
context, modifying records, initiating clinical workflows, or performing actions with limited
opportunities for human review.

We recommend that CDRH place particular emphasis on high-volume, clinically-informed,
trajectory-level evaluation in both premarket and postmarket settings.

Generative AI systems can produce variable outputs in response to similar inputs, and agentic systems
introduce further complexity because small differences in an early decision may propagate through
subsequent actions. Evaluating a small number of isolated examples is therefore unlikely to adequately
characterize the range of potential failures.

Manufacturers should evaluate realistic multi-step scenarios, including variations in the order of
information, incomplete patient histories, conflicting inputs, changing clinical conditions, and
extended interactions in which important safety information may become less salient.

A central concern is context degradation. An agent may initially recognize a clinically significant
contraindication but subsequently fail to account for it when performing an action many steps later.
Likewise, an agent may correctly execute individual tool calls while producing an unsafe overall
outcome because it has lost track of the clinical context or failed to reassess whether its planned actions
remain appropriate.

These risks are not hypothetical properties of tool use alone. They arise from the interaction between
the underlying generative system, its context-management mechanisms, the tools it can access, and the
clinical consequences of its decisions.

Our experience conducting trajectory-based evaluations in behavioral health has reinforced the
importance of assessing complete interactions. The order in which information is introduced, the
progression of the user's circumstances, and the accumulated conversational context can materially
affect how safety safeguards are invoked and whether a system maintains appropriate boundaries.

We recommend that acceptance criteria therefore address not only the correctness of individual
responses or tool calls, but also the safety of the complete action sequence.

Relevant competencies should include persistent recognition of clinically significant information,
adherence to authorized scope, appropriate human-approval checkpoints, resistance to prompt
injection, handling of erroneous tool outputs, and the ability to stop or safely recover when
circumstances change.
Equally important is the scientific reliability of the evaluation process itself. Agentic systems produce
complex, open-ended trajectories, but this does not mean their performance must be assessed through
subjective or irreproducible judgments.

Evaluation methods can incorporate clinical context and nuanced behavioral criteria while producing
structured, quantitative, and auditable results. We recommend that manufacturers demonstrate how
their scoring methods distinguish clinically significant failures, how reliably those methods agree with
qualified clinical judgment, and whether repeated evaluation of identical trajectories produces
consistent results.

A zero-variance evaluator should assign the same score to an identical trajectory. Such reproducibility is
particularly valuable for identifying safety regressions across device versions and postmarket updates.
Generative LLM-as-a-judge approaches may provide useful qualitative assessments, but their
variability, systematic biases, and potential correlated errors should be characterized before they are
relied upon for safety-critical determinations. A changing evaluation method can make it difficult to
determine whether a device has actually improved or deteriorated.

Finally, we recommend that agentic device evaluation combine large-scale synthetic testing with
clinically representative confirmation and continuing postmarket reassessment. Testing should be
sufficiently broad to explore complex and uncommon trajectories, while statistical claims should
account for the representativeness and dependence of generated scenarios.

The objective should be to establish reproducible evidence that clinically significant safeguards persist
across complex action sequences, changing contexts, and evolving system configurations.

As generative AI becomes increasingly capable of acting rather than merely providing information, the
importance of measuring these properties will only grow.