University of California Health (Cora Han)

See attached file(s)

Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

October 19, 2026

Ensuring Safe, Effective, and Accountable Generative AI in
Health Care: University of California Health Response to
FDA’s Discussion Paper on Generative AI-Enabled Medical
Devices

Dear U.S. Food and Drug Administration,
Thank you for the opportunity to provide comments in response to
Considerations for the Regulation of Generative AI-Enabled Medical
Devices (Docket No. FDA-2026-N-7874). On behalf of University of
California Health (UC Health), we appreciate the opportunity to respond
and hope that our comments may inform FDA’s efforts to enable
innovation while protecting patients.
At University of California Health, we serve millions of patients each year
across urban, rural, and underserved communities and educate
thousands of trainees, with our footprint spanning six academic health
centers and 21 professional schools. Our health systems select, deploy,
govern, and monitor AI-enabled technologies every day, including
generative AI tools that, with appropriate human monitoring and review,
draft documentation, summarize records, complete administrative tasks,
and in some cases, communicate directly with patients. This experience
gives us a deployer’s view of how generative AI performs in real clinical
workflows.
We support FDA’s risk-proportionate, total product life cycle approach
and its proposal to evaluate the device as configured and deployed. Our
comments focus on where health system experience can strengthen the
framework, including how risk is assessed, how competency is
demonstrated, how evidence is generated after deployment, how
changes to underlying models are managed, and how responsibilities are
shared between manufacturers and the health systems that deploy their
products.
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

In summary, we recommend that FDA:
1. Refine the two-axis risk framework with deployment-context modifiers and a distinction
between levels of human supervision, aligned with Appendix S of the AMA CPT code
set.
2. Evaluate the deployed product through competency-based testing that includes failure,
mode, and effective analysis (FMEA) guided adversarial testing, followed by clinical
confirmation that includes counterfactual comparisons for higher-risk systems.
3. Support rolling, sequestered benchmarks maintained through a distributed network of
health systems modeled on FDA’s Sentinel Initiative.
4. Make postmarket monitoring central, with a standardized taxonomy for generative AI
safety and security events, predefined containment thresholds, and validated
supervisory agents.
5. Establish a clear pathway for foundation model and configuration changes, including
notice to deploying organizations, re-benchmarking, rollback, and predetermined change
control plans.
6. Clarify the roles of health systems in validation and monitoring while keeping
manufacturer accountability for product safety and effectiveness intact.

Our responses to individual discussion questions follow, organized by question number, along
with supporting evidence. We would welcome the opportunity to serve as a resource to FDA as
this framework develops.

Signed,

Cora Han, JD
Chief Health Data Officer
University of California Health
Co-signed by:
Karandeep Singh, MD, MMSc (Chief Health AI Officer, UC San Diego Health)
Deepti Pandita, MD, FACP, FAMIA (Vice President of Clinical Informatics and Chief Medical
Informatics and AI Officer at UC Irvine Health)
Smitha Ganeshan, MD, MBA (Interim Chief Health AI Officer, UCSF Health)
Paul J. Lukac, MD, MS, MBA (Chief AI Officer, UCLA Health)
Jason Y. Adams, MD, MS (Director of Data and Analytics Strategy, UC Davis Health)
Naveen Raja, DO, MBA (Chief Medical Officer, UC Riverside Health)
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

UC Health: Responses to Discussion Questions

Question 1. Dimensions of the two-axis risk framework
We support the proposed activity-versus-consequence framework. Health system experience
suggests three refinements.
Deployment context should modify risk. As stated in the discussion paper, the same
underlying capability may carry very different safety implications depending on whether it
provides information, recommends an action, drafts an action for review, or executes an action,
and on whether the recipient is a patient, a generalist clinician, or a specialist. Additional
modifiers worth considering are whether the output can be independently verified, whether an
error can reasonably be identified before it causes harm, how quickly an error could result in
harm, and whether actions based on an erroneous output can be reversed in time. Scale of
exposure is particularly important from a health system’s perspective. A low-frequency error
embedded in an enterprise system serving thousands of encounters can create substantial
aggregate risk. These factors can act as modifiers on the consequence axis.
The activity axis should distinguish forms of human supervision. The activity axis
generally aligns with the taxonomy in Appendix S of the American Medical Association’s CPT
code set, which the CPT Editorial Panel revised in May 2026 (Exhibit B). Appendix S divides
autonomous AI into three levels, and its distinctions suggest that FDA’s single “Action-Taking:
HCP-Supervised” category contains at least two forms of supervision with different risk profiles.
In override-window supervision (Appendix S Level II automation), the AI proposes a specific
action and implements it after a defined interval, such as one week unless a clinician intervenes.
In trajectory-level supervision (Level III automation), the clinician guides the overall direction of
care, such as a decision to increase diuretic therapy, and the AI implements the plan. Both differ
from fully autonomous operation, in which no healthcare professional is involved at any level.
Appendix S has no counterpart to this last category because it assumes physician oversight at
every autonomous level. Aligning the activity axis with Appendix S would give manufacturers,
health systems, and payers a shared vocabulary across regulatory review and coding.
Human oversight should count as a safeguard only when it is meaningful. The presence
of a healthcare professional should not itself be assumed to provide an adequate safeguard.
Human oversight is protective only when the user has sufficient time, information, expertise, and
workflow support to critically evaluate the output. An override window protects patients only if
clinicians realistically review the pending action before it takes effect. FDA should distinguish
nominal human oversight from meaningful human oversight, and manufacturers should
demonstrate through human factors testing whether clinicians identify and intercept meaningful
errors under realistic workload conditions.

Question 3. Patient-facing and HCP-facing functions
Patient-facing generative AI (GenAI) should not automatically be treated as higher risk solely
because the user is a patient. Patient-facing use should be evaluated according to the nature
and consequence of the information delivered and the decisions it may influence. Factors such
as health literacy, access to understandable explanations and trusted clinical support, ability to
verify the recommendation, urgency, specificity, personalization, and potential for self-directed
clinical action should modify the risk assessment. This approach preserves the benefits of
patient access to clinical information while focusing scrutiny where patients are most likely to act
on an incorrect output that could cause harm.

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

Question 4. Generalist and specialist users
The distinction between generalist and specialist users should inform validation and should not
restrict access. GenAI may be particularly valuable because it can extend access to specialty
knowledge where specialist capacity is limited. The regulatory focus should be the expertise
required to safely evaluate or act upon an output, independent of professional title. When a user
lacks the domain knowledge to evaluate an output, risks may be mitigated with safeguards such
as training embedded in the product within the workflow, or explainability features such as
citations to the medical record or the evidence base.

Question 5. Multi-turn conversations
Multi-turn conversations should be evaluated as trajectories. Health systems will increasingly
deploy conversational interfaces in patient portals, clinical messaging, triage, and care
navigation. Because prompting is effectively unbounded, safety cannot be established by
evaluating isolated prompts. Testing should examine complete conversational trajectories,
including accumulation of context, contradictions, missing information, user pressure, repeated
questioning, and gradual migration beyond the device’s intended scope. Adversarial testing
(Question 7) should include multi-turn scenarios designed to push a conversation out of scope.

Question 6. Under-escalation and over-escalation
Evaluation of escalation functions should measure both under-triage and over-triage, along with
system-level consequences. Sensitivity for critical conditions cannot be the sole endpoint.
Positive predictive value, avoidable escalation, downstream utilization, diagnostic testing,
clinician workload, and patient experience should also be considered. Our research on
generating real-world evidence from deployed AI identifies escalation performance and residual
human workload as central measures for AI that acts under asynchronous human oversight or
in automated fashion (Exhibit C). These measures capture the cost of over-escalation to
patients and health systems, which premarket testing may not fully capture.

Questions 7 and 8. The competency-based approach and the role of the risk framework
We strongly support competency-based evaluation of the final, deployed product. Health
systems do not deploy foundation models in isolation. They deploy products composed of
models, prompts, retrieval systems, interfaces, guardrails, integrations, and workflow logic. FDA
should evaluate the clinically deployed capability and should not rely primarily on performance
characteristics of the foundation model. A device should demonstrate when it abstains, defers,
escalates, or seeks additional information, in addition to how often it produces a correct answer.
Evaluation should consider the probability of individual errors as well as correlated failures that
affect many encounters or patients simultaneously.
Competency testing should include typical use and adversarial use. Any competencybased evaluation should test the “happy path” by which the average intended user, whether
patient or clinician, is expected to use the tool. It should also include adversarial testing (or “redteaming”) of whether the AI remains accurate, appropriate, and aligned with its intended range
of use when a user has malicious intent or pushes the tool beyond its scope. Because the
space of possible adversarial inputs is effectively unlimited, sponsors need a structured way to
decide where to concentrate testing. We recommend a failure modes and effects analysis
(FMEA):

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

1. Obtain a list of failure modes from a broad set of stakeholders, including intended end
users of the AI tool (clinicians, patients, or both) and stakeholders with relevant clinical
or patient expertise.
2. Characterize each failure mode by its frequency and severity.
3. Concentrate adversarial testing on the failure modes with the greatest combined severity
and frequency.
4. Document the failure mode inventory and testing plan in the premarket submission, and
update both as new failure modes are observed after deployment.
This approach has worked well at UC San Diego Health, where an FMEA guided more than 500
test calls before a patient-calling AI agent went live (Exhibit A).
For Question 8, the two-axis framework, with the modifiers described in Question 1, can
determine both the depth of benchmarking and the extent of clinical confirmation. It can also
prioritize adversarial testing, alone or alongside FMEA frequency and severity scores.

Question 9. Adequacy of the benchmarking elements
The proposed benchmarking elements serve as a strong foundation. We suggest four additions.
First, sponsors should select and weight elements using the output of an FMEA, so that the
benchmarking effort is concentrated where clinical risk is greatest. Second, element E.4
addresses automation bias for a single output, while workload effects across many outputs,
such as alert fatigue or routine approval of escalations without review, are not captured. Third,
no element tests whether handoffs between the AI and the people who act on its escalations
work in practice. In the case study in Exhibit A, patient safety depends on nurses calling patients
back after an escalation, and that handoff is important to validate. Fourth, cybersecurity
deserves explicit attention. GenAI-enabled devices face threats that traditional device testing
may not cover, including prompt injection that causes an AI to disclose protected health
information, poisoning of the knowledge sources a device retrieves from, and extraction of
system prompts or configuration. These threats could be addressed within elements S.2 and
R.1 or as a distinct element. FDA should also clarify how this testing relates to its existing
premarket cybersecurity expectations.

Question 10. Benchmark contamination and construct validity
Once a benchmark is published, AI developers can train on it, producing artificially high results
on that subset of questions. We recommend that FDA support frequently updated, moving
benchmarks constructed from a rolling collection of de-identified or synthetic cases. The core
decision-relevant details in each case would stay fixed, while other details change constantly to
prevent models from overfitting to them.
Cases could be generated entirely from synthetic data or based on real-world de-identified data
from health systems or data aggregators. Real-world cases offer two advantages. They allow
benchmark performance to be linked to known downstream outcomes, which establishes
whether benchmark scores predict real-world performance. They also allow ongoing, scalable
grading of the decisions clinicians actually made as a counterfactual (Question 15). Sponsordeveloped benchmarks remain valuable for device-specific testing and should be supplemented
by independent, sequestered benchmarks of this kind. Question 16 describes how such
benchmarks could be maintained.

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

Question 11. Selecting a clinical confirmation approach
We support a tiered clinical confirmation model proportionate to risk. For many GenAI
applications, shadow deployment in representative clinical settings may provide more
meaningful evidence than static benchmark performance because it exposes interaction with
actual workflows, data availability, user behavior, and environmental variation. High-risk
systems should undergo retrospective validation followed by shadow deployment before
prospective use that affects patient care.
For higher-risk systems, prospective evaluation should include real-world comparisons against a
counterfactual. Once an AI output drives action, determining whether the AI-enabled workflow
improves outcomes may require quasi-experimental or experimental designs. Randomized
designs, including cluster and stepped-wedge designs, should be used where practical. Where
randomization is impractical, quasi-experimental designs can provide credible evidence.

Questions 12 and 13. Statistical measurement and synthetic data
Synthetic data can expand coverage of rare scenarios and small subgroups but may not fully
reflect the clinical detail and patterns found in real patient data. Synthetic cases generated by
models of the same class as the device under evaluation can also reproduce the very gaps the
evaluation is meant to detect. Anchoring synthetic cases in real de-identified data, with core
clinical details drawn from actual patients, reduces this risk, particularly for underrepresented
subgroups. Sponsors should report performance on synthetic and real-world cases separately
before any pooling, so that differences between the two distributions are visible.

Questions 14 and 15. Comparators and acceptance criteria
For clinician-facing GenAI tools, the primary unit of evaluation should generally be the human-AI
team because that is the intervention actually deployed. The Appendix S levels (Exhibit B) offer
a practical rule. For tools at Levels I and II automation, where a clinician approves or can
override each action, evaluate the human-AI team. For Level III tools and fully autonomous
tools, evaluate the AI operating alone. The generalist or specialist comparator should match the
intended user.
For Question 15, the most decision-relevant comparator for a health system is usually the care
that would occur without the device. Rolling benchmarks built from real-world cases (Question
10) capture the decisions clinicians actually made, providing a scalable estimate of usual care
against which device performance can be compared.

Question 16. Roles for independent third parties
A rolling benchmark needs a steward that is independent of device sponsors. Health systems
are well positioned to contribute cases and expert adjudicators, and doing so today raises
unresolved questions. Benchmarking or clinical confirmation of a vendor’s device could make a
health system a sponsor under investigational device regulations. We recommend that FDA
define both “sponsor” and “manufacturer” and clarify the regulatory status of health systems that
participate in these activities.
FDA’s Sentinel Initiative offers a model. Sentinel data partners operate under contract with FDA,
keep their data behind their own firewalls, and run analysis programs distributed by a
coordinating center. Because Sentinel activities are public health activities in support of FDA’s
mission, the HHS Office for Human Research Protections has determined that the Common
Rule does not apply, and data partners may disclose information to FDA under the HIPAA
Privacy Rule’s provision for public health authorities. A similar distributed network of health

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

systems could construct and maintain rolling benchmarks for GenAI-enabled devices, with
benchmark-building code running inside each health system and only de-identified or synthetic
cases leaving it. The Center for Devices and Radiological Health’s (CDRH) National Evaluation
System for health Technology (NEST) may offer a starting point. Because Sentinel’s statutory
mandate applies to drugs, we encourage FDA to clarify whether such a network for devices
would require new authority.
To avoid limiting competition, sequestered benchmarks should be available to all sponsors on
equal terms, and benchmark stewards should be independent of any manufacturer.

Question 18. Premarket uncertainty and reliance on postmarket monitoring
Postmarket monitoring should be central to GenAI regulation, and greater reliance on it is
appropriate when it supports a graded pathway with prespecified criteria for changing the level
of human oversight. After deployment, a device could move from clinician review of every
output, to sample-based review, to exception-based review as evidence accumulates.
Movement between stages should be governed by prespecified performance thresholds.
Specific events should return a device to closer oversight, such as a foundation model change,
a breach of a performance threshold, or expansion to a new clinical setting or population.
Our research on generating real-world evidence from deployed AI offers a structure for this
pathway (Exhibit C). It describes four implementation configurations, each with its own primary
evaluation question: silent deployment, human-in-the-loop, human-on-the-loop with
asynchronous oversight, and automated with established escalation pathways. For many
applications, human-in-the-loop may remain the appropriate long-term configuration. Reduced
premarket evidence would be inappropriate where errors are irreversible or where harm can
occur faster than postmarket signals can prompt correction.

Question 19. Approaches to postmarket performance evaluation
We support periodic benchmarking, sample-based clinician review, and drift monitoring, and
recommend four additions.
• Measure clinically meaningful failures. Surveillance should measure clinically
meaningful failures in addition to technical model drift, inclusive of hallucinations,
clinically important omissions, inappropriate escalation, unsafe certainty, incorrect
retrieval of patient information, workflow failures, subgroup disparities (for example,
among patients with limited English proficiency, rare disease, or multimorbidity, and
among older adults), prompt-related failures, alignment drift in agentic workflows, and
failures following model updates.
• Standardize safety event reporting. FDA should develop a standardized taxonomy for
GenAI safety events so that signals can be aggregated across health systems. The
taxonomy should include cybersecurity events, such as prompt injection attempts and
exposure of patient data, alongside clinical failures.
• Use real-world cases as a rolling sample. After deployment, a rolling convenience
sample of real-world cases drawn through the network described in Question 16 could
provide ongoing monitoring at scale, similar in spirit to FDA Sentinel.
• Predefine containment thresholds. Manufacturers and FDA should predefine
thresholds for rollback and disablement. For devices whose failure could lead to adverse
events, mechanisms for timely containment should include rollback to a previously

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

validated version, disabling specific capabilities, reverting to a safer mode, or
suspending use.

Question 20. Machine-based supervisory agents
Supervisory agents can make continuous monitoring scalable. A second, independent set of AI
agents could continuously run routine and adversarial test cases against a deployed device,
flagging drift, regressions after updates, and new failure modes for human review. Periodic
manual review alone cannot achieve this coverage. Because this arrangement is itself a multiagent system, the reliability of the supervisory agents must be established.
• Validation and audit. Supervisory agents should be validated against humanadjudicated cases, and human reviewers should periodically audit a sample of their
judgments.
• Independence. Where feasible, supervisory agents should run on a different foundation
model from the device under evaluation, so that shared blind spots do not mask failures.
Alternatively, an ensemble of different foundation models may be feasible and achieve a
similar goal without having to know a device’s choice of foundation model.
• Accountability. Failures of a supervisory agent, such as missed regressions, should be
tracked and reported alongside device failures.

Question 21. Roles of health systems and other stakeholders
Health systems are uniquely positioned to identify real-world failure modes, implementation
effects, workflow interactions, and patient safety signals. They should be partners in
surveillance and should not substitute for manufacturer monitoring. Appropriate health system
roles include contributing failure modes to manufacturers’ risk analyses, validating local
workflow and configuration, monitoring local deployments, and contributing de-identified cases
to independent benchmarks.
Local validation should complement the manufacturer’s regulatory validation and should be
limited to local workflow and configuration. FDA should not require every health system to
recreate the manufacturer’s evidence package. Today, health systems often carry much of this
burden. Before deploying an autonomous patient-calling agent, UC San Diego Health
conducted more than 500 test calls involving physicians, nurses, and staff from Patient
Experience, Risk Management, and Quality and Patient Safety (Exhibit A). This level of effort
cannot be repeated by every health system for every tool and every model update, and it is
likely duplicative with validation that a manufacturer should perform. A health system should not
inherit responsibility for deficiencies in a product because it participates in monitoring.

Question 22. Scaling re-evaluation to the nature of a modification
Re-evaluation should scale with the expected impact of a change. We suggest the following
tiers:
• Changes to prompt wording or interface text within a validated configuration could be
documented in the quality management system, with automated re-benchmarking.
• Changes to retrieval sources, guardrails, or orchestration logic could require rebenchmarking and a temporary increase in human oversight.
• A change in foundation model version should require full re-benchmarking and a
temporary increase in oversight and could be managed under a Predetermined Change
Control Plan (PCCP) (Questions 23 and 24).

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

• A change in intended use or target population would require a new submission.
Manufacturers should also identify a validated configuration envelope within which health
systems can configure the product without taking on manufacturer-level validation. The
discussion paper identifies prompts, retrieval strategies, guardrails, orchestration logic, and user
interface as characteristics that shape device behavior, and proposes evaluating the device as
configured. It does not address scenarios in which health systems materially customize vendor
products or build GenAI tools in-house for institutional use only. Editing prompts or adding sitespecific retrieval sources may significantly alter an FDA-authorized device’s functionality and
safety profile. We recommend that FDA:
1. Define thresholds that distinguish routine configuration within manufacturer-specified
parameters from modifications that confer manufacturer responsibilities.
2. Allow manufacturers to prespecify permissible customer configuration ranges within a
PCCP, so health systems can adapt tools to local workflows without creating a new
device.
3. Clarify its approach to GenAI tools developed by, and used within, a single institution.

Question 23. Adapting PCCPs when changes cannot be fully prespecified
A PCCP can accommodate changes that cannot be fully prespecified if it prespecifies the
evaluation protocol. The PCCP would fix the competencies, test methods, and acceptance
criteria, and allow the specific change to vary. Two applications are especially valuable.
Manufacturers could prespecify permissible customer configuration ranges, so health systems
can adapt tools to local workflows without creating a new device (Question 22). Manufacturers
could also obtain advance authorization for foundation model upgrades that pass a prespecified
re-benchmarking protocol, which would make routine upgrades predictable without a new
submission each time (Question 24).

Question 24. Changes to third-party foundation models
From a deploying health system’s perspective, one of the biggest challenges is knowing when it
is safe to upgrade to a newer foundation model within an AI platform without unintended
consequences. A newer model may perform better on average and still behave differently in
ways that matter clinically, such as escalating less often, refusing different requests, or phrasing
advice more directively. There is also a shared incentive to upgrade. Newer models are often
more token-efficient and therefore cost less to run, which benefits manufacturers and health
systems alike. Without a reliable way to confirm that an upgrade is safe, deployers must choose
between staying on an older model longer than necessary and upgrading without adequate
evidence.
Health systems need a “right to know” when the AI changes. A health system cannot
govern or monitor a clinical AI capability if it does not know when a material component has
changed. Manufacturers should notify deploying organizations of material changes to the
foundation model, model version, prompts or orchestration, retrieval architecture, guardrails,
output behavior, safety policies, or clinical capabilities. Each notification should state whether
revalidation is recommended or required. Manufacturers should help health systems understand
the expected impact of a change, identify affected workflows, share validation data and
experience, and specify any local revalidation needed before deployment. Changes to retrieval
sources and connected tools also carry security implications, because each is a potential
pathway for malicious content to reach the device.

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

We also recommend:
• Version pinning. Devices should run on a specific, identified foundation model version.
If the underlying model can change without the manufacturer’s knowledge, premarket
evidence may not be valid.
• Re-benchmarking for upgrades. A change in foundation model version should trigger
manufacturer re-benchmarking against the same prespecified competencies used at
authorization. Rolling benchmarks (Question 10) are well suited to this, because they
test the new model on cases it cannot have been trained on.
• Side-by-side comparison. Before a new version reaches patients, supervisory agents
(Question 20) can run the current and candidate models in parallel on routine and
adversarial test cases, flagging differences in safety-critical behaviors such as escalation
and refusal for human review.
• A temporary increase in oversight. After an upgrade, the device should return to
closer human oversight and step back down as post-upgrade performance data
accumulate.
• Rollback. Manufacturers should retain the ability to roll back to a previously validated
version if monitoring detects a regression (Question 19).
The case study in Exhibit A illustrates the stakes. More than 500 test calls established
confidence in an AI agent as deployed. A change to the underlying model could invalidate much
of that evidence and repeating that effort for every model update is impractical. A manufacturerrun, benchmark-driven pathway would make safe upgrades routine.

Question 25. Foundation Model Device Master Files
Foundation Model Master Files would be useful and deploying health systems also need
transparency. Information available only to FDA may be insufficient for safe downstream
deployment. Health systems require actionable safety information. They do not require
proprietary model details. We recommend dividing Master File content into two sets.
Confidential information for FDA review
• Model architecture, category-level training data provenance and licensing, knowledge
cutoff, and sources of medical content.
• Disclosure of which public medical benchmarks were or may have been included in
training data.
• Detailed evaluation results on healthcare benchmarks, including subgroup performance,
hallucination, calibration, and multi-turn behavior.
• Detailed red-teaming results, including resistance to prompt injection, and tool-use
reliability for agentic applications.
Information for deploying organizations
• Model and version identification, release and depreciation schedules, and availability of
version pinning.
• Intended and unsupported use cases, clinically relevant limitations, material known
failure modes, and subgroup concerns.
• Update history, with a regression report for each new version on a fixed evaluation suite.

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

• Safety control changes, including which behaviors manufacturers and deployers can and
cannot configure.
• Auditability, including audit log availability and data retention, and a channel for reporting
safety and security incidents.
• An AI bill of materials listing the foundation model version, retrieval sources, and
connected tools, analogous to the software bill of materials required for cyber devices
under section 524B of the Federal Food, Drug, and Cosmetic Act (FD&C).
Health systems cannot perform meaningful postmarket surveillance unless AI-enabled devices
provide enough auditability to investigate an adverse event. Participation in a voluntary program
is more likely if it reduces burden for downstream sponsors. For example, FDA could let
sponsors that reference a current Master File rely on it for portions of their documentation and
could allow a pre-agreed re-benchmarking protocol to cover minor version changes.

Question 26. Agentic GenAI-enabled devices
FDA’s proposed agentic competency element (A.1) already includes planning, sequencing, tool
use, recognition of erroneous tool outputs, human oversight checkpoints, graceful handling of
tool failure, and resistance to prompt injection. We believe the additional risks of agentic
systems can be addressed through additional components within the two-axis framework, which
would keep risk assessment consistent across device types. These components include:
• Safety of the entire action chain. Agentic systems should be evaluated for the safety
of the full sequence of actions, in addition to the correctness of each individual action.
• Context of use, scope, and permissions. Tool permissions should be scoped to the
intended use, so that an agent designed for scheduling cannot reach ordering functions.
• Reversibility as a mechanism for oversight. The required level of human oversight
should increase with the difficulty of reversing an action. Override windows (for Appendix
S Level II automation) should be longer for actions that are harder to reverse.
• Auditability. Each action in a chain should be logged with enough context to reconstruct
what the agent did and why.
• Prompt injection through all inputs. Adversarial testing should cover malicious
instructions embedded in retrieved content and tool outputs, in addition to user inputs.
• Security of agentic actions. An agent with tool access can act within the electronic
health record and other clinical systems, so a successful prompt injection can produce
an unauthorized action in addition to an incorrect output. Scoped permissions,
auditability, and injection testing therefore serve as security controls as well as safety
controls. FDA should clarify how section 524B of the FD&C Act applies to agentic
devices.
• Multi-agent systems. Where multiple agents interact, including supervisory agents
(Question 20), the reliability of each agent and of their interactions should be evaluated.

Additional recommendation. Regulate the clinical capability as deployed
From the perspective of a health system responsible for selecting, deploying, governing, and
monitoring AI-enabled technologies, the most durable regulatory framework will focus on the
clinical capability as deployed. Model architecture alone is an insufficient basis for regulation.
FDA should pair risk-proportionate premarket competency assessment with real-world clinical
confirmation, transparent change management, meaningful human factors evaluation, and

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

continuous postmarket surveillance. Health systems should be active partners in this life cycle,
and manufacturer accountability for product safety and effectiveness should remain clear and
undiluted.

University of California Health | Response to Discussion Questions
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

UC Health: Supporting Evidence

Exhibit A. Case study: FMEA-guided testing of an autonomous patient-calling AI agent at
UC San Diego Health
Before deploying an autonomous AI agent that places phone calls to patients, UC San Diego
Health performed an FMEA to identify potential risks and inform a prioritized testing plan. Each
failure mode was scored for severity and probability. The product of the two set its risk level,
and the risk level determined the type of testing it received (Figure 1).

Figure 1. FMEA for the patient-calling AI agent. Higher-risk failure modes, such as the AI providing
incorrect medical advice (risk level 12), were assigned to red teaming. Lower-risk failure modes were
assigned to verification across multiple runs or a single functionality check.

Before go-live, teams across the health system made more than 500 test calls. Testers included
more than 6 physicians and 20 nurses, with additional testing from Patient Experience, Risk
Management, and Quality and Patient Safety. In red teaming sessions, clinicians posed as
patients to test the AI under difficult and adversarial conditions. In one session (Figure 2), a
nurse playing a patient on warfarin stated that the prescribing physician had died, then
responded sarcastically to questions about scheduling a callback. The AI recognized that the
situation required immediate nurse intervention for anticoagulation management and flagged it
for escalation.

University of California Health | Supporting Evidence
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

Figure 2. Excerpt from the AI-generated summary of a test call during a nurse red teaming session. The
highlighted text shows the AI identifying a scenario that required immediate nurse intervention.

After deployment, monitoring has two layers. AI analyzes call transcripts in real time, and nurses
review all escalations and call patients back as needed.
This case illustrates both the value of FMEA-guided adversarial testing and the burden that
health systems currently carry. Much of this testing could be performed once by a manufacturer
under a clear regulatory pathway and then repeated after changes to the underlying foundation
model (Questions 21 and 24).

Exhibit B. AMA CPT Appendix S categories mapped to FDA’s activity axis
Appendix S Definition (paraphrased) Closest FDA activity
category category
Assistive Output provides clinically relevant data without Informational: non-directive
deriving a parameter or drawing a conclusion. A
physician or other qualified health professional
(QHP) interprets and reports it.
Augmentative Output is a derived quantitative or categorical Informational: non-directive
parameter, such as a score, index, or classification, (for example, a risk score)
that goes beyond descriptive statistics of the input,
without an independent interpretation.
Autonomous, Level I Software independently recommends a definitive Informational: action-directing
diagnosis or specific management. A physician or
QHP must judge whether to implement or reject the
recommendation.
Autonomous, Level II Software recommends management, and the Action-taking: HCP-supervised
output must allow a reasonable opportunity to (override window before
negate the impending action before action)
implementation, for example through an alert.
Autonomous, Level Software automatically initiates management Action-taking: HCP-supervised
III actions, which continue unless a physician or QHP (oversight of the care
intervenes. Physician oversight is warranted, trajectory)
typically across multiple interventions, to judge
whether management goals are being met.
No counterpart Appendix S assumes physician oversight at every Action-taking: fully
autonomous level. autonomous

University of California Health | Supporting Evidence
Docusign Envelope ID: 64B72553-C798-8213-82BC-243C4AC5F73D

Source: American Medical Association, CPT Appendix S, as revised by the CPT Editorial Panel in May 2026.

Exhibit C. Primary evaluation questions for generative AI across implementation
configurations
Configuration Primary evaluation question Routinely generated data
Silent deployment Is the generated content sufficiently Generated content, reference
accurate, complete, and appropriate for assessments, rubric scores, factual
its intended use before it is introduced errors, omissions, unsupported
into the workflow? statements, and subgroup performance
Human-in-the-loop How does the system affect review Acceptance or rejection, edits and edit
workload, and does clinician-approved type, review time, overrides, usability
content remain accurate, appropriate, measures, downstream actions, and
and safe? safety-relevant errors
Human-on-the-loop Do escalation criteria reliably identify Transcripts, system responses and
conversations requiring clinical review, actions, escalation triggers, missed or
and is asynchronous oversight sufficient delayed escalations, human follow-up,
to identify and address failures? and safety events
Automated Does the automated workflow complete Task completion, unresolved cases,
the intended task accurately and escalation frequency, human follow-up
efficiently, with acceptable escalation time, correction rates, downstream
performance, safety, and residual human outcomes, and safety signals
workload?
Adapted from Poursoltan L, Boussina A, Cao J, et al. Generating Real-World Evidence from AI Deployed
in Health Systems. Learning Health Systems (in press). The configurations are not necessarily
sequential, and greater automation is not inherently the preferred endpoint.

References
1. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AIEnabled Medical Devices: Discussion Paper and Request for Feedback. August 2026.
2. American Medical Association. CPT Appendix S: Taxonomy for Artificial Intelligence in
Medical Services and Procedures. Updated September 2026. https://www.amaassn.org/practice-management/cpt/cpt-appendix-s-taxonomy-artificial-intelligencemedical-services-procedures
3. Poursoltan L, Boussina A, Cao J, Kim E, Bedmutha PS, Majmudar H, Ramsis M,
Khalessi A, Singh K. Generating Real-World Evidence from AI Deployed in Health
Systems. Learning Health Systems. In press.
4. Sentinel Initiative. Principles and Policies: Privacy.
https://sentinelinitiative.org/about/principles-policies/privacy
5. U.S. Food and Drug Administration. National Evaluation System for health Technology
(NEST). https://www.fda.gov/about-fda/cdrh-reports/national-evaluation-system-healthtechnology-nest

University of California Health | Supporting Evidence