Sudha Jayaraman

See attached file(s)

Comment to FDA — Docket FDA-2026-N-7874
Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback (CDRH Digital Health Center of Excellence,
18 August 2026)

Submitted by:

Sudha Jayaraman MD MSc FACS, Medical Director, Clinical Strategy and Research, Suki AI
Arooshi Kumar, MD Director of AI and Innovation, Neuroscience, Rush University Medical
Center
Erin Palm MD MBA FACS, Medical Lead, Infinitus AI
(as individual contributors)

Title:
Lessons from work on safety of patient facing agentic AI

Questions addressed:
1. Does the two-axis risk framework, organized around device activity and the
consequence of relying on an incorrect output, appropriately capture the dimensions
most relevant to the risk of a GenAI-enabled software function? If there are additional
dimensions—such as the reversibility of a resulting action, the availability of downstream
safeguards, the time pressure of the deployment setting, or the traceability of the output
(i.e., to primary source materials)—that should be represented in a risk framework,
please describe and provide examples of how they should be represented.

We support the two-axis framework as the primary basis for risk assessment, but recommend
that exposure, the intended deployment volume of the function, be added as a third input. The
consequence axis measures severity per incident. It does not capture how many incidents will
occur, and for GenAI-enabled device software functions that omission matters for two reasons.
First, aggregate harm scales with volume. Second, device errors are correlated in a way
clinician errors are not.

A group of us representing diverse stakeholders formed the Healthcare Risk Management for
Voice AI Taskforce, an informal, multidisciplinary group, in mid 2025 to develop shared
frameworks for the safe deployment of voice AI in healthcare. We met over the last year to
discuss and derive consensus around the safety risks of autonomous patient-facing agentic AI.
During this process, we found that existing frameworks did not directly apply to agentic AI.
Therefore, we drafted a risk framework based on ISO 31000, mapped to the Joint Commission
and Coalition for Health AI guidance and the NIST AI Risk Management Framework. We
included likelihood of occurrence and potential for clinical impact as the main dimensions. We
applied this framework to 5 illustrative clinical use-cases we developed to derive types of risks.
We then conducted a Delphi study to assess how individual risks could map to this framework
(see pre-prints here and here). A summary white paper can be found here. The formal
two-round modified Delphi study included 14 experts spanning clinical medicine, informatics,
health system leadership, payors, and industry rated 21 risks against each use case on an ISO
31000 likelihood and impact matrix, reaching consensus on all 105 risk-by-use-case
combinations.

Through this process, we realized that we had not included volume as a key dimension in our
framework. One of the five cases involved application of agentic AI by a payor to meet HEDIS
measures at a population level, unlike the other cases which involved specific patient
sub-groups, intentionally of varying clinical acuity. In this example, the potential for
magnification of an error across a massive population meant that a risk that had a low likelihood
of occurrence and low potential for clinical impact would still translate into enormous societal
repercussions. This was a key lesson that we recommend CDRH consider carefully. Point
solutions may not have repercussions at scale due to lack of market share, whereas those
solutions with population-wide reach have unique potential for harm at a scale that will be
difficult to mitigate and therefore may need to be vetted and monitored more carefully. We
therefore suggest a multi-variable risk assessment framework that includes the probabilities
related to the likelihood of occurrence and impact as well as intended patient volume in addition
to the degree of agent autonomy to determine risk.

Second, device errors are correlated in a way clinician errors are not. A single prompt, guardrail,
or underlying model change alters behavior for every user simultaneously, so a defect is a
widespread event. The two-axis framework assumes risk accrues one patient at a time.

Factors such as time pressure, ability to mitigate and reversibility contribute to these
dimensions. Lack of traceability in itself may not increase risk in as much as decrease the
reversibility and/or ability to rescue or mitigate the effects. In summary, based on lessons from
our group’s work, we recommend a more nuanced assessment of AI risks in order to inform
efforts to prevent or mitigate harm that could occur from these risks since the relationship
between risks and harm is non-linear beyond informational functions.

5. For multi-turn conversational GenAI-enabled devices that may migrate from providing
“non-directive” information to “action-directing” information over the course of an
exchange, how should risk be assessed across realistic Conversational trajectories?
How could the intended use of such a device be characterized when its behavior is
emergent across a conversation?

The following response applies to our work focused on patient facing agentic AI. We identified
four risk levels in patient facing agentic AI: agent, data, patient, clinical, based on five illustrative
clinical use cases. Each risk level has multiple attendant risks and each risk requires dedicated
mitigation strategies. Some of the key risks relating to conversational trajectories exist at the
patient level and result from patient factors such as limited English proficiency, low literacy, no
smartphone access or sensory impairments. Second, not all patients or conditions will be
amenable to agentic AI interactions, such as acute postop complications. Third, there may be
specific patient symptoms such as aphasia in stroke patients or confusion, due to infection or
medications, in postop patients which may affect the ability of AI agents from accurately
interacting with patients, exposing them to risks of misinterpretation or failure to recognise
red‑flag symptoms. Lastly, the ability for an AI agent to escalate appropriately based on the
conversation needs to be considered. For example, an agent conducting post stroke follow-up
would need to recognize high risk symptoms that would mandate further clinical evaluation.
Furthermore, a function that offers information instead of suggesting an immediate action could
be safer by getting the HCP to consider more information.
We suggest assessing risks across these levels at a minimum, and at the patient-level to be
specific. While establishing risk thresholds may not be realistic without benchmarking data,
CDRH could consider asking developers for data according to these risk levels and explicit
strategies to monitor or mitigate the attendant risks, in order to determine the specific risk profile
of each GenAI device under review. Using explicit risk levels will offer vendors the opportunity to
incorporate considerations of such risks in the product development cycle and would inspire
trust in clinicians and patients from much-needed transparency in an opaque product space.

6. For GenAI-enabled care escalation functions, CDRH is considering how the evaluation
may account for both under-escalation and over-escalation. How could manufacturers
characterize and weigh these two directions of error, given that they may not be
commensurable and that acceptable trade-offs may vary by clinical context?

Under-escalation and over-escalation should not be combined into a single score. They should
be measured separately, each with its own ceiling, and the acceptable ratio between them
should be set by the clinical context, not by a universal rule. The two errors must be measured
in different places, so we recommend that every escalation function be evaluated in both an
approved simulation environment and a real-world pilot before authorization.

Clinical care requires an assessment of disease severity and therefore is subject to under or
over triage in routine clinical practice. Some specialty specific accrediting bodies often set
these thresholds to ensure patient safety, transparency and trust (e.g., American College of
Surgeons Trauma Quality Improvement Program monitors under and overtriage rates for trauma
patients and all Level 1 Trauma Centers). Trauma triage has accepted for decades that missing
a badly injured patient is far worse than over-activating the trauma team. Similarly, in stroke
triage, all patients with large vessel occlusion symptoms are routed to the
thrombectomy-capable center on a positive screen, because a missed occlusion means
permanent disability. GenAI-enabled care escalation functions will need to follow similar
processes and standards and balance the tradeoffs across precision and recall which will vary
by use case and clinical context.

In our patient facing agentic AI work, we propose several strategies that may be worth
considering including setting calibrated thresholds starting with low under‑triage tolerance and
accepting higher over‑triage while calibrating models for safety which can then be adjusted over
time based on audited outcomes and resource capacity. Second, patient‑specific risk scores
could be used to tailor thresholds to reduce false positives among low‑risk individuals while
maintaining high recall for high‑risk patients. Third, dedicated cost‑benefit analyses may need to
be conducted to evaluate the clinical, economic and operational impact of over‑triage to balance
resource utilization with safety and help adjust workflows to handle false alarms efficiently. In
reality, some combination of these strategies will probably be required depending on the use
case and the resources available to implementers. Fourth, an ongoing monitoring process to
assess the under and over triage rates will be needed as well as safe harbor mechanisms to
share the lessons with manufacturers and improve the capabilities of these GenAI-enabled care
escalation functions. We think this likely will require human review through existing
mechanisms such as process improvement programs in clinical departments.

7. Is the competency-based approach described above, i.e., device benchmarking
followed by clinical confirmation, a useful and appropriate framework for evaluating
GenAI-enabled devices?

And
9. Would a benchmarking structure such as the one described above be likely to provide
adequate evidence of clinical knowledge, analytic capabilities, safety behavior,
communication, and generalizability to support a reasonable assurance of safety and
effectiveness? Are there elements that are missing, redundant, or inappropriately
categorized? Are there externally developed standards that could be leveraged?

Benchmarking followed by Clinical Confirmation as described in the Discussion paper may be
sufficient for low-risk use cases. However, for high-risk use cases, such as an AI agent whose
scope includes autonomous titration of medications, we think that Clinical Confirmation needs a
higher bar than proposed. Real world clinical validation, such as through prospective clinical
trials, may be needed to prove efficacy because anything short of that will result in
heterogeneous outcomes due to variations in clinical practice, professional opinion, and
benchmark design. Additionally the benchmarks for each use case will need to be
independently established and validated ideally by a third party as described in the paper.

8. How might the two-axis risk framework described in Section IV be considered within
the competency-based approach described above to help determine the level of evidence
needed for a premarket submission?

We suggest modifications to the risk framework as noted above in Q1 and explicit reporting by
level and risk (Q5). We also think that validation of the risk framework prior to becoming a
governance standard will be essential to understand its feasibility and limitations both for
manufacturers and implementers. Based on our experience above in conducting a Delphi of the
risk framework and use cases, we found that there are substantial nuances in each risk across
each clinical use-case which will need to be understood by both stakeholders and adopting a
framework without applying it may cause confusion and risk stifling innovation without materially
increasing safety and quality.

19. Please comment on the potential approaches to postmarket performance evaluation,
including periodic re-benchmarking, sample-based clinician review, and performance
degradation monitoring. What additional approaches should CDRH consider, and how
should the cadence and triggering events for reassessment be determined?

Specific metrics and thresholds cannot be standardized; they depend on the use case, the
population, and the capacity of the care team receiving the device’s outputs. What can be
standardized is the structure of a monitoring plan: required elements, responsibilities, and
triggers.

We suggest that CDRH include a prespecified monitoring plan in the premarket package,
developed with at least one deploying institution and finalized per site before go-live. The
manufacturer owns the plan and remains accountable. The implementer runs the parts only it
can run (sampling real conversations, linking escalations to EHR outcomes, measuring clinician
response) and returns the data. This is how post-approval studies already work.

On the three approaches: re-benchmarking is a regression test and should be event-triggered,
not calendar-driven. Clinician review needs a sampling plan that oversamples escalations,
missed escalations with bad outcomes, and flagged subgroups. Degradation monitoring should
track behavior (escalation rate, refusal rate, drift toward directive output), which moves in near
real time, not only accuracy, which lags.
What is missing is the receiving side. Our Delphi panel rated clinician-level risks highest of four
risk levels (median likelihood × impact 10.5 versus 9.5 for agent-level), pointing to the clinician
interface rather than the agent. Our interpretation: a better agent may increase this risk by
sending more correctly flagged escalations into saturated teams. Plans should measure time to
clinician response, fraction of escalations acted on, and override rates. A device whose
escalations are ignored is unsafe regardless of its accuracy.

Cadence should scale with risk tier and exposure. Triggers should include any change to model,
prompts, guardrails, or orchestration; any third-party model version change (requiring version
pinning and notification rights); crossing an exposure threshold; a shift in any leading indicator;
and new sites, populations, or languages. The device must not change silently.

20. Please comment on whether the proposed approaches to postmarket monitoring can
be facilitated by machine-based supervisory agents. What considerations, including the
evaluation and reliability of the supervisory agent itself, should CDRH take into account
for such an approach?

Based on our experience in healthcare and industry, we think that postmarket monitoring will
likely need at least one, if not multiple supervisory agents working together to determine risk, in
conjunction with human reviewers. Until supervisory agents are adequately validated, human
reviewers will be essential for safety. Additionally, we should rely on proven ML methods for
monitoring risk since more deterministic classifiers are likely to be safer as the first line.
However we also recognize that humans have already been shown to be inaccurate due to
automation bias (Vinas et al,. 2026 and May et al., 2026) As these technologies mature, we
anticipate that they will become more accurate and therefore may result in error rates that may
progress beyond human standards of measurement. As such, we think that each agent would
be subject to the same risk assessment at a minimum. A supervisory agent is a device
component and should be subject to the same risk assessment and the same independence
requirements as the device it monitors. CDRH already states this principle for adjudicators in
Section V.B.2, including when the adjudicator is itself an LLM. In our view, independence
requires a different model family, not a different prompt on the same model, and even that
reduces rather than eliminates correlated error: LLM evaluators favor their own outputs
(Panickssery et al., NeurIPS 2024) and those of architectural siblings (Philipp et al., 2026,
preprint), and a model does not reliably correct its own reasoning without an external signal
(Huang et al., ICLR 2024).

Finally, supervisory agents should be developed and governed with independence from both the
sponsor and the foundation model developer, through the third-party mechanisms CDRH
describes in Section V.D.2, so that monitoring is not optimized toward any party’s interest.

21. What roles might clinicians, healthcare institutions, professional societies,
standards-setting bodies, and other stakeholders appropriately play in postmarket
monitoring, and how can these roles be structured without diffusing manufacturer
accountability?

One of the main findings of the work from Healthcare Risk Management for Voice AI Taskforce
is concern for transfer of accountability and liability to clinicians. We clearly saw that the
clinician level risks superseded the rest in the results of our Delphi. We think the emphasis of
“human in the loop" potentially places additional burden on clinicians who are already a
shrinking, overburdened and burnt-out workforce. Therefore, relying on humans to monitor
technologies that are increasingly accurate and highly sophisticated but which clinicians are not
well-trained in, may exacerbate the pressures on this workforce and therefore on implementers.
We think this is an additional reason why multi-agent evaluation processes will become
necessary over time. We encourage CDRH to carefully consider the repercussions of
advocating or requiring human reviewers for all monitoring due to these concerns.

We therefore recommend one structuring principle: neither using a device nor participating in its
monitoring transfers accountability for device performance from the manufacturer. First, CDRH
should not consider clinician review as an acceptable risk control for a device-level failure
unless the sponsor has validated, through human factors testing with representative clinicians
under realistic workload, that reviewers can detect that failure. This applies FDA’s existing
human factors guidance to GenAI devices. Second, labeling should state what the device is
validated to do and what human review is and is not expected to catch. Third, obligations
assigned to deploying institutions or clinicians, whether by labeling or by contract, should not
count toward the manufacturer’s monitoring responsibilities. This principle is consistent with
AMA policy that liability should rest with those best positioned to know and avert an AI system’s
risks (H-480.939), with analysis showing that under current law clinicians bear the exposure
when AI advice departs from the standard of care (Price, Gerke, Cohen, JAMA 2019), and with
the finding that tools relying on a human in the loop may increase liability risk for hospitals and
practitioners because errors become attached to the human-computer interaction (Mello and
Guha, NEJM 2024).

Finally, accountability needs to be shared because data on quality and performance is not
wholly owned by any stakeholder but is shared across developers and implementers. Close
collaboration for ongoing performance evaluation is essential for any AI technology but
especially agentic AI.

Scope of This Submission:

This public comment discusses risks relevant to patient facing agentic AI, suggests dimensions
to include in a framework and offers considerations for real world applications. It does not
evaluate clinical efficacy or provide legal advice, and nothing in this submission is meant to
express an opinion about whether any particular products or technologies should be considered
to be medical devices. The outputs and the participants of the Healthcare Risk Management for
Voice AI Taskforce are listed: here, here and here.