Regavise (Markus Yap)

Regavise appreciates the opportunity to comment on CDRH’s discussion paper on generative AI (GenAI)-enabled medical devices [1, 2]. The paper is among the most technically serious regulatory documents yet published on this subject anywhere. It is expressly exploratory—it is not draft guidance and does not resolve questions of legal authority—and we respond in that spirit. Its proposed two-axis risk heuristic, competency-based premarket evaluation, risk-proportionate postmarket monitoring, voluntary Foundation Model Master Files, and its candor about the limits of current methods give industry a credible starting point. Our comments aim to make those concepts operational, measurable, and predictable for sponsors, and to identify the places where we believe the framework needs an additional dimension, a sharper definition, or a guardrail.

Regavise | Comment on Docket No. FDA-2026-N-7874

PUBLIC COMMENT | SUBMITTED ELECTRONICALLY VIA REGULATIONS.GOV

Building a Predictable, Evidence-Proportionate Pathway
for Generative AI-Enabled Medical Devices
Comments of Regavise on FDA’s Discussion Paper, “Considerations for the Regulation of
Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback”

Docket No.: FDA-2026-N-7874
To: Dockets Management Staff (HFA-305), U.S. Food and Drug Administration, 5630 Fishers Lane, Rm. 1061, Rockville,
MD 20852
Attention: Digital Health Center of Excellence (DHCoE), Center for Devices and Radiological Health (CDRH)
Submitted by: Regavise | Markus Yap, PhD, MPH, RAC (US) - Chief Technology Officer | markus@regavise.com
Date: October 19, 2026

Regavise appreciates the opportunity to comment on CDRH’s discussion paper on generative AI (GenAI)-enabled
medical devices [1, 2]. The paper is among the most technically serious regulatory documents yet published on
this subject anywhere. It is expressly exploratory—it is not draft guidance and does not resolve questions of legal
authority—and we respond in that spirit. Its proposed two-axis risk heuristic, competency-based premarket
evaluation, risk-proportionate postmarket monitoring, voluntary Foundation Model Master Files, and its candor
about the limits of current methods give industry a credible starting point. Our comments aim to make those
concepts operational, measurable, and predictable for sponsors, and to identify the places where we believe
the framework needs an additional dimension, a sharper definition, or a guardrail.

We respond to all 26 discussion questions (Section 4) and offer a set of cross-cutting proposals (Section 3) that
we believe would strengthen the framework as a whole. We also address three requests for feedback that
appear in the body of the paper rather than in the numbered questions: GenAI-enabled measurement, signalprocessing, and in vitro diagnostic functions (Section IV; see our Question 1 response); the use of real-world
data, data collected outside the United States, and the applicability of 21 CFR Parts 50, 56, and 812 to clinical
confirmation (Section V.C.1; see Question 11); and passive or continuous “model-evolution changes” (Section
VI.C; see Questions 22–23). A crosswalk mapping each question to our recommendations appears in Appendix A,
and grouped references follow in Section 7.

How to read our proposals
The named constructs in these comments (for example, the AI Configuration Bill of Materials, the
Directiveness Ladder, Entrustable Device Activities, and the Monitoring-Readiness Criteria) are offered as
illustrative, voluntary analytical tools that FDA could test through guidance development, worked examples,
public workshops, or pilots. They are not proposed as new binding requirements, new device classifications, or
substitutes for the statutory standard of reasonable assurance of safety and effectiveness. Where we refer to
existing legal requirements (for example, FD&C Act §524B, the QMSR, PCCP authorization, or 21 CFR Part 812),
we identify them as such and do not suggest that our proposals expand them.

Page 1 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Contents

1. About Regavise and Our Perspective ................................................................................................................ 3
2. Executive Summary .......................................................................................................................................... 4
3. Cross-Cutting Proposals .................................................................................................................................... 6
3.1 The regulated object is a configuration: Configuration Identity and the AI-CBOM ............................................................. 6
3.2 Architecture is a risk control: recognize demonstrated “architectural containment” ........................................................ 7
3.3 Traceability is a safety property, not a feature .................................................................................................................... 8
3.4 Competence must be demonstrated in interaction, not only in knowledge ....................................................................... 8
3.5 Predictability for sponsors: structured, voluntary Pre-Submission feedback on GenAI evaluation plans ........................... 8
4. Responses to CDRH’s Discussion Questions .................................................................................................... 10
4.1 Risk Assessment (Section IV; Questions 1–6) .................................................................................................................... 10
Question 1 — Does the two-axis framework capture the most relevant risk dimensions? ........................................................ 10
Question 2 — How to account for the spectrum between “non-directive” and “action-directing” information? ..................... 12
Question 3 — Patient-facing versus HCP-facing informational functions ................................................................................... 13
Question 4 — Generalist versus specialist HCP-facing functions ................................................................................................ 14
Question 5 — Multi-turn conversations and emergent intended use ........................................................................................ 15
Question 6 — Weighing under-escalation and over-escalation .................................................................................................. 16
4.2 Competency-Based Premarket Evaluation (Section V; Questions 7–17) ........................................................................... 17
Question 7 — Is benchmarking followed by clinical confirmation a useful framework? ............................................................ 17
Question 8 — Using the two-axis framework to set the level of evidence ................................................................................. 18
Question 9 — Adequacy of the benchmarking structure; missing elements; external standards .............................................. 19
Question 10 — Establishing that benchmark performance predicts real-world behavior.......................................................... 21
Question 11 — Selecting and justifying a clinical confirmation approach .................................................................................. 22
Question 12 — Statistically meaningful performance measurement; synthetic data; combining evidence .............................. 24
Question 13 — Where synthetic data fits, where it fails, and safeguards .................................................................................. 26
Questions 14 and 15 — Comparators, acceptance criteria, and the counterfactual care pathway ........................................... 26
Question 16 — Roles for independent third parties ................................................................................................................... 28
Question 17 — Applying the approach to other architectures (VLMs, generative/predictive world models) ........................... 28
4.3 Postmarket Monitoring (Section VI; Questions 18–21) ..................................................................................................... 29
Question 18 — Accepting greater premarket uncertainty through postmarket monitoring ...................................................... 29
Question 19 — Postmarket approaches, additional methods, cadence and triggers ................................................................. 31
Question 20 — Machine-based supervisory agents .................................................................................................................... 31
Question 21 — Roles of the broader ecosystem without diffusing manufacturer accountability .............................................. 32
4.4 Change Control and Third-Party Models (Section VI.C; Questions 22–24) ........................................................................ 33
Question 22 — Scaling re-benchmarking to the nature of a modification.................................................................................. 33
Question 23 — Adapting PCCPs when future modifications cannot be fully prespecified ......................................................... 34
Question 24 — Detecting and responding to third-party foundation model changes................................................................ 35
4.5 Foundation Model MAFs and Agentic AI (Section VII; Questions 25–26) .......................................................................... 36
Question 25 — Practicality and content of voluntary Foundation Model MAFs ........................................................................ 36
Question 26 — Additional considerations for agentic GenAI-enabled devices........................................................................... 38
5. Additional Recommendations Beyond the Discussion Questions .................................................................... 40
5.1 Transparency of the public record ..................................................................................................................................... 40
5.2 A clear GenAI function-boundary decision tree................................................................................................................. 40
5.3 Least burdensome implementation for small and mid-sized sponsors ............................................................................. 40
5.4 Machine-readable regulation and interoperable evaluation reports ................................................................................ 40
5.5 International convergence ................................................................................................................................................. 41
6. Conclusion and Offer of Collaboration ............................................................................................................ 42
7. References ..................................................................................................................................................... 43
Appendix A. Crosswalk of Discussion Questions to Regavise Recommendations ................................................ 48

Page 2 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

1. About Regavise and Our Perspective
Regavise is developing an expert-guided AI platform intended to connect authoritative FDA sources (statutes,
regulations, guidance documents, recognized consensus standards, classification and product-code data, and
public decision summaries) with device-specific regulatory planning from early development through postmarket
compliance. The platform is being built by medical device regulatory professionals drawing on independent
consulting experience in private industry. Our day-to-day work sits at the point where FDA policy becomes a
sponsor’s concrete plan: pathway selection, Q-Submission strategy, evidence planning, PCCP design, and
postmarket surveillance programs.

This vantage point gives us three relevant perspectives:

• The planner’s perspective. Sponsors—particularly small and mid-sized companies—do not struggle with
high-level principles; they struggle with translating them into a defensible test plan, sample size,
acceptance criterion, and change-control strategy before they have spent their runway. Where we can,
our comments therefore propose specific, testable mechanisms rather than general endorsements.
• The builder’s perspective. The Regavise platform under development is itself a GenAI system grounded in
authoritative regulatory sources. It is a regulatory-planning tool, not a medical device. Building it has
highlighted practical engineering questions: whether an output is faithful to its cited primary source, how
prompt and model changes silently shift behavior, how to keep nonbinding or superseded sources from
being treated as binding requirements, and how to design evaluation and human expert review into the
workflow from the outset. These observations motivate our proposals; they are not offered as clinical
validation data.
• The lifecycle perspective. Our platform spans the total product life cycle (TPLC). We strongly support
CDRH’s TPLC framing [1, 3] and focus our recommendations on making premarket evidence, change
control, and postmarket monitoring one connected system rather than three separate obligations.
Disclosures
• Commercial interest. Regavise has a commercial interest in clear, predictable regulatory frameworks for
digital health and AI-enabled devices. Regavise is developing an expert-guided AI regulatory platform
designed to help medical-device innovators navigate FDA requirements through source-linked, device- and
indication-specific guidance. Its planned services encompass early regulatory planning, identification of
testing and evidence gaps, submission preparation, documentation quality and consistency review, and postmarket compliance support.
• Independence. These comments reflect the independent professional views of the author and Regavise, draw
solely on publicly available information and the author’s professional experience, and do not represent the
views of FDA.
• Author. The author, Markus Yap, PhD, MPH, RAC (US), served at FDA from 2009 to 2017. As an FDA
Commissioner’s Fellow (2009–2011), he evaluated, tested and developed software applications and tools for
managing marketing submissions, including business-process-management applications for reviewers’
workflows. He subsequently served at CDER as IT project manager for development of a master data
management data warehouse. At CDRH, he served as Branch Chief for Nuclear Medicine and Radiation
Therapy, supervising premarket and postmarket review. He later served as Deputy Director of CDRH’s
Division of International Compliance Operations, overseeing postmarket compliance, import and export
operations, and foreign facility inspections. Since 2017, as Principal of YLS Regulatory Consultants, he has
advised medical device, diagnostics, biotechnology, pharmaceutical, and law firm clients on FDA premarket
and postmarket submissions.

Page 3 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Use of AI. Prepared with the assistance of Claude Opus 5.5. The author reviewed the final text and is
responsible for its content.

2. Executive Summary
We support the overall architecture of CDRH’s approach: risk-based, intended-use-driven, evaluating the final,
user-facing device as configured rather than the foundation model in isolation, and built on a TPLC foundation.
The timing is right. FDA’s list of AI-enabled medical devices (of all kinds, not only GenAI) now exceeds 1,600
entries [13]; FDA has cleared a patient-facing conversational device with a PCCP (K253281) that its manufacturer
publicly describes as the first FDA-cleared patient-facing clinical LLM, although FDA’s decision summary does not
itself identify the conversation service as GenAI [14, 97]; and peer regulators are moving quickly—the UK
National Commission into the Regulation of AI in Healthcare issued 44 recommendations in September 2026, all
accepted by the UK Government [38, 39], while IMDRF has circulated a draft AI life-cycle technical framework
[33]. U.S. leadership now depends on converting the discussion paper’s concepts into predictable, leastburdensome expectations.

Our ten principal recommendations are:

1. Evaluate and control the configuration, not just the model. Define a GenAI device’s Configuration
Identity—model and version, system prompts, retrieval corpus and version, guardrails, orchestration logic,
tools, and inference parameters—and encourage sponsors to document it in a risk-scaled AI Configuration
Bill of Materials (AI-CBOM) within existing design and change controls. The AI-CBOM is a proposed
documentation practice, analogous to—but legally distinct from—the SBOM required for cyber devices
under FD&C Act §524B [11]. Safety-relevant configuration changes should trigger a documented changeimpact assessment (Sections 3.1, 4.4).
2. Keep the two axes; document four contextual risk factors. Retain device activity × consequence as the
primary heuristic, and consider four factors that materially change risk: (a) reversibility and time-to-harm;
(b) independent verifiability and source traceability (which can raise as well as lower risk, e.g., for opaque
measurement and IVD functions); (c) exposure scale and error correlation; and (d) architectural
containment (e.g., a “generative shell around a deterministic clinical core”). These factors should inform
the risk analysis; a lower risk estimate should be claimed only with evidence that the associated control
works in the actual setting of use (Q1).
3. Measure directiveness; don’t infer it from wording. Pilot a five-level Directiveness Ladder (D0–D4) as a
common descriptive vocabulary and characterize devices by their Observed Directiveness Profile—the
upper tail of directiveness actually produced across realistic conversations—rather than by labeling or
disclaimers (Q2, Q5).
4. Describe intended use for conversational devices as a “Conversational Trajectory Envelope.” As a
testable behavioral specification, intended use should specify the permitted functions, the permitted
transitions between them, and the required escalation/exit behaviors, and be tested with trajectory
sampling across conversation lengths and adversarial personas (Q5).
5. Add “Entrustable Device Activities” to the competency-based approach. Borrowing directly from
competency-based medical education [80, 81], offer an optional activity-and-supervision map that
decomposes intended use into discrete activities, each authorized at a defined supervision level.
Progression to greater autonomy should occur only where the modification, methods, and impact
assessment are adequately specified in an FDA-authorized PCCP, or otherwise through a new marketing
submission (Q7, Q8, Q18, Q23).

Page 4 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

6. Strengthen benchmarking with six missing elements and risk-proportionate validity evidence. Add
Source Fidelity and Traceability; Sycophancy and User-Pressure Resistance; Privacy and Confidentiality;
Knowledge Currency; Human–AI Team Performance; and Stochastic Stability and Configuration Sensitivity.
For benchmarks that support consequential claims, provide evidence of construct and content validity and
contamination control—and predictive validity where it can practicably be established (Q9, Q10).
7. Make statistics explicit for stochastic devices. Distinguish per-output variability, per-scenario reliability
across repeated generations, and prevalence-weighted real-world risk; use repeated sampling per
scenario, clustered uncertainty estimates, a clearly defined consistency measure for safety-critical
behaviors, zero-failure sample-size planning with explicit independence assumptions, and predictionpowered inference [68] where grader calibration has been demonstrated (Q12, Q20).
8. Pair absolute safety floors with realistic comparators. Non-compensatory safety gates (e.g., underescalation of time-critical emergencies) should never be traded off against average performance; benefit
should be judged against the realistic counterfactual care pathway with any noninferiority margin
prespecified and clinically justified (Q6, Q14, Q15).
9. Manage residual uncertainty postmarket only when detection and protective action can precede serious
harm. Postmarket monitoring complements—but cannot substitute for—premarket controls where harm
can occur faster than it can be detected. We offer six Monitoring-Readiness Criteria as decision aids,
together with a tiered change taxonomy, envelope-based PCCP concepts, epoch-gated verification of
passive or continuous “model-evolution changes,” and risk-based controls for upstream foundation-model
changes (Q18–Q24).
10. Make the Foundation Model MAF interoperable by design. Explore a voluntary field-by-field mapping
between MAF content and the EU AI Act Article 53/Annex XII downstream-provider documentation and
the UK Commission’s proposed opt-in Master File [35, 36, 38], to reduce duplicative effort without
implying that one dossier satisfies each jurisdiction’s distinct legal requirements; pair it with risk-based
supplier controls under the QMSR (Q25).

What we ask CDRH to do next
• Use the GenAI workstream identified on CDRH’s FY 2027 guidance agenda (“Risk Assessment and
Evidentiary Considerations for Generative Artificial Intelligence-Enabled Devices,” listed as Under
Construction) [98] to publish risk-scaled, function-level worked examples across the two-axis grid,
sample hazard-to-evidence maps, and a GenAI function-boundary decision tree reconciled with the
January 2026 CDS and General Wellness guidances [5, 6]—coordinated with the separate A-list draft
guidance on GenAI conversational devices for mental disorders and the AI lifecycle guidance [7].
• Pilot a standardized, voluntary Pre-Submission feedback template for GenAI evaluation plans within
the existing Q-Submission Program [103], documenting points of alignment, unresolved questions, and
assumptions to revisit—while preserving the ordinarily nonbinding nature of Pre-Submission feedback.
• Clarify, with examples, how 21 CFR Parts 50, 56, and 812 apply to shadow and clinician-gated clinical
confirmation of GenAI devices, and what bridging evidence FDA expects when relying on data collected
outside the United States [99, 101].
• Explore voluntary MDDT qualification of suitable GenAI benchmark suites and simulators within a
specified context of use, and consider an ASCA-like pilot for independent GenAI evaluation laboratories
if validated methods and statutory authority support it.
• Publish the benchmarking taxonomy as a machine-readable schema and add a GenAI/LLM flag to the
AI-Enabled Medical Devices List and public decision summaries where GenAI functionality is confirmed.

Page 5 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

3. Cross-Cutting Proposals
Before responding question by question, we offer five cross-cutting proposals. Each is referenced in multiple
answers in Section 4.

3.1 The regulated object is a configuration: Configuration Identity and the AI-CBOM
The discussion paper correctly notes that GenAI-enabled devices can change through “the underlying model,
prompts, retrieval strategies, guardrails, orchestration logic, and/or user interface” [1]. In our experience,
sponsors and reviewers frequently treat the model as the regulated artifact, when in practice clinically relevant
behavior emerges from the full configuration of the device software function. A one-line change to a system
prompt, a refreshed retrieval index, or a vendor’s silent update to a hosted model endpoint can create hard-todetect, safety-relevant differences. Behavior drift in hosted LLMs over a few months is well documented [70],
and even nominally deterministic inference (temperature 0) can yield different outputs because of server-side
batching effects [71].

We recommend that FDA encourage sponsors to maintain risk-appropriate configuration records—an AI
Configuration Bill of Materials (AI-CBOM)—within applicable quality-management processes. This is a proposed
documentation practice, not an independent statutory submission requirement. FD&C Act §524B separately
requires an SBOM for products that meet the statutory definition of a cyber device, and an AI-CBOM should not
be characterized as required by that provision [11]. Supplier controls apply in accordance with the QMSR, which
incorporates ISO 13485:2016 [10].

Implementation detail. The AI-CBOM should be a minimum change-impact record, not an exhaustive catalog of
every internal parameter. Its core should link the device software function, clinically relevant configuration
versions, responsible owner, evaluated baseline, change date, safety-impact rationale, and required verification
and validation evidence. Sponsors should be able to refer to existing design-control, cybersecurity, and supplier
records rather than duplicate them, and to generate the record automatically using emerging machine-readable
formats such as the SPDX 3.0 AI and Dataset profiles and the CycloneDX machine-learning BOM [102].

Table 1. Proposed minimum AI-CBOM content for GenAI-enabled devices (illustrative)

Component What to record Why it matters

Foundation model(s) Provider; model name; immutable version or Upstream changes are the most
snapshot identifier (where the provider makes one common unplanned change source
available); hosting mode (vendor API, private cloud, (Q24)
on-premises); fine-tuning lineage; MAF reference (if
any)

Adaptation layer Fine-tuning/adapter datasets and versions; Determines domain behavior and
reward/preference data provenance bias profile

Prompts and policies System prompts, templates, few-shot exemplars, Small edits can shift directiveness
content policies (version-hashed) and escalation behavior

Retrieval (RAG) Corpus sources, curation rules, index version/date, Governs knowledge currency and
chunking/embedding model, ranking logic; source- source traceability
status metadata (binding, final guidance, draft,
superseded)

Page 6 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Component What to record Why it matters

Guardrails and classifiers Input/output filters, safety classifiers, escalation Primary control for S.1–S.3
rules, refusal logic (versions) behaviors

Orchestration and tools Agent graph, tool inventory and permissions, Defines action authority (Q26)
deterministic services invoked, tool-call schema
validators

Inference parameters Temperature, top-p, max tokens, seed policy, Drives reproducibility (R.1)
batching/determinism settings

User interface Output rendering, citations display, disclaimers, Affects comprehension and
escalation affordances automation bias (E.4)

Evaluation baseline Pointer to the frozen benchmark/clinical-confirmation Anchor for re-benchmarking
baseline used for authorization (Section VI.C)

A hash over the AI-CBOM yields a “configuration fingerprint” that can be logged with outputs to attribute postmarket events to specific
configurations. Logging should capture the minimum information necessary for traceability, with documented retention limits and
protection of patient data and trade secrets. A manifest fingerprint records what the sponsor deployed; it is not proof that a hosted
upstream model endpoint is immutable.

3.2 Architecture is a risk control: recognize demonstrated “architectural containment”
Not all GenAI-enabled devices let the generative model decide. FDA’s public decision summary for K253281
describes a patient-facing “Conversation Service (UpDoc Agent)” that collects patient-reported data and delivers
instructions, while insulin instructions are “computed … based on the healthcare provider-defined treatment
parameters,” and the authorized PCCP requires that modifications “maintain deterministic insulin dosing logic
without altering core clinical decision-making” [14]. The summary reports human factors and software
documentation but no non-clinical performance or clinical studies, and it does not use the words “generative,”
“LLM,” or “language model”; the LLM characterization comes from the manufacturer’s public statements [97],
and one cannot infer the device’s GenAI status from the FDA record alone.

Where an implementation uses GenAI for interaction while reserving clinical computation to validated
deterministic services, it forms a generative shell around a deterministic clinical core. Its residual GenAI risks
(mis-capture of data, mis-communication of an instruction, failure to escalate a symptom, scope drift in
conversation) are real but different from those of a device in which the model itself generates the clinical
recommendation. We recommend that the risk framework consider demonstrated architectural containment as
a contextual safeguard (see Q1), while requiring evidence that three engineering invariants actually hold:

1. Input/state schema isolation: the generative layer extracts candidate clinical parameters into a strictly
typed schema that requires explicit user confirmation or deterministic range validation before entering
the clinical calculation core.
2. Output immutability: deterministic clinical outputs (e.g., calculated doses, scores, contraindication alerts)
are rendered through protected templates or verified token-for-token by a deterministic post-processor,
so the language model cannot paraphrase, round, or soften them.
3. Parallel deterministic escalation: rule-based red-flag and threshold checks run in parallel with, or before,
the generative model, so conversational context cannot suppress an emergency escalation.

Page 7 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

We also recommend that public decision summaries disclose when GenAI is present and what role it plays, so
that the field can learn from precedents (see Section 5.1).

3.3 Traceability is a safety property, not a feature
The paper’s Question 1 lists “traceability of the output (i.e., to primary source materials)” as a candidate risk
dimension. We believe this is one of the most important ideas in the paper. When a clinician (or a sufficiently
literate patient) can see and verify the source of an assertion, the device’s error becomes detectable before
reliance, which is the logic underlying the independent-review criterion of FD&C Act §520(o)(1)(E) and FDA’s
January 2026 CDS guidance [5]. But citations are only protective if they are faithful: an automated evaluation of
seven LLMs across ~58,000 statement–source pairs found that 50–90% of responses were not fully supported by
the sources they cited, and even a web-search-enabled model left roughly 30% of individual statements
unsupported [78]. A citation that does not support the claim is worse than no citation, because it manufactures
false confidence. Equally, a clickable citation does not by itself establish an ability to independently review the
basis of a recommendation: the relevance, sufficiency, and comprehensibility of the underlying basis must be
demonstrated for the intended user and the decision time available.

We therefore recommend that traceability be credited as a risk-reducing factor only when source fidelity is
demonstrated (new benchmarking element S.4, Section 4.2) and when the user interface makes the source
readily inspectable within the time available for the decision. For retrieval-augmented clinical content,
evaluation should cover not only citation existence and statement-level entailment but also source authority,
version and date, conflicting guidance, withdrawn or superseded materials, and the device’s behavior when
sources are insufficient.

3.4 Competence must be demonstrated in interaction, not only in knowledge
A randomized, preregistered study of 1,298 members of the public found that LLMs alone identified relevant
conditions in 94.9% of scenarios, yet participants using the same LLMs identified relevant conditions in fewer
than 34.5% of cases and chose the correct disposition in fewer than 44.2%—no better than a control group—and
standard benchmarks and simulated users did not predict these failures [59]. Exam-style accuracy can also
overstate reasoning: replacing correct MedQA answers with “none of the other answers” reduced accuracy for
every frontier model tested [63], and a systematic review of 519 LLM evaluation studies found only 5% used real
patient-care data [64].

These findings map onto Miller’s pyramid of clinical competence (“knows → knows how → shows how → does”)
[81]. Benchmarks largely test the lower levels; clinical confirmation must test the upper levels. We recommend
that FDA make this mapping explicit, and that human–AI team performance be a first-class benchmarking
element wherever a human is in the loop (Section 4.2, Q9 and Q14).

3.5 Predictability for sponsors: structured, voluntary Pre-Submission feedback on GenAI
evaluation plans
The competency-based approach asks sponsors to define test scope, rubrics, adjudicators, and acceptance
criteria themselves [1]. That flexibility is valuable, but it also creates the single largest source of regulatory risk
for sponsors: discovering at the review stage that FDA views the benchmark scope, comparator, or threshold
differently. We recommend that CDRH pilot a standardized, voluntary feedback template for GenAI evaluation
plans within the existing Q-Submission Program [103]. Pre-Submission feedback is ordinarily nonbinding, and we
do not ask FDA to promise binding review outcomes through this mechanism. Rather, a five-part template—

Page 8 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

covering (i) intended-use boundaries and two-axis placement, (ii) principal hazards, (iii) benchmark and clinicalconfirmation design, (iv) statistical analysis plan, and (v) proposed change-control and monitoring approach—
would record what FDA did and did not address, points of alignment, and assumptions that may need to be
revisited as evidence and knowledge evolve. This single mechanism would do more to reduce burden and
uncertainty than any individual test method.

Page 9 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

4. Responses to CDRH’s Discussion Questions
Each response restates the question in brief (shaded grey), provides our analysis, and closes with a
recommendation box. Question numbers follow Appendix B of the discussion paper.

4.1 Risk Assessment (Section IV; Questions 1–6)
Question 1 — Does the two-axis framework capture the most relevant risk dimensions?
Does the two-axis risk framework (device activity × consequence of relying on an incorrect output) capture the
dimensions most relevant to risk? Should additional dimensions—reversibility, downstream safeguards, time
pressure, traceability—be represented, and how?

Yes, as the primary heuristic—provided it is supplemented by explicit modifiers. The two-axis structure is
familiar to industry because it echoes the IMDRF SaMD categorization (significance of information × state of the
healthcare situation) [34] and the model-influence × decision-consequence matrices of FDA’s computational
modeling credibility guidance and ASME V&V 40 [21, 22]. That continuity is a strength: sponsors can reuse
existing risk-management reasoning under ISO 14971 and AAMI CR34971 [47].

However, two devices at the same grid position can differ by an order of magnitude in real risk. We recommend
keeping the grid simple and documenting a small, closed set of contextual risk factors (“modifiers”) to guide
function-specific risk assessment. They should inform—rather than automatically change—a function’s
placement: the discussion paper does not define a validated numeric band scale, and an unverified safeguard
should never be used as a discount. A closed set preserves predictability; an open-ended list would invite
inconsistency between review teams.

Risk-analysis safeguard. Consistent with ISO 14971 [47], sponsors should analyze hazard severity, probability of
occurrence and of reliance, detectability, exposure scale, and correlated failure patterns separately. A proposed
modifier should reduce estimated residual risk only with evidence that the associated control is effective in the
actual setting of use.

Table 2. Proposed risk modifiers for the two-axis framework

Modifier Why it changes risk Evidence needed to claim a Example
reduction

M1. Reversibility and Harm that unfolds slowly and can Clinical rationale for harm Weekly titration suggestion
time-to-harm be reversed allows detection and latency; demonstrated reviewed by HCP vs.
correction; irreversible or rapid detection/correction pathway immediate thrombolysis
harm does not (e.g., scheduled clinician order set
review before next dose)

M2. Independent If the user can verify the basis S.4 source-fidelity results; UI Guideline-grounded answer
verifiability and before relying on it, errors are evidence that sources are with verified citations for a
source traceability interceptable (cf. §520(o)(1)(E)); if inspectable within decision clinician vs. unreferenced
(bidirectional) the output is opaque (e.g., time; user-type-specific human patient advice; generative
generative measurement, signal- factors data; for opaque waveform denoising or
processing, or IVD functions, raw-signal/image virtual staining
transformations), undetected overlays, localized uncertainty
reliance becomes more likely maps, and analytical validation
against a reference standard

Page 10 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Modifier Why it changes risk Evidence needed to claim a Example
reduction

M3. Exposure scale A GenAI device replicates the Correlated-failure analysis; Direct-to-consumer triage
and error correlation same error across thousands of deployment scale estimate; at national scale vs. singleusers simultaneously; correlated diversity of safeguards; department pilot
errors defeat “many eyes” monitoring sensitivity to
safeguards systematic error (warrants
heightened attention, not a
downward adjustment)

M4. Architectural When clinical computations are Design evidence and Conversational intake with
containment deterministic and the generative adversarial testing that the deterministic dosing core
layer cannot alter them, generative layer cannot (cf. K253281)
generative error is bounded modify, paraphrase, or
override deterministic outputs

Downstream safeguards and deployment-setting time pressure, also named in Question 1, are captured within M1 (time-to-harm) and M2
(whether an independent check can occur before reliance). Any lower residual-risk estimate should be supported by verified safeguards and
clinical rationale; M3 should only raise attention, never lower it.

Measurement, signal-processing, and IVD functions (Section IV). CDRH asks whether GenAI-enabled in vitro
diagnostic, measurement, and signal-processing functions—which yield non-directive outputs the user typically
cannot independently evaluate—should migrate higher on the consequences axis. We agree that such functions
warrant heightened scrutiny, and that this is consistent with the independent-review logic of §520(o)(1)(E) noted
in the paper’s footnote 13. Our suggestion concerns where that scrutiny is represented. Under ISO 14971 [47],
the user’s inability to inspect the basis of an output does not change the clinical severity of the resulting harm; it
increases the probability that a hazardous output escapes detection and is relied upon. Representing opacity as
higher severity would blur the vertical axis across device types. We therefore suggest representing it through M2
operating in both directions: verified source traceability can support a lower residual-risk estimate, while an
opaque generative measurement or transformation (e.g., generative waveform denoising, virtual histological
staining, synthetic image sequences) warrants a higher one unless mitigated by side-by-side raw-signal or image
display, localized uncertainty quantification, and analytical validation against an established reference standard.
The practical outcome is the same as CDRH’s proposal—more evidence for opaque functions—but with a cleaner
risk rationale.

Two further observations. First, the activity axis mixes two distinct concepts—how directive an output is and
how much human supervision exists before action. We suggest the guidance explicitly separate “informational
directiveness” (Q2) from “action authority and supervision” (Q26) as sub-scales of the same axis, because
conversational devices can move along the first without changing the second. Second, consequences should be
assessed as the reasonably foreseeable severity of harm for relevant uses and vulnerable subgroups, with
likelihood, exposure, and impact reported separately. Aggregating harm into a population average must not hide
low-frequency catastrophic outcomes; at the same time, not every patient-facing function should default to
“severe” when the foreseeable harms are in fact limited.

Recommendation (Q1)
• Retain the two-axis grid; document a closed set of four contextual risk factors (Table 2) that inform the
risk analysis without automatic band changes or unsupported risk reduction.

Page 11 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Represent the opacity of GenAI measurement, signal-processing, and IVD functions as increased
probability of undetected reliance (M2 operating in both directions), rather than as increased severity
on the consequences axis.
• Split the activity axis into two sub-scales—informational directiveness and action authority/supervision.
• Publish 10–15 worked examples placing hypothetical devices on the grid, including how contextual
factors are documented, to promote consistency across review divisions.

Question 2 — How to account for the spectrum between “non-directive” and “action-directing”
information?
What output characteristics—wording, specificity, personalization, context—could modify risk? What would
give manufacturers clarity and predictability, recognizing directiveness is a continuum?

We agree with CDRH that directiveness depends on “the substance and context of the output, not solely on
whether it uses words such as ‘recommend,’ ‘should,’ or ‘consider,’” and that disclaimers such as “talk to your
doctor” do not reduce directiveness [1]. The lisinopril example in the paper already describes a natural ordinal
scale. We propose piloting it as a nonbinding, descriptive Directiveness Ladder, scored on the combination of
four observable features: specificity (general → patient-specific), personalization (population → this user’s data),
actionability (information → executable instruction), and endorsement (neutral → device-endorsed).

Table 3. Proposed Directiveness Ladder (using CDRH’s own lisinopril illustration)

Level Description Illustration (from discussion paper) Typical features

D0 General reference (Patient education about hypertension, no Population-level; no action
information treatment content) content

D1 General practice “Lisinopril dosages are sometimes increased Population-level; describes
information when blood pressure remains above the practice
treatment goal.”

D2 Contextualized practice “In situations similar to this, clinicians often Linked to user’s situation;
increase the lisinopril dosage.” implicit action

D3 Endorsed action “I recommend increasing the lisinopril dosage.” Personalized; device endorses
an action

D4 Executable instruction “Increase the lisinopril from 10 mg to 20 mg Specific, quantitative,
daily.” immediately actionable

For GenAI devices, the critical insight is that directiveness is a property of the output distribution, not of the
design intent. The same device will produce D1 answers most of the time and D3–D4 answers in some fraction
of conversations—especially under user pressure. We recommend that sponsors report an Observed
Directiveness Profile: the distribution of directiveness levels across a prespecified, realistic scenario set
(including adversarial and “insistent user” personas), with the device characterized by its upper tail (e.g., the
level not exceeded in ≥99% of outputs, with confidence bounds). Classification can be performed by blinded
clinician raters, with validated automated raters used for scale (see Q12 and Q20 on statistically valid use of
automated graders).

Page 12 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Where a sponsor intends a directiveness ceiling (e.g., “never above D2”), that ceiling becomes a design
requirement verified under benchmarking element S.2 (scope maintenance), and its breach rate becomes a
postmarket key risk indicator.

The ladder could also help sponsors reason about the device/non-device boundary, although it cannot replace
the statutory analysis. Patient-facing functions fall outside the §520(o)(1)(E) CDS exclusion, which applies to
functions intended for health care professionals. For HCP-facing functions, the ladder could make the January
2026 CDS guidance’s criteria—including its treatment of single recommendations and time-critical use—easier to
apply to conversational outputs whose directiveness varies from turn to turn [5]. We encourage FDA to include
worked examples showing how ladder levels, observed directiveness, and source fidelity (S.4) bear on each CDS
criterion. Separately, the ladder should keep endorsing an action (D3) distinct from executing one, which belongs
on the action-authority sub-scale (Q26).

Recommendation (Q2)
• Pilot a five-level Directiveness Ladder (D0–D4) defined by specificity, personalization, actionability, and
endorsement, as a common descriptive vocabulary.
• Characterize devices by an Observed Directiveness Profile (upper-tail behavior), not by labeling,
wording conventions, or disclaimers.
• Treat a sponsor-declared directiveness ceiling as a verifiable design requirement and a postmarket
metric.
• Publish worked examples relating ladder levels to the §520(o)(1)(E) criteria and the January 2026 CDS
guidance for HCP-facing functions.

Question 3 — Patient-facing versus HCP-facing informational functions
When might delivery of clinical information to patients present different or higher risks, while recognizing
benefits of patient empowerment? What characteristics or safeguards mitigate risk without underestimating
patient capability?

We recommend reframing the question from who the user is to what verification capacity the user has in the
moment of reliance. Credential category is only a proxy. A board-certified cardiologist is a weak verifier of a
pediatric dermatology output; a patient with long-standing type 2 diabetes may be a strong verifier of a glucoselog summary but a weak verifier of an insulin titration. This framing respects patient capability, avoids medical
parentalism, and remains consistent with the independent-review logic of §520(o)(1)(E) [5].

The evidence supports caution for patient-facing, action-directing functions. In addition to the human–LLM
interaction gap described in Section 3.4 [59], a structured evaluation of a consumer health chatbot reported
under-triage of about half of gold-standard emergencies and over-triage of about two-thirds of non-urgent
cases, with symptom minimization in prompts markedly increasing the odds of a less urgent recommendation
[76]. That study has been methodologically challenged on the grounds that exam-style protocols suppress
clarifying questions [77]—which itself reinforces our recommendation that patient-facing devices be evaluated
in naturalistic, multi-turn conditions with realistic users.

Mental health illustrates both sides of the balance CDRH describes, and was the focus of the November 2025
Digital Health Advisory Committee meeting [4]. A randomized trial of a GenAI therapy chatbot reported
significant symptom reductions versus waitlist [90], while an evaluation of general-purpose and therapy-branded

Page 13 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

LLMs documented stigma and inappropriate responses to serious scenarios such as delusions and suicidal
ideation [89]. Purpose-built, evaluated, and monitored devices are precisely how the benefits can be captured
while the risks are controlled.

Mitigations that genuinely reduce risk for patient-facing functions (and that sponsors could cite as modifiers
M1/M2) include:

• Verification-capacity-matched design: health-literacy-adaptive explanations, teach-back confirmation for
any D3–D4 output, and plain-language uncertainty statements (benchmark E.4).
• Structural escalation affordances: one-tap connection to a clinician or emergency services that is always
visible, rather than buried in text.
• Clinician-in-the-loop gating for D3–D4 outputs above a consequence threshold (e.g., dose changes), with
service-level timing commitments.
• Asymmetric guardrails that default toward escalation for red-flag symptoms, while measuring overescalation (Q6).
• Human factors validation with representative lay users across literacy levels, consistent with FDA’s
human factors guidance (finalized for submission content in May 2026) [19].

Finally, the patient-facing question cannot be separated from the regulatory boundary. General-purpose
chatbots are already used by large numbers of consumers for health questions, and several states have enacted
laws restricting AI-delivered mental-health therapy or AI that implies professional licensure [95]. A framework
that imposes evidence expectations only on products that voluntarily seek authorization, while functionally
similar general-purpose tools operate without oversight, would disadvantage responsible sponsors. We
encourage CDRH to clarify, with examples, how objective intended use (21 CFR 801.4) applies when generalpurpose models are marketed with health-specific features (see Section 5.2).

Recommendation (Q3)
• Use the user’s verification capacity at the moment of reliance as the operative risk variable, with
patient/HCP status as a default proxy that sponsors may rebut with evidence.
• Treat patient-facing D3–D4 functions as warranting heightened evidence by default, rebuttable by
demonstrated safeguards (teach-back, gating, escalation affordances, human factors data).
• Require naturalistic, multi-turn, lay-user testing for patient-facing functions; do not rely on exam-style
or single-turn benchmarks.

Question 4 — Generalist versus specialist HCP-facing functions
How might the generalist–specialist distinction inform risk? When does risk change if an HCP lacking specialist
knowledge receives specialist-area information? What safeguards mitigate this?

This is the same verification-capacity problem as Question 3, applied within the clinician population, and it is no
longer hypothetical: 81% of U.S. physicians surveyed in early 2026 reported using AI professionally [94].
Automation-bias evidence is directly relevant: in a mammography reader study, when AI-suggested BI-RADS
categories were incorrect, the accuracy of inexperienced readers fell to 19.8% (from 79.7% when the AI was
correct), while very experienced readers fell to 45.5% (from 82.3%) [72]. Less domain expertise means more

Page 14 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

susceptibility to a fluent, authoritative—but wrong—output. FDA’s January 2026 CDS guidance also added
discussion of automation bias under Criterion 4 [5].

We recommend that risk increase when (a) the intended user population includes generalists, (b) the output
falls in a domain where specialist contextualization is needed to detect errors, and (c) the consequence of
reliance is moderate or severe. Mitigations include domain-of-competence labeling, explicit “specialist-referral
triggers,” output formats that expose the reasoning and key discriminating features (supporting independent
review), and benchmark testing with generalist users as part of human–AI team evaluation (H.1, Section 4.2).
The comparator should match the claim: if a device claims to extend specialist knowledge to generalists, the
relevant question is whether generalist + device approaches specialist performance without unacceptable new
failure modes (Q14).

Recommendation (Q4)
• Treat “specialist-dependent outputs delivered to generalist users” as a risk-raising condition when
consequence is moderate or severe.
• Require human–AI team testing with the actual intended clinician population (generalist and specialist
strata) for such functions.
• Credit mitigations that expose reasoning, discriminating findings, and specialist-referral triggers.

Question 5 — Multi-turn conversations and emergent intended use
How should risk be assessed across realistic conversational trajectories for devices that migrate from nondirective to action-directing information? How can intended use be characterized when behavior is emergent?

Multi-turn behavior is where GenAI devices most clearly break single-input/single-output assumptions. Across
more than 200,000 simulated conversations and 15 LLMs, performance dropped by an average of 39% when task
information was spread over several turns, mostly due to increased unreliability rather than lost capability [85].
Sycophancy compounds the problem: LLMs complied with illogical medical requests at high rates unless
explicitly prompted to reject them [65].

We propose that the intended use of a conversational device be expressed as a Conversational Trajectory
Envelope—a specification, akin to a state diagram, of:

1. Permitted functions (states), each with its two-axis placement and directiveness ceiling (e.g., “symptom
intake, D0–D1”; “treatment-plan explanation, D2”; “dose instruction from deterministic core, D4 verbatim
only”).
2. Permitted transitions between functions, and the conditions for each (e.g., “may move from education to
triage only after red-flag screen”).
3. Mandatory exits: escalation, refusal with redirection, or human hand-off when a conversation would cross
the envelope boundary.
4. Session-level constraints: maximum conversation length before re-grounding, memory/persistence rules,
and handling of contradictory user statements.

Page 15 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Risk is then assessed at the level of the envelope (its highest-risk reachable state) and verified by trajectory
sampling: large numbers of simulated multi-turn conversations generated from clinician-authored scenario
seeds and diverse user personas (including minimizing, insistent, distressed, low-literacy, and adversarial users),
with metrics such as envelope-escape rate, drift-to-directive rate (transitions to a higher directiveness level than
permitted), and time-to-escalation in evolving presentations. This approach also gives the PCCP and postmarket
program a concrete object to monitor.

Because multi-turn reliability degrades as conversations lengthen [85], trajectory sampling should be stratified
by conversation length (for example, 5, 10, 20, and 50 turns) and across session boundaries, to confirm that
context truncation or summarization does not drop red-flag history. Where a device retains information across
sessions, a memory schema should define what may persist (structured, verifiable fields rather than free-text
inferences), for how long, and how users or clinicians can inspect and correct it—so that stale symptoms from a
prior session cannot silently contaminate a new triage episode.

Recommendation (Q5)
• Consider describing conversational intended use as a Conversational Trajectory Envelope (functions,
transitions, mandatory exits, session constraints)—a testable behavioral specification.
• Assess risk across credible trajectories, including reasonably foreseeable envelope escapes and the
most severe reachable states; verify with trajectory sampling stratified by conversation length and
adversarial personas; report envelope-escape and drift-to-directive rates with confidence bounds.
• Where cross-session memory is used, validate a bounded memory schema with inspection, correction,
and expiry rules.
• For products with meaningful conversational scope boundaries, consider envelope-escape rate as a
device-specific postmarket key risk indicator.

Question 6 — Weighing under-escalation and over-escalation
How could manufacturers characterize and weigh under- and over-escalation, given that they may not be
commensurable and acceptable trade-offs vary by context?

We agree that both directions of error matter. The history of symptom checkers is instructive: an audit of 23
symptom checkers found appropriate triage advice in only 57% of standardized vignettes, with performance
much better for emergent cases (80%) than for self-care cases (33%)—i.e., tools tended toward over-escalation
[75]. Recent consumer-chatbot evaluations show the opposite risk is also real [76]. Because the two errors are
not commensurable, we recommend a two-tier structure rather than a single blended metric:

1. Tier 1 — Non-compensatory safety floor. For a prespecified list of time-critical conditions relevant to the
intended use (e.g., chest pain with high-risk features, stroke symptoms, sepsis, DKA, suicidal ideation with
intent), the device must meet a minimum escalation sensitivity, demonstrated with a lower confidence
bound at or above a justified threshold. No amount of over-triage reduction may offset a floor failure.
2. Tier 2 — Weighted trade-off above the floor. For the remaining distribution, sponsors prespecify harm
weights for each error type and acuity stratum (with clinical and patient input) and report net benefit
using decision-curve methods [86] across a plausible range of weights, so that conclusions are robust to
the exact weighting.

Page 16 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Results should be reported as a full acuity-stratified confusion matrix (not a single accuracy figure), alongside the
corresponding performance of the realistic comparator (e.g., existing nurse advice lines or usual care) so that
over-escalation is judged against current practice rather than an idealized zero. Postmarket, sponsors should
monitor downstream signals of both errors—e.g., emergency visits within 72 hours after “self-care” advice
(under-escalation) and the proportion of escalations judged unnecessary on clinician review (over-escalation).

Two further points respond to CDRH’s concern that over-escalation can erode trust and reduce appropriate careseeking over time. First, sponsors should report positive predictive value and the number of escalations needed
to identify one true emergency, calibrated to the acuity mix of the intended population rather than to a
balanced test set. Second, for longitudinal or chronic-care devices, sponsors should monitor “alert fatigue”—
whether repeated unnecessary escalations reduce adherence when a true red-flag escalation occurs. Finally, rare
time-critical scenarios should be tested with adjudicated sentinel cases and prespecified minimum performance
criteria, without claiming that strong sensitivity on synthetic stress tests predicts equally high sensitivity in the
deployed population; even a small number of severe under-escalations should trigger case-level review
regardless of aggregate accuracy.

Recommendation (Q6)
• Use a non-compensatory safety floor for time-critical conditions plus a weighted net-benefit analysis
above the floor.
• Require acuity-stratified confusion matrices and comparator-relative reporting; prespecify harm
weights with clinician and patient input and show robustness across weights.
• Carry both error directions into postmarket monitoring with outcome-linked indicators, prevalencecalibrated PPV, and alert-fatigue monitoring for longitudinal devices.

4.2 Competency-Based Premarket Evaluation (Section V; Questions 7–17)
Question 7 — Is benchmarking followed by clinical confirmation a useful framework?
Is the competency-based approach—device benchmarking followed by clinical confirmation—a useful and
appropriate framework for evaluating GenAI-enabled devices?

Yes. It is the most promising structure we have seen for devices whose input space cannot be exhaustively
sampled, and it rightly evaluates “the final user-facing device, as configured and intended to be deployed” rather
than the foundation model alone [1]. The clinician-credentialing analogy proposed by Patel and Blumenthal and
by Bergman, Wachter, and Emanuel [55, 56] is apt, and CDRH is right to adapt it to device law rather than adopt
licensure wholesale, especially given the accountability concerns raised by Freyer et al. [57]. Keeping the
approach product-centric is also wise: FDA’s Software Pre-Cert pilot concluded that an organization-level
appraisal model would require new legislative authority [23].

We recommend importing one more concept from competency-based medical education that the paper does
not yet use: Entrustable Professional Activities (EPAs) [80]. In medical training, competence is not certified
globally; trainees are entrusted with specific activities at specific supervision levels, and entrustment expands as
evidence accumulates. Applied to devices, Entrustable Device Activities (EDAs) would decompose the intended
use into discrete activities (e.g., “summarize medication history,” “generate differential diagnosis,” “advise on ED
attendance”), each authorized at an explicit supervision level:

Page 17 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Table 4. Entrustable Device Activities: supervision levels (adapted from EPA entrustment scales)

Level Supervision Device implication

E1 Observation only (shadow) Outputs recorded, not shown; used for clinical confirmation or new-site
validation

E2 Direct supervision Every output reviewed/approved by a qualified clinician before
reaching the user or the record

E3 Indirect supervision Outputs delivered; clinician reviews asynchronously within a defined
window; sampling + rules-based review

E4 Oversight by exception Outputs act autonomously within envelope; human review triggered by
flags, uncertainty, or red-flag content

E5 Autonomous within scope Fully autonomous action only within a validated, legally authorized
intended use and applicable safety controls

Each EDA has its own two-axis placement, acceptance criteria, and supervision level. Progression (e.g., E2 → E3) could be proposed within a
PCCP only if the modification, the development and validation methods, and the impact assessment are sufficiently specified and
authorized; greater autonomy outside an authorized PCCP may require a new marketing submission (Q18, Q23).

EDAs offer three practical benefits: they make intended use granular and testable; they let a sponsor launch a
device with mixed autonomy (low-risk activities at E4, higher-risk ones at E2); and they create a principled,
evidence-gated path for expanding autonomy that aligns with the UK National Commission’s recommendation
for staged authorization and expanded PCCPs [38]. We offer EDAs as an optional activity-and-supervision map,
not as a new authorization category.

Recommendation (Q7)
• Adopt the benchmarking + clinical confirmation structure.
• Offer sponsors an optional activity-and-supervision map (E1–E5); any progression in autonomy should
require a suitably prespecified, FDA-authorized PCCP or, where applicable, a new submission.
• Map benchmark elements to Miller’s pyramid levels so sponsors and reviewers can see which levels of
competence each piece of evidence actually supports.

Question 8 — Using the two-axis framework to set the level of evidence
How might the two-axis risk framework be considered within the competency-based approach to determine
the level of evidence needed for a premarket submission?

We propose an illustrative Risk–Evidence Matrix that shows how grid position and contextual factors might
inform evidence expectations. The bands below are not FDA classifications and are not proposed as mandatory
study designs; evidence should be determined case by case, informed by FDA feedback through the existing PreSubmission process (Section 3.5). Publishing illustrative examples of this kind would be the single most useful
predictability tool CDRH could provide.

Page 18 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Table 5. Illustrative Risk–Evidence Matrix (not FDA classifications or mandatory study designs)

Illustrative risk Benchmarking scope Clinical confirmation (example) Postmarket & change control
grouping

Band 1: Low (e.g., Core elements S.1–S.3, E.1, E.4, Retrospective evaluation on Periodic re-benchmarking;
D0–D1, limited R.1; sponsor benchmarks real inputs or standardized QMS-documented changes
consequence) acceptable with validity dossier interactions within envelope

Band 2: Moderate All applicable elements incl. S.4– Clinician adjudication of real Sample-based clinician review +
(e.g., D2–D3 or S.5, H.1, R.2–R.3; at least one cases and/or shadow drift monitoring; PCCP with reHCP-supervised sequestered/independent test deployment; human–AI team benchmark gates
action, moderate set study where HCP in loop
consequence)

Band 3: High (e.g., Full element set; independent Gated or limited deployment Monitoring-Readiness Criteria
D4 or autonomous or sequestered testing where and/or prospective study (Q18); any autonomy
action, moderate– justified; adversarial red- (randomized where feasible) progression only within an
severe teaming authorized PCCP; FDA reporting
consequence) as required

Band 4: Critical As Band 3 plus closed- Clinical evidence suitable for Pre-specified stopping rules;
(autonomous, loop/agentic competencies the high-consequence claim; a real-time safety supervision;
severe/irreversible, (A.1–A.4) prospective controlled study limited initial scope
time-critical) when scientifically warranted

Recommendation (Q8)
• Publish illustrative risk–evidence examples linking intended use, consequences, evaluation, and
postmarket controls—without establishing mandatory study designs by numerical band.
• Encourage sponsors to discuss alternative evidence plans through existing, ordinarily nonbinding PreSubmission interactions.

Question 9 — Adequacy of the benchmarking structure; missing elements; external standards
Would the benchmarking structure provide adequate evidence? Are elements missing, redundant, or
miscategorized? Are there external standards to leverage?

The proposed elements (S.1–S.3, E.1–E.4, R.1–R.2, A.1) are well chosen. We see six gaps and one
recategorization:

Table 6. Proposed additional benchmarking elements

Proposed element What it would test Evidence basis / rationale

S.4 Source fidelity and Whether every cited source exists, is authoritative Citation unfaithfulness is common even
traceability and current, and actually supports the specific with retrieval [78]; required to credit
claim (statement-level entailment); whether modifier M2
retrieval surfaces the right source; handling of
conflicting, withdrawn, or superseded sources and
of failed retrieval

Page 19 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Proposed element What it would test Evidence basis / rationale

S.5 Sycophancy and user- Whether the device maintains correct, safe High compliance with illogical medical
pressure resistance positions when users assert false premises, push requests [65]; symptom minimization
back, minimize symptoms, or request unsafe shifted triage [76]. Currently only partly
actions covered in S.1

S.6 Privacy and Leakage of PHI across sessions/users, Unique to generative systems with
confidentiality memorization, inappropriate disclosure in shared memory and retrieval; complements
settings, prompt-based exfiltration §524B cybersecurity [11]

E.5 Knowledge currency Whether outputs reflect current guidelines, Clinical knowledge changes continuously;
labeling, and safety communications as of a stated supports change triggers (Q19)
knowledge date; behavior when asked about
post-cutoff developments

H.1 Human–AI team Performance of the intended user with the device LLM-alone performance did not transfer to
performance and vs. without; error propagation when the device is users [59]; automation bias [72]. Links to
automation bias wrong; over-reliance and under-reliance IEC 62366-1 [48] and human factors
guidance [19]

R.3 Stochastic stability Consistency of safety-critical behavior across Nondeterminism at temperature 0 [71];
and configuration repeated runs (consistency@k), inference drift across versions [70]; inconsistent
sensitivity settings, hardware/batching, and minor race-based outputs across runs [73]
configuration changes

On calibration (S.3). Across 12 models on a benchmark containing unanswerable and fabricated questions, LLMs
seldom recognized the limits of their knowledge and answered confidently when no correct option existed [82].
S.3 should therefore include unanswerable and out-of-scope items, and sponsors should be encouraged (not
required) to use measurable uncertainty methods—such as semantic-entropy-based confabulation detection
[67] or conformal prediction sets with statistical coverage guarantees [91]—to trigger deferral.

Refinements to existing elements. Three refinements would make the proposed elements harder to “game.” (1)
S.2 and S.3 each have two failure directions, as Appendix A of the paper recognizes: under-refusal or underdeferral compromises safety, while excessive refusal or deferral makes a device clinically useless and can push
users toward unregulated general-purpose tools. Sponsors should therefore report both together—safetyrefusal sensitivity alongside in-scope task completion—rather than a refusal rate alone. (2) For E.2, multi-turn
simulations should score the quality and sequencing of follow-up questions and resistance to premature closure
when early benign information masks an evolving serious condition. (3) For E.3, because token-by-token
generation is brittle for dosing arithmetic and unit conversion, we recommend that higher-risk devices delegate
clinical calculations and reference-range lookups to deterministic tools or verifiers (linking E.3 to architectural
containment and A.2), and that E.3 test whether that delegation occurs reliably.

Recategorization. The automation-bias content currently in E.4 (communication quality) is better located in H.1,
because it is a property of the human–device system, not of the text. E.4 can then focus on comprehension,
empathy, literacy, and non-manipulative language. For agentic devices, we suggest splitting A.1 into A.1
planning and task decomposition, A.2 tool-use integrity, A.3 authority and permission adherence, and A.4
recovery and rollback (see Q26), so acceptance criteria can be set separately for each.

Standards to leverage. No single standard covers GenAI device evaluation, but several provide ready-made
scaffolding: ISO 14971 with AAMI CR34971 (FDA-recognized) for ML-specific risk management [47]; IEC 62304,

Page 20 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

IEC 62366-1, and IEC 82304-1 for software life cycle, usability, and health software [48]; AAMI CR515:2025 on ML
cybersecurity, recognized by FDA in March 2026 [51]; ISO/IEC TS 8200 on controllability of automated AI
systems, relevant to human hand-off and override [50]; ISO/IEC 42001 and 23894 for AI management and risk
processes [49]; the NIST AI 600-1 Generative AI Profile, whose risk catalogue (e.g., confabulation, human–AI
configuration, value-chain and component integration) maps closely to the benchmark elements [46]; ASME V&V
40 credibility concepts for simulators and virtual patients [21]; IEEE P3926, a recommended practice in
development for LLM-driven patient-oriented healthcare services [52]; and the CHAI Applied Model Card for
transparency [53]. We recommend FDA publish a crosswalk from each benchmark element to these standards
and prioritize recognition of test-method standards as they mature.

Recommendation (Q9)
• Add S.4, S.5, S.6, E.5, H.1, and R.3; move automation bias from E.4 to H.1; split A.1 into A.1–A.4.
• Report over- and under-refusal together for S.2/S.3; score premature-closure resistance in E.2; for
higher-risk devices, test reliable delegation of clinical calculations to deterministic tools in E.3.
• Publish a crosswalk of benchmark elements to existing standards and prioritize recognition of GenAI
test-method standards.

Question 10 — Establishing that benchmark performance predicts real-world behavior
How should a sponsor establish that benchmark performance predicts safe and effective real-world behavior?
What evidence should support construct validity? What role for sponsor-developed benchmarks?

The concern is well founded. Public medical benchmarks derived from licensing exams show significant
construct-validity gaps when compared with real clinical data [62]; FDA’s own scientists have warned about
“train-to-the-test” overfitting [58]; and large benchmark efforts such as HealthBench (5,000 multi-turn
conversations, 262 physicians) and MedHELM (121 clinician-defined tasks) illustrate both the value of clinicianauthored, rubric-based evaluation and the need to refresh benchmarks continually as models improve [60, 61].
We recommend that benchmarks relied upon for a material performance claim be supported by riskproportionate validity evidence, which we summarize as a Benchmark Validity Dossier (sponsors should be able
to assemble it from existing records):

Table 7. Benchmark Validity Dossier — minimum content

Validity facet Evidence expected Practical method

Construct validity Explicit mapping from intended use and EDAs to the Task taxonomy and traceability
constructs measured; justification that the construct is matrix (intended use → EDA →
the clinically relevant one element → items)

Content validity Coverage of the input distribution, acuity mix, Coverage map with stratum counts;
subgroups, and edge cases; clinician authorship/review independent clinician panel review
of items

Rubric/scoring validity Inter-rater reliability among clinicians; agreement of ICC/kappa; automated-grader
any automated grader with clinicians, by stratum validation (e.g., [69]); bias checks
for self-preference

Page 21 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Validity facet Evidence expected Practical method

Contamination control Evidence that test items were not in training or tuning Sequestered splits; item freezing
data and did not influence model/prompt selection before final configuration lock;
canary strings;
paraphrase/perturbation delta tests
(cf. [63])

Headroom (non- Items capable of discriminating acceptable from Item-difficulty analysis; inclusion of
saturation) unacceptable performance hard and adversarial strata

Predictive validity Evidence that benchmark results correlate with clinical- Bridging analysis between
confirmation results for this device class (where benchmark strata and clinical
demonstrable; it may not yet be estimable for novel confirmation outcomes; accumulate
devices) across submissions

Sponsor-developed benchmarks should be welcome—they are often the only way to test a narrow intended
use—provided four safeguards are met: (1) items are prespecified and frozen before the final configuration is
locked; (2) a sequestered portion is held by an independent custodian or placed in benchmark escrow (e.g.,
hashed and time-stamped with FDA or a third party), so the sponsor cannot iteratively optimize against it; (3)
items are periodically refreshed to prevent postmarket overfitting; and (4) at least one independent or public
benchmark is reported alongside for calibration. Over time, FDA could aggregate (confidentially) the relationship
between benchmark and clinical-confirmation results across submissions to build empirical evidence of
predictive validity for each element—an important regulatory-science asset.

A lighter-weight alternative: a risk-to-evidence traceability ledger. Rather than an additional standalone
dossier, sponsors could present a single ledger generated from existing records: intended use and function →
clinical hazard → risk control → evaluation item and data provenance → acceptance criterion → observed result
and uncertainty → residual risk → postmarket indicator and owner. This gives reviewers a line of sight from
clinical risk to evidence and maps naturally onto eSTAR and Pre-Submission discussions.

Recommendation (Q10)
• Request risk-proportionate benchmark validity evidence (construct, content, scoring, contamination,
headroom, and predictive validity where demonstrable) for consequential performance claims, and
permit reuse of existing evidence records.
• Consider a risk-to-evidence traceability ledger as a practical format.
• Accept sponsor-developed benchmarks with freezing, escrow/sequestration, refresh, and calibration
safeguards.
• Have CDRH’s regulatory science program accumulate cross-submission evidence on the predictive
validity of benchmark elements.

Question 11 — Selecting and justifying a clinical confirmation approach
How might a sponsor select and justify a confirmation approach proportionate to risk? For which device types
would approaches be insufficient? How should the real-world input distribution be considered? Are there other
methods?

Page 22 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

We agree that a prospective study should not be required in every case. Selection should be driven by four
questions: (1) What is the risk band (Table 5)? (2) Can errors be observed by the evaluator in the chosen setting?
(3) Is the input distribution in the confirmation setting representative of deployment? (4) Are the inputs
endogenous to the device’s outputs?

The fourth question is, in our view, the most important and least discussed. For conversational devices, each
user turn depends on the device’s previous outputs, so historical transcripts are not valid test inputs for a
different device—the conversation would have unfolded differently. Retrospective evaluation and blinded
clinician review of real cases therefore work well for single-shot functions (e.g., summarizing a record, drafting
an impression) but are inherently limited for interactive ones. For interactive devices, we recommend usersimulators calibrated against real transcripts for scale, followed by live-user methods for confirmation.

We also propose two additional confirmation methods for the list in Section V.C.1:

• Gated (clinician-in-the-loop) deployment. The device runs live with real patients, but every output is
reviewed and, if necessary, corrected by a qualified clinician before release (EDA level E2). This captures
real, endogenous inputs and real user behavior with essentially no unreviewed exposure, and the
edit/override rate itself becomes a powerful performance measure. It is the natural confirmation method
for patient-facing conversational devices, where true shadow deployment is impossible because the
patient must see an output to continue the conversation.
• Staged, cluster-randomized rollout (e.g., stepped-wedge). For devices deployed across sites, staggered
introduction allows concurrent comparison with usual care and is compatible with health-system
operations.

Regarding the input distribution: sponsors should present an Input Distribution Coverage Map—the anticipated
deployment mix by acuity, population, language/literacy, and use context—and report performance both by
stratum and re-weighted to the deployment mix. Approaches are insufficient on their own when: standardized
patient interactions are used as the sole evidence for high-consequence patient-facing triage (actors cannot
reproduce real distress, ambiguity, or minimization); retrospective evaluation is used for interactive devices
(endogeneity); or shadow deployment is used where the device’s value depends on changing user behavior.

Real-world data and data collected outside the United States (Section V.C.1). CDRH specifically invites input on
these sources. For GenAI devices, OUS data raise domain-transfer issues that differ from those of most hardware
devices: anatomy and physiology are largely shared across borders, but GenAI outputs are tightly coupled to
local clinical guidelines, drug formularies and names, measurement units, language, and care-delivery pathways.
Building on 21 CFR 812.28 and FDA’s guidance on accepting clinical data from outside the United States [99,
101], we suggest that sponsors relying on OUS confirmation data provide a three-part GenAI OUS bridging
justification: (1) guideline and formulary concordance—reference standards, medication names and availability,
and units (e.g., mmol/L vs. mg/dL) match U.S. practice, or outputs were re-adjudicated by U.S. clinicians; (2)
language and health-system pragmatics—idioms, dialects, and escalation routing (for example, the UK’s
999/111/GP/A&E pathways versus U.S. 911/primary care/urgent care/ED) do not confound escalation (S.1) or
comprehension (E.4); and (3) workflow equivalence—documentation, EHR, and agentic workflows reflect U.S.
practice. For real-world data generally, consistent with FDA’s real-world evidence guidance [100], records should
capture the configuration fingerprint (Section 3.1) and the full multi-turn context needed to attribute outcomes
to a specific configuration.

Page 23 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

21 CFR Parts 50, 56, and 812 (footnote 20). Uncertainty about investigational-device requirements often stalls
the very confirmation methods the paper describes. We encourage CDRH to clarify, with examples, (a) when
silent shadow deployment—where outputs are never displayed and cannot affect care—may fall within an IDE
exemption (for example, 21 CFR 812.2(c)(3) for qualifying diagnostic devices) or otherwise be treated as nonsignificant risk; (b) when clinician-gated deployment, in which a qualified clinician reviews every output before
release, would ordinarily be considered non-significant risk under 21 CFR 812.2(b); and (c) when an IRB may
waive or alter informed consent for such minimal-risk investigations under 21 CFR 50.22 [99]. These
determinations remain fact-specific, but worked examples would materially reduce delay and cost.

Recommendation (Q11)
• Select confirmation methods using risk band, error observability, input representativeness, and input
endogeneity.
• Add gated (clinician-in-the-loop) deployment and staged cluster-randomized rollout to the list of
confirmation methods.
• Require an Input Distribution Coverage Map and deployment-weighted performance estimates.
• For OUS confirmation data, expect a GenAI bridging justification covering guideline/formulary
concordance, language and care-pathway pragmatics, and workflow equivalence; for RWD, require
configuration-level attribution.
• Clarify with examples how IDE exemptions, non-significant-risk determinations, and minimal-risk
consent waivers may apply to shadow and clinician-gated confirmation.

Question 12 — Statistically meaningful performance measurement; synthetic data; combining evidence
How should sponsors achieve statistically meaningful measurement? How should differences between
synthetic and real distributions be handled? When can benchmarking and clinical confirmation evidence be
combined?

Stochastic, open-ended outputs require some adjustments to familiar practice. A foundational clarification:
safety claims should distinguish (i) per-output variability, (ii) per-scenario reliability across repeated generations,
and (iii) prevalence-weighted real-world risk, with uncertainty reported separately for each. Repeated outputs
for the same prompt measure within-scenario stochasticity; they are not independent patient encounters, and
arbitrarily generated prompts are not a sample of the patient population. We recommend that guidance address
five points:

1. Repeated sampling and clustering. Each scenario should be run multiple times (k ≥ 5 is a reasonable
default for safety-critical scenarios) under deployed inference settings, with uncertainty estimated by
scenario-clustered methods (e.g., cluster bootstrap or hierarchical models). Treating repeated runs as
independent observations overstates precision.
2. Consistency@k for safety-critical behavior. For behaviors such as escalation and refusal, report the
proportion of scenarios in which all k runs behave correctly (we use “consistency@k” for this defined
measure; it is not yet a standardized metric), not the proportion in which any run does (pass@k). Element
R.1 already states that variation in safety-critical behavior is a failure; consistency@k operationalizes it.
3. Zero-failure planning for critical failure modes. Where acceptance requires that a critical failure rate be
below a threshold, sample sizes follow directly from binomial reasoning (the “rule of three” [92]); see
Table 8.

Page 24 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

4. Prediction-powered inference (PPI) for automated grading. When an LLM-based or other automated
grader scores large volumes of outputs, a randomly selected subset should be double-scored by clinicians
and combined using PPI, which yields valid confidence intervals regardless of grader quality and narrower
intervals when the grader is accurate [68]. Automated graders can approach clinician agreement on some
tasks [61, 69], but PPI removes the need to assume this. PPI improves precision only when its assumptions
hold and the human reference labels are of adequate quality; it cannot remove bias in the ground truth
itself. Where clinician adjudicators legitimately disagree on open-ended outputs, uncertainty models
should separate scenario, repeated-run, and rater variance (e.g., crossed random-effects models) so that
rater disagreement is not mistaken for device error.
5. Synthetic data as a separate stratum. Estimates from synthetic inputs should be reported separately from
real-input estimates. Combining them is appropriate only when exchangeability is supported—for
example, by showing no significant performance difference on a matched real/synthetic bridging subset—
and should use prespecified methods that down-weight synthetic evidence (e.g., Bayesian power priors or
importance weighting toward the real distribution). Safety-floor claims (Q6) should rest on real or clinically
representative inputs.

Table 8. Number of independent scenario-trials with zero critical failures needed to demonstrate an upper
confidence bound on the failure rate

Maximum acceptable failure rate n at 95% confidence n at 99% confidence

5% 59 90

2% 149 228

1% 299 459

0.5% 598 919

0.1% 2,995 4,603

Computed as n = ln(1 − confidence) / ln(1 − p). The calculation assumes independent, identically distributed trials, a prespecified failure
definition, and zero observed failures. The figures bound the failure rate on the tested scenario distribution, not the prevalence of harm in
deployment; rare-harm rates in real populations cannot be established from convenient synthetic stress tests. Cluster effects, subgroup
uncertainty, selection bias, and endpoint validity require separate treatment. Scenarios, not repeated runs of the same scenario, are the
independent unit. These counts are feasible for automated benchmarking but not for clinician-adjudicated real-world confirmation—
another reason to combine methods using PPI and stratified designs (which can reduce annotation burden but do not replace clinical
confirmation).

Benchmarking and clinical confirmation evidence answer different questions (capability vs. real-world
performance) and should generally be reported separately. A combined estimate is reasonable only for a
specific, shared endpoint measured with the same rubric in both settings, where a bridging analysis shows
comparable performance on overlapping strata. In that case, a prespecified Bayesian hierarchical model or
discounted (power) prior could borrow strength from large benchmark strata to supplement a smaller real-world
confirmation cohort for rare safety-critical presentations—with the degree of borrowing justified and subject to
sensitivity analysis.

Recommendation (Q12)

Page 25 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Distinguish per-output, per-scenario, and prevalence-weighted risk; use repeated sampling with
scenario-clustered uncertainty, a clearly defined consistency measure for safety-critical behaviors, and
zero-failure sample-size planning with explicit independence assumptions for critical failure modes.
• Consider PPI or other statistically justified grader-assisted estimation only when grader calibration,
grader error, and human reference-label quality have been demonstrated; model rater variance
explicitly.
• Report synthetic and real estimates separately; combine only with demonstrated exchangeability and
prespecified down-weighting.

Question 13 — Where synthetic data fits, where it fails, and safeguards
For which domains, functions, or subpopulations is synthetic data well-suited or inadequate? What safeguards
mitigate the risk that synthetic data generated by same-class models reproduces the performance gaps the
evaluation seeks to detect?

Well-suited: rare but critical presentations (stress-testing escalation); adversarial and prompt-injection testing;
linguistic variation (paraphrase, dialect, colloquialism, literacy, translation); quantitative perturbations (unit
errors, implausible values, missing units—element E.3); long and branching conversational trajectories (Q5); and
privacy-preserving development.

Inadequate as primary evidence: estimating real-world prevalence, calibration, or positive predictive value;
subgroup performance claims, where the generator may share the device’s blind spots (LLMs have reproduced
race-based medical misconceptions [73]); real multimodal artifacts (image noise, device-specific acquisition,
ambient audio); and the social and emotional realities of patient interactions. Recursively generated data is also
known to lose the tails of the original distribution [74]—precisely where underrepresented subgroups and rare
emergencies reside.

Safeguards we recommend: (1) generator independence—use generators from a different model family than
the device, and preferably several; (2) clinician-authored seeds and clinician review of a random sample of
generated items; (3) a realism test—blinded clinicians or a discriminator attempt to distinguish synthetic from
real cases, with results reported; (4) distributional comparison against real data on key features; (5) provenance
tagging of every item; and (6) a rule that subgroup claims require at least a real-data bridging sample for each
claimed subgroup.

Recommendation (Q13)
• Use synthetic data primarily for stress, robustness, and coverage testing—not as sole evidence for
prevalence-dependent metrics or subgroup claims.
• Require generator independence, clinician seeding/review, realism testing, distributional comparison,
provenance tagging, and real-data bridging for subgroups.

Questions 14 and 15 — Comparators, acceptance criteria, and the counterfactual care pathway
Q14: How should comparators and acceptance criteria be selected for open-ended outputs (standard of care
vs. median clinician; generalist vs. specialist; human–AI team vs. device alone)? Q15: Can performance be

Page 26 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

assessed relative to what is likely to occur absent the device (unaided judgment, delayed specialist review, no
intervention)?

We recommend separating three roles that a “comparator” plays, because conflating them causes much of the
confusion in this area:

Table 9. Three distinct reference roles for open-ended GenAI outputs

Role Question it answers Recommended reference

Correctness reference Is this individual output acceptable? Rubric grounded in guidelines and standard of
care, defined by a consensus panel of clinicians
at the level of expertise the claim implies
(specialist panel when specialist-level outputs
are claimed)

Safety floor Is the device safe enough to be used at all? Absolute, non-compensatory thresholds for
critical failure modes (Q6), independent of any
comparator

Benefit comparator Is the device better (or not unacceptably The realistic counterfactual care pathway in the
worse) than what would otherwise happen? intended setting (Q15): usual care, unaided
clinician, delayed specialist access, nurse line,
or—where appropriate—no access

On the specific sub-questions: the median clinician in practice can be a reasonable benefit comparator for noninferiority in some settings—but not as a universal standard, and any noninferiority margin should be
prespecified and clinically justified—while the standard of care should define correctness, because the median
clinician includes known practice variation. Generalist vs. specialist should follow the claim and the users: a
device marketed to generalists for specialty questions should be compared with what generalists achieve today
(benefit) while being graded against specialist standards (correctness). Human–AI team performance should be
the primary basis whenever labeling places a human in the loop, because team performance can differ sharply
from device-alone performance in both directions [59, 72]; device-alone performance remains necessary to
characterize failure modes the human may miss.

On Q15, we strongly support counterfactual comparators—they are the most honest measure of benefit,
especially for access-expanding devices in underserved settings. But they must be paired with the safety floor:
“better than nothing” must never justify a device that fails critical escalation scenarios. The counterfactual
should be documented with data (e.g., wait times for specialist review, usage of unregulated alternatives) and
discussed with FDA through the Pre-Submission process.

Recommendation (Q14–Q15)
• Distinguish correctness reference, safety floor, and benefit comparator; define each explicitly in the
submission.
• Use standard of care for correctness, the realistic counterfactual pathway for benefit, and absolute
floors for safety.
• Make human–AI team performance primary whenever a human is in the loop, with device-alone
failure-mode characterization as supporting evidence.

Page 27 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Question 16 — Roles for independent third parties
Should independent third parties be involved in benchmarking and clinical confirmation? What qualifications
and independence criteria should apply? What safeguards prevent limiting competition or innovation?

Yes, as an option that sponsors may choose and that FDA could consider for selected higher-risk contexts
within appropriate legal authority—not as a mandatory gate for all devices. Independent parties are most
valuable for (1) custody of sequestered test sets, (2) independent adjudication panels, and (3) accredited
execution of FDA-recognized test methods. We caution that private assurance infrastructure has struggled when
it preceded agreed test methods: the health-AI “assurance lab” model proposed in 2024 did not materialize as
planned, reportedly due in part to business-model and conflict-of-interest challenges [96]. The sequence
matters: standardize methods first, then accredit labs to run them—exactly the ASCA logic [25].

Recommended program features:

• Accreditation of GenAI evaluation laboratories under an ASCA-like pilot, if validated methods and
statutory authority support it, against FDA-recognized test methods, with ISO/IEC 17025-style competence
requirements.
• Voluntary MDDT qualification of suitable benchmark suites, user simulators, and virtual-patient tools
within a specified context of use [26], with periodic re-qualification to address saturation and
contamination.
• Independence criteria: no financial relationship with the sponsor or its foundation model provider for the
evaluated device; no development consulting on the same device within a defined look-back; disclosure of
all relationships; rotation of adjudicators.
• Competition safeguards: multiple accredited bodies; published fee schedules and turnaround times; open
test specifications with sequestered items; no exclusivity; small-business fee relief; and the right of
sponsors to perform recognized methods in-house with FDA audit, so third-party use remains a choice.
• Data stewardship: sequestered datasets must be refreshed and retired on a schedule, with leakage
monitoring.

Recommendation (Q16)
• Explore an optional third-party pathway built on recognized test methods, voluntary MDDT-qualified
tools, and—where authority and validated methods support it—an ASCA-like accreditation pilot.
• Adopt explicit independence and competition safeguards, including in-house testing with recognized
methods as an alternative.

Question 17 — Applying the approach to other architectures (VLMs, generative/predictive world models)
Are there aspects of the competency-based approach that might be ineffective for multimodal vision-language
models and generative or predictive world models? What additional considerations apply?

The structure transfers, but the element content must change:

Page 28 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Vision-language models (VLMs): add visual grounding tests (does the text correspond to what is in the
image?), cross-modal consistency, and robustness to acquisition variation. VLMs are vulnerable to
instructions hidden in images: in a study of 594 attacks, including near-invisible text embedded in
oncology images, all four models tested were susceptible [66]. Element S.2 should therefore explicitly
include multimodal prompt injection.
• Generative image reconstruction, enhancement, and synthesis: the key failure mode is plausible
hallucinated anatomy or erased pathology. Task-based image quality evaluation (reader studies on the
clinical task), lesion insertion/removal tests, and hallucination maps are more informative than pixelsimilarity metrics.
• Generative or predictive world models (e.g., disease-trajectory or physiological simulators): these are
closer to computational models than to conversational agents. FDA’s credibility framework for
computational modeling and ASME V&V 40 [21]—context of use, model risk, verification, validation, and
applicability—should be the primary lens, supplemented by counterfactual validity and temporalconsistency testing.
• Speech and ambient interfaces: add automatic speech recognition error propagation and accent/dialect
robustness. Ambient documentation tools still produced clinically significant inaccuracies in randomized
evaluation [84], and clinician-annotated studies report non-trivial hallucination and omission rates [83].

Recommendation (Q17)
• Keep the competency structure; publish architecture-specific element supplements (VLM, generative
imaging, world models, speech/ambient).
• Apply the computational-modeling credibility framework to generative/predictive world models.

4.3 Postmarket Monitoring (Section VI; Questions 18–21)
Question 18 — Accepting greater premarket uncertainty through postmarket monitoring
Under what conditions might it be appropriate to accept greater premarket uncertainty through greater
reliance on postmarket monitoring? What monitoring characteristics would justify reduced premarket
evidence? Where would this be inappropriate?

We support this concept, which is consistent with FDA’s benefit-risk uncertainty guidance [15] with programs
such as the TEMPO pilot that pair earlier access with real-world data collection [28], and with the regulatorysandbox concept in America’s AI Action Plan [29]. The evidence base also argues for it: AI-enabled devices
without reported clinical validation have been associated with more recalls [87], and most historically cleared AI
devices were evaluated only retrospectively [88]—so premarket evidence alone has not been a reliable
guarantee. But the trade only works if the monitoring program can actually catch problems before they cause
serious harm. We propose six Monitoring-Readiness Criteria (MRC) for judging whether specified residual
uncertainties can responsibly be managed postmarket. These are proposed decision aids—not current legal
prerequisites, and not a substitute for an adequate showing of reasonable assurance of safety and effectiveness:

Page 29 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Table 10. Proposed Monitoring-Readiness Criteria

Criterion Requirement Illustrative evidence

MRC-1 Observability Complete, privacy-protected logging of inputs, Logging architecture; data-use
outputs, configuration fingerprint (Section 3.1), and agreements with deploying sites
user actions; linkage to downstream outcomes
where feasible

MRC-2 Detection before Expected detection latency for a clinically meaningful Simulation of detection power and
harm degradation is shorter than the harm latency for the latency for prespecified degradation
affected function (modifier M1) scenarios

MRC-3 Controllability Ability to pin versions, roll back, throttle, restrict Documented rollback/kill-switch
scope, or disable a function rapidly; deterministic procedures, tested in drills
safe fallback

MRC-4 Reversibility The harms that might go undetected premarket are Hazard analysis under ISO 14971 with
predominantly reversible or mitigable harm-latency annotation

MRC-5 Prespecified Statistical stopping/escalation rules (e.g., CUSUM or Statistical monitoring plan with
decision rules sequential probability ratio tests [93]) and pre- operating characteristics
agreed actions

MRC-6 Independent Risk-appropriate independent clinical review of Reporting template; adjudication
oversight and reporting samples; any reporting to FDA follows applicable charter
law, authorization conditions, or study protocols

Two clocks. The decisive test is the comparison between time-to-harm and time-to-detection-plus-intervention.
The postmarket plan should identify which failures are observable, how long detection and protective action
take, and when a clinician or independent safety layer can interrupt harm before it occurs. Where harm can
occur faster than it can be detected and acted upon, postmarket monitoring complements—but cannot
replace—prospective safety validation and preventive design controls.

Inappropriate settings include: functions at Band 4 (autonomous, severe or irreversible, time-critical); closedloop control of therapy; neonatal and pediatric emergency triage; populations that cannot readily report harm
(e.g., cognitive impairment); and functions whose harms are latent over long periods (e.g., cumulative
medication errors) where MRC-2 cannot be met. A practical alternative is the EDA model (Q7): launch at higher
supervision levels with prespecified evidence collection; any further autonomy must remain within an authorized
PCCP or follow the applicable new-submission pathway.

Recommendation (Q18)
• Consider specified residual uncertainties for postmarket management only where reasonable assurance
of safety and effectiveness remains supported, using the six Monitoring-Readiness Criteria as decision
aids.
• Exclude Band 4 functions and settings where detection and protective action cannot precede harm.
• Consider staged supervision; any autonomy progression must satisfy the boundaries of an authorized
PCCP or applicable new-submission requirements.

Page 30 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Question 19 — Postmarket approaches, additional methods, cadence and triggers
Comment on periodic re-benchmarking, sample-based clinician review, and performance-degradation
monitoring. What additional approaches should be considered, and how should cadence and triggering events
be determined?

All three proposed approaches are appropriate and complementary. They are also consistent with themes raised
in FDA’s 2025 request for comment on real-world performance evaluation of AI devices [12]. We recommend
five additions:

1. Production canary probes. A fixed, sequestered set of synthetic “canary” scenarios is run against the live
production configuration on a schedule (e.g., daily). Behavioral changes on canaries detect silent upstream
or configuration changes within hours, long before clinical signals emerge—particularly valuable for thirdparty models (Q24).
2. Leading behavioral indicators. Escalation rate, refusal rate, guardrail-trigger rate, directiveness-ceiling
breach rate, envelope-escape rate, citation-fidelity spot checks, and average conversation length,
monitored by subgroup. Changes in these often precede measurable clinical harm.
3. Input-distribution monitoring. Embedding-space and topic drift on inputs, new languages or populations,
and new use contexts, detecting off-label or unanticipated use (with data minimization, defined retention,
and privacy protections for any logged interaction content).
4. Outcome-linked surveillance. Where data access permits, linkage to downstream events (e.g., ED visits
after “self-care” advice; medication changes after recommendations).
5. Structured complaint and feedback mining. User feedback and clinician overrides classified (with
validated automated assistance) and routed into complaint handling under the QMSR [10] and, where
applicable, Medical Device Reporting under 21 CFR Part 803 [31].

Cadence should be risk-banded (e.g., continuous leading indicators for all bands; monthly sample-based review
for Band 2 and weekly for Band 3; quarterly full re-benchmarking for Bands 2–3). Triggering events for
unscheduled reassessment should include: any AI-CBOM change; an upstream model notification or a canary
deviation; a statistical alarm on a leading indicator; a material clinical guideline or labeling change in the device’s
domain (knowledge currency, E.5); entry into a new site, population, or language; and any serious adverse event
or credible safety signal.

Recommendation (Q19)
• Supplement the three proposed approaches with canary probes, leading behavioral indicators, inputdistribution monitoring, outcome linkage, and structured complaint mining.
• Set risk-banded cadences and a standard list of triggering events tied to the AI-CBOM and the clinical
knowledge environment.

Question 20 — Machine-based supervisory agents
Can postmarket monitoring be facilitated by machine-based supervisory agents? What considerations—
including the evaluation and reliability of the supervisory agent itself—should apply?

Page 31 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Yes—supervisory agents are probably the only way to review GenAI outputs at production scale, and evidence
shows automated graders can reach clinician-level agreement on some evaluation tasks [61, 69]. But a
supervisor that shares the device’s blind spots provides false assurance. We recommend these conditions:

• Independence and diversity: the supervisor should use a different model family (and ideally provider)
than the monitored device, so that errors are less correlated.
• Validation for its specific role: documented sensitivity for the safety-critical event classes it is meant to
flag, measured against clinician adjudication, with acceptance criteria.
• Statistically valid estimation: aggregate performance rates derived from supervisor labels should be
corrected using a random clinician-reviewed subset (prediction-powered inference [68]).
• Human review: all supervisor-flagged critical events, plus a random sample of unflagged interactions (to
estimate false negatives).
• Configuration control: the supervisor’s own configuration is versioned and change-controlled within the
QMS; its drift is monitored (“monitor the monitor”).
• Regulatory boundary: a supervisor that only observes and reports is a QMS tool; one that intervenes in
real time (blocking or rewriting outputs) is part of the device and must be evaluated as such.
• No automatic ground truth: no supervisory agent should be treated as ground truth without independent
validation against qualified human adjudication.

Recommendation (Q20)
• Permit supervisory agents subject to independence, role-specific validation, PPI-based estimation,
human review of flags plus random samples, and configuration control.
• Clarify that real-time intervening supervisors are device components.

Question 21 — Roles of the broader ecosystem without diffusing manufacturer accountability
What roles might clinicians, institutions, professional societies, standards bodies, and others play in
postmarket monitoring, and how can these be structured without diffusing manufacturer accountability?

The ecosystem is already organizing. The Joint Commission and CHAI’s 2025 guidance asks health systems to
establish AI governance, conduct local validation, monitor quality, and voluntarily report AI safety events [54].
The UK National Commission recommended that all risk controls be contractually allocated between
manufacturer and provider so that frontline clinicians do not become “liability sinks” [38]. Meanwhile, ONC’s
HTI-1 decision-support source-attribute requirements—which helped give deployers transparency—have been
proposed for removal in HTI-5 [30], which increases the importance of FDA labeling as the authoritative
transparency channel.

We recommend that FDA keep accountability anchored with the manufacturer by using labeling to define a set
of Conditions of Safe Deployment: the deployer prerequisites the manufacturer’s safety case depends on (e.g.,
local validation steps, configuration choices permitted to the deployer, user training, monitoring data the
deployer must share, and event-reporting channels). A deployer that departs from those conditions changes the
device’s risk profile, but the manufacturer remains responsible for specifying them, verifying they are achievable,
and monitoring adherence. Professional societies are best placed to maintain clinical rubrics and standards-ofPage 32 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

care references for benchmarking; standards bodies to maintain test methods; and patients should have a clear,
in-product route to report problems that feeds both the manufacturer’s complaint system and FDA MedWatch.
An FDA-convened collaborative community on GenAI device monitoring could align these roles.

Recommendation (Q21)
• Anchor accountability with the manufacturer via labeled Conditions of Safe Deployment and an
accountability (RACI-style) map across the TPLC.
• Leverage professional societies for rubrics, standards bodies for test methods, and health systems for
local validation and data sharing.
• Convene a collaborative community on GenAI postmarket monitoring.
• Distinguish voluntary shared safety signals from legal reporting duties (e.g., Medical Device Reporting),
and make clear who holds each.

4.4 Change Control and Third-Party Models (Section VI.C; Questions 22–24)
Question 22 — Scaling re-benchmarking to the nature of a modification
How might re-benchmarking be scaled to the nature and impact of a modification? Which changes might be
managed within the QMS, require premarket review, or be appropriate for a PCCP?

Using the AI-CBOM (Section 3.1) as the reference, we propose a tiered taxonomy. The key analytical tool is
differential testing: run the prior and modified configurations on the same frozen benchmark and report the flip
rate—the proportion of items whose outcome changes in either direction—by element and stratum. A
modification can improve average performance while creating new critical failures; flip analysis exposes that.

Table 11. Proposed change taxonomy for GenAI-enabled devices

Tier Examples Evidence Regulatory mechanism

T0 — No Infrastructure scaling, logging, Regression verification; canary Applicable QMS change
behavioral cosmetic UI that does not alter probes unchanged controls
impact content or escalation affordances

T1 — Bounded Prompt wording refinements, Targeted re-benchmark of affected QMS documentation
behavioral guardrail threshold tuning, retrieval- elements; zero new critical failures; where the change does
change within corpus refresh from pre-approved non-inferiority on safety elements not require a submission;
envelope sources, bug fixes; epoch-gated otherwise an authorized
releases from continuous retrieval PCCP or a marketing
ingestion or bounded local adaptation submission as required for
that modification

T2 — Foundation model version update Full re-benchmark on frozen Authorized PCCP if the
Component within a family; new embedding baseline + flip analysis; limited substitution and
substitution model; new fine-tune on same task clinical re-confirmation when Band ≥ evaluation method were
within same 2 prespecified; otherwise
class assess the need for a new
submission

Page 33 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Tier Examples Evidence Regulatory mechanism

T3 — New model family; new EDA or higher New benchmarking and clinical Assess applicable newCapability or supervision level outside PCCP; new confirmation proportional to new submission requirements;
scope change population, language, or care setting; risk band an authorized PCCP covers
higher directiveness ceiling only adequately described
modifications

Tier labels are an analytical aid for scaling verification evidence; they do not replace the legal determination of whether a modification
requires a new marketing submission.

Passive or continuous “model-evolution changes” (Section VI.C). CDRH distinguishes discrete sponsor updates,
third-party model updates, and changes that occur “more passively, continuously, or incrementally” because a
device learns or adapts during use—for example, continuous retrieval-corpus ingestion, adaptive local
calibration, persistent cross-session memory, or automated prompt optimization from user feedback. Allowing
effective clinical behavior to change continuously in production, without discrete verification checkpoints, makes
failure attribution and rollback nearly impossible. We recommend an epoch-gated release approach: adaptive
updates are accumulated into discrete, version-identified candidate snapshots (e.g., daily or weekly) that must
automatically pass the frozen canary suite and the non-compensatory safety floors before promotion, under an
authorized PCCP where one applies. Patient-specific adaptation, such as cross-session memory, should be
confined to a bounded memory schema with deterministic validation and user or clinician inspectability (Q5).

Recommendation (Q22)
• Adopt a four-tier GenAI change taxonomy keyed to AI-CBOM components and behavioral impact.
• Require differential (flip-rate) analysis on a frozen baseline for T1–T2 changes, with a rule of zero new
critical failures.
• Govern passive or continuous model-evolution changes through epoch-gated releases with automated
canary and safety-floor verification before promotion, and bounded memory schemas for patientspecific adaptation.

Question 23 — Adapting PCCPs when future modifications cannot be fully prespecified
How might PCCP concepts or other change-control approaches be adapted for GenAI-enabled devices when the
nature or scope of future modifications cannot be fully prespecified?

Current PCCPs, consistent with FDA’s final guidance [8] and the international guiding principles [9], enumerate
specific modifications. The K253281 PCCP is a good example: six categories of discrete, largely deterministic
changes [14]. For generative components, enumerating every prompt or retrieval change is impractical. We
recommend that FDA consider an envelope-based PCCP whose Description of Modifications specifies bounds
rather than instances—while recognizing that the planned modifications, the methods to develop and validate
them, and the impact assessment must still be described specifically enough for FDA to review and authorize
them [8]:

• Configuration bounds: which AI-CBOM components may change and within what limits (e.g., “foundation
model versions within family X, subject to the model-substitution protocol”; “retrieval corpus limited to
listed authoritative sources”).

Page 34 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Behavioral envelope: the Conversational Trajectory Envelope, directiveness ceilings, and EDA supervision
levels that may not be exceeded.
• Performance envelope: acceptance criteria for every benchmark element, including non-compensatory
safety floors, that each modified configuration must meet on the frozen baseline plus refreshed items.
• Modification Protocol: differential testing, flip analysis, canary verification, staged rollout (e.g., limitedpercentage release with monitoring before full deployment), and rollback criteria; for passive or
continuous model-evolution mechanisms, the epoch cadence, zero-critical-flip gating criteria, and
automatic rollback triggers.
• Entrustment progression (optional): prespecified real-world evidence thresholds that permit an EDA to
move to a lower-supervision level, with defined FDA review checkpoints where appropriate and a clear
route to a new submission if the change exceeds what was authorized.

This approach aligns with the UK Government’s commitment to publish draft PCCP guidance permitting change
“boundaries,” including for adaptive AI, by December 2026 [39], and with Health Canada’s PCCP approach for
ML-enabled devices [43]. International alignment on envelope-based PCCPs would be highly valuable to
sponsors.

Recommendation (Q23)
• Consider envelope-based PCCPs defined by configuration, behavioral, and performance bounds with a
rigorous modification protocol and staged rollout, provided modifications, methods, and impact
assessments are specified sufficiently for authorization.
• Permit entrustment progression within a PCCP only where adequately specified and authorized, with
review checkpoints and an off-PCCP route to a new submission.
• Require epoch-gated verification within the PCCP for any passive, continuous, or incremental modelevolution mechanism.
• Pursue alignment with MHRA and Health Canada on envelope-based PCCP concepts.

Question 24 — Detecting and responding to third-party foundation model changes
How can a manufacturer detect, evaluate, and respond in a timely manner to changes initiated by the thirdparty foundation model developer? What mechanisms—contractual, technical, PCCP—could provide
reasonable assurance?

Upstream change is, in our view, the single most under-controlled risk in GenAI device development today,
compounded by the practice of hosted-model providers periodically deprecating model versions on timelines
that can be shorter than a typical premarket review cycle. We recommend that sponsors apply risk-based
supplier evaluation and controls to foundation-model services under the QMSR (ISO 13485:2016, clause 7.4)
[10], with the depth of control proportionate to the supplier’s effect on device conformity and patient risk, using
a layered control set:

Page 35 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Table 12. Layered controls for third-party foundation model changes

Layer Controls

Contractual Access to immutable, version-pinned model snapshots; advance notice of changes and deprecations
with a minimum window sufficient for re-benchmarking; incident and safety-relevant behaviorchange notification; audit or information rights (or MAF reference, Q25); data-use and retention
terms compatible with HIPAA and device logging

Technical Version pinning by default; behavioral fingerprinting via production canary probes (Q19) to detect
silent changes; configuration fingerprint and any provider-supplied model version or snapshot
identifier logged with outputs; validated fallback (alternate model or deterministic safe mode);
automated failover to fallback on canary deviation

Procedural / QMS Supplier qualification and periodic re-evaluation; an unannounced upstream behavior change is
handled as a nonconformance with impact assessment; documented model-substitution protocol

Regulatory (PCCP) Where appropriate, an FDA-authorized model-substitution protocol (T2 in Table 11) specifying rebenchmark scope, flip-analysis thresholds, and staged rollout, so that timely migration is possible
when a provider deprecates a version

We also encourage FDA to recognize that some sponsors will self-host open-weight models specifically to control
change; guidance should not inadvertently favor one deployment model over another, but should hold each to
the same assurance outcome.

Small-sponsor implementation. Contractual leverage over dominant foundation-model providers may be
limited, so FDA should accept demonstrably equivalent controls—such as immutable self-hosted weights,
monitored endpoint snapshots, provider attestations, validated synthetic canaries, and a documented fail-safe
fallback. The expectation should address the manufacturer’s demonstrated control outcome rather than
presume negotiable audit rights.

Recommendation (Q24)
• Apply risk-based supplier controls to foundation-model services, accepting effective combinations of
contractual, technical, procedural, and authorized-PCCP controls.
• For safety-relevant hosted-model dependencies, consider version pinning, canary-based change
detection, and a validated fallback, in proportion to actual device risk and vendor capability; a
prespecified model-substitution protocol can prevent forced downtime or unverified emergency
migrations when a provider deprecates a version.

4.5 Foundation Model MAFs and Agentic AI (Section VII; Questions 25–26)
Question 25 — Practicality and content of voluntary Foundation Model MAFs
Would voluntary Foundation Model MAFs be practical to provide and keep current? What content? What
would make the program useful given limited incentives to disclose? Are there alternative mechanisms?

We strongly support the concept. The Device Master File program already provides a confidential, reference-byauthorization mechanism [24], and the UK National Commission independently recommended an opt-in Master
File for general-purpose models, alongside mandatory reporting by manufacturers of dependencies on such

Page 36 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

models [38]; the UK Government has committed to consider a pilot [39]. The practicality question is mostly
about incentives and reuse—along with confidentiality tiers, update procedures, downstream reliance,
revocation and model retirement, and designs that smaller foundation-model suppliers can also use:

• Reuse of documentation already required elsewhere. Since August 2, 2025, providers of general-purpose
AI models placed on the EU market must prepare technical documentation and information for
downstream providers under Article 53 and Annexes XI–XII of the EU AI Act, supported by the Model
Documentation Form in the GPAI Code of Practice [35, 36]. If FDA structures MAF content to map onto
these artifacts, the marginal cost to developers falls sharply. A voluntary trilateral (FDA–MHRA–EU) field
mapping would be better still—although each jurisdiction’s legal obligations would remain distinct,
devices would still need device-specific evidence, and providers may have legitimate constraints on
sharing proprietary information.
• Timely change notification. FDA could consider an electronic notification mechanism through which a
MAF holder’s new model snapshot, safety-relevant update, or deprecation notice reaches the device
sponsors it has authorized to reference the file.
• Commercial incentive. A MAF referenced by many device sponsors reduces each customer’s regulatory
burden and makes the model more attractive to the regulated health market. FDA could, with the holder’s
consent, list MAF holders publicly.
• Reviewer efficiency. Consistent information across submissions reduces deficiency letters about the same
model.

Table 13. Proposed tiered content for Foundation Model MAFs

Tier Content Update expectation

Core (all holders) Model identity and version scheme; architecture class; training Periodic update and prompt notice of
data provenance summary; intended and out-of-scope uses; material changes; exact cadence to be
known limitations and failure modes in healthcare contexts; agreed in the MAF scheme
safety behaviors and content policies; version-pinning and
deprecation policy; change-notification commitments

Health Results on healthcare-relevant benchmarks (with validity Per major version
evaluation notes), subgroup performance, calibration and uncertainty
behavior, hallucination and citation-fidelity evaluations, redteam summaries relevant to clinical misuse

Operational Determinism/reproducibility characteristics, inference-stack On change
changes that can alter behavior, audit-log availability, incidentnotification process, security posture

Consistent with the discussion paper, a MAF would not authorize the model for any intended use; sponsors remain responsible for devicelevel safety and effectiveness.

Alternatives and complements, useful where a provider declines to file a MAF: (1) supplier-qualification
packages shared under NDA with the sponsor; (2) independent third-party model audits against a recognized
documentation standard; (3) a requirement—mirroring UK Recommendation 8—that the device submission
disclose foundation-model dependencies, associated risks, and continuity plans [38]; and (4) greater sponsorside evaluation and change-detection burden (Q24) in proportion to the information gap.

Recommendation (Q25)

Page 37 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Proceed with voluntary Foundation Model MAFs using a tiered template with a voluntary field mapping
to EU AI Act Annex XII and the GPAI Model Documentation Form, without implying that one dossier
satisfies each jurisdiction’s distinct requirements.
• Encourage manufacturer transparency about clinically significant foundation-model dependencies and
continuity plans in proportion to device risk and applicable requirements, regardless of MAF availability.
• Scale sponsor-side evaluation and monitoring burden inversely to the information available about the
model.

Question 26 — Additional considerations for agentic GenAI-enabled devices
Are there additional considerations for agentic devices? How should elevated risk from autonomous multi-step
action, tool use, and reduced human review be reflected in acceptance criteria and oversight?

Agentic devices introduce three properties that conversational devices do not: error compounding across steps,
side effects in external systems, and indirect prompt injection via retrieved content and tool outputs. The
compounding effect is easy to underestimate: under a simplifying (and usually unrealistic) independence
assumption, if each step of a 20-step workflow succeeds 98% of the time, the whole task succeeds only about
67% of the time (0.98²⁰ ≈ 0.67). Real failures may be correlated, and an unsafe-action rate is not interchangeable
with task-success probability, but the direction of the effect is clear. In a realistic FHIR-based EHR environment,
the best model tested completed about 70% of 300 clinician-written agentic tasks [79]. Acceptance criteria must
therefore be set at the trajectory level, not the step level.

Table 14. Proposed action-authority levels for agentic devices

Level Authority Typical controls Two-axis implication

AA0 Read-only (retrieve, summarize) Least-privilege read scopes; provenance Informational
logging

AA1 Draft/propose (orders, messages) Mandatory human checkpoint; diff view of Action-taking, HCPfor human sign-off proposed change supervised

AA2 Execute reversible actions within Undo window; rate limits; scope allow-lists; Action-taking; modifier M1
scope post-hoc review may lower

AA3 Execute irreversible or high- Pre-action human confirmation required Action-taking, autonomous;
consequence actions unless a specific autonomous function has Band 3–4
independently established authorization;
dual control for severe consequence

AA4 Control another medical device Device-level interoperability verification; Highest; cf. DSF-MMA
closed-loop safety limits; independent Example #1 [17]
watchdog

Recommended considerations for acceptance criteria and oversight:

• Trajectory-level metrics: task success, unsafe-action rate, near-miss rate, unnecessary-action rate, and
recovery success after injected tool failures—measured in high-fidelity sandboxes (e.g., synthetic EHRs)
before any live use.

Page 38 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

• Least privilege and authority adherence: every tool permission justified by an EDA; testing that the agent
never exceeds granted authority even when instructed by users, retrieved documents, or tool outputs
(A.3).
• Indirect prompt-injection resistance: adversarial content planted in records, documents, and tool
responses (A.2, S.2) [66].
• Technically enforced tool boundaries: every tool invocation (e.g., HL7 FHIR read/write operations) passes
through a deterministic schema, permission, and parameter-range validator before execution; hard limits
on steps, tokens, and run time prevent runaway loops; and multi-step sequences that fail midway trigger
compensating rollback actions, which should themselves be tested.
• Human checkpoints before irreversible actions, with tests that checkpoints cannot be bypassed and that
the information presented at the checkpoint is sufficient for meaningful review (ISO/IEC TS 8200
controllability concepts [50]).
• Auditability: complete action logs attributable to the agent identity and configuration fingerprint, enabling
reconstruction of any trajectory.
• Multi-agent interactions: where devices interact with other agents (including non-device agents in the
same workflow), test for emergent behavior at the interfaces.

Recommendation (Q26)
• Define action-authority levels (AA0–AA4) and require trajectory-level acceptance criteria that account
for error compounding.
• Expect least privilege, injection resistance, non-bypassable human checkpoints for irreversible actions,
deterministic tool-call validation, execution limits, tested rollback, and full auditability.
• Split agentic benchmarking into A.1–A.4 (planning, tool-use integrity, authority adherence, recovery).

Page 39 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

5. Additional Recommendations Beyond the Discussion Questions
5.1 Transparency of the public record
Learning across the field depends on precedents being visible. FDA has stated it will explore ways to identify
devices that incorporate foundation models on its AI-Enabled Medical Devices List and encourages sponsors to
describe LLM functionality in public summaries [13]. We recommend making this systematic: a GenAI/LLM flag
on the list and in the 510(k)/De Novo/PMA databases, and a short, standardized GenAI disclosure section in
public decision summaries (role of the generative component, two-axis placement, supervision level, high-level
evaluation approach, and existence of a PCCP). The K253281 decision summary, for example, describes a
“Conversation Service (UpDoc Agent)” but does not identify it as GenAI or an LLM—a characterization that
comes only from the manufacturer [97]—or describe how it was evaluated [14], which limits its value as a
precedent for other sponsors. Any such flag should be applied only where GenAI functionality is confirmed,
recognizing that most AI-enabled devices do not use GenAI. We also support “living” model cards and dynamic
labeling that reflect the current configuration and latest re-benchmark results, consistent with FDA’s
transparency guiding principles and draft AI lifecycle guidance [7, 20].

5.2 A clear GenAI function-boundary decision tree
Sponsors today must reconcile the discussion paper’s activity axis with the January 2026 CDS guidance (including
its new enforcement-discretion position for single recommendations and its treatment of time-critical use under
Criterion 4), the January 2026 General Wellness guidance, the Multiple Function guidance, and the DSF-MMA
guidance [5, 6, 16, 17]. None of the CDS or wellness guidances addresses GenAI specifically. We recommend that
the forthcoming GenAI guidance include a function-boundary decision tree with GenAI-specific examples—e.g.,
ambient documentation that remains under clinician review versus documentation that drives coding or orders
(an issue the MHRA addressed in its July 2026 ambient voice technology guidance [41]); patient-facing education
versus personalized D3–D4 advice; and general-purpose assistants marketed with health features. Such a
decision tree would also reduce the risk of uneven treatment between products that seek authorization and
functionally similar products that do not.

5.3 Least burdensome implementation for small and mid-sized sponsors
• Provide templates: AI-CBOM, Benchmark Validity Dossier, Conversational Trajectory Envelope, Risk–
Evidence justification, and a GenAI module for eSTAR, scaled to the Basic/Enhanced documentation levels
of FDA’s software submissions guidance [18].
• Fund or qualify shared public test infrastructure (MDDT-qualified simulators and sequestered
benchmarks) so small sponsors are not forced to build bespoke evaluation stacks.
• Prioritize predictable Pre-Submission turnaround and reusable public templates for small businesses,
rather than introducing a separate agreement mechanism.
• Accept modular evidence reuse: when multiple devices share a foundation model and MAF, permit reuse
of model-level characterization while requiring device-level confirmation.

5.4 Machine-readable regulation and interoperable evaluation reports
From our vantage point as builders of regulatory-intelligence tooling, we see a large, untapped opportunity:
much of the friction sponsors experience comes from translating prose guidance into checklists, test plans, and

Page 40 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

submission sections by hand. FDA has itself adopted generative and agentic AI internally for review support [27].
We recommend that CDRH publish the benchmarking taxonomy, risk-band definitions, and AI-CBOM fields as a
versioned, machine-readable schema, and accept evaluation reports in a corresponding structured format (as
eSTAR did for submissions). This would make submissions more consistent, enable automated completeness
checks on both sides, allow cross-submission regulatory-science analyses (e.g., the predictive validity of
benchmark elements, Q10), and reduce the cost of compliance for every sponsor.

A realistic first step is a versioned public vocabulary (e.g., JSON or CSV) with completed example templates, not
an immediate eSTAR mandate. That vocabulary should include source-status metadata identifying whether each
referenced requirement derives from statute, regulation, final guidance (nonbinding recommendations), draft
guidance, a discussion paper, a recognized standard, scientific literature, or an individual device decision
summary—and whether it has been withdrawn or superseded. This distinction matters to sponsors and is
essential for any regulatory-intelligence system, including the one we are building, to avoid treating nonbinding
or superseded sources as binding requirements.

5.5 International convergence
The timing for convergence is unusually favorable. IMDRF’s draft AI life-cycle technical framework (N93) was
open for consultation through July 2026 [33]; IMDRF finalized harmonized GMLP principles in January 2025 [32];
Korea’s MFDS issued a GenAI-specific device guideline in January 2025 [42]; Singapore refreshed its AI-inhealthcare guidelines in March 2026 with GenAI provisions [45]; the UK has committed to a broad lifecycle
reform program [38, 39] informed by MHRA’s AI Airlock sandbox, which has tested LLM-based clinical
summarisation and patient-facing results explanation [40]; WHO has issued governance guidance on large multimodal models [44]; the EU has confirmed that AI Act requirements for medical devices can be integrated into
existing MDR/IVDR quality systems and technical documentation [37]; and the European Commission confirms
that the 2026 AI Omnibus entered into force on July 27, 2026, setting August 2, 2028 as the application date for
high-risk rules covering AI embedded in Annex I products, which include medical devices (other AI Act
obligations follow different timelines) [35], creating a window to align technical expectations before they bind.
We encourage CDRH to bring the two-axis framework, competency-based benchmarking elements, AI-CBOM,
and Foundation Model MAF template to IMDRF as candidate harmonized work items.

Page 41 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

6. Conclusion and Offer of Collaboration
CDRH’s discussion paper asks the right questions at the right time. Our central message is that the framework
will succeed if it becomes measurable and predictable while remaining grounded in existing law: a regulated
object defined by its configuration; risk placed on a familiar grid with a small number of evidence-backed
modifiers; directiveness and conversational scope measured rather than asserted; competence demonstrated in
interaction and entrusted activity by activity; statistics suited to stochastic systems; and premarket, change
control, and postmarket monitoring connected through a single evaluation baseline.

Regavise would welcome the opportunity to support CDRH’s next steps—for example, by participating in a public
workshop, contributing to public discussions of risk-scaled configuration records and benchmark-evidence
templates, or helping pilot a machine-readable representation of the benchmarking taxonomy linked to
authoritative FDA sources. Thank you for considering these comments.

Respectfully submitted,

Markus Yap, PhD, MPH, RAC (US)

Chief Technology Officer, Regavise | markus@regavise.com | Regavise.com

Regavise - AI regulatory intelligence for the medical-device lifecycle

Page 42 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

7. References
References are grouped by category. Web resources were accessed between October 18 and 19, 2026.

A. FDA and U.S. Government sources
[1] U.S. Food and Drug Administration (FDA), CDRH. Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback. August 2026. Docket FDA-2026-N-7874.
https://www.fda.gov/media/194242/download; landing page: https://www.fda.gov/medical-devices/digital-health-centerexcellence/considerations-regulation-generative-ai-enabled-medical-devices-discussion-paper-and-request
[2] FDA. News release: FDA Seeks Public Feedback to Inform Regulatory Approach for Generative AI-Enabled Medical Devices.
August 18, 2026. https://www.fda.gov/news-events/press-announcements/fda-seeks-public-feedback-inform-regulatoryapproach-generative-ai-enabled-medical-devices
[3] FDA. Executive Summary for the Digital Health Advisory Committee Meeting: Total Product Lifecycle Considerations for
Generative AI-Enabled Devices. November 20–21, 2024. https://www.fda.gov/media/182871/download
[4] FDA. Digital Health Advisory Committee Meeting: Generative AI-Enabled Digital Mental Health Medical Devices. November 6,
2025. https://www.fda.gov/advisory-committees/advisory-committee-calendar/november-6-2025-digital-health-advisorycommittee-meeting-announcement-11062025
[5] FDA. Clinical Decision Support Software: Guidance for Industry and FDA Staff. January 2026. https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/clinical-decision-support-software
[6] FDA. General Wellness: Policy for Low Risk Devices. Guidance. January 2026. https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/general-wellness-policy-low-risk-devices
[7] FDA. Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission
Recommendations. Draft guidance. January 2025. https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing
[8] FDA. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled
Device Software Functions. Final guidance; current version August 2025 (originally issued December 2024).
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendationspredetermined-change-control-plan-artificial-intelligence
[9] FDA, Health Canada, MHRA. Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding
Principles. October 2023. https://www.fda.gov/medical-devices/software-medical-device-samd/predetermined-changecontrol-plans-machine-learning-enabled-medical-devices-guiding-principles
[10] FDA. Medical Devices; Quality System Regulation Amendments (Quality Management System Regulation). Final rule. 89 Fed.
Reg. 7496 (Feb. 2, 2024); effective February 2, 2026. https://www.federalregister.gov/documents/2024/02/02/202401709/medical-devices-quality-system-regulation-amendments
[11] FDA. Cybersecurity in Medical Devices: Quality Management System Considerations and Content of Premarket Submissions.
Guidance, February 2026; and Federal Food, Drug, and Cosmetic Act §524B (21 U.S.C. 360n-2).
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/cybersecurity-medical-devices-qualitymanagement-system-considerations-and-content-premarket
[12] FDA. Request for Public Comment: Measuring and Evaluating Artificial Intelligence-Enabled Medical Device Performance in the
Real World. September 30, 2025. Docket FDA-2025-N-4203. https://www.fda.gov/medical-devices/digital-health-centerexcellence/request-public-comment-measuring-and-evaluating-artificial-intelligence-enabled-medical-device
[13] FDA. Artificial Intelligence-Enabled Medical Devices (list and program page). Updated September 2026.
https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices
[14] FDA. 510(k) Substantial Equivalence Determination Decision Summary, K253281 (UpDoc, UpDoc Inc.). Decision date December
23, 2025. https://www.accessdata.fda.gov/cdrh_docs/reviews/K253281.pdf
[15] FDA. Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo
Classifications, and Humanitarian Device Exemptions. Guidance. August 2019.
https://www.fda.gov/media/131930/download
[16] FDA. Multiple Function Device Products: Policy and Considerations. Guidance. 2020. https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/multiple-function-device-products-policy-and-considerations
[17] FDA. Policy for Device Software Functions and Mobile Medical Applications. Guidance. September 2022.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/policy-device-software-functions-andmobile-medical-applications

Page 43 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

[18] FDA. Content of Premarket Submissions for Device Software Functions. Guidance. June 2023. https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/content-premarket-submissions-device-software-functions
[19] FDA. Applying Human Factors and Usability Engineering to Medical Devices (February 2016); and Content of Human Factors
Information in Medical Device Marketing Submissions (final guidance, May 2026). https://www.fda.gov/regulatoryinformation/search-fda-guidance-documents/content-human-factors-information-medical-device-marketing-submissions
[20] FDA, Health Canada, MHRA. Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles. June 2024.
https://www.fda.gov/medical-devices/software-medical-device-samd/transparency-machine-learning-enabled-medicaldevices-guiding-principles
[21] FDA. Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions. Guidance. November
2023; and ASME V&V 40-2018, Assessing Credibility of Computational Modeling through Verification and Validation:
Application to Medical Devices.
[22] FDA. Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological
Products. Draft guidance. January 2025. https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological
[23] FDA. Digital Health Software Precertification (Pre-Cert) Pilot Program: Tailored Total Product Lifecycle Approaches and Key
Findings. September 2022. https://www.fda.gov/medical-devices/digital-health-center-excellence/digital-health-softwareprecertification-pre-cert-pilot-program
[24] FDA. Master Files (Device Master Files). https://www.fda.gov/medical-devices/premarket-approval-pma/master-files
[25] FDA. Accreditation Scheme for Conformity Assessment (ASCA) Program. https://www.fda.gov/medical-devices/premarketsubmissions-selecting-and-preparing-correct-submission/accreditation-scheme-conformity-assessment-asca
[26] FDA. Medical Device Development Tools (MDDT) Program. https://www.fda.gov/medical-devices/medical-devicedevelopment-tools-mddt
[27] FDA. News release: FDA Expands Artificial Intelligence Capabilities with Agentic AI Deployment. December 1, 2025 (following
agency-wide launch of the “Elsa” generative AI tool, June 2025). https://www.fda.gov/news-events/pressannouncements/fda-expands-artificial-intelligence-capabilities-agentic-ai-deployment
[28] FDA. TEMPO pilot for digital health devices in connection with the CMS ACCESS Model. Federal Register notice, December
2025.
[29] The White House. Winning the Race: America’s AI Action Plan. July 23, 2025.
[30] Office of the National Coordinator for Health IT (ASTP/ONC). HTI-1 Final Rule, 89 Fed. Reg. 1192 (Jan. 9, 2024) (decision
support interventions, 45 CFR 170.315(b)(11)); and HTI-5 Proposed Rule, 90 Fed. Reg. 60970 (Dec. 29, 2025).
[31] 21 CFR Part 803 (Medical Device Reporting) and 21 CFR Part 806 (Reports of Corrections and Removals).

B. International regulators and bodies
[32] International Medical Device Regulators Forum (IMDRF). Good Machine Learning Practice for Medical Device Development:
Guiding Principles. IMDRF/AIML WG/N88 FINAL:2025. January 2025. https://www.imdrf.org/documents
[33] IMDRF. Technical Framework for Artificial Intelligence Life Cycle Management. IMDRF/AIML WG/N93 DRAFT:2026. April 2026
(consultation closed July 2026). https://www.imdrf.org/consultations
[34] IMDRF. Software as a Medical Device: Possible Framework for Risk Categorization and Corresponding Considerations.
IMDRF/SaMD WG/N12FINAL:2014. September 2014.
[35] Regulation (EU) 2024/1689 (Artificial Intelligence Act), esp. Art. 6(1), Art. 53, Annex I and Annexes XI–XII; as amended by the AI
Omnibus (Regulation (EU) 2026/1744), which entered into force July 27, 2026. Per the European Commission, high-risk rules
for AI embedded in Annex I products apply from August 2, 2028 and for Annex III systems from December 2, 2027; other
obligations follow different timelines. European Commission, “AI Omnibus enters into force,” July 27, 2026. https://eurlex.europa.eu/eli/reg/2024/1689/oj; https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force
[36] European Commission, AI Office. General-Purpose AI Code of Practice (Transparency chapter and Model Documentation
Form). July 10, 2025. https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
[37] Medical Device Coordination Group & AI Board. MDCG 2025-6 / AIB 2025-1: FAQ on Interplay between the MDR & IVDR and
the Artificial Intelligence Act. June 2025.
[38] National Commission into the Regulation of AI in Healthcare (UK). Recommendations for a Future Regulatory Framework.
September 10, 2026 (44 recommendations; incl. Rec. 6 expanded PCCPs, Rec. 7 opt-in Master File for general-purpose
models, Rec. 8 dependency reporting, Rec. 14 staged authorisation, Rec. 28 contractual allocation of risk controls).
https://www.gov.uk/government/publications/national-commission-into-the-regulation-of-ai-in-healthcarerecommendations-for-a-future-regulatory-framework

Page 44 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

[39] UK Department of Health and Social Care / MHRA. Government Response to the National Commission’s Recommendations on
the Regulation of AI in Healthcare. October 6, 2026. https://www.gov.uk/government/publications/government-responseto-the-national-commissions-recommendations-on-the-regulation-of-ai-in-healthcare
[40] MHRA. AI Airlock sandbox pilot programme report and Phase 2 cohort. October 2025.
https://www.gov.uk/government/publications/ai-airlock-phase-2-cohort
[41] MHRA. Guidance on Ambient Voice Technology-Enabled Products. July 2026.
https://www.gov.uk/government/publications/ambient-voice-technology-enabled-products
[42] Ministry of Food and Drug Safety (Republic of Korea). Guideline on Review and Approval of Generative AI-Enabled Medical
Devices. January 2025.
[43] Health Canada. Pre-market Guidance for Machine Learning-Enabled Medical Devices. February 2025 (revised April 2026).
https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/applicationinformation/guidance-documents/pre-market-guidance-machine-learning-enabled-medical-devices.html
[44] World Health Organization. Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models.
January 2024. https://www.who.int/publications/i/item/9789240084759
[45] Singapore Ministry of Health, Health Sciences Authority & Synapxe. Artificial Intelligence in Healthcare Guidelines (AIHGle 2.0).
March 2026.

C. Standards and consensus frameworks
[46] National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial
Intelligence Profile (NIST AI 600-1). July 2024. doi:10.6028/NIST.AI.600-1
[47] ISO 14971:2019, Medical devices—Application of risk management to medical devices; AAMI CR34971:2022 (FDA-recognized)
and AAMI TIR34971:2023, Guidance on the application of ISO 14971 to artificial intelligence and machine learning.
[48] IEC 62304:2006+A1:2015, Medical device software—Software life cycle processes; IEC 62366-1:2015+A1:2020, Application of
usability engineering to medical devices; IEC 82304-1:2016, Health software—General requirements for product safety.
[49] ISO/IEC 42001:2023, Artificial intelligence—Management system; ISO/IEC 23894:2023, Artificial intelligence—Guidance on risk
management; ISO/IEC 42005:2025, AI system impact assessment.
[50] ISO/IEC TS 8200:2024, Information technology—Artificial intelligence—Controllability of automated artificial intelligence
systems.
[51] AAMI CR515:2025, Cybersecurity for machine learning-enabled medical devices (added to FDA recognized consensus
standards, March 2026).
[52] IEEE P3926 (project authorized March 2026). Recommended Practice for Large Language Model-Driven Patient-Oriented
Intelligent Healthcare Services (in development). https://standards.ieee.org/ieee/3926/12519/
[53] Coalition for Health AI (CHAI). Applied Model Card. 2025. https://registry.chai.org/applied-model-card
[54] The Joint Commission and Coalition for Health AI. Guidance on the Responsible Use of AI in Healthcare (RUAIH). September
2025.

D. Scientific and policy literature
[55] Patel B, Blumenthal D. A novel approach to overseeing the clinical application of generative AI. JAMA Health Forum.
2026;7(3):e256947. doi:10.1001/jamahealthforum.2025.6947
[56] Bergman A, Wachter RM, Emanuel EJ. A licensure framework for autonomous clinical AI. JAMA. 2026;335(20):1751-1754.
doi:10.1001/jama.2026.5483
[57] Freyer O, Jayabalan S, Kather JN, Gilbert S. Overcoming regulatory barriers to the implementation of AI agents in healthcare.
Nat Med. 2025;31(10):3239-3243. doi:10.1038/s41591-025-03841-1
[58] Garcia V, Sidulova M, Badano A. Performance assessment strategies for language model applications in healthcare. Artif Intell
Life Sci. 2026;9:100162. Preprint: arXiv:2509.08087
[59] Bean AM, Payne RE, Parsons G, et al. Reliability of LLMs as medical assistants for the general public: a randomized
preregistered study. Nat Med. 2026;32(2):609-615. doi:10.1038/s41591-025-04074-y
[60] Arora RK, Wei J, Hicks RS, et al. HealthBench: evaluating large language models towards improved human health.
arXiv:2505.08775. 2025.
[61] Bedi S, Cui H, Fuentes M, et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nat Med.
2026;32:943-951. doi:10.1038/s41591-025-04151-2
[62] Alaa A, Hartvigsen T, Golchini N, et al. Position: medical large language model benchmarks should prioritize construct validity.
Proc 42nd Int Conf Machine Learning (ICML), PMLR 267; 2025. arXiv:2503.10694

Page 45 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

[63] Bedi S, et al. Fidelity of medical reasoning in large language models. JAMA Netw Open. 2025;8(8):e2526021.
doi:10.1001/jamanetworkopen.2025.26021
[64] Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: a systematic
review. JAMA. 2025;333(4):319-328. doi:10.1001/jama.2024.21700
[65] Chen S, Gao M, Sasse K, et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic
behavior. npj Digit Med. 2025;8:605. doi:10.1038/s41746-025-02008-z
[66] Clusmann J, Ferber D, Wiest IC, et al. Prompt injection attacks on vision language models in oncology. Nat Commun.
2025;16:1239. doi:10.1038/s41467-024-55631-x
[67] Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature.
2024;630:625-630. doi:10.1038/s41586-024-07421-0
[68] Angelopoulos AN, Bates S, Fannjiang C, Jordan MI, Zrnic T. Prediction-powered inference. Science. 2023;382(6671):669-674.
doi:10.1126/science.adi6000
[69] Croxford E, et al. Evaluating clinical AI summaries with large language models as judges. npj Digit Med. 2025;8.
doi:10.1038/s41746-025-02005-2
[70] Chen L, Zaharia M, Zou J. How is ChatGPT’s behavior changing over time? Harv Data Sci Rev. 2024;6(2).
doi:10.1162/99608f92.5317da47
[71] He H; Thinking Machines Lab. Defeating nondeterminism in LLM inference. Connectionism. September 2025.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
[72] Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS
suggestions on reader performance. Radiology. 2023;307(4):e222176. doi:10.1148/radiol.222176
[73] Omiye JA, Lester JC, Spichak S, et al. Large language models propagate race-based medicine. npj Digit Med. 2023;6:195.
doi:10.1038/s41746-023-00939-z
[74] Shumailov I, Shumaylov Z, Zhao Y, et al. AI models collapse when trained on recursively generated data. Nature. 2024;631:755759. doi:10.1038/s41586-024-07566-y
[75] Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study.
BMJ. 2015;351:h3480. doi:10.1136/bmj.h3480
[76] Ramaswamy A, Tyagi A, Hugo H, et al. ChatGPT Health performance in a structured test of triage recommendations. Nat Med.
2026;32:1671-1675. doi:10.1038/s41591-026-04297-7. PMID 41731097.
[77] Fraile Navarro D, Magrabi F, Coiera E. Evaluation format, not model capability, drives triage failure in the assessment of
consumer health AI. arXiv:2603.11413. 2026 (preprint).
[78] Wu K, Wu E, Wei K, et al. An automated framework for assessing how well LLMs cite relevant medical references. Nat
Commun. 2025;16:3615. doi:10.1038/s41467-025-58551-6
[79] Jiang Y, Black KC, Geng G, et al. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents. NEJM AI. 2025.
Preprint: arXiv:2501.14654
[80] ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39(12):1176-1177.
doi:10.1111/j.1365-2929.2005.02341.x
[81] Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63-S67.
doi:10.1097/00001888-199009000-00045
[82] Griot M, Hemptinne C, Vanderdonckt J, Yuksel D. Large language models lack essential metacognition for reliable medical
reasoning. Nat Commun. 2025;16:642. doi:10.1038/s41467-024-55628-6
[83] Asgari E, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digit
Med. 2025;8:274. doi:10.1038/s41746-025-01670-7
[84] Lukac PJ, Turner W, Vangala S, et al. Ambient AI scribes in clinical practice: a randomized trial. NEJM AI. 2025;2(12).
doi:10.1056/AIoa2501000. PMID 41497288.
[85] Laban P, Hayashi H, Zhou Y, Neville J. LLMs get lost in multi-turn conversation. International Conference on Learning
Representations (ICLR) 2026. arXiv:2505.06120
[86] Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making.
2006;26(6):565-574. doi:10.1177/0272989X06295361
[87] Lee B, et al. Early recalls and clinical validation gaps in artificial intelligence-enabled medical devices. JAMA Health Forum.
2025;6(8):e253172. doi:10.1001/jamahealthforum.2025.3172
[88] Wu E, Wu K, Daneshjou R, et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of
FDA approvals. Nat Med. 2021;27(4):582-584. doi:10.1038/s41591-021-01312-x

Page 46 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

[89] Moore J, Grabb D, Agnew W, et al. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental
health providers. Proc ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25). 2025:599-627.
[90] Heinz MV, Mackin DM, Trudeau BM, et al. Randomized trial of a generative AI chatbot for mental health treatment. NEJM AI.
2025;2(4). doi:10.1056/AIoa2400802
[91] Quach V, Fisch A, Schuster T, et al. Conformal language modeling. International Conference on Learning Representations (ICLR)
2024.
[92] Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA.
1983;249(13):1743-1745.
[93] Page ES. Continuous inspection schemes. Biometrika. 1954;41(1/2):100-115; Wald A. Sequential tests of statistical hypotheses.
Ann Math Stat. 1945;16(2):117-186.
[94] American Medical Association. AI usage among doctors doubles as confidence in technology grows (2026 physician survey).
March 2026. https://www.ama-assn.org/press-center/ama-press-releases/ama-ai-usage-among-doctors-doublesconfidence-technology-grows
[95] State laws on AI in health care, including Illinois Wellness and Oversight for Psychological Resources Act (HB 1806, 2025); Utah
HB 452 (2025); Nevada AB 406 (2025); California AB 489 (2025); and Texas SB 1188 and HB 149 (2025).
[96] Fierce Healthcare. Inside CHAI’s failed assurance labs. February 2026. https://www.fiercehealthcare.com/ai-and-machinelearning/inside-chais-failed-assurance-labs

E. References added in final edit
[97] HLTH. UpDoc debuts first FDA-cleared patient-facing clinical LLM for insulin management. July 2, 2026 (reporting the
manufacturer’s June 25, 2026 announcement). https://hlth.com/insights/news/updoc-debuts-first-fda-cleared-patientfacing-clinical-llm-for-insulin-management
[98] FDA, CDRH. CDRH Proposed Guidance for Fiscal Year 2027 (FY 2027) (A-list, B-list, and Under Construction list). October 2026.
https://www.fda.gov/medical-devices/guidance-documents-medical-devices-and-radiation-emitting-products/cdrhproposed-guidances-fiscal-year-2027-fy-2027; https://www.fda.gov/media/195054/download
[99] 21 CFR Part 812 (Investigational Device Exemptions), including §812.2(b) (abbreviated requirements for non-significant-risk
studies), §812.2(c) (exempted investigations), and §812.28 (acceptance of data from clinical investigations conducted
outside the United States); 21 CFR Part 50, including §50.22 (IRB waiver or alteration of informed consent for minimal-risk
clinical investigations); and 21 CFR Part 56 (Institutional Review Boards).
[100] FDA. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices. Final guidance (fiscal year
2026).
[101] FDA. Acceptance of Clinical Data to Support Medical Device Applications and Submissions: Frequently Asked Questions.
Guidance. February 2018.
[102] SPDX 3.0 specification (Linux Foundation, 2024), AI and Dataset profiles https://spdx.dev; and CycloneDX Bill of Materials
standard (ECMA-424), including machine-learning BOM support https://cyclonedx.org.
[103] FDA. Requests for Feedback and Meetings for Medical Device Submissions: The Q-Submission Program. Guidance.
https://www.fda.gov/media/114034/download

Page 47 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Appendix A. Crosswalk of Discussion Questions to Regavise
Recommendations
Q Topic Core recommendation(s) Section

1 Two-axis framework Keep grid; four documented contextual factors without automatic 3.2, 3.3, 4.1
band changes (reversibility/time-to-harm, verifiability/traceability,
exposure scale, architectural containment); split activity axis;
opacity of measurement/IVD functions treated as higher
probability of undetected reliance, not higher severity

2 Directiveness spectrum Pilot Directiveness Ladder D0–D4; Observed Directiveness Profile; 4.1
verifiable directiveness ceiling; CDS worked examples

3 Patient-facing functions Verification capacity as operative variable; heightened evidence 4.1, 5.2
by default for patient-facing D3–D4; naturalistic lay-user testing

4 Generalist vs. specialist Risk-raising condition for specialist-dependent outputs to 4.1
generalists; human–AI team testing

5 Multi-turn conversation Conversational Trajectory Envelope; trajectory sampling by 4.1
conversation length; envelope-escape metric; bounded memory
schema

6 Escalation errors Non-compensatory safety floor + weighted net benefit; stratified 4.1
confusion matrices; prevalence-calibrated PPV; alert-fatigue
monitoring; sentinel cases

7 Competency-based Adopt; optional Entrustable Device Activities (E1–E5); map to 3.4, 4.2
approach Miller’s pyramid

8 Risk-to-evidence Illustrative risk–evidence examples; standardized, nonbinding Pre- 3.5, 4.2
Submission feedback template

9 Benchmark elements Add S.4, S.5, S.6, E.5, H.1, R.3; split A.1; standards crosswalk; two- 4.2
sided refusal reporting; premature closure; deterministic
calculation delegation

10 Benchmark validity Risk-proportionate validity evidence; escrow/sequestration for 4.2
sponsor benchmarks; risk-to-evidence traceability ledger

11 Clinical confirmation Input endogeneity test; gated deployment; stepped-wedge 4.2
rollout; Input Distribution Coverage Map; GenAI OUS bridging
justification; Part 50/812 clarifications

12 Statistics Three levels of risk; repeated sampling; clustered CIs; 4.2
consistency@k; zero-failure planning with stated assumptions; PPI
where validated; rater variance; synthetic as separate stratum

13 Synthetic data Use for stress/coverage; generator independence; realism tests; 4.2
real-data bridging for subgroups

14–15 Comparators Separate correctness reference, safety floor, benefit comparator; 4.2
counterfactual care pathway

16 Third parties Optional; methods first, then possible ASCA-like pilot; voluntary 4.2
MDDT; competition safeguards

Page 48 of 49
 Regavise | Comment on Docket No. FDA-2026-N-7874

Q Topic Core recommendation(s) Section

17 Other architectures Architecture-specific supplements; CM&S credibility for world 4.2
models; multimodal injection

18 Premarket uncertainty Six Monitoring-Readiness Criteria as decision aids; “two clocks” 4.3
limit; autonomy progression only within authorized PCCP or new
submission

19 Postmarket methods Canary probes; leading indicators; input drift; outcome linkage; 4.3
complaint mining; triggers

20 Supervisory agents Independence, validation, PPI, human sampling, configuration 4.3
control; real-time supervisors are device components

21 Shared ecosystem Labeled Conditions of Safe Deployment; accountability map; 4.3
collaborative community

22 Scaling re-benchmarking Four-tier change taxonomy (analytical aid only); differential (flip- 3.1, 4.4
rate) testing; epoch-gated releases for model-evolution changes

23 PCCP adaptation Envelope-based PCCP with configuration, behavioral, and 4.4
performance bounds, still sufficiently specified for authorization

24 Third-party model changes Risk-based supplier controls; version pinning; canary detection; 4.4
validated fallback; equivalent controls for small sponsors

25 Foundation Model MAF Tiered template with voluntary mapping to EU Annex XII; change 4.5
notification; dependency transparency

26 Agentic AI Action-authority levels AA0–AA4; trajectory-level criteria; least 4.5
privilege; non-bypassable checkpoints; tool-call validation and
execution limits

Page 49 of 49