Abridge (Michelle Lease)

Please see attached comments from Abridge.

October 19, 2026

Michelle E. Tarver, M.D., Ph.D.
Director
Center for Devices and Radiological Health
U.S. Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993

Re: FDA-2026-N-7874

Dear Dr. Tarver:

Abridge appreciates the opportunity to provide comments in response to the U.S. Food and
Drug Administration’s (FDA) Center for Devices and Radiological Health (CDRH) discussion
paper titled “Considerations for the Regulation of Generative AI-Enabled Medical Devices”
(FDA-2026-N-7874).

Abridge was founded in 2018 by physicians, for physicians, with a mission to empower deeper
understanding in healthcare. Abridge has developed a clinical intelligence platform to
support physicians at every step of patient and clinical care. Our first product uses
generative artificial intelligence (AI) to transform clinical conversations into structured
documentation, integrating information into patient electronic health records (EHRs). We
have since expanded into clinical decision support (CDS)—providing evidence-based
information directly into clinician workflows—as well as nursing, clinical trial matching, and
pre-visit chart preparation.

Today, Abridge is deployed across more than 300 health systems and is expected to support
more than 100 million clinical conversations annually. Burnout among clinicians in ambulatory
clinics fell from 52% to 39% after 30 days with Abridge, with significant gains in their
attention to patients.1 This experience deploying generative AI in clinical environments
informs our perspective on several of the questions CDRH raised. Abridge supports FDA
efforts to consider how its existing risk-based medical device framework should apply to
generative AI.

As FDA recognizes, generative AI raises new questions due to its open-ended inputs and
outputs, evolving models, and reliance on third-party foundation models. At the same time,
the discussion paper appropriately does not assume that using generative AI itself
Olson et al., “Use of Ambient AI Scribes to Reduce Administrative Burden and Professional Burnout,’
JAMA (Oct. 2025), https://pmc.ncbi.nlm.nih.gov/articles/PMC12492056/.
determines whether a software function is a medical device or its risk level. This distinction
should remain foundational to any future framework.

FDA should preserve the statutory distinctions established under Section 520(o) of the
Federal Food, Drug, and Cosmetic Act (FDCA), including applicable exclusions for certain
administrative support, electronic health record, and clinical decision support functions. The
incorporation of generative AI into an otherwise excluded software function should not, by
itself, trigger medical device regulation. FDA should first evaluate the specific function and
intended use and, where the function is subject to device regulation, apply regulatory
requirements proportionate to the risks presented by that function.

As the FDA considers stakeholder feedback on the discussion paper and charts a new path
forward for regulating generative AI-enabled devices, Abridge encourages the Agency to
preserve a function-specific, risk-based approach. We appreciate the FDA’s consideration of
two key risk elements: the potential consequences of an incorrect output and the degree of
autonomy of a software function from that of a clinician. Those factors should be considered
alongside other factors such as the function’s intended use, clinical context, intended users,
detectability of errors, the presence of meaningful healthcare professional oversight, and
safeguards built into the product or the clinical workflow.

Rapidly evolving AI technologies will benefit from increased reliance on post-market
monitoring that pre-market testing alone cannot fully replicate, as the FDA recognizes in the
discussion paper. Relying primarily on pre-market testing for an improving technology could
be infeasible and burdensome. Monitoring production notes and gathering feedback from
early adopters of the product who identify needed improvements are valuable techniques for
complementing pre-market evidence and identifying meaningful changes in real-world
performance over time.

Acknowledging the specific questions posed by the FDA in the discussion paper, appended
to this comment letter are specific responses to the FDA’s questions.

A Function-Specific Approach to Risk

Generative AI is increasingly incorporated into a wide variety of healthcare tasks, from
technologies that synthesize documentation and information to those that recommend,
direct, or independently execute clinical actions. With these diverse use cases, the FDA is
appropriately planning to evaluate software in a risk-based way. In doing so, FDA should
consider primarily what a particular function does, who uses it, and what happens if it
produces an incorrect output, rather than on the technical architecture used to generate
that output.

The FDA’s discussion paper provides a useful foundation for this approach by distinguishing
among informational, action-directing, and action-taking functions, and by identifying the
degree of independence a technology may have from a clinician as an important risk
consideration. The FDA also recognizes meaningful differences between patient-facing and
clinician-facing functions, generalist and specialist-facing users, and
continuously-supervised and autonomous applications. As Abridge has seen from the
development of our new nursing product, the workflows and needs of non-physician
clinicians present an additional dimension of difference. These distinctions should remain
central to any future framework. A clinician-facing application that organizes, summarizes, or
documents information for professional review presents a fundamentally different risk
severity than a system that independently recommends a medication change or executes a
clinical action.

In addition, the FDA should consider risk from an incorrect output in the context of the actual
clinical workflow, and update its risk framework to consider more broadly the risk of an
incorrect output impacting patient health. In its current risk framework, the FDA assumes a
device has made an error, and then compares the severity of harm if the incorrect output is
relied upon, to device activity (i.e., a measure of the degree of automation of the device). In
doing so, the FDA is assuming that the device activity correlates with the likelihood that an
incorrect output will affect patient health, but it is an incomplete proxy. Several additional
factors specific to the device and the clinical setting contribute to that probability, and FDA
should consider updating the X-Axis of its framework to more directly assess that probability
on a spectrum, rather than apparently discrete degrees of automation.

For example, the FDA should also examine the detectability of an error in its X-Axis
assessment. If a CDS solution recommends treating a chronic headache with treatment or
support that is not consistent with standard of care, such as brain surgery, that is starkly
different than treating a chronic headache with a medication that is contraindicated for the
patient, because the clinical workflow is much more likely to detect the error before brain
surgery takes place.2 It is also critical to ensure that the FDA distinguishes between where an
error ‘can’ be reviewed and whether an error is ‘easy to review and catch.’ For example,
identifying an omission in a clinical note captured during a previous patient visit on the same
day may be considered straightforward, however identifying an omission from a pre-visit
summary from a patient visit weeks or months in the past may prove more difficult. Abridge
urges the FDA to use its risk framework to assess the likely clinical consequence if an error is
not identified.

Detectability depends on product design as much as on the underlying model. Design
features such as linking outputs to source material, flagging uncertainty, presenting outputs
clearly for review, and making them easy to edit or reject can make errors easier to catch.
Accordingly, FDA should consider error detectability, and the design features that improve it,
as one component of an overall risk assessment, rather than as a standalone testing or
As with the FDA’s Discussion Paper, this document illustrates its point with examples that do not
necessarily fall within the statutory definition of a device.
validation requirement applicable to all generative AI-enabled functions. The relevance of
detectability will necessarily depend on the intended use, the clinical significance of the
output, and the opportunities for correction before an error could affect patient care.

Meaningful clinician oversight should be recognized as an important risk-mitigating factor,
with consideration given to the nature of the software function, the clinical context, and the
potential consequences of an incorrect output. FDA should distinguish between workflows in
which professional review is necessary to mitigate meaningful clinical risk and those in which
additional review requirements would provide limited safety benefits while increasing
administrative burden. Relevant considerations may include whether an output is clearly
presented for review; whether a clinician can readily inspect, edit, reject, or correct it;
whether clinical action occurs before or after that review; and whether the system preserves
clinician decision-making authority. The presence, timing, and nature of clinician oversight
should inform a function-specific risk assessment rather than operate as a universal
regulatory prerequisite; other safeguards may play a larger role as technology and evidence
mature. This approach would allow FDA to recognize meaningful human oversight without
relying solely on a nominal “human-in-the-loop” designation that may not accurately reflect
how a technology functions in practice.

Evaluating Performance in the Context of Intended Use: Pre-market to Post-market

The open-ended nature of generative AI makes it difficult, and potentially impossible, to
evaluate every conceivable input and output before deployment. Abridge therefore supports
the FDA’s exploration of competency-based approaches that combine appropriate
benchmarking with clinical confirmation and other evaluation methods proportionate to risk.
Abridge also welcomes the FDA’s consideration of accepting greater pre-market uncertainty
when it can be addressed through strong post-market monitoring.

A competency-based approach may be particularly well suited to generative AI because it
can focus evaluation on whether a particular software function can reliably perform a
specific intended task, rather than attempt to characterize every capability or possible
output of the underlying generative model.

Benchmarks: Within such a framework, FDA should recognize the value of
sponsor-developed benchmarks, particularly when they closely reflect the anticipated use
case, intended population, clinical context, and actual conditions in which the function will
operate. Standardized or independent benchmarks can provide useful comparative
information, but they can diverge from real-world conditions and should complement, not
displace, evaluations designed around the specific function and its intended use.
Independent organizations may not have access to real clinical conversations, or access may
be limited to a single healthcare facility or hospital system whose patient base is
unrepresentative. Independent benchmarks also often do not test products end-to-end, and
so may miss factors such as how a noisy environment affects a documentation tool.
Abridge’s own evaluation practices illustrate this approach. Abridge rates potential errors in
generated notes on two dimensions: how well each statement is supported by the
underlying conversation, and the severity of its potential clinical impact if left uncorrected.
Abridge has also built a range of internal benchmarks that together cover more than 10,000
realistic clinical encounters, supported by over a thousand hours of annotation and
validation by board-certified physicians, and validates its automated evaluators against
clinician annotations before relying on them.3

Evidentiary Standard: An appropriate evidentiary standard should depend on the function’s
intended use, clinical context, users, degree of autonomy, and potential consequences of
error. For lower-risk clinician-facing functions, retrospective evaluation, representative
real-world data, clinician adjudication, workflow testing, and shadow deployment may
provide meaningful evidence of performance. Technologies capable of independently
directing or executing consequential clinical actions may appropriately require additional
evidence and additional post-market monitoring, scaled to their risk.

Product-Lifecycle Approach: Generative AI also challenges traditional distinctions between
pre-market and post-market evaluation because these technologies may evolve more rapidly
than conventional medical devices. Abridge supports the FDA’s consideration of a
total-product-lifecycle approach that combines appropriate pre-market evaluation with
robust, risk-proportionate monitoring of real-world performance, including the approaches
FDA identifies: periodic benchmarking, sample-based clinician review, real-world
performance monitoring, and monitoring for performance degradation.

Transparency: Product-level transparency also supports oversight across the lifecycle.
Companies building AI products, beyond those making frontier models, have begun making
model cards for their products available. For example, Abridge makes model cards and other
documentation available to customers, giving health systems and clinicians information they
need to deploy a tool appropriately and monitor it locally. Documentation of this kind can also
create a record of how a product performs across versions. FDA could recognize
product-level documentation as a way that sponsors demonstrate ongoing performance,
without prescribing a single format.

Independent Evaluators: The discussion paper also identifies independent evaluators and
adjudicators as another potential approach. We believe that third-party evaluation could be
most successful if it provides high-quality, standardized evaluations; includes competition
amongst evaluators and those that certify evaluators; follows clearly described
See Samir Khan, Catherine Chen, Michael Oberst, Alex Chouldechova, “AI Evaluation at Abridge”
(Aug. 17, 2026), https://tech.abridge.com/blog/ai-evaluation-at-abridge; Abridge, Pioneering the
Science of AI Evaluation; Abridge, The Science of Confabulation Elimination Toward Hallucination-Free
AI-Generated Clinical Notes, https://www.abridge.com/whitepapers.
conflict-of-interest requirements; are transparent to FDA staff and regulated industry
stakeholders; protects proprietary and patient data; and is used to inform – but not alone
determine – FDA determinations on clearance of generative AI-enabled devices, as the
sponsor should retain responsibility for whether a device is meeting its intended purpose
and use for the intended populations.

Post-Market Monitoring: For rapidly evolving AI technologies, trying to eliminate all
uncertainty through increasingly burdensome pre-market testing may be neither feasible nor
the most effective way to protect patients. For generative AI in particular, monitoring
real-world performance may provide important signals that pre-market testing alone cannot
fully replicate, such as use of large language model (LLM) judges (LLJs) to monitor
production notes,4 collection of feedback from early adopters of the product who identify
needed improvements, and analysis of clinician review data (e.g. time spent editing an AI
output). FDA should therefore consider how validated, risk-appropriate online monitoring
approaches can complement pre-market evidence and identify meaningful changes in
real-world performance over time.

Foundation Models and Modifications

A framework that sets appropriate initial evidence requirements combined with monitoring
clinically meaningful performance in real-world use could give regulators and developers
better information about how a system performs over time. For example, a modification to an
underlying foundation model should not automatically be treated as a clinically meaningful
modification to every downstream application that relies upon it. In Abridge’s experience,
foundation model changes are deliberate rather than unplanned, which allows each change
to be evaluated before deployment.

Instead, the relevant question for FDA is whether the foundation model change (as
implemented by the device) materially affects the performance, intended use, or risk profile
of the regulated software function. Periodic benchmarking and online performance
monitoring are alternate ways to measure whether a change materially affects performance.
Such a risk-based model would align with FDA’s rules on software updates for medical
devices, which require new pre-market submissions if a change “could significantly affect
the safety or effectiveness of the device.”5

Predetermined acceptance criteria, appropriate validation, and Predetermined Change
Control Plans could help demonstrate that an application continues to perform within
An LLJ is an evaluator that uses an LLM to score an input according to a set of instructions. LLJs are
commonly used to evaluate GenAI outputs, from ambient scribe outputs to non-clinical applications
such as chatbot responses.
21 C.F.R. § 807.81(a)(3)(i); see also FDA, Deciding When to Submit a 510(k) for a Software Change to
an Existing Device (2017).
established parameters after changes. This would preserve appropriate oversight while
recognizing the iterative nature of AI development.

The FDA’s proposed voluntary Foundation Model Device Master File concept may also offer a
useful mechanism to reduce duplicative submissions and allow downstream sponsors to
reference technical information without disclosing proprietary information. However,
regulatory responsibility should remain aligned with control over the regulated function and
its intended use. A general-purpose foundation-model developer may have little or no
visibility into a downstream application's design, intended use, workflow integration, or
clinical risk.

The FDA’s emphasis on evaluating the final user-facing product, rather than the underlying
foundation model alone, is particularly important. The performance of a clinical GenAI
application may depend on much more than the foundation model, including retrieval
systems, prompts, workflow design, user interfaces, application-specific guardrails, and
post-processing. Additionally, performance evaluations may differ depending on what
literature the system has access to, the agent architecture, and how information is reflected
or shown to users. The same underlying model could therefore support applications with
very different intended uses and risk profiles. FDA itself recognizes that different functions
on a broader AI platform may warrant different regulatory treatment.

Accordingly, the FDA should focus primarily on the performance of the specific software
function as deployed for its intended use, rather than assessing a foundation model as a
proxy for the performance or risk of every downstream application. Evaluating every iteration
of a foundation model independently would not keep pace with model development and
would say little about the safety of any application.

Preserving a Risk-Based Regulatory Gradient

Ultimately, Abridge encourages the FDA to avoid developing a single regulatory paradigm for
“generative AI” in health care. The category encompasses technologies with profoundly
different functions and risk profiles. A more useful framework would instead recognize a
regulatory gradient ranging from informational and documentation support to
clinician-directed decision support, to action-directing functions, to action-taking or agentic
systems, and ultimately to autonomous clinical action. The appropriate level of regulatory
scrutiny should increase as a function becomes more autonomous, the consequences of
incorrect output become more significant, and safeguards against error diminish.

Where a broader product or platform includes functions that fall at different points along this
gradient, the FDA should evaluate the particular function and intended use rather than
assigning the entire product the regulatory characteristics of its highest-risk potential
capability. An informational or documentation-support function should not necessarily
become subject to requirements appropriate for an autonomous clinical function simply
because both are offered through the same platform or rely on the same underlying
foundation model.

Similarly, a software function may not always fit neatly into a single category. An interaction
that begins by providing information, for example, may subsequently direct or facilitate an
action. In such circumstances, the FDA should focus on the function being performed, the
degree of independence involved, and the pathway through which the output can affect
patient care. This functional approach can accommodate evolving capabilities without
requiring the FDA to regulate an entire technology according to the most consequential
action it could theoretically perform.

Such an approach would build on the FDA’s proposed two-axis framework and provide
developers, clinicians, health systems, and patients with greater regulatory predictability. It
would also help ensure that requirements designed for autonomous technologies are not
inadvertently applied to clinician-facing assistive applications merely because both employ
generative AI.

Conclusion

Abridge appreciates the FDA’s thoughtful consideration of these issues and its decision to
seek stakeholder input before developing or proposing a new regulatory approach. The
discussion paper provides an important opportunity to establish principles that can remain
durable even as the underlying technology continues to evolve.

Abridge encourages the FDA to maintain a framework grounded in a technology’s actual
function and intended use, rather than the mere presence of generative AI. The potential
consequences of error, the likelihood that an error is caught, degree of autonomy, clinical
context, intended users, safeguards built into a product and its workflow, and the pathway
through which an output can affect patient care should collectively inform the appropriate
level of regulatory oversight.

To evaluate performance, the FDA should similarly focus on whether the regulated function
demonstrates the competencies needed to perform its intended task safely and effectively.
This approach can accommodate rapidly evolving foundation models without requiring the
FDA to evaluate every underlying model or every possible output. A function-specific,
competency-based, and risk-proportionate framework, supported by appropriate lifecycle
monitoring, can protect patients while preserving the ability of clinicians and health systems
to adopt technologies that reduce administrative burden, support clinical workflows, and
give health care professionals more time for patients.

Abridge welcomes the opportunity to continue engaging with the FDA as the Agency
evaluates stakeholder feedback and stands ready to provide further technical feedback.
Sincerely,

Michelle Lease
Head of Policy and Government Affairs
Abridge AI Inc.

Encl.: Abridge Responses to FDA Questions for Consideration
Appendix A: Abridge Responses to FDA Questions for Consideration

Response to Section IV, Question 1

The two-axis risk framework could be improved by broadening the X-Axis from device activity
to “likelihood of harm.” As described above (see: A Function-Specific Approach to Risk),
several additional factors influence the likelihood of harm:
●​ Detectability: a clinical output that is incorrect in an easily detectable way may be
lower risk, while a subtly incorrect output could be higher risk.
●​ Review Workflow: Some errors may also be more easily reviewed and caught, such as
identifying an omission from a patient note on the same day as the clinical visit versus
identifying an omission in a pre-visit summary from a months-old patient visit.
●​ Reversibility: The recourse for omissions or errors identified.
●​ Traceability: The ability to identify or trace an output, such as through Linked
Evidence in Abridge’s clinical notes, can reduce the likelihood of harm.

Design features in products and clinical workflows that make errors easier to detect and
correct should be supported in order to lower the overall risk, for any degree of device
activity.

Response to Section IV, Question 2

Outputs that offer considerations for a clinician, such as differential diagnoses or alternative
treatment possibilities, are less directive than a single, specific recommendation and
generally carry less risk. A calibrated assessment of confidence could also help a clinician
judge how much to rely on an output, though generative AI models are still improving to a
point where this can be consistently and accurately provided. Specific wording,
personalization, or context therefore may be less effective at reducing risk than presenting a
set of clinical considerations to the clinician.

Response to Section IV, Question 3

The FDA should consider different thresholds of risk as part of its framework, depending on
the end user, i.e. whether clinical information is being presented to a patient only or
healthcare provider only. In addition, safeguards such as a clear disclosure that AI has been
used in information provided, or, where appropriate, clinician review prior to documentation
being handed off to the patient, could represent additional risk mitigation.

Response to Section IV, Question 4

The distinction between a generalist healthcare provider or specialized healthcare provider
may be relevant in some contexts, such as when a generalist receives specialist-level
information, and could be considered as one factor in the assessment of risk.
Response to Section IV, Question 5

Risk should be assessed at a turn-by-turn level, accounting for the entire conversational
history or exchange to ensure systems are not evaluated based on a narrow subset of use,
such as individual question-and-answer turns, when multi-turn use is supported.

For single-turn evaluations, Abridge utilizes rubrics and gold responses.

Response to Section IV, Question 6

Evaluation of generative AI-enabled care escalation functions should weigh
under-escalation and over-escalation in the context of each case. For example, failing to
escalate a possible malignancy during post-oncological treatment surveillance carries a very
different risk than failing to escalate a minor incidental finding. When evaluating these risks,
the FDA should consider over- or under-escalation against a status quo baseline of a
clinician not employing generative AI, since over- and under-escalation also occur in current
practices that do not use AI.

Response to Section V, Question 7

Abridge supports the tenets of the competency-based approach. However, specifics around
device benchmarking merit additional time and attention, such as how benchmarks will be
kept current, where the responsibility lies to ensure device benchmarking is complete, and
who should conduct audits in light of potential incentivization and capability challenges. A
further question is how often benchmarking should be repeated and whether that timeline
should parallel requirements for software updates to conventional devices, such as an MRI
machine.

Response to Section V, Question 8

High-risk, consequential clinical activities may require greater levels of evidence when
considered using the two-axis risk framework. The level of evidence required should also
account for the rapid pace of innovation within AI development and growth of generative
AI-enabled medical devices. Unlike drugs or traditional medical devices, where a property of a
drug’s active pharmaceutical ingredients (APIs) or a medical device’s mechanical parts
remain the same between batches or production runs and can be examined or tested, AI
systems do not remain static. In the time between testing to FDA approval, an AI system may
have been able to make significant improvements. These factors make expeditious reviews
especially valuable by increasing the accrual of safety and efficacy data on an AI system, and
the relevance of such evidence to the latest AI capabilities.

Response to Section V, Question 9
Internal and external benchmarking can both be valuable, such as EHR retrieval benchmarks,
which are important to understanding whether AI systems are identifying relevant content
and information well. Accuracy, safety, fairness, and usability are all reasonable to assess
prior to deployment of a generative AI-enabled medical device, such as through
benchmarking, back-testing, red-teaming, and silent deployments. In addition, many of the
tools used for pre-market evaluation may also be repurposed for post-market monitoring.

Response to Section V, Question 10

Sponsor-developed benchmarks have an important role to play, as they can represent higher
levels of construct validity due to the inclusion of real-world data or queries from end users.
Concerns about overfitting to the benchmark can be addressed through practices such as
held-out test sets and prespecified analysis plans. Abridge supports the FDA’s efforts to
consider publicly available benchmarking assets as part of the overarching framework and
welcomes further discussion in this area. Any movement toward making benchmarking
assets publicly available should involve discussion among a diverse group of stakeholders
and a commitment to the highest standards of realism.

Response to Section V, Questions 12 and 13

Real patient data should be the primary basis for evaluating a generative AI-enabled medical
device, and clearance based largely on synthetic data should be viewed with caution.
However, there may be utility in use of synthetically generated inputs in cases where real
patient data are limited, such as during pre-marketing and pre-testing of an LLM-powered
question and answer tool that has not been asked questions by real users. Where synthetic
or semi-synthetic data are used, sponsors should be able to establish its similarity to the real
patient data it supplements.

Response to Section V, Questions 14 and 15

There are two key questions that must be answered when assessing performance for
open-ended device outputs: 1) what is being compared? and, 2) what is it being compared
to?

Regarding the first question, when deciding the appropriate comparator, it is important to
note that the variance from clinician to clinician may be large. For example, in the clinical note
application, one way to assess performance may be clinician revisions; however, one clinician
may revise a visit summary to include a certain call out, whereas another clinician may revise
a visit summary to highlight a different detail. This may be true across the generalist
healthcare provider and specialist healthcare provider spectrum, regardless of what is being
compared. For this reason, acceptance criteria based on clinically meaningful errors may be
more useful than agreement with any single reference.
Regarding the second question on what is it being compared to, the most appropriate
comparison may be clinicians using the tool versus clinicians practicing as they do today. This
consideration is analogous to studies of new drugs, where a potential new drug is tested for
comparison against the existing standard of care. Unaided clinical judgment is becoming
less common with the advent of new AI solutions and other decision-support tools, and may
not represent a realistic baseline in the future. We also urge careful consideration of
additional factors, such as clinical setting, AI training, and AI fluency.

Response to Section V, Question 16

Qualified, independent third parties may play an important role in benchmarking and clinical
confirmation as an independent arbiter for both developers and hospitals or health systems.
For example, hospitals and health facilities may sometimes decline to provide certain coding
or claims data for fear of improper use or sharing confidential information, but may trust a
third party to receive and develop benchmarks using appropriately deidentified health
system data.

Third-party involvement would be most successful if it provides high-quality, standardized
evaluations; includes competition amongst evaluators and certifiers of evaluators; follows a
transparent process with clear conflict-of-interest requirements; protects proprietary and
patient data; and informs, but does not alone determine, FDA determinations.

There should also be a clear definition for what ‘real data’ means in the context of
back-testing or internal benchmarks. While Abridge exclusively uses de-identified data to
back-test or internally benchmark, the same definition may not hold true for another
developer, and clarity by the FDA would be welcome to ensure a standardized definition.

Response to Section VI, Question 18

Abridge believes it reasonable to have greater reliance on post-market monitoring when
pre-market evaluations cannot accurately reflect post-market conditions. This might include
staged rollouts, which progress from ‘silent release’ to full roll-out of a solution’s use under
intended use conditions. Additional safeguards, such as review requirements or a lower
threshold on escalation, could be employed to reduce risk for systems intended to operate
autonomously.

Response to Section VI, Question 19

As stated in response to Section V, Question 7, benchmarks should evolve alongside new
inputs or new forms of LLM errors. Reassessment could be triggered by events such as a
change in foundation model or a foundation model generation upgrade, a shift in the patient
population or clinical setting, or a change in monitored performance.
Response to Section VI, Question 20

Machine-based supervisory agents such as LLM judges (or “machine-based supervisory
agents”)are a critical part of Abridge’s continual monitoring work. Manual human review
cannot match the scale, rate, and volume of patient encounters, particularly for solutions
such as those Abridge offers which are expected to be used in over 100 million encounters
annually. However, ensuring supervisory agents have high validity and reliability while
challenging is necessary. For example, in our experience, some LLM judges align better with
clinician judgment for some applications than others. If machine-based supervisory agents
are used, they must be validated and continually revalidated, particularly as the systems
under supervision evolve.

Response to Section VI, Question 21

A wide range of stakeholders, including clinicians, health systems, professional societies,
patient advocacy groups, developers, and standards bodies should play a role in post-market
monitoring, such as sharing best practices. These stakeholders can help develop evaluation
methods centered around real data. Appropriate transparency about post-market
monitoring information can also build trust with patients and the public.

Response to Section VI, Question 24

Sponsors can manage changes to third-party foundation models through a combination of
contractual, technical, and procedural safeguards. Contractually, sponsors can seek
commitments from foundation model developers such as continued access to fixed model
versions, advance notice of changes or deprecations, and transition periods sufficient to
evaluate a replacement. Technically, sponsors can use fixed model versions so that adopting
a new version is a deliberate decision, and can evaluate a candidate version against the
device’s established performance baseline before deployment, using the same
benchmarking, back-testing, and silent deployment tools described in our response to
Question 9. After deployment, staged rollouts and ongoing performance monitoring can
detect unexpected changes, and sponsors can retain the ability to revert to a prior version if
performance degrades. A PCCP could prespecify these steps, including the benchmarks and
acceptance criteria a new model version must meet before release. FDA could recognize
these mechanisms as providing reasonable assurance without requiring a new submission
for each model version that meets prespecified criteria.

Response to Section VII, Question 26

Agentic functions should be evaluated case-by-case based on their intended use and risk, as
with any other function. Agentic actions can in consequence severity, and additional
requirements should not apply to all agentic systems by default. Evaluation should reflect
what actions a system can take, whether they can be reversed, and what safeguards are in
place, such as limits on available actions, confirmation steps before consequential actions,
and monitoring. Because agentic systems perform multi-step tasks where errors can
compound, evaluation should also assess performance across complete task sequences,
rather than individual steps alone. For auditability, logs of the inputs, tool calls, and actions a
system takes provide a reliable record of its behavior.