Omnalis (Liz Wells)

Omnalis appreciates the opportunity to comment on CDRH’s discussion paper on the regulation of generative AI-enabled medical devices. Omnalis is an applied AI research lab for regulatory reasoning, starting with the FDA. We build regulatory validation infrastructure, the evidence layer that makes AI-generated regulatory work trustworthy, beginning with medical devices. We combine automated cross-document validation with human-in-the-loop review by credentialed former FDA reviewers to identify cross-document inconsistencies, missing studies, and citation gaps in regulatory packages before they reach the Agency, and we are building toward benchmarks and measures of calibration and reviewer agreement for regulatory reasoning. Our current build extends to generative-AI medical devices. We support CDRH’s objective of a nimble, risk-proportionate, least-burdensome approach that gives patients timely access to safe and effective devices while preserving FDA’s sole regulatory authority.
Our interest is twofold: as a developer of validation infrastructure, and as an early-stage company that would be affected by the evidentiary burdens placed on both device sponsors and independent evaluators. We note a disclosure of interest: Omnalis could participate in the independent third-party ecosystem the paper describes in Section V.D.2. We have therefore tried to confine our comments to design principles that would apply to any qualified party.

Our comments focus on the questions where our operating experience is most directly relevant: benchmark validity (Question 10), performance comparators and acceptance criteria (Question 14), independent third parties (Question 16, including whether analogous ASCA and MDDT frameworks can be leveraged and how the independence and qualification considerations apply to LLM adjudicators), machine-based supervisory agents (Question 20), and third-party foundation model changes (Question 24). We organize them around one central recommendation that we believe is material to whether a competency-based approach succeeds:

The competency-based approach will only be as credible as the standardization of its evidence record. If each sponsor, and each independent party, applies its own benchmark design, its own definition of a failure, and its own reporting format, FDA will be left reconciling heterogeneous, non-comparable evaluations of variable quality. We therefore recommend that CDRH define, from the outset, a structured standard for what a benchmarking and adjudication record contains and how each result traces to a hazard, a test item, and a named, qualified adjudicator, independent of who performed the evaluation.

That recommendation recurs below. The same record that makes a premarket benchmark reviewable also makes it usable as the baseline for postmarket re-benchmarking and for evaluating changes to third-party foundation models.

Summary of our principal recommendations (a fuller list appears in Section E):

Standardize the evidence record. Define a structured, risk-scaled Benchmarking and Adjudication Record so that evaluations are comparable, auditable, and machine-checkable.

Judge benchmarks by validity, and set acceptance criteria relative to clinician-panel variability (Questions 10 and 14).

Use independent third parties for adjudication, sequestered datasets, and benchmark maintenance, building on MDDT-style qualification and ASCA-style accreditation in phases (Question 16).

Apply the independence and qualification principles to LLM adjudicators, including a different model family from the device and requalification on change (Section V.B.2).

Condition reliance on postmarket monitoring on validated supervisory agents, and prespecify the response to third-party foundation-model changes (Questions 20 and 24).

October 19, 2026

Dockets Management Staff (HFA-305)

Food and Drug Administration

5630 Fishers Lane, Room 1061

Rockville, MD 20852

Re: Docket No. FDA-2026-N-7874: Considerations for the Regulation of Generative
AI-Enabled Medical Devices; Discussion Paper and Request for Feedback

Dear Dr. Abramson and the CDRH Review Team:

Omnalis appreciates the opportunity to comment on CDRH's discussion paper on the regulation
of generative AI-enabled medical devices. Omnalis is an applied AI research lab for regulatory
reasoning, starting with the FDA. We build regulatory validation infrastructure, the evidence
layer that makes AI-generated regulatory work trustworthy, beginning with medical devices. We
combine automated cross-document validation with human-in-the-loop review by credentialed
former FDA reviewers to identify cross-document inconsistencies, missing studies, and citation
gaps in regulatory packages before they reach the Agency, and we are building toward
benchmarks and measures of calibration and reviewer agreement for regulatory reasoning. Our
current build extends to generative-AI medical devices. We support CDRH's objective of a
nimble, risk-proportionate, least-burdensome approach that gives patients timely access to safe
and effective devices while preserving FDA's sole regulatory authority.

Our interest is twofold: as a developer of validation infrastructure, and as an early-stage
company that would be affected by the evidentiary burdens placed on both device sponsors and
independent evaluators. We note a disclosure of interest: Omnalis could participate in the
independent third-party ecosystem the paper describes in Section V.D.2. We have therefore
tried to confine our comments to design principles that would apply to any qualified party.

Our comments focus on the questions where our operating experience is most directly relevant:
benchmark validity (Question 10), performance comparators and acceptance criteria (Question
14), independent third parties (Question 16, including whether analogous ASCA and MDDT
frameworks can be leveraged and how the independence and qualification considerations apply
to LLM adjudicators), machine-based supervisory agents (Question 20), and third-party
foundation model changes (Question 24). We organize them around one central
recommendation that we believe is material to whether a competency-based approach
succeeds:

The competency-based approach will only be as credible as the standardization of its
evidence record. If each sponsor, and each independent party, applies its own
benchmark design, its own definition of a failure, and its own reporting format, FDA will
be left reconciling heterogeneous, non-comparable evaluations of variable quality. We
therefore recommend that CDRH define, from the outset, a structured standard for what a
benchmarking and adjudication record contains and how each result traces to a hazard, a test
item, and a named, qualified adjudicator, independent of who performed the evaluation.

That recommendation recurs below. The same record that makes a premarket benchmark
reviewable also makes it usable as the baseline for postmarket re-benchmarking and for
evaluating changes to third-party foundation models.

Summary of our principal recommendations (a fuller list appears in Section E):

●​ Standardize the evidence record. Define a structured, risk-scaled Benchmarking and
Adjudication Record so that evaluations are comparable, auditable, and
machine-checkable.​

●​ Judge benchmarks by validity, and set acceptance criteria relative to
clinician-panel variability (Questions 10 and 14).​

●​ Use independent third parties for adjudication, sequestered datasets, and
benchmark maintenance, building on MDDT-style qualification and ASCA-style
accreditation in phases (Question 16).​

●​ Apply the independence and qualification principles to LLM adjudicators, including
a different model family from the device and requalification on change (Section V.B.2).​

●​ Condition reliance on postmarket monitoring on validated supervisory agents, and
prespecify the response to third-party foundation-model changes (Questions 20 and 24).​

A. Premarket Evaluation of GenAI-Enabled
Devices (Section V)
1. Benchmark Validity and Sponsor-Developed
Benchmarks
Question 10. CDRH recognizes that publicly available benchmarking assets may be subject to
data contamination, saturation, and limited real-world representativeness. How should a
sponsor establish that performance on a given benchmark predicts safe and effective real-world
behavior for the device's intended use? What evidence should support the construct validity of a
benchmark used to gate device evaluation, and what role should sponsor-developed
benchmarks play given potential concerns around independence and optimization to the test?
A benchmark should be accepted as evidence only to the extent a sponsor can show that it
predicts the failures that matter for the device's intended use. We recommend that construct
validity be supported in three ways:

●​ Hazard traceability: each benchmark item or item family maps to a specific hazard or
failure mode in the device's risk analysis and to the relevant element in Appendix A (for
example, S.1 escalation or E.3 dosing), so coverage gaps are visible rather than implicit.​

●​ Predictive validity: the sponsor shows that benchmark performance correlates with
independently adjudicated performance on real or clinically representative inputs, using
approaches already described in Section V.C such as retrospective evaluation or
shadow deployment. Where the correlation is weak, the benchmark should not gate the
device.​

●​ Contamination and saturation controls: held-out items that neither the sponsor nor
the model developer has seen, refreshed on a defined cadence, with the gap between
public-benchmark and held-out performance reported.​

Sponsor-developed benchmarks should play a meaningful role, because public assets are
rarely tailored to a specific intended use, and we would not exclude them. Concerns about
independence and optimization to the test are better addressed by process than by
prohibition: prespecified and versioned test sets, a held-out portion scored by independent
adjudicators, disclosure of the full development history, and reporting of reviewer agreement on
the scoring rubric.

2. Comparators and Acceptance Criteria
Question 14. For open-ended device outputs, how should performance comparators and
acceptance criteria be selected? When a panel of qualified clinicians serves as the comparator,
how should the applicable standard (for example, the standard of care versus the performance
of a median clinician in practice) be defined and justified? Should generalist or specialist
physicians be used as a performance standard? When should human-AI team performance,
rather than the device operating alone, serve as the basis for evaluation?

When a clinician panel is the comparator, the panel's own variability should be measured and
used to set acceptance criteria. We recommend:

●​ Define the standard by prespecified panel qualifications and consensus, not an ad
hoc median. Training, experience, and domain should be documented, and the panel
should be independent of the sponsor and the model developer.​

●​ Measure inter-clinician agreement and use it as a noise floor. If qualified clinicians
disagree on a case type, a device cannot meaningfully be held to a single correct
 answer. Acceptance can be framed as performance within the range of qualified-clinician
variability, with stricter requirements for safety-critical behaviors, where the paper
already treats variation as a failure (R.1).​

●​ Match the comparator to the task. Use specialists for tasks within a specialty. For
generalist-facing devices, report performance against both generalist and specialist
panels, because the comparisons mean different things: generalist parity shows no
added risk, while specialist parity shows meaningful access to expertise.​

●​ Choose human-AI team versus standalone evaluation based on the intended
workflow. If the device is intended to be used with clinician review, evaluate the
combined team, but also report standalone performance so reviewers can see how
much weight the human check is carrying and the exposure to automation bias noted in
E.4.​

●​ Validate any LLM adjudicator. Consistent with the paper's note that independence
considerations apply when the adjudicator is an LLM, an LLM judge should be validated
against human expert adjudication on the same items, with agreement metrics reported,
and should not be drawn from the same model family as the device under test.​

3. Independent Third Parties
Question 16. Should independent third parties be involved in some or all aspects of a
competency-based approach, including device benchmarking and clinical confirmation? If so, in
what ways might qualified, independent third-party participation contribute to this approach, and
what qualifications and independence criteria should apply? Are there aspects of the
assessment for which third-party involvement would be impractical or inadvisable? What
safeguards or program design features would be critical to prevent third-party participation from
limiting competition or preventing innovation?

Independent third parties can add real value, particularly for adjudication and sequestered
datasets, provided independence is defined by structure and incentives rather than by label.
The roles where they add the most are:

●​ Independent expert adjudication. Rubric-based scoring of open-ended outputs
requires qualified clinicians and regulatory experts who are not paid on the basis of the
result. This is the component a sponsor can least credibly self-generate.​

●​ Sequestered evaluation sets. Sets held outside the reach of the sponsor and the
model developer reduce the risk of optimization to the test.​
 ●​ Benchmark maintenance. Independent parties can refresh and version test items to
counter contamination and saturation.​

We recommend the following independence criteria:

●​ No equity, contingent fee, or other financial interest in the sponsor's marketing outcome;
fees should not depend on the result of the assessment.​

●​ Structural independence from both the device sponsor and, where applicable, the
foundation model developer, consistent with the paper's own principle in Section V.B.2.​

●​ Disclosure of prior and concurrent engagements with the sponsor and the model
developer, with recusal procedures.​

●​ Documented, verified adjudicator qualifications matched to the clinical domain.​

●​ Reported inter-adjudicator agreement, so FDA can see how reliable the adjudication
itself was.​

●​ A complete audit trail of what was tested, what was scored, and what changed between
rounds.​

To prevent third-party participation from limiting competition or innovation, FDA should avoid a
single gatekeeper and allow multiple qualified parties, as well as sponsor-developed tests
alongside third-party assets. A recognition model analogous to ASCA, in which the method and
the party's qualifications are accredited rather than individual outcomes, would preserve
competition. Sponsors should disclose prior assessments of the same device configuration so
results cannot be selectively presented. The assessment should remain an input to FDA's
review, not a substitute for it.

Third-party involvement would be impractical or inadvisable for determinations of intended use
and benefit-risk, and for real-time postmarket intervention, which remain the manufacturer's and
FDA's responsibility.

To make third-party work comparable, we recommend FDA define a structured Benchmarking
and Adjudication Record, a schema rather than a free-form narrative, in which each result
carries a consistent set of fields:

●​ Hazard anchor: the risk-analysis hazard and Appendix A element the test item
addresses​

●​ Test item and version: the exact item, rubric, and benchmark version used​
 ●​ Device configuration: the exact device configuration and foundation model version
evaluated​

●​ Result and severity: the outcome, with severity assigned from a standardized taxonomy​

●​ Adjudication: adjudicator qualifications, independence disclosures, and agreement with
other adjudicators​

●​ Sponsor disposition: any change to the device or claim made in response, with rationale​

●​ Reviewer identity, version, and timestamp​

A structured record makes evaluations comparable across parties, auditable, and
machine-checkable for completeness, so gaps in the evaluation itself are visible to FDA.

4. Whether a New Third-Party Framework Is Needed or
Analogous Frameworks Can Be Used
Question 16 (related, Section V.D.2). Does a new third-party framework need to be developed to
support the assessment of GenAI-enabled devices, or can analogous third-party frameworks,
such as the Accreditation Scheme for Conformity Assessment (ASCA) Program and the Medical
Device Development Tool (MDDT) program, be utilized?

We recommend that CDRH build on existing frameworks first and develop new program
elements only where a demonstrated gap remains. Each existing program fits part of the
problem, and neither fits all of it. Our reading of the programs is as follows, and we would defer
to CDRH where we have mischaracterized them.

ASCA, as a model for accrediting evaluators. ASCA is built around conformity to
FDA-recognized consensus standards and test methods. FDA-recognized accreditation bodies
accredit testing laboratories under ISO/IEC 17025 together with ASCA program specifications
for each eligible standard, and FDA generally accepts results from ASCA-accredited
laboratories, supported by a declaration of conformity. Two features transfer well: accrediting the
organization's competence and impartiality, and FDA's reliance on the resulting determination
without repeating the testing. What does not transfer directly is the premise of a recognized
standard and test method with defined pass/fail logic. For GenAI-enabled devices, recognized
consensus standards for benchmarking and adjudication do not yet exist, and much of the work
involves expert judgment on open-ended outputs rather than conformity to a fixed method.
ASCA's architecture is therefore a good template for accrediting independent evaluators and
adjudication panels, but its scope would need to extend from test methods to adjudication
processes, and standards development would need to proceed in parallel.
MDDT, as a model for qualifying evaluation tools. MDDT qualifies a tool for a specific
context of use, after which sponsors may use it in submissions without re-establishing its
validity each time. This is a strong fit for the reusable assets in Section V.B: benchmark sets and
scoring rubrics, simulation tools including virtual patient avatars, and validated LLM
adjudicators, each qualified for a defined intended-use class and risk tier. Qualification once,
reuse many times, would reduce per-submission burden and give small sponsors access to
credible evaluation assets. MDDT currently recognizes non-clinical assessment models (which
include computational models), biomarker tests, and clinical outcome assessments. Simulation
tools and virtual patient avatars may fit within non-clinical assessment models, but benchmark
sets, scoring rubrics, and adjudication tools are less clearly covered, and CDRH should
consider whether clarified guidance or an additional category is warranted. Once a tool is
qualified for a context of use, FDA's stated expectation is that reviewers accept it for that use
without reconfirming its suitability, which is the property that makes qualification valuable here.

Gaps that analogous frameworks do not cover. These are the elements we believe would
require new design:

●​ Living assets. Benchmarks saturate and become contaminated, and models change. A
qualified benchmark needs a versioning and requalification cadence, which is different
from a one-time qualification.​

●​ Adjudication of open-ended outputs. Accreditation should attach to the adjudicator
qualification, independence, and agreement process, not only to a laboratory's test
methods.​

●​ Independence from the foundation model developer. Existing programs address
independence from the sponsor but not from an upstream model provider.​

●​ Custody of sequestered data. Rules are needed for who holds held-out sets, how
access is logged, and how sets are retired after exposure.​

●​ Machine adjudicators. Qualification and requalification criteria for LLM adjudicators,
discussed below.​

We therefore recommend a phased approach: use MDDT-style qualification for reusable
benchmark and adjudication tools now, apply an ASCA-style accreditation model to independent
evaluators in a pilot, and use what the pilot reveals to decide whether any additional program is
warranted. This is faster and lower-burden than a new bespoke framework and keeps the focus
on the gaps that actually matter. The structured Benchmarking and Adjudication Record
described above would serve as the common reporting format across both mechanisms.
5. LLM Adjudicators and the Independence and
Qualification Considerations
Section V.B.2. Where expert adjudication is warranted, it would include adjudicators that are
structurally independent from the device sponsor and, where the device incorporates a
third-party model, from the developer of that model, and possess qualifications aligned with the
clinical domain and complexity of the outputs under review. These considerations would still be
applicable when the expert adjudicator is itself an LLM. What are the implications of applying
these considerations to LLM adjudicators?

We agree that independence and qualification should apply to LLM adjudicators, and we
recommend that CDRH state what that means in practice, because the human-oriented criteria
do not translate automatically.

Independence from the sponsor. The sponsor should not control the judge configuration. The
judge model, version, prompt, rubric weighting, and decoding settings should be prespecified
and locked before testing, and the sponsor should not select among multiple judge runs after
seeing results. Judge outputs should not be used to tune the device, or if they are, a separate
held-out judge and test set should be used for the final evaluation.

Independence from the foundation model developer. An LLM judge drawn from the same
developer or model family as the device's foundation model is not independent in any
meaningful sense. Shared training data, shared architecture, and known self-preference effects
can produce correlated errors, so the judge may fail to detect the very failures the device
exhibits. We recommend that judges come from a different model family than the device, with
the lineage disclosed. Because the market may offer few independent frontier options, CDRH
should accept a practical alternative where full independence is not achievable: multiple judges
from different families, anchored by human expert adjudication on a sampled subset, with
disagreement among judges treated as a signal for human review.

Qualifications for an LLM. For a human, qualifications are credentials and experience. For an
LLM, we recommend qualification be defined as demonstrated agreement with a qualified,
independent human expert panel on a calibration set matched to the clinical domain and the
complexity of the outputs under review. Agreement should be reported by domain, difficulty, and
severity, with particular attention to safety-critical items where errors matter most. Qualification
should be scoped to a context of use (as in the MDDT model) and should not be assumed to
carry over to other specialties or output types.

Requalification on change. Providers update models, sometimes without notice. A judge
qualified under one model version is not automatically qualified under the next, so judge
versions should be pinned and requalification triggered by any change, using the same
change-control logic as the device.
Adversarial robustness. Because the judge reads the device's outputs, it can be influenced by
them. Device outputs may contain text that functions as instructions to the judge, whether by
design or by accident. Judge qualification should include testing for prompt injection through the
content under review, and for sensitivity to the length, fluency, and format of outputs, which can
inflate scores independent of clinical quality.

Accountability and human roles. An LLM can provide scalable scoring against a rubric, but it
should not serve as the sole adjudicator for safety-critical findings, and it should not establish
ground truth for contested or evolving clinical questions. Human experts should develop and
validate the rubric, resolve judge disagreements, and review a stratified sample of judge
decisions, including decisions where the judge found no problem. A named, accountable human
reviewer should remain responsible for each adjudication outcome.

Reproducibility and disclosure. The record should capture the judge model and version,
prompt, settings, number of repeated runs, and run-to-run stability. Disclosure that an LLM
contributed to adjudication should be a required field in the Benchmarking and Adjudication
Record.

B. Postmarket Monitoring and Change
Control (Section VI)
1. Machine-Based Supervisory Agents
Question 20. Please comment on whether the proposed approaches to postmarket monitoring
can be facilitated by machine-based supervisory agents. What considerations, including the
evaluation and reliability of the supervisory agent itself, should CDRH take into account for such
an approach?

Supervisory agents can make postmarket monitoring practical at scale, particularly for the
repeatable, checkable layer of the three approaches in Section VI.A. They should be treated as
components whose own performance is evidenced, and they should not be the sole line of
review. Importantly, the premise of Question 18, that stronger postmarket monitoring can
justify greater premarket uncertainty, holds only to the extent the monitor can reliably
detect failures. A supervisory agent's validated detection performance should therefore be a
condition of any reduced premarket evidence, not an assumption.

Where supervisory agents fit well and where they do not

●​ Periodic re-benchmarking: a strong fit. Agents can run the standing benchmark and
regression suite on schedule and on trigger events, compare results to the premarket
baseline, and report against prespecified thresholds.​
 ●​ Performance degradation monitoring: a strong fit. Statistical drift detection across
input populations, outputs, and safety-critical behaviors such as escalation and refusal is
repeatable and checkable.​

●​ Triage for sample-based clinician review: a good fit with safeguards. Agents can
prioritize interactions for human review, but sampling should not depend solely on what
the agent flags.​

●​ Determining standard of care, adjudicating novel or contested cases, and judging
the materiality of a model change: a poor fit. These require expert judgment and
should remain with qualified humans and the manufacturer.​

Considerations for the evaluation and reliability of the supervisory agent itself

●​ Evaluate the supervisor against a human-adjudicated reference. Report sensitivity
and specificity for safety-critical failures, with confidence intervals, stratified by clinical
subgroup and by failure type, rather than overall accuracy alone. Acceptance thresholds
for the supervisor should be prespecified.​

●​ Correlated failure. A supervisor built on the same model family as the monitored device
may share its blind spots. Independence in model family, prompts, and data is
preferable, and the same independence principles discussed above for LLM adjudicators
should apply.​

●​ Sample the supervisor's negatives. Humans should audit a stratified sample of cases
the supervisor did not flag, since those errors are otherwise invisible. Without this, the
monitoring program cannot estimate its own miss rate.​

●​ Alert burden. A supervisor that over-flags can overwhelm reviewers and erode attention
to real problems, while one that under-flags creates false assurance. Both error rates,
and the time to escalation, should be reported.​

●​ Supervisor change control. The supervisor can drift or be updated, particularly if it
relies on a third-party model. It should be version-pinned and requalified on change,
using the same logic as the device.​

●​ Security and privacy. The supervisor reads real-world patient interactions, so it requires
appropriate data protection. It is also exposed to prompt injection through the content it
monitors, which should be part of its qualification testing.​

●​ Bounded autonomy and accountability. The supervisor should flag and escalate and
should not itself modify or disable the device. A named, accountable human should be
responsible for decisions informed by supervisor output, which avoids an unbounded
 chain of machines monitoring machines.​

●​ Auditability. Every flag, dismissal, and override should be reconstructable from a
structured record that identifies the supervisor version, the evidence reviewed, and the
human disposition.​

Describing the supervisor, its qualification evidence, and its performance thresholds in the
manufacturer's postmarket monitoring plan, and treating it as part of the device's quality system,
would give FDA a reviewable basis for relying on it.

2. Third-Party Foundation Model Changes
Question 24. For devices built on third-party foundation models, changes to the underlying
model may be initiated by the third-party model developer rather than the device manufacturer.
How can a manufacturer detect, evaluate, and respond to such changes in a timely manner?
What mechanisms (for example, contractual, technical, or through a PCCP) could provide
reasonable assurance that third-party developer-initiated changes do not compromise the safety
or effectiveness of the device?

Manufacturers should be expected to detect and evaluate foundation-model changes through a
defined, prespecified process, because contractual commitments alone will not catch silent
updates. We recommend:

●​ Version pinning and change notification. Where the provider permits, pin model
versions and obtain contractual notification commitments. Where a provider does not
offer them, that should be documented as a risk factor.​

●​ Behavioral regression testing. Maintain a standing regression suite drawn from the
premarket benchmark, especially elements S.1 through S.3 and R.1, run on a schedule
and on any detected change, so that silent updates are caught by behavior and not only
by notice.​

●​ Materiality thresholds tied to safety-critical elements. The extent of re-benchmarking
should scale with the change. Any shift in escalation, refusal, or dosing behavior should
trigger full re-benchmarking and review, while changes affecting only non-safety-critical
phrasing could be managed within the quality system.​

●​ The premarket benchmark as the baseline. We agree with the paper's concept in
Section VI.C that the initial benchmark can anchor later assessment, which is a further
reason to design it carefully up front, with versioned test sets and prespecified
thresholds.​
 ●​ PCCP design. For GenAI devices, a PCCP may be more workable if it prespecifies the
re-evaluation protocol and acceptance thresholds that any change, including a
provider-initiated one, must meet, rather than attempting to enumerate the changes
themselves. This also addresses the concern raised in Question 23, where the scope of
future modifications cannot be fully described.​

●​ Foundation Model MAFs (Question 25). These would be useful but are not sufficient.
A model card describes a model at a point in time, and sponsors would still need their
own evidence on the final deployed device.​

C. Trade-offs and Additional
Considerations
1. Trade-offs and Opposing Views
We recognize that the approaches we recommend carry costs and that reasonable stakeholders
may disagree. We address the principal ones directly.

●​ Standardization versus flexibility and least-burdensome principles. A common
record schema adds up-front cost for sponsors and could ossify as methods evolve. We
would mitigate this by scaling the depth of the record to risk using CDRH's own two-axis
framework, keeping the schema modular and versioned, and starting with higher-risk
functions rather than imposing a uniform burden. A sponsor with a lower-risk,
non-directive function should not be required to produce the full record.​

●​ Third-party involvement versus cost, time, and capacity. Independent evaluation
adds expense and could become a bottleneck or a barrier for small sponsors. We would
make third-party roles optional for lower-risk devices, and rely on MDDT-style
qualification so that the cost of validating a benchmark or adjudication tool is shared
across many submissions rather than repeated in each one. A reasonable alternative for
some sponsors is sponsor-conducted testing combined with an independently scored
held-out set.​

●​ Independence versus availability of expertise. Strict independence criteria could
exclude the few qualified specialists in narrow clinical areas, many of whom have prior
ties to industry. We would favor disclosure, documented recusal, and reporting of
adjudicator agreement over blanket exclusion, so that independence is demonstrated
rather than presumed.​
 ●​ LLM adjudicators: scale versus circularity. LLM judges make high-volume scoring
feasible, but they can share blind spots with the devices they assess and can change
without notice. For this reason we recommend that they supplement, and not replace,
human adjudication for safety-critical findings.​

●​ Greater reliance on postmarket monitoring. Some stakeholders will argue that
premarket evidence should remain the primary gate, particularly for high-consequence or
autonomous functions, because harm may reach patients before a monitor detects it. We
agree this concern is strongest at the severe end of the consequences axis. We
therefore suggest that any reliance on postmarket monitoring be conditioned on the
validated detection performance of the monitoring system, as discussed in our response
to Question 20.​

Economic considerations. We have not quantified costs, and we think rigorous quantification
is difficult before standards exist. Qualitatively, we expect up-front costs from structured records,
held-out datasets, and re-benchmarking, offset over time by reusable qualified assets, less
duplicative testing across sponsors, and fewer review cycles caused by non-comparable
evidence. We would welcome the opportunity to contribute to a cost analysis as the framework
develops.

Who bears the costs and who benefits. The fixed costs of benchmarking, held-out datasets,
and independent adjudication fall disproportionately on small and early-stage sponsors, while
larger sponsors can spread them across many products. The benefits of more reliable
evaluation accrue to patients and clinicians, and especially to subgroups for whom performance
failures (element R.2) are otherwise least visible. Those equity benefits are difficult to monetize
but should be weighed. Shared, qualified evaluation assets, as discussed in our response on
MDDT-style qualification, are the main lever for narrowing this gap, and we encourage CDRH to
consider small-sponsor access explicitly in any framework design.

2. Additional Considerations
Three considerations follow from adopting, or failing to adopt, a common standard early:

●​ Inter-evaluator consistency. CDRH should establish, and test, how it will determine
whether different sponsors and different independent parties evaluate the same device
consistently. A shared, de-identified benchmark corpus with a sequestered held-out
portion would be the natural instrument, and it would allow cross-comparison without
exposing sponsor intellectual property.​

●​ Confidentiality. Sponsor data used in benchmarking and adjudication should remain
confidential, segregated, access-controlled, and auditable, and should not be used for
model training or tooling development absent the sponsor's explicit authorization. This
 should be a baseline requirement for any qualified third party.​

●​ Software-assisted evaluation. Where software or AI materially contributes to a
benchmarking or adjudication finding, FDA should specify what is disclosed: the
contribution, the traceability of each finding to source evidence, and a named
accountable human reviewer.​

Taken together, these point to an opportunity larger than a premarket framework. With a
common standard for findings, evidence, and provenance, the competency-based approach can
become the baseline for a continuously learning regulatory system, in which premarket
benchmarks, postmarket monitoring, and FDA determinations remain traceable to one another
across a device's total product life cycle.

D. Requests for Clarification
Several details would help stakeholders respond more precisely as CDRH develops this
approach:

1.​ How the two-axis risk framework would translate into specific evidence expectations, and
whether CDRH could publish worked examples across the activity and consequences
grid.​

2.​ How CDRH would define "structurally independent" and "qualified" adjudicators, for both
human and LLM adjudicators, including any thresholds.​

3.​ Whether results from independently qualified benchmarks or tools, or from accredited
evaluators, would be accepted in review without re-evaluation, and under what
conditions.​

4.​ Which marketing pathways (for example 510(k), De Novo, and PMA) the
competency-based approach is intended to inform, and how it would interact with
existing predicate and special-controls frameworks.​

5.​ Who would hold and govern held-out benchmark sets, and how versions would be
retired after exposure.​

6.​ How re-benchmarking triggers would relate to the content of a PCCP.​

E. Summary of Recommendations
In summary, we recommend that CDRH:

1.​ Standardize the evidence record. Define a structured, risk-scaled Benchmarking and
Adjudication Record so that evaluations are comparable, auditable, and
machine-checkable for completeness.​

2.​ Require benchmark validity, not just benchmark scores. Ask sponsors to show
hazard traceability, predictive validity against adjudicated real-world performance, and
contamination controls, and manage sponsor-developed benchmarks through process
rather than prohibition (Question 10).​

3.​ Set acceptance criteria relative to clinician-panel variability. Use prespecified panel
qualifications, treat inter-clinician agreement as a noise floor, match comparators to the
task, and report both human-AI team and standalone performance (Question 14).​

4.​ Use independent third parties where they add the most value. Focus on
adjudication, sequestered datasets, and benchmark maintenance, with defined
independence criteria and safeguards against gatekeeping (Question 16).​

5.​ Build on existing frameworks in phases. Use MDDT-style qualification for reusable
evaluation tools and an ASCA-style accreditation model for independent evaluators, then
assess remaining gaps before creating any new program (Question 16).​

6.​ Apply independence and qualification principles to LLM adjudicators. Require a
different model family from the device, locked and disclosed configurations,
agreement-based qualification, requalification on change, and human accountability for
safety-critical findings (Section V.B.2).​

7.​ Permit supervisory agents with evidence. Condition any reliance on postmarket
monitoring on the validated detection performance, independence, and bounded
autonomy of the supervisory agent (Question 20).​

8.​ Prespecify the response to foundation-model changes. Use version pinning,
standing regression testing, materiality thresholds tied to safety-critical elements, and a
PCCP that specifies the re-evaluation protocol (Question 24).​

9.​ Create a shared, de-identified benchmark corpus with a sequestered portion, to test
consistency across evaluators without exposing sponsor intellectual property, and
require baseline confidentiality protections for any qualified third party.​

10.​Require disclosure of software and AI contribution to evaluation findings, with
traceability to source evidence and a named accountable human reviewer.​
Omnalis thanks CDRH for the opportunity to comment and for its openness in seeking early
input. We would welcome the opportunity to provide additional input, including on the design of
a standardized benchmarking and adjudication record or a shared evaluation benchmark, as
these efforts develop.

Respectfully submitted,

Liz Wells, MPH

Chief Executive Officer & Co-Founder

Omnalis

October 19, 2026

Dockets Management Staff (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Room 1061
Rockville, MD 20852

Re: Docket No. FDA-2026-N-7874: Considerations for the Regulation of Generative AIEnabled Medical Devices; Discussion Paper and Request for Feedback

Dear Dr. Abramson and the CDRH Review Team:
Omnalis appreciates the opportunity to comment on CDRH's discussion paper on the regulation
of generative AI-enabled medical devices. Omnalis is an applied AI research lab for regulatory
reasoning, starting with FDA. We build regulatory validation infrastructure, the evidence layer
that makes AI-generated regulatory work trustworthy, beginning with medical devices. We
combine automated cross-document validation with human-in-the-loop review by credentialed
former FDA reviewers to identify cross-document inconsistencies, missing studies, and citation
gaps in regulatory packages before they reach the Agency, and we are building toward
benchmarks and measures of calibration and reviewer agreement for regulatory reasoning. Our
current build extends to generative-AI medical devices. We support CDRH's objective of a
nimble, risk-proportionate, least-burdensome approach that gives patients timely access to safe
and effective devices while preserving FDA's sole regulatory authority.
Our interest is twofold: as a developer of validation infrastructure, and as an early-stage
company that would be affected by the evidentiary burdens placed on both device sponsors and
independent evaluators. We note a disclosure of interest: Omnalis could participate in the
independent third-party ecosystem the paper describes in Section V.D.2. We have therefore tried
to confine our comments to design principles that would apply to any qualified party.
Our comments focus on the questions where our operating experience is most directly relevant:
benchmark validity (Question 10), performance comparators and acceptance criteria (Question
14), independent third parties (Question 16, including whether analogous ASCA and MDDT
frameworks can be leveraged and how the independence and qualification considerations apply
to LLM adjudicators), machine-based supervisory agents (Question 20), and third-party
foundation model changes (Question 24). We organize them around one central recommendation
that we believe is material to whether a competency-based approach succeeds:
The competency-based approach will only be as credible as the standardization of its
evidence record. If each sponsor, and each independent party, applies its own benchmark
design, its own definition of a failure, and its own reporting format, FDA will be left
reconciling heterogeneous, non-comparable evaluations of variable quality. We therefore
recommend that CDRH define, from the outset, a structured standard for what a benchmarking
and adjudication record contains and how each result traces to a hazard, a test item, and a named,
qualified adjudicator, independent of who performed the evaluation.

Omnalis | Docket No. FDA-2026-N-7874 | Page 1 of 13
That recommendation recurs below. The same record that makes a premarket benchmark
reviewable also makes it usable as the baseline for postmarket re-benchmarking and for
evaluating changes to third-party foundation models.
Summary of our principal recommendations (a fuller list appears in Section E):
• Standardize the evidence record. Define a structured, risk-scaled Benchmarking and
Adjudication Record so that evaluations are comparable, auditable, and machinecheckable.
• Judge benchmarks by validity, and set acceptance criteria relative to clinician-panel
variability (Questions 10 and 14).
• Use independent third parties for adjudication, sequestered datasets, and
benchmark maintenance, building on MDDT-style qualification and ASCA-style
accreditation in phases (Question 16).
• Apply the independence and qualification principles to LLM adjudicators, including
a different model family from the device and requalification on change (Section V.B.2).
• Condition reliance on postmarket monitoring on validated supervisory agents, and
prespecify the response to third-party foundation-model changes (Questions 20 and 24).

A. Premarket Evaluation of GenAI-Enabled Devices (Section V)

1. Benchmark Validity and Sponsor-Developed Benchmarks
Question 10. CDRH recognizes that publicly available benchmarking assets may be subject to
data contamination, saturation, and limited real-world representativeness. How should a
sponsor establish that performance on a given benchmark predicts safe and effective real-world
behavior for the device's intended use? What evidence should support the construct validity of a
benchmark used to gate device evaluation, and what role should sponsor-developed benchmarks
play given potential concerns around independence and optimization to the test?
A benchmark should be accepted as evidence only to the extent a sponsor can show that it
predicts the failures that matter for the device's intended use. We recommend that construct
validity be supported in three ways:
• Hazard traceability: each benchmark item or item family maps to a specific hazard or
failure mode in the device's risk analysis and to the relevant element in Appendix A (for
example, S.1 escalation or E.3 dosing), so coverage gaps are visible rather than implicit.
• Predictive validity: the sponsor shows that benchmark performance correlates with
independently adjudicated performance on real or clinically representative inputs, using
approaches already described in Section V.C such as retrospective evaluation or shadow
deployment. Where the correlation is weak, the benchmark should not gate the device.
• Contamination and saturation controls: held-out items that neither the sponsor nor the
model developer has seen, refreshed on a defined cadence, with the gap between publicbenchmark and held-out performance reported.

Omnalis | Docket No. FDA-2026-N-7874 | Page 2 of 13
Sponsor-developed benchmarks should play a meaningful role, because public assets are rarely
tailored to a specific intended use, and we would not exclude them. Concerns about
independence and optimization to the test are better addressed by process than by prohibition:
prespecified and versioned test sets, a held-out portion scored by independent adjudicators,
disclosure of the full development history, and reporting of reviewer agreement on the scoring
rubric.

2. Comparators and Acceptance Criteria
Question 14. For open-ended device outputs, how should performance comparators and
acceptance criteria be selected? When a panel of qualified clinicians serves as the comparator,
how should the applicable standard (for example, the standard of care versus the performance
of a median clinician in practice) be defined and justified? Should generalist or specialist
physicians be used as a performance standard? When should human-AI team performance,
rather than the device operating alone, serve as the basis for evaluation?
When a clinician panel is the comparator, the panel's own variability should be measured and
used to set acceptance criteria. We recommend:
• Define the standard by prespecified panel qualifications and consensus, not an ad
hoc median. Training, experience, and domain should be documented, and the panel
should be independent of the sponsor and the model developer.
• Measure inter-clinician agreement and use it as a noise floor. If qualified clinicians
disagree on a case type, a device cannot meaningfully be held to a single correct answer.
Acceptance can be framed as performance within the range of qualified-clinician
variability, with stricter requirements for safety-critical behaviors, where the paper
already treats variation as a failure (R.1).
• Match the comparator to the task. Use specialists for tasks within a specialty. For
generalist-facing devices, report performance against both generalist and specialist
panels, because the comparisons mean different things: generalist parity shows no added
risk, while specialist parity shows meaningful access to expertise.
• Choose human-AI team versus standalone evaluation based on the intended
workflow. If the device is intended to be used with clinician review, evaluate the
combined team, but also report standalone performance so reviewers can see how much
weight the human check is carrying and the exposure to automation bias noted in E.4.
• Validate any LLM adjudicator. Consistent with the paper's note that independence
considerations apply when the adjudicator is an LLM, an LLM judge should be validated
against human expert adjudication on the same items, with agreement metrics reported,
and should not be drawn from the same model family as the device under test.

3. Independent Third Parties
Question 16. Should independent third parties be involved in some or all aspects of a
competency-based approach, including device benchmarking and clinical confirmation? If so, in
what ways might qualified, independent third-party participation contribute to this approach,
and what qualifications and independence criteria should apply? Are there aspects of the
assessment for which third-party involvement would be impractical or inadvisable? What

Omnalis | Docket No. FDA-2026-N-7874 | Page 3 of 13
safeguards or program design features would be critical to prevent third-party participation
from limiting competition or preventing innovation?
Independent third parties can add real value, particularly for adjudication and sequestered
datasets, provided independence is defined by structure and incentives rather than by label. The
roles where they add the most are:
• Independent expert adjudication. Rubric-based scoring of open-ended outputs requires
qualified clinicians and regulatory experts who are not paid on the basis of the result.
This is the component a sponsor can least credibly self-generate.
• Sequestered evaluation sets. Sets held outside the reach of the sponsor and the model
developer reduce the risk of optimization to the test.
• Benchmark maintenance. Independent parties can refresh and version test items to
counter contamination and saturation.
We recommend the following independence criteria:
• No equity, contingent fee, or other financial interest in the sponsor's marketing outcome;
fees should not depend on the result of the assessment.
• Structural independence from both the device sponsor and, where applicable, the
foundation model developer, consistent with the paper's own principle in Section V.B.2.
• Disclosure of prior and concurrent engagements with the sponsor and the model
developer, with recusal procedures.
• Documented, verified adjudicator qualifications matched to the clinical domain.
• Reported inter-adjudicator agreement, so FDA can see how reliable the adjudication itself
was.
• A complete audit trail of what was tested, what was scored, and what changed between
rounds.
To prevent third-party participation from limiting competition or innovation, FDA should avoid
a single gatekeeper and allow multiple qualified parties, as well as sponsor-developed tests
alongside third-party assets. A recognition model analogous to ASCA, in which the method and
the party's qualifications are accredited rather than individual outcomes, would preserve
competition. Sponsors should disclose prior assessments of the same device configuration so
results cannot be selectively presented. The assessment should remain an input to FDA's
review, not a substitute for it.
Third-party involvement would be impractical or inadvisable for determinations of intended use
and benefit-risk, and for real-time postmarket intervention, which remain the manufacturer's and
FDA's responsibility.
To make third-party work comparable, we recommend FDA define a structured Benchmarking
and Adjudication Record, a schema rather than a free-form narrative, in which each result
carries a consistent set of fields:
• Hazard anchor: the risk-analysis hazard and Appendix A element the test item addresses
• Test item and version: the exact item, rubric, and benchmark version used

Omnalis | Docket No. FDA-2026-N-7874 | Page 4 of 13
 • Device configuration: the exact device configuration and foundation model version
evaluated
• Result and severity: the outcome, with severity assigned from a standardized taxonomy
• Adjudication: adjudicator qualifications, independence disclosures, and agreement with
other adjudicators
• Sponsor disposition: any change to the device or claim made in response, with rationale
• Reviewer identity, version, and timestamp
A structured record makes evaluations comparable across parties, auditable, and machinecheckable for completeness, so gaps in the evaluation itself are visible to FDA.

4. Whether a New Third-Party Framework Is Needed or Analogous Frameworks Can Be
Used
Question 16 (related, Section V.D.2). Does a new third-party framework need to be developed to
support the assessment of GenAI-enabled devices, or can analogous third-party frameworks,
such as the Accreditation Scheme for Conformity Assessment (ASCA) Program and the Medical
Device Development Tool (MDDT) program, be utilized?
We recommend that CDRH build on existing frameworks first and develop new program
elements only where a demonstrated gap remains. Each existing program fits part of the
problem, and neither fits all of it. Our reading of the programs is as follows, and we would defer
to CDRH where we have mischaracterized them.
ASCA, as a model for accrediting evaluators. ASCA is built around conformity to FDArecognized consensus standards and test methods. FDA-recognized accreditation bodies accredit
testing laboratories under ISO/IEC 17025 together with ASCA program specifications for each
eligible standard, and FDA generally accepts results from ASCA-accredited laboratories,
supported by a declaration of conformity. Two features transfer well: accrediting the
organization's competence and impartiality, and FDA's reliance on the resulting determination
without repeating the testing. What does not transfer directly is the premise of a recognized
standard and test method with defined pass/fail logic. For GenAI-enabled devices, recognized
consensus standards for benchmarking and adjudication do not yet exist, and much of the work
involves expert judgment on open-ended outputs rather than conformity to a fixed method.
ASCA's architecture is therefore a good template for accrediting independent evaluators and
adjudication panels, but its scope would need to extend from test methods to adjudication
processes, and standards development would need to proceed in parallel.
MDDT, as a model for qualifying evaluation tools. MDDT qualifies a tool for a specific
context of use, after which sponsors may use it in submissions without re-establishing its
validity each time. This is a strong fit for the reusable assets in Section V.B: benchmark sets and
scoring rubrics, simulation tools including virtual patient avatars, and validated LLM
adjudicators, each qualified for a defined intended-use class and risk tier. Qualification once,
reuse many times, would reduce per-submission burden and give small sponsors access to
credible evaluation assets. MDDT currently recognizes non-clinical assessment models (which
include computational models), biomarker tests, and clinical outcome assessments. Simulation
tools and virtual patient avatars may fit within non-clinical assessment models, but benchmark

Omnalis | Docket No. FDA-2026-N-7874 | Page 5 of 13
sets, scoring rubrics, and adjudication tools are less clearly covered, and CDRH should consider
whether clarified guidance or an additional category is warranted. Once a tool is qualified for a
context of use, FDA's stated expectation is that reviewers accept it for that use without
reconfirming its suitability, which is the property that makes qualification valuable here.
Gaps that analogous frameworks do not cover. These are the elements we believe would
require new design:
• Living assets. Benchmarks saturate and become contaminated, and models change. A
qualified benchmark needs a versioning and requalification cadence, which is different
from a one-time qualification.
• Adjudication of open-ended outputs. Accreditation should attach to the adjudicator
qualification, independence, and agreement process, not only to a laboratory's test
methods.
• Independence from the foundation model developer. Existing programs address
independence from the sponsor but not from an upstream model provider.
• Custody of sequestered data. Rules are needed for who holds held-out sets, how access
is logged, and how sets are retired after exposure.
• Machine adjudicators. Qualification and requalification criteria for LLM adjudicators,
discussed below.
We therefore recommend a phased approach: use MDDT-style qualification for reusable
benchmark and adjudication tools now, apply an ASCA-style accreditation model to independent
evaluators in a pilot, and use what the pilot reveals to decide whether any additional program is
warranted. This is faster and lower-burden than a new bespoke framework and keeps the focus
on the gaps that actually matter. The structured Benchmarking and Adjudication Record
described above would serve as the common reporting format across both mechanisms.

5. LLM Adjudicators and the Independence and Qualification Considerations
Section V.B.2. Where expert adjudication is warranted, it would include adjudicators that are
structurally independent from the device sponsor and, where the device incorporates a thirdparty model, from the developer of that model, and possess qualifications aligned with the
clinical domain and complexity of the outputs under review. These considerations would still be
applicable when the expert adjudicator is itself an LLM. What are the implications of applying
these considerations to LLM adjudicators?
We agree that independence and qualification should apply to LLM adjudicators, and we
recommend that CDRH state what that means in practice, because the human-oriented criteria do
not translate automatically.
Independence from the sponsor. The sponsor should not control the judge configuration. The
judge model, version, prompt, rubric weighting, and decoding settings should be prespecified
and locked before testing, and the sponsor should not select among multiple judge runs after
seeing results. Judge outputs should not be used to tune the device, or if they are, a separate heldout judge and test set should be used for the final evaluation.

Omnalis | Docket No. FDA-2026-N-7874 | Page 6 of 13
Independence from the foundation model developer. An LLM judge drawn from the same
developer or model family as the device's foundation model is not independent in any
meaningful sense. Shared training data, shared architecture, and known self-preference effects
can produce correlated errors, so the judge may fail to detect the very failures the device exhibits.
We recommend that judges come from a different model family than the device, with the lineage
disclosed. Because the market may offer few independent frontier options, CDRH should accept
a practical alternative where full independence is not achievable: multiple judges from different
families, anchored by human expert adjudication on a sampled subset, with disagreement among
judges treated as a signal for human review.
Qualifications for an LLM. For a human, qualifications are credentials and experience. For an
LLM, we recommend qualification be defined as demonstrated agreement with a qualified,
independent human expert panel on a calibration set matched to the clinical domain and the
complexity of the outputs under review. Agreement should be reported by domain, difficulty,
and severity, with particular attention to safety-critical items where errors matter most.
Qualification should be scoped to a context of use (as in the MDDT model) and should not be
assumed to carry over to other specialties or output types.
Requalification on change. Providers update models, sometimes without notice. A judge
qualified under one model version is not automatically qualified under the next, so judge
versions should be pinned and requalification triggered by any change, using the same changecontrol logic as the device.
Adversarial robustness. Because the judge reads the device's outputs, it can be influenced by
them. Device outputs may contain text that functions as instructions to the judge, whether by
design or by accident. Judge qualification should include testing for prompt injection through the
content under review, and for sensitivity to the length, fluency, and format of outputs, which can
inflate scores independent of clinical quality.
Accountability and human roles. An LLM can provide scalable scoring against a rubric, but it
should not serve as the sole adjudicator for safety-critical findings, and it should not establish
ground truth for contested or evolving clinical questions. Human experts should develop and
validate the rubric, resolve judge disagreements, and review a stratified sample of judge
decisions, including decisions where the judge found no problem. A named, accountable human
reviewer should remain responsible for each adjudication outcome.
Reproducibility and disclosure. The record should capture the judge model and version,
prompt, settings, number of repeated runs, and run-to-run stability. Disclosure that an LLM
contributed to adjudication should be a required field in the Benchmarking and Adjudication
Record.

B. Postmarket Monitoring and Change Control (Section VI)

1. Machine-Based Supervisory Agents
Question 20. Please comment on whether the proposed approaches to postmarket monitoring
can be facilitated by machine-based supervisory agents. What considerations, including the
evaluation and reliability of the supervisory agent itself, should CDRH take into account for
such an approach?

Omnalis | Docket No. FDA-2026-N-7874 | Page 7 of 13
Supervisory agents can make postmarket monitoring practical at scale, particularly for the
repeatable, checkable layer of the three approaches in Section VI.A. They should be treated as
components whose own performance is evidenced, and they should not be the sole line of
review. Importantly, the premise of Question 18, that stronger postmarket monitoring can
justify greater premarket uncertainty, holds only to the extent the monitor can reliably
detect failures. A supervisory agent's validated detection performance should therefore be a
condition of any reduced premarket evidence, not an assumption.
Where supervisory agents fit well and where they do not
• Periodic re-benchmarking: a strong fit. Agents can run the standing benchmark and
regression suite on schedule and on trigger events, compare results to the premarket
baseline, and report against prespecified thresholds.
• Performance degradation monitoring: a strong fit. Statistical drift detection across
input populations, outputs, and safety-critical behaviors such as escalation and refusal is
repeatable and checkable.
• Triage for sample-based clinician review: a good fit with safeguards. Agents can
prioritize interactions for human review, but sampling should not depend solely on what
the agent flags.
• Determining standard of care, adjudicating novel or contested cases, and judging
the materiality of a model change: a poor fit. These require expert judgment and should
remain with qualified humans and the manufacturer.
Considerations for the evaluation and reliability of the supervisory agent itself
• Evaluate the supervisor against a human-adjudicated reference. Report sensitivity
and specificity for safety-critical failures, with confidence intervals, stratified by clinical
subgroup and by failure type, rather than overall accuracy alone. Acceptance thresholds
for the supervisor should be prespecified.
• Correlated failure. A supervisor built on the same model family as the monitored device
may share its blind spots. Independence in model family, prompts, and data is preferable,
and the same independence principles discussed above for LLM adjudicators should
apply.
• Sample the supervisor's negatives. Humans should audit a stratified sample of cases the
supervisor did not flag, since those errors are otherwise invisible. Without this, the
monitoring program cannot estimate its own miss rate.
• Alert burden. A supervisor that over-flags can overwhelm reviewers and erode attention
to real problems, while one that under-flags creates false assurance. Both error rates, and
the time to escalation, should be reported.
• Supervisor change control. The supervisor can drift or be updated, particularly if it
relies on a third-party model. It should be version-pinned and requalified on change,
using the same logic as the device.
• Security and privacy. The supervisor reads real-world patient interactions, so it requires
appropriate data protection. It is also exposed to prompt injection through the content it
monitors, which should be part of its qualification testing.

Omnalis | Docket No. FDA-2026-N-7874 | Page 8 of 13
 • Bounded autonomy and accountability. The supervisor should flag and escalate and
should not itself modify or disable the device. A named, accountable human should be
responsible for decisions informed by supervisor output, which avoids an unbounded
chain of machines monitoring machines.
• Auditability. Every flag, dismissal, and override should be reconstructable from a
structured record that identifies the supervisor version, the evidence reviewed, and the
human disposition.
Describing the supervisor, its qualification evidence, and its performance thresholds in the
manufacturer's postmarket monitoring plan, and treating it as part of the device's quality system,
would give FDA a reviewable basis for relying on it.

2. Third-Party Foundation Model Changes
Question 24. For devices built on third-party foundation models, changes to the underlying
model may be initiated by the third-party model developer rather than the device manufacturer.
How can a manufacturer detect, evaluate, and respond to such changes in a timely manner?
What mechanisms (for example, contractual, technical, or through a PCCP) could provide
reasonable assurance that third-party developer-initiated changes do not compromise the safety
or effectiveness of the device?
Manufacturers should be expected to detect and evaluate foundation-model changes through a
defined, prespecified process, because contractual commitments alone will not catch silent
updates. We recommend:
• Version pinning and change notification. Where the provider permits, pin model
versions and obtain contractual notification commitments. Where a provider does not
offer them, that should be documented as a risk factor.
• Behavioral regression testing. Maintain a standing regression suite drawn from the
premarket benchmark, especially elements S.1 through S.3 and R.1, run on a schedule
and on any detected change, so that silent updates are caught by behavior and not only by
notice.
• Materiality thresholds tied to safety-critical elements. The extent of re-benchmarking
should scale with the change. Any shift in escalation, refusal, or dosing behavior should
trigger full re-benchmarking and review, while changes affecting only non-safety-critical
phrasing could be managed within the quality system.
• The premarket benchmark as the baseline. We agree with the paper's concept in
Section VI.C that the initial benchmark can anchor later assessment, which is a further
reason to design it carefully up front, with versioned test sets and prespecified thresholds.
• PCCP design. For GenAI devices, a PCCP may be more workable if it prespecifies the
re-evaluation protocol and acceptance thresholds that any change, including a
provider-initiated one, must meet, rather than attempting to enumerate the changes
themselves. This also addresses the concern raised in Question 23, where the scope of
future modifications cannot be fully described.

Omnalis | Docket No. FDA-2026-N-7874 | Page 9 of 13
 • Foundation Model MAFs (Question 25). These would be useful but are not sufficient.
A model card describes a model at a point in time, and sponsors would still need their
own evidence on the final deployed device.

C. Trade-offs and Additional Considerations

1. Trade-offs and Opposing Views
We recognize that the approaches we recommend carry costs and that reasonable stakeholders
may disagree. We address the principal ones directly.
• Standardization versus flexibility and least-burdensome principles. A common
record schema adds up-front cost for sponsors and could ossify as methods evolve. We
would mitigate this by scaling the depth of the record to risk using CDRH's own two-axis
framework, keeping the schema modular and versioned, and starting with higher-risk
functions rather than imposing a uniform burden. A sponsor with a lower-risk, nondirective function should not be required to produce the full record.
• Third-party involvement versus cost, time, and capacity. Independent evaluation adds
expense and could become a bottleneck or a barrier for small sponsors. We would make
third-party roles optional for lower-risk devices, and rely on MDDT-style qualification so
that the cost of validating a benchmark or adjudication tool is shared across many
submissions rather than repeated in each one. A reasonable alternative for some sponsors
is sponsor-conducted testing combined with an independently scored held-out set.
• Independence versus availability of expertise. Strict independence criteria could
exclude the few qualified specialists in narrow clinical areas, many of whom have prior
ties to industry. We would favor disclosure, documented recusal, and reporting of
adjudicator agreement over blanket exclusion, so that independence is demonstrated
rather than presumed.
• LLM adjudicators: scale versus circularity. LLM judges make high-volume scoring
feasible, but they can share blind spots with the devices they assess and can change
without notice. For this reason we recommend that they supplement, and not replace,
human adjudication for safety-critical findings.
• Greater reliance on postmarket monitoring. Some stakeholders will argue that
premarket evidence should remain the primary gate, particularly for high-consequence or
autonomous functions, because harm may reach patients before a monitor detects it. We
agree this concern is strongest at the severe end of the consequences axis. We therefore
suggest that any reliance on postmarket monitoring be conditioned on the validated
detection performance of the monitoring system, as discussed in our response to Question
20.
Economic considerations. We have not quantified costs, and we think rigorous quantification is
difficult before standards exist. Qualitatively, we expect up-front costs from structured records,
held-out datasets, and re-benchmarking, offset over time by reusable qualified assets, less
duplicative testing across sponsors, and fewer review cycles caused by non-comparable
evidence. We would welcome the opportunity to contribute to a cost analysis as the framework
develops.

Omnalis | Docket No. FDA-2026-N-7874 | Page 10 of 13
Who bears the costs and who benefits. The fixed costs of benchmarking, held-out datasets, and
independent adjudication fall disproportionately on small and early-stage sponsors, while larger
sponsors can spread them across many products. The benefits of more reliable evaluation accrue
to patients and clinicians, and especially to subgroups for whom performance failures (element
R.2) are otherwise least visible. Those equity benefits are difficult to monetize but should be
weighed. Shared, qualified evaluation assets, as discussed in our response on MDDT-style
qualification, are the main lever for narrowing this gap, and we encourage CDRH to consider
small-sponsor access explicitly in any framework design.

2. Additional Considerations
Three considerations follow from adopting, or failing to adopt, a common standard early:
• Inter-evaluator consistency. CDRH should establish, and test, how it will determine
whether different sponsors and different independent parties evaluate the same device
consistently. A shared, de-identified benchmark corpus with a sequestered held-out
portion would be the natural instrument, and it would allow cross-comparison without
exposing sponsor intellectual property.
• Confidentiality. Sponsor data used in benchmarking and adjudication should remain
confidential, segregated, access-controlled, and auditable, and should not be used for
model training or tooling development absent the sponsor's explicit authorization. This
should be a baseline requirement for any qualified third party.
• Software-assisted evaluation. Where software or AI materially contributes to a
benchmarking or adjudication finding, FDA should specify what is disclosed: the
contribution, the traceability of each finding to source evidence, and a named accountable
human reviewer.
Taken together, these point to an opportunity larger than a premarket framework. With a
common standard for findings, evidence, and provenance, the competency-based approach can
become the baseline for a continuously learning regulatory system, in which premarket
benchmarks, postmarket monitoring, and FDA determinations remain traceable to one another
across a device's total product life cycle.

D. Requests for Clarification
Several details would help stakeholders respond more precisely as CDRH develops this
approach:
1. How the two-axis risk framework would translate into specific evidence expectations,
and whether CDRH could publish worked examples across the activity and consequences
grid.
2. How CDRH would define "structurally independent" and "qualified" adjudicators, for
both human and LLM adjudicators, including any thresholds.
3. Whether results from independently qualified benchmarks or tools, or from accredited
evaluators, would be accepted in review without re-evaluation, and under what
conditions.

Omnalis | Docket No. FDA-2026-N-7874 | Page 11 of 13
 4. Which marketing pathways (for example 510(k), De Novo, and PMA) the competencybased approach is intended to inform, and how it would interact with existing predicate
and special-controls frameworks.
5. Who would hold and govern held-out benchmark sets, and how versions would be retired
after exposure.
6. How re-benchmarking triggers would relate to the content of a PCCP.

E. Summary of Recommendations
In summary, we recommend that CDRH:
1. Standardize the evidence record. Define a structured, risk-scaled Benchmarking and
Adjudication Record so that evaluations are comparable, auditable, and machinecheckable for completeness.
2. Require benchmark validity, not just benchmark scores. Ask sponsors to show hazard
traceability, predictive validity against adjudicated real-world performance, and
contamination controls, and manage sponsor-developed benchmarks through process
rather than prohibition (Question 10).
3. Set acceptance criteria relative to clinician-panel variability. Use prespecified panel
qualifications, treat inter-clinician agreement as a noise floor, match comparators to the
task, and report both human-AI team and standalone performance (Question 14).
4. Use independent third parties where they add the most value. Focus on adjudication,
sequestered datasets, and benchmark maintenance, with defined independence criteria
and safeguards against gatekeeping (Question 16).
5. Build on existing frameworks in phases. Use MDDT-style qualification for reusable
evaluation tools and an ASCA-style accreditation model for independent evaluators, then
assess remaining gaps before creating any new program (Question 16).
6. Apply independence and qualification principles to LLM adjudicators. Require a
different model family from the device, locked and disclosed configurations, agreementbased qualification, requalification on change, and human accountability for safetycritical findings (Section V.B.2).
7. Permit supervisory agents with evidence. Condition any reliance on postmarket
monitoring on the validated detection performance, independence, and bounded
autonomy of the supervisory agent (Question 20).
8. Prespecify the response to foundation-model changes. Use version pinning, standing
regression testing, materiality thresholds tied to safety-critical elements, and a PCCP that
specifies the re-evaluation protocol (Question 24).
9. Create a shared, de-identified benchmark corpus with a sequestered portion, to test
consistency across evaluators without exposing sponsor intellectual property, and require
baseline confidentiality protections for any qualified third party.
10. Require disclosure of software and AI contribution to evaluation findings, with
traceability to source evidence and a named accountable human reviewer.

Omnalis | Docket No. FDA-2026-N-7874 | Page 12 of 13
Omnalis thanks CDRH for the opportunity to comment and for its openness in seeking early
input. We would welcome the opportunity to provide additional input, including on the design of
a standardized benchmarking and adjudication record or a shared evaluation benchmark, as these
efforts develop.
Respectfully submitted,
Elizabeth Wells
Chief Executive Officer & Co-Founder
Omnalis

Omnalis | Docket No. FDA-2026-N-7874 | Page 13 of 13