Alterra Labs

Alterra Labs, an independent evaluator of GenAI models and devices for healthcare, submits the attached comments on CDRH’s discussion paper on generative AI-enabled medical devices (FDA-2026-N-7874). We recommend that CDRH (1) evaluate these devices in realistic, interactive clinical environments; (2) judge them against expert clinician judgment, with critical errors treated as disqualifying; (3) base regulatory decisions on rotating held-out evaluations across the total product life cycle; and (4) give qualified independent evaluators a defined role in designing and running those assessments.

October 19, 2026

Food and Drug Administration
Center for Devices and Radiological Health (CDRH)

RE: Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback (FDA-2026-N-7874)

Dear Dr. Tarver and Dr. Abramson,

Alterra Labs greatly appreciates the opportunity to respond to the Digital Health Center of Excellence's
discussion paper on the regulation of generative artificial intelligence (GenAI)‑enabled medical devices.

We are a model behavior analysis and alignment company, focused exclusively on AI safety for
healthcare and the life sciences. Our clinicians, healthcare and life sciences domain experts, and
machine learning researchers study how GenAI models and devices perform in realistic clinical settings,
where they fail, and why. We then leverage what we learn about those failures to train and align models to
be useful, trustworthy, and safe for patients.

Our work gives us a particular view on the questions posed in CDRH’s discussion paper. We are an
independent evaluator of frontier models and GenAI-enabled devices and applications for model
developers, device makers, and the health systems that deploy them. We share the Agency's view that
these devices could help deliver abundant, affordable, high-quality care to anyone, anywhere in the world.
Whether they will achieve this potential depends largely on the question at the center of the discussion
paper: how to know, before a device reaches patients and for as long as it stays in use, whether it is safe,
reliable, and trustworthy for the patients it serves. We design our evaluations and methodology
accordingly, as we describe in this comment.

In our comments, we describe how we approach a problem in our own work and then translate that
experience into specific recommendations for CDRH. Our intent is not to ask CDRH to adopt our
particular implementation, but to propose principles and requirements that any rigorous evaluation of a
GenAI-enabled device should meet.

Our summarized recommendations:
1.​ Evaluate devices in interactive environments grounded in realistic clinical data and
situations. Test GenAI-enabled devices, especially multi-turn, longitudinal, and agentic ones, in
interactive environments where they take actions rather than answer static questions.
2.​ Judge performance against expert clinical judgment, with serious errors treated as
disqualifying. Use robust clinician-authored rubrics that distinguish critical from non-critical
criteria, so that a single error with serious consequences for the patient fails the evaluation no
matter how well the device performs otherwise. Measure whether a device succeeds consistently
across repeated runs, not just once.
3.​ Use rotating held-out evaluations across the device’s total product life cycle, from
premarket review through postmarket use. Keep the evaluations used for regulatory decisions
unseen by the device during training. These should be rotated periodically to reflect updated
clinical standards and model capabilities, and to replace evaluation components that may have
leaked into model training data.
4.​ Give independent evaluators a defined role. Qualified independent evaluators should design
and run assessments independently of the sponsor, while CDRH sets the acceptance thresholds.
A. OVERVIEW OF EVALUATION ENVIRONMENTS

We support the discussion paper's choice to evaluate the final user-facing device rather than the
foundation model alone, since tools, prompts, retrieval, and guardrails all change behavior. However, we
also believe the foundation model should be evaluated, since evaluating a model once can surface failure
modes that every device built on it inherits. Model-level results also help attribute a device’s failures to the
right source, whether the model or the device’s own configuration, and provide a baseline for detecting
when a foundation model update upstream of a device changes how that device interacts with patients.

We therefore recommend that CDRH, as it develops the proposed voluntary Foundation Model Device
Master File (Q25), ask foundation model developers to include results from clinically grounded, held-out
evaluations of the base model, including documented healthcare-specific failure modes. Separately, when
a sponsor’s device is built on a third-party foundation model, CDRH should consider asking the sponsor
to report results for both the foundation model alone and the configured device on the same evaluation
tasks. Paired results would let reviewers separate inherited failures from device-specific ones. They would
also create the baseline needed to assess upstream model updates (Q24).

For many GenAI devices, especially when they are multi-turn, longitudinal, or agentic, answering this
question requires an evaluation environment. These devices are especially difficult to evaluate with static
test sets, because their most consequential failures emerge over a sequence of interactions rather than in
any single output. As a result, their performance can only be properly assessed in an environment that
tracks changes over time, changes itself in response to the device’s actions, and observes those actions
directly (see Figure 1).

Figure 1. Comparison of static benchmarks and environment-based evaluations.

For devices toward the action-taking end of the discussion paper’s proposed independence axis
(supervised or autonomous action), we recommend that CDRH expect device benchmarking to include
interactive, environment-based evaluation. Each evaluation environment has multiple components:
 ●​ Tasks: clinician-authored instructions anchored to a specific scenario and the associated scope
of actions or role that a model must take
●​ Data: de-identified, longitudinal patient records, either real data or thoughtfully designed synthetic
data.
●​ Simulation Environment: the simulated clinical workspace, software systems, tools, and
simulated actors the device interacts with, all of which respond to its actions.
●​ Rubrics: clinician-authored criteria for judging open-ended outputs, such as a note or a message
to a patient, with each criterion weighted by its clinical consequence.
●​ Verifiers: automated checks that read the environment’s state after the device acts, such as
which orders were placed, which messages were sent, and what was written to the chart, to
confirm what the device actually did.
●​ Checkpoints: evaluations at defined points within a task so that unsafe intermediate actions are
caught when they occur, even if the device later recovers.

B. DESIGN OF EVALUATIONS AND BENCHMARKS

The discussion paper asks how a sponsor should establish that benchmark performance predicts
real-world behavior. We believe construct validity comes from task provenance (who wrote the task, from
what data, for which user, and in which setting) and from the fidelity of the task, data, and environment to
the real world. Below we describe how we build tasks and environments to achieve this; Section C
addresses how performance on them is scored.

Realism starts with pairing every task with a specific patient and an environment to act in. An instruction
like “reconcile this patient’s medications before discharge” means little without a specific patient, because
the right answer depends on what is in their record. And without an environment, the device can only
describe what it would do rather than do it.

Realism also depends on setting. We build tasks that represent the real work a device will do in the
setting where it will be deployed. The same nominal task looks different across local contexts, since
patient mix, order sets, formularies, referral pathways, and staffing all change what counts as a correct
action and what information is available. To capture this, we draw tasks from records contributed by data
partners across a range of healthcare contexts, including academic, community, and rural sites.

We recommend that CDRH, or independent evaluators acting on its behalf, build regulatory benchmarks
from records spanning the care settings in a device’s intended use. Tasks should be authored by
practicing clinicians, including clinicians from those settings, and performance should be reported by
setting type rather than as a single pooled score. This would extend the discussion paper’s
generalizability competency to variation across care settings.

Real and Synthetic Data
Environments are only as realistic as the data they are seeded with. We often build tasks from real,
longitudinal patient data, sourced through data partners and de-identified in a way which preserves these
features while removing direct identifiers. Real records delineate the distributions and characteristics that
devices will actually face, such as outdated medication lists, duplicated entries, notes that contradict lab
results, and information scattered across years of encounters. These conditions test whether a device can
recognize when information conflicts or is missing, weigh which source to trust, and express uncertainty
or seek clarification rather than confidently acting on one version of the record. An environment built on
clean, idealized records will likely overstate how well a device performs in real settings.
Real records are also internally coherent in a way that is difficult to manufacture. Once a record is altered,
every downstream clinical state that depends on the change, from labs and medications to notes, must be
re-reviewed by an expert to stay consistent. Done carelessly, perturbation or synthetic data generation
can produce patient charts that look realistic, but do not reflect clinical reality. We therefore use perturbed
or synthetic data only where real data does not exist, and even then we build from real clinical records
and have experts review the results for clinical coherence. For example, testing whether a device flags a
sulfonamide ordered for a patient with prior Stevens-Johnson syndrome requires adding that history to a
real record, since such orders rarely reach the chart.

We recommend that CDRH expect evaluations used for regulatory decisions to be built primarily on real
clinical records. Perturbed or synthetic data should be used only to fill gaps, and clinical experts should
review it for coherence.

Clinician or Expert Design of Tasks
In our process, clinicians review real patients in a purpose-built clinical environment, define a clinical task,
and specify the actions a competent clinician would take. Clinicians are essential to ensuring that tasks
reflect realistic clinical conditions and variation (atypical presentations, medically complex patients, and
vulnerable populations). Where data on a population or care setting is thin, such as in rural or
resource-constrained settings, contributions from clinicians who practice there matter especially to create
tasks and rubric criteria that reflect clinical realism.

Held-Out Evaluations
A device that has seen an evaluation’s tasks during training can score well by recalling them rather than
by doing the work, much as a student who has seen the answer key can pass an exam without mastering
the material. This risk is higher for devices built on foundation models, which are trained on vast amounts
of data that may include published benchmarks, shared datasets, or close paraphrases of both, often
without the sponsor’s knowledge. The risk materializes when the device is used to treat patients it has
never seen before, failing to safely respond to edge cases.

For this reason, a set of tasks remains a valuable evaluation instrument only when neither the device nor
anyone with a stake in its score has been able to train on it (i.e., it is “held out”). Regulatory gating,
meaning the decision of whether a device can proceed to market, should rest on assets like these.
Held-out status also erodes over time, since tasks can leak through publication, reuse, or repeated
testing. These sets should be refreshed periodically, with retired tasks replaced by new ones that reflect
updated clinical guidelines and model capabilities.

As CDRH considers shared, standardized benchmarks, we recommend that any benchmark used for
regulatory gating be held by CDRH or an independent evaluator, kept inaccessible to sponsors and
foundation model developers, and refreshed on a defined schedule. A separate public set could show
sponsors the format and competencies tested without exposing the gating set.

C. RUBRICS, VERIFIERS, AND CHECKPOINTS

This section addresses how a device’s work is judged: who sets the standard, how errors are weighted,
how grades are applied and estimated, and what results are compared against. Rubric, verifier, and
checkpoint design is arguably the most consequential part of building a good evaluation.

Clinical Experts Set the Standard
Much of medicine has no ground truth to check against, and for many open-ended tasks, several
responses are acceptable. We believe that standard lives with experienced clinicians who have
developed judgment and intuition through training and experience about what to look for, which
information to trust, what can wait, and what cannot. There are so many unique inputs that may be
relevant to clinical judgment in different settings, and clinicians are required to identify which information
forms the basis for clinical decision-making. This judgment matters most where guidelines do not apply
cleanly, such as a patient with several interacting conditions or an ambiguous finding.

Decisions recorded in medical records should not be treated as ground truth. Medical records show what
happened, but not why a decision was made or whether it was the best one available. Records also
contain mistakes, omissions, and decisions made under constraints that were never documented. A
device graded only against historical behavior will be rewarded for reproducing those shortcomings. We
recommend that CDRH adopt the discussion paper’s suggested panel of qualified clinicians as the default
reference standard for open-ended tasks, instead of historical decisions in the record. Rubrics used for
regulatory decisions should be reached by clinician consensus rather than by averaging ratings.

Defining Rubrics
For each task, our rubrics are written and validated by panels of our in-house clinical team, including
physicians and advanced practice providers depending on the scope of the task. Because care varies
from clinician to clinician, a single “correct” path is rarely realistic, and one clinician’s answer is
insufficient. Our tooling lets clinicians generate variants from a base task, so one patient template yields
multiple valid trajectories. The rubrics credit valid alternatives across all trajectories, while still identifying
choices that would lead to avoidable risk or harm.

Rubric development is iterative and continues on both sides of running the device. Before any device
outputs exist, each draft rubric is reviewed by multiple clinicians who were not its original authors. When
reviewers disagree, we do not average their answers. We ask them to discuss the task and rubric until
they reach consensus, and that discussion often reveals an unclear task, a weak criterion, or genuine
clinical uncertainty that the device itself should recognize.

After the device has been run on the task, clinicians review a sample of its graded outputs. They red-team
the rubric, looking for cases where a clearly wrong output scored well or a clearly right one scored poorly,
and confirm that failures are attributed to the device rather than to the task or environment. Gaps found at
either stage are fed back into the rubric, task, or environment, and the cycle repeats until the rubric
reliably captures clinical judgment.

Weighting Errors by Criticality
Overall accuracy is not sufficient to establish safety, since a device can be right in most cases and still
make rare but catastrophic errors. In our rubrics, every criterion carries a clinician-assigned risk score
describing what happens if the device misses it (see Figure 2). When a device is run on a task and its
outputs are graded, a trial passes only if every critical criterion is met, regardless of how well the device
scored elsewhere. Beyond the critical criteria, a passing trial must also earn a minimum share of the
available points, with the threshold set for each task by the clinicians who write its rubric. If an action was
not completed, the device simply doesn’t earn points. Points are deducted only when the device does
something incorrect or harmful, such as inventing a finding, placing an unsupported order,
over-escalating, or expressing false certainty. Whether or not this specific scale is used, we recommend
that CDRH require that failure on any critical criterion fail the trial outright. CDRH should also consider
linking criterion severity to the consequence axis of its proposed two-axis risk framework.
Figure 2. Example clinical severity scale for rubric criteria. Clinicians assign each criterion a score from 1
to 6 based on the consequence to the patient if the device misses it. Criteria scored 4 or higher are
critical, and missing any one of them fails the trial.

Scoring Trajectories and Checkpoints
For devices that take actions, the final state does not capture everything that matters. Harm can occur
mid-task, a correct outcome can rest on flawed reasoning, and a correct decision can be executed
incorrectly. Our evaluations therefore examine the full sequence of actions, not only where the device
ends up, using checkpoints within each task. When a result is wrong or departs from clinical judgment,
checkpoints coupled with visibility into the device’s reasoning can help evaluators find the step where the
device diverged and understand how to fix it, the same kind of root-cause analysis used for patient-safety
events. In the rubric in Figure 3, for example, the Core and Act layers form a checkpoint between
interpretation and action: Core criteria check whether the device correctly read the image, and Act criteria
check what it did next. If the device fails to escalate, the checkpoint shows whether it missed the
enlarging effusion or saw it and failed to act.

Trajectory scoring does not mean penalizing every deviation along the way, since a device that recovers
from an early misstep or takes a valid alternative path should still receive credit. However, some actions
are unsafe the moment they occur, and recovery afterward does not undo them. Flawed reasoning should
also be scored, because a device that reaches the right answer for the wrong reasons is likely to reach
the wrong answer on the next patient. For agentic competency, we recommend that CDRH score full
action trajectories rather than final outputs alone and treat any unauthorized irreversible action as a
critical failure, regardless of the eventual outcome.
Figure 3. Example rubric for a chest radiograph follow-up task. The rubric evaluates whether a
GenAI-enabled device correctly interprets a chest radiograph (Core) and takes appropriate follow-up
actions (Act), such as messaging the care team and ordering drainage. Each criterion is marked critical or
non-critical according to the clinical severity scale in Figure 2.

Verifiers and Graders
Because a single evaluation can involve thousands of device responses over multiple trials, rubrics may
be implemented through rules-based or model-based graders, with clinicians writing the criteria and
checking the results. Rule-based checks, called verifiers, inspect the simulated record directly to confirm
what the device did, such as whether an order was placed or a message was sent. They give the same
answer every time and cannot be misled by how a device describes its own actions, so they should be
used wherever the clinical question allows. Model-based graders, in which a separate AI model reads the
device’s response and scores it against the rubric, are needed for open-ended outputs such as clinical
notes or patient messages that rules cannot easily assess. Because how these graders are built and
maintained is itself a consequential design choice, we recommend that CDRH require model-based
graders, including the LLM adjudicator the discussion paper contemplates, to be validated against
clinician judgment, documented, and independent of the device and its foundation model.
Performance Estimates and Comparators
Generative devices are stochastic, so the same device can pass or fail the same case by chance. We run
each task a prespecified number of times (usually 10 trials) and report per-case pass rates and variance.

Pass@k, the probability that a device succeeds on at least one of k attempts, is common in AI
benchmarking but overstates reliability in healthcare contexts, since a patient gets only one attempt.
Pass^k, the probability that a device succeeds on all k attempts, is the right measure for critical criteria. A
device that escalates a stroke presentation in nine of ten runs scores perfectly on pass@10 but has a
failure mode that pass^10 exposes. Pass^k shows how consistently a device performs and can be used
to quantify the risk of a critical failure once the device is used across many patients. In response to the
question on sample sizing (Q12), we recommend that CDRH require repeated trials per task and report
metrics such as pass^k on critical criteria, with consistency thresholds that scale with the device’s risk.

The CDRH discussion paper asks how performance comparators should be selected for open-ended
outputs. We believe comparators themselves need to evolve. Comparing a device to a clinician
performing today’s tasks in today’s workflows assumes the device’s job is to replicate existing care.
However, we believe much of the promise of GenAI lies in redesigning how care is delivered, through
reaching patients who currently go without review, closing gaps such as incidental imaging findings whose
recommended follow-up is never completed, and creating workflows that are more effective and
accessible than the ones they replace. A comparator anchored to the status quo can understate that
value, or fail to apply at all when no clinician performs the task today.

We therefore encourage CDRH to allow comparators that reflect high-quality care, including new
workflows with no direct precedent, and to evaluate whether those workflows are safe in their own right.
That evaluation should still hold the device to expert clinician-level performance on critical safety criteria,
since a new workflow that improves care on average can still be unsafe if it misses cases a clinician
would have caught.

D. CONSIDERATIONS FOR POSTMARKET MONITORING

GenAI-enabled devices change faster than regulatory processes were built for, especially as the
underlying foundation models update every as quickly as every few weeks. Newer models are not
uniformly better than the one it replaces. It may improve at certain tasks while degrading on other tasks.
Because these trade-offs are hard to predict, every model update needs to be re-evaluated rather than
assumed to be an improvement.

In response to the discussion paper’s question on third-party foundation model changes (Q24), we
recommend that CDRH treat any change to the underlying foundation model as a trigger for re-running
the device’s held-out benchmark before the updated version reaches patients. PCCPs for GenAI-enabled
devices should specify the evaluation suite, acceptance criteria, and rollback plan that apply when an
upstream model changes. CDRH should also expect sponsors to secure advance notice of model
updates from their foundation model developers. The update notification commitments contemplated in
the Foundation Model MAF could support this.

We recommend that CDRH make this division of roles between premarket and postmarket evaluations.
We view premarket evaluation to serve as the national minimum-threshold floor against a device’s
intended use, with postmarket monitoring tailored to where a device is actually deployed, since the same
device can behave differently across health systems with different workflows and patient populations.
We support the three monitoring approaches described in the discussion paper, and evaluation
environments offer a practical way to carry them out. For periodic benchmarking and performance
degradation monitoring, CDRH or independent evaluators should maintain a reference suite of held-out
cases with clinician-validated rubrics, run against the production device at regular intervals so that any
change in outputs reveals drift in the device or its underlying model. The reference suite should be
updated as clinical guidelines change and model capabilities advance, with each update versioned and
run alongside the previous version for a period so results stay comparable. Monitoring cadence should
scale with risk, and reassessment should also be triggered by known changes to the model, prompts,
retrieval sources, guardrails, or deployment environment.

The discussion paper asks whether evidence expectations could shift from premarket to postmarket
(Q18). We recommend that any such shift be conditioned on the sponsor having this monitoring
infrastructure in place, so that reduced premarket evidence is never accepted without a credible way to
detect failure after deployment. The discussion paper also asks about machine-based supervisory agents.
We believe they can usefully extend monitoring coverage. We recommend that CDRH treat them as a
complement to, not a replacement for, clinician review and held-out reference suites, and that they be
validated against clinician judgment in the same way as model-based graders.

Postmarket monitoring infrastructure can be costly, and most critical access hospitals and publicly funded
clinics cannot build it themselves, even though these are the settings where clinical AI may do the most to
improve access and quality. We encourage CDRH to make clear that postmarket monitoring is the
sponsor’s responsibility wherever a device is deployed, to ask sponsors to report performance by
deployment setting so that results in resource-limited sites are not obscured by averages, and to share
findings with deploying sites. Standardized, FDA-recognized monitoring tools could further reduce the
burden on sites that lack the resources to monitor on their own.

E. INDEPENDENT THIRD-PARTY EVALUATORS

We strongly support qualified third parties conducting competency assessments that result in a
determination CDRH could consider in its review of the device. Independent evaluators should support
premarket testing, postmarket monitoring, maintenance of sequestered evaluation sets, adjudication of
sponsor-submitted testing results, and standards development.

We believe independent evaluators should design and run the assessment for a given device, translating
its intended use into concrete scenarios, simulated environments, and case distributions that reflect the
clinical setting, user population, and workflow in which it will be deployed. Intended use also includes the
safeguards the device will operate under, such as clinician review of its outputs or required approval
before certain actions, and the environment should reproduce those safeguards so the device is tested as
it will actually be used.

Because independent evaluators maintain their own evaluation assets, a single evaluation can serve
several purposes. Devices with similar intended uses can be assessed against a shared evaluation
environment, which lowers costs for sponsors and lets CDRH compare results across devices. The same
evaluation can also be run on the full device, on its foundation model alone, and on clinicians working with
and without the device, which helps attribute failures to their source and supports the comparisons
discussed in Section C.

However, we recommend that CDRH should remain responsible for setting criteria for passing, which is
the regulatory bar. We suggest that the sponsor, with CDRH’s agreement through the pre-submission
process, establish the device’s intended use, risk profile, applicable elements, and acceptance
thresholds. Independent evaluators would then design and regularly refresh the assessment, run the
device in environments configured to match its deployment context, adjudicate the results, and issue the
determination. The sponsor would know which competencies will be tested and what performance is
required, but would never see or run the test itself.

To state the division of roles explicitly, CDRH or qualified independent evaluators would build, maintain,
and run all evaluation assets described in Sections B and C, including tasks, data, rubrics, graders, and
held-out sets, which sponsors would not author, submit, or see. Sponsors would not author, submit, or
see these assets. Because standards and thresholds remain public and FDA-recognized, independent
evaluators would compete on the quality and clinical fidelity of their assessments, which also addresses
the Agency’s concern that third-party participation could limit competition.

For this model to work, independence must be defined by what an evaluator’s assets have touched, not
only by its ownership. At a minimum, an independent evaluator should have no financial interest in
whether a device is cleared, should disclose all commercial relationships with the sponsor and its model
developer, and should be able to document that its evaluation assets have not been exposed to either.
Assets used to gate a device should never be used or licensed to train that device or its foundation
model, and CDRH should consider requiring evaluators to maintain a firewall between regulatory
evaluation and any model development work. CDRH should also set clear expectations for the access
sponsors must provide, so that evaluators can assess the device as it will actually be deployed, including
its configuration, guardrails, tool integrations, and version history.

Not every device needs the full model described above. The degree of third-party involvement should
scale with the device’s scope and risk, as oversight already does across device classes. Narrow, low-risk,
or clinician-mediated functions could rely on sponsor-run testing with independent adjudication, while full
independent assessment would be reserved for broad or action-taking devices. Consistent with least
burdensome principles, independent assessments can be leveraged where the potential for harm is
greatest, without slowing lower-risk devices that can benefit patients sooner.

F. CONCLUSION

The discussion paper’s competency-based approach, which evaluates the final user-facing device
through benchmarking and clinical confirmation across its total product life cycle, is the right frame for
GenAI-enabled devices, and its effectiveness will depend on how realistically those competencies are
measured.

To that end, we urge CDRH to evaluate GenAI-enabled devices in realistic and interactive environments,
judge them against expert clinical judgment weighted by consequence, rely on held-out evaluations
throughout the device’s lifecycle, and give qualified independent evaluators a defined role in that work.
We are not asking CDRH to adopt any single organization’s methods. We are asking it to set clear
expectations for realism, clinical grounding, consequence-weighted scoring, and independence that any
qualified sponsor or evaluator can meet.​

We thank the Agency and CDRH for a thoughtful discussion paper and for its consideration of our
comments. We also encourage CDRH to schedule periodic future discussions on this topic, since
evaluation methods for GenAI systems are constantly evolving. Any standards set now should be built to
be revised. Alterra Labs would welcome the chance to discuss our comments, share our methods in more
detail, and contribute to pilot work on evaluation or postmarket monitoring.