Rubrum Advising

Please see the attached document for our comments on FDA’s “Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback.”

October 19, 2026

Dockets Management Staff (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Rm. 1061
Rockville, MD 20852

Re: Docket No. FDA-2026-N-7874, "Considerations for the Regulation of Generative AI-Enabled
Medical Devices: Discussion Paper and Request for Feedback"

1. Introduction
Rubrum Advising is a healthcare advising and solutions firm with extensive federal regulatory
experience that helps clients and customers bring innovative health solutions, therapies, and
technologies to people sooner. We work with innovators, manufacturers, providers, investors,
policy makers, patients and other stakeholders to develop practical end-to-end strategies that
address federal requirements and healthcare system realities to achieve sustainable
implementation and patient access.

Where the interests of device manufacturers and the interests of patients could diverge, we
have taken the view that accuracy, ethics, and patient protections are not negotiable, and we
have not advanced positions that would require compromising them.

These comments are not exhaustive. We address only the questions on which our experience is
most relevant.

2. Summary of positions
• Benchmarking is an appropriate standard for safety and baseline efficacy, but it should
not be viewed as the basis for evidence of clinical applicability. That determination
should rest on clinical confirmation, which provides stronger evidence of a device’s
ability to perform in real-world clinical contexts.
• The intended end-user's level of training should be a recognized risk modifier, and
companies should be able to specify their intended end-user. End-user education is
therefore an important component of any Generative AI-enable medical device.
• A patient should not be able to negotiate an output with a device. A request to adapt an
output to a patient's circumstances should trigger deferral to a provider.
• A device's risk stratification should direct the degree of evidence and the intensity of
testing required in its clinical confirmation.
• Drift should be tolerated within pre-approved bounds. Drift in excess of those bounds
should trigger a re-review.
 • A manufacturer should be able to specify a device’s intended use and intended user
while remaining accountable for the device’s safety and performance within those
specifications.
• Where a healthcare institution authorizes or permits use beyond the above
specifications, the institution should bear responsibility for that expanded use, without
relieving the manufacturer of its own regulatory obligations.

3. Risk Assessment (Section IV; Questions 1, 2, 3, 4)

A. Directiveness depends on the intended user and their level of training (Q1, Q2, Q4)
One overlooked characteristic of a device's users is their level of training and clinical
background. The distinction between an intern or resident, an attending physician or advanced
practice nurse is significant. A fully trained and licensed physician is more likely to disregard a
wrong output, whereas a physician in training is more likely to question their own judgment
rather than the output, even when the output is incorrect and the physician’s judgment is
sound. Certain devices are also intended for nurses, nurse practitioners, physician assistants,
medical assistants, and others. These individuals have varying levels of training, and the
interpretability of a result changes with the end-user even when all of them are healthcare
providers. As a result, the user's level of training should be treated as a risk modifier. Provisions
are necessary, where applicable, that allow companies to specify their intended end-user. If a
healthcare system or institution decides to allow broader access to a device, responsibility for
that use should ultimately fall on the institution. These factors increase the importance of the
manufacturer or health system providing appropriate training within their setting.

The democratization of specialty-specific information to generalists can significantly improve
accessibility. It becomes unclear, however, whether an output will confirm a generalist's
preexisting clinical judgment or address something the generalist has limited experience with.
Information-focused outputs can broaden a generalist's scope of knowledge. Action-directing
outputs should have a threshold between what a generalist may be advised to do, based on risk
stratification, and what should remain a specialist's responsibility, including the need to refer to
a specialist. Stratification can be based on the perceived likelihood of side effects, treatment
efficacy, and similar factors. Among action-directing outputs, this is where the severity of
consequences becomes relevant. Advice that carries increasing activity and/or risk should have
appropriate text-based safeguards, such as "if uncertain, please consult with the relevant
service." Increasing consequences should call for increased safeguards, whereas limited
consequences should allow for minimal safeguards. To be clear, this applies to provider-facing
outputs, where the safeguard directs a generalist to a specialist. It is not the patient-facing
disclaimer the paper questions, where a "talk to your doctor" statement may not make an
output any less directive. These safeguards should be proportionate to the device's risk
classification and should supplement, rather than substitute for, appropriate risk controls.

B. Patient-facing outputs: no negotiation with the device (Q2, Q3)
A patient should not have the ability to negotiate an output with a device. Providers frequently
modify prescriptions or other treatment options based on their understanding of a patient's
circumstances and the likelihood that the patient will adhere to the assigned regimen. A
clinician knows what is clinically necessary, what concessions are possible, and what
alternatives are available if a patient is unable to comply and can weigh these factors against
the patient's clinical needs in a way the device may not be equipped to do. This risk is much
higher in patient-facing outputs because negotiation can lead to an override of safety
mechanisms. In these situations, the device should recommend that the patient consult a
provider.

This creates two relevant areas of concern. First, under Question 2, "personalization" should be
treated as a risk modifier that can move a patient-facing function higher on the consequences
axis. Personalization is where clinician judgment comes into play beyond objective clinical
evidence: the best choice for a patient may not be the therapy which has the strongest
evidence of efficacy. Second, under Appendix A element S.3 (p. 24), the appropriate device
behavior is deferral. A patient's request to modify an output based on their circumstances
should be viewed as meeting that criterion, and the device should direct the patient to a
provider. Accordingly, S.3 should be a required benchmarking element for patient-facing
functions, including tests of whether the device appropriately defers when a patient seeks
individualized clinical guidance.

4. Premarket Evaluation (Section V; Questions 7, 8, 9–10, 11, 12–13, 14–15, 16)

A. Overall design of the competency-based approach (Q7, Q8)
The proposed dual requirement is important, but it can be unnecessarily prescriptive if applied
in full to every generative AI device. It needs to scale with the intended user and the intended
use of the device. A device meant to understand and interpret a patient's chart has a
dramatically different risk stratification than a device that produces clinical output. The former
may need only benchmarking for safety and accuracy; the latter should require comprehensive
clinical testing. Every device should complete benchmarking, but the degree and intensity of
clinical confirmation should scale with risk, because clinical confirmation is where the evidence
for real-world scenarios and ambiguous situations comes from. This should be iterative since
providers may use a device differently than intended.

B. Benchmarking as a screen, clinical confirmation as the evidence (Q10, Q12–13)
Benchmarking cannot be viewed as a marker of true clinical applicability. Under the proposed
framework, benchmarking is similar to licensing exams or board questions. Board questions
have key words and question stems, and a device can pass them without that pass assessing its
clinical judgment. The paper already recognizes that public benchmarks can suffer from "data
contamination, saturation, and lack of representativeness of real-world conditions". For these
reasons, benchmarking should not be viewed as an appropriate test of clinical judgment.

Clinical confirmation is the more appropriate test. Questions 12 and 13 raise the need to
correct for differences between synthetic and real patients. Rather than attempting to
compensate for the limitations of synthetic data through greater reliance on benchmarking,
sponsors should use a more intensive clinical confirmation approach, selected from the five
proposed strategies, when the device's risk profile warrants it.

Where benchmarking is used, there can be a lot of ambiguity about the correct treatment
choice. Evaluation methods should accommodate clinically reasonable alternatives rather than
score every output against a single prescriptive answer. Otherwise the evaluation may penalize
appropriate clinical reasoning and give disproportionate weight to one preferred response,
particularly in cases where clinical acumen requires balancing competing considerations (Q9).

C. Performance standard and comparators (Q14, Q15)
Testing should be based on the intended audience. This is particularly applicable when
distinguishing information tools from action-oriented tools, and both are further differentiated
by whether they are patient-facing or provider-facing. Evaluation classification needs to follow
these distinctions.

Both a panel of clinicians and the median clinician should be used, but for different jobs. A
panel of clinicians should define the acceptable outputs and serve as the comparator. The
median clinician's rate of acceptable output should set the threshold. A minimum safety floor
must be met regardless. The paper presents these as alternatives, but they work better
together.

A panel is the right comparator because not every output can be directly linked to an outcome,
and a less frequently chosen option is not necessarily a wrong one. For example, more medicine
subspecialty clinicians use normal saline than lactated Ringer's, even though normal saline can
be detrimental in certain contexts. The median clinician is the right threshold because it
prevents a bottleneck in approval. However, it should be calibrated to the type of question, the
intended clinical context, and the applicable quality and safety standards, rather than set an
arbitrary cutoff. Additionally, if a device is to be used in a unique context, such as one that
would otherwise have minimal care, a percentile-based standard is more appropriate and fair.
Any such alternative standard should be justified prospectively and remain subject to the
applicable minimum safety floor.

D. Procedure: choosing a confirmation method and third-party reviewers (Q11, Q16)
Companies should have the ability to propose which of the five clinical confirmation
approaches should apply to their device. Confirmation by an independent reviewer, whether
FDA review staff or an FDA-recognized third party, should be sufficient to proceed with that
approach. If there is disagreement, a second independent reviewer should adjudicate which
approach is necessary. This gives companies predictability on the standard they will be held to
before they generate the evidence. In select situations, a device may need to meet more than
one of the five approaches. Independent reviewing organizations should be required to meet
timeline standards to be approved, so that they do not become a bottleneck.

To make this process predictable and proportionate, the selected approach and its evidentiary
requirements should be documented before the sponsor undertakes clinical confirmation. Any
subsequent change to those requirements should be justified by a material concern about the
device's risk or the adequacy of the proposed evidence.

5. Postmarket Monitoring and Modifications (Section VI; Questions 19, 21, 22, 23, 24)

A. Change control through a pre-approved drift tolerance (Q22, Q23, Q19)
A PCCP could incentivize improvements by removing hesitation over the amount of retesting a
change would require. There should also be a standard acceptable drift, defined as a preapproved variation from the baseline benchmark. In practice, the sponsor would prespecify a
minimum performance level for each benchmarking element and an allowed variation around it
and justify both in the PCCP. As an illustration only, if every element is at a minimum of 95%,
performance could be allowed to vary by 3% under periodic low-intensity retesting. Deviations
larger than the allowed variation should trigger larger and fuller retesting. A standard
justification within the PCCP, documenting why a change will not cause harm or a significant
change in performance, should be required. Periodic retesting, regardless of updates, should
also pick up drift or greater-than-expected changes in output. Where fuller retesting is
necessary, the premarket approaches should be considered; clinical confirmation such as
shadow testing, for example, is strong. This builds on the paper's own idea that the premarket
assessment becomes the baseline for re-benchmarking after a change.

Drift tolerances should be established separately for each relevant benchmarking element and
justified according to the element's clinical significance. A uniform numerical tolerance may be
inappropriate where a small performance change could materially affect patient safety. The
PCCP should also specify the monitoring frequency, the method for measuring drift, the triggers
for escalation, and the corrective actions required when a threshold is exceeded. The numerical
example above is illustrative, not a proposed universal threshold. Where drift exceeds the preapproved bounds, the sponsor should conduct the level of additional testing specified in the
PCCP and seek further review where required by the nature of the change and the applicable
regulatory framework. This approach is consistent with our broader view, expressed in prior
comments, that greater reliance on postmarket surveillance rather than additional premarket
review may be appropriate for certain AI-enabled technologies.

B. Third-party foundation model changes (Q24)
Model developers should be required to be transparent about all changes. Where that is
outside the scope of FDA regulation, device companies should demonstrate in any device
submission how they plan to obtain or enforce information about third-party model changes,
whether by contract, technical controls, or a PCCP (Q24).

At a minimum, sponsors should have a documented process for identifying material changes to
third-party foundation models, assessing their potential impact on the device's safety and
performance, and determining whether additional verification, validation, or clinical
confirmation is needed. Reliance on a third-party model should not eliminate the sponsor's
responsibility to evaluate the effects of relevant changes on the regulated device.

C. Responsibility for use beyond intended use (Q21)
Physicians frequently use drugs off-label, and that responsibility does not fall on the
pharmaceutical company. Responsibility for the decision to deploy the AI beyond its intended
user or context should rest with the health system. This does not remove manufacturer
accountability but instead enforces a boundary (Q21). Within the pre-approved use context, the
manufacturer is responsible for safety and efficacy and therefore should be liable for
negligence related to using the AI as intended. Beyond that use, the health system assumes
responsibility. This makes it paramount that any approval submission clearly states the
intended user and context. The same limits that apply to drugs should apply here: the
manufacturer cannot market or design the device toward use beyond its labeled population,
and known misuse cannot be ignored. Postmarket monitoring should capture and report
detected off-label use so that it is folded into design and labeling rather than treated as a
deferral of responsibility. There should be a mechanism for other health systems and providers
to learn from observations and safety concerns in any one system.

The allocation of responsibility should distinguish between a manufacturer's obligations for the
device as designed, labeled, and marketed and an institution's decision to deploy the device
outside its specified intended use or user population. An institution that authorizes such use
should be responsible for its deployment decision and for the associated clinical governance,
training, and oversight. This allocation should not absolve a manufacturer of responsibility for
defective design, inadequate labeling, misleading claims, or failure to address known safety
concerns.

Accordingly, each submission should clearly identify the intended user, intended use, and
relevant clinical context including education related to its use. Manufacturers should not
promote or design a device for use outside its authorized intended use, and they should not
disregard reasonably foreseeable misuse or known safety signals. Postmarket monitoring
should identify and assess evidence of use beyond the specified intended use and inform
appropriate changes to risk controls, design, labeling, and user training. Such monitoring should
complement and not replace, the responsibilities of both manufacturers and healthcare
institutions.

6. Closing
The FDA's framework should preserve a clear distinction between benchmarking, which
establishes baseline performance, and clinical confirmation, which evaluates performance in
clinically relevant and potentially ambiguous situations. Requirements should be scaled with
the device's intended use, intended user, and risk profile. Patient-facing systems should defer
requests that require individualized clinical judgment, while post market controls should permit
bounded improvements without compromising safety. Finally, responsibilities should be
allocated clearly between manufacturers and institutions, without diminishing either party's
existing obligations.

We appreciate the opportunity to provide these comments and encourage the FDA to adopt a
risk proportionate framework that supports innovation while maintaining rigorous standards
for patient safety and clinical performance.