Raidium (Pierre Manceron)

Comment from Raidium SAS (Paris, France) on docket FDA-2026-N-7874, "Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback".

Raidium is developing a radiology foundation model and a PACS viewer intended to draft radiology reports (CT, MRI, X-ray) for review and signature by radiologists. The product is under development. Our detailed answers to the 26 discussion questions are provided in the attached document. They reflect the questions we are working through while designing the device and its evaluation, and examples refer to radiology unless stated otherwise. The main points are summarized below.

Risk assessment (Section IV). We agree with the two-axis framework, with two additions. First, a single multi-indication device operates at several risk levels at once, so the framework should be applied per function or indication, with a prespecified tiering of findings by clinical consequence (ex: ACR critical and actionable findings) and evidence, thresholds and monitoring proportionate to each tier. Second, a raw LLM output cannot be non-directive; a device exposing it should by default sit at the highest activity level reachable in a conversation, and only the manufacturer’s harness and structured output can bring a function down to the non-directive level. The harness, not the underlying model, should be the object of risk assessment and evaluation. We also propose a controlled "patient mode" and "non-specialist mode" of HCP-cleared devices, whose evidentiary burden should be set against the real-world counterfactual, i.e. the unsupervised use of generalist LLM platforms.

Premarket evaluation (Section V). We support the competency-based approach. Its value lies in changing what is benchmarked: not one powered study per finding type, but a general verification of competencies, sampled across the intended use, as a board examination does, scored per finding with a weight per tier. Such benchmarks rely on human-judged quality and are expensive to build; we recommend that medical societies build and host them, and that sponsor benchmarks be published. Performance should be measured against two bounds on the same cases: routine clinicians without the device (lower bound, directly available from the original signed reports of retrospective exams) and expert consensus (upper bound). For functions reviewed by a specialist, the signed report is what counts: team performance is measured in routine use, standalone performance at the benchmark.

Postmarket monitoring (Section VI). We support accepting greater premarket uncertainty in exchange for stronger postmarket monitoring, for functions reviewed by a specialist. Premarket evidence would be limited to benchmarking and retrospective standalone evaluation; clinical confirmation would be produced in routine use by comparing the device output with the signed report on every exam, using the same per-finding metric as the benchmark, and following cases to downstream ground truth where it exists. This is more powerful than a one-off MRMC study and, for multi-indication devices, the only way to reach statistical power on rare findings. Conditions: monitoring specified at authorization together with the PCCP (metric, thresholds, timelines), built into the software with no burden for radiologists, conditional authorization with withdrawal if criteria are not met, and an FDA-endorsed methodology. Machine-based judges, validated against human adjudication and frozen in a versioned configuration, are what make such monitoring viable. The PCCP should be defined by what is verified (benchmark, thresholds, monitoring) rather than by a list of modifications.

Other topics (Section VII). Foundation Model MAFs are useful if the process is adapted to the release cadence of foundation models, with a lightweight file per model release. For agentic devices, we propose a human checkpoint before any irreversible action, full logging of tool calls, and acceptance criteria on action sequences, illustrated with an agentic report drafting example.

Contact: Pierre Manceron, President, Raidium SAS, pierre.manceron@raidium.eu

Raidium answers to the FDA GenAI Discussion Paper

Raidium is developing a radiology foundation model and a PACS viewer intended to draft radiology
reports (CT, MRI, X-ray) for review and signature by radiologists. The product is under development.
The questions below were read in that light: our answers reflect the questions we are working
through while designing the device and its evaluation, and examples refer to radiology unless stated
otherwise.

Section IV: Considerations for the Assessment of Risk for GenAI-Enabled
Devices

Q1. Does the two-axis risk framework, organized around device activity and the consequence of
relying on an incorrect output, appropriately capture the dimensions most relevant to the risk of a
GenAI-enabled software function? If there are additional dimensions (such as the reversibility of a
resulting action, the availability of downstream safeguards, the time pressure of the deployment
setting, or the traceability of the output, i.e., to primary source materials) that should be
represented in a risk framework, please describe and provide examples of how they should be
represented.
We agree with the two-axis framework.
An important additional consideration is that a single device may operate at multiple risk levels. A
radiology report drafting device covers all findings of an exam, and missing a pulmonary embolism, a
cancer, a fracture or an atelectasis has very different consequences. Risk also depends on the
direction of the error: a missed embolism and a hallucinated cancer lead to different harms.
Classifying the whole device at the level of its most severe finding would be disproportionate for
general-purpose devices. The two-axis framework should therefore be applied per function or
indication, each with its own position on the chart, rather than once for the whole device. We
propose a prespecified tiering of findings by clinical consequence, based on external references such
as the ACR guidance on critical and actionable findings, with evidence requirements, acceptance
criteria and postmarket monitoring proportionate to each tier, instead of a blanket requirement
driven by the most severe finding.
Two of the additional dimensions listed by CDRH should also be part of the risk assessment:
downstream safeguards (systematic specialist review) and traceability (each finding can be checked
in the image).
We also argue that there is a continuum between Informational / Action-directing and Actiontaking / HCP-supervised, mainly because of automation bias. A report drafting device suggests a
report (Informational / Action-directing), but automation bias can turn it into a de facto decision if
the report is signed without reviewing the exam. Automation bias is a long-standing issue for CAD
functions, so far addressed mainly through labeling. Paradoxically, the more accurate an actiondirecting device, the stronger the automation bias: users tend to accept a 95% accurate output
systematically, but independently re-check a 75% accurate one, so the less accurate device can be
safer in practice. Accuracy alone is therefore not a sufficient risk modifier for action-directing
functions.
An additional dimension for us is HCP involvement in the analysis. If the HCP still has to perform the
analysis, and the report can only be signed while the HCP is reviewing the exam, the device stays on
the Informational / Action-directing side. For example, the draft is only displayed once the
radiologist has opened the images, and each finding is linked to its location in the exam. These
mechanisms remain imperfect, as it is hard to verify that the HCP actually reviewed the image and
looked at the lesion. New methods to measure and limit automation bias are needed, such as
logging which linked findings were opened before signature.
Another dimension is the scope of functionalities. A well-known issue with LLMs is that their breadth
makes it easy to use them outside of their intended use, for example asking an orthopedic surgery
question to a radiology report drafting device, or worse, an accounting question. This mainly
correlates with the complexity of the task: the more complex the task, the more ways there are to
interact with the model. Compare a model limited to detecting the presence of a lesion with a model
that characterizes the lesion.

Q2. CDRH seeks input on how to account for the spectrum of GenAI-enabled informational
functions that vary in the degree to which they direct a user to a particular action (i.e., between
"non-directive" and "action-directing"). What characteristics of an output, such as its wording,
specificity, personalization, or context, could be considered as modifiers of the risk associated with
an informational function, after accounting for the device's overall functionality and intended use?
We argue that this continuum exists by construction. The underlying LLM can become actiondirecting over the course of a multi-turn conversation, especially when a doubtful user asks for
reassurance: the LLM then becomes more directive to address the user's doubt.
There are two ways to look at this. Unconstrained LLMs cannot be purely non-directive, so nondirective cannot exist as a risk category for them. Alternatively, an LLM can be forced to remain nondirective, with heavy constraints. This is done primarily through a dedicated harness design,
including flow architecture, fail-safes, pre-validation and post-validation. In report drafting, for
example, a structured output limited to findings and measurements, with follow-up
recommendations only taken from guidelines such as Fleischner.
A raw LLM output cannot be non-directive. A device exposing raw LLM output therefore cannot
claim the non-directive risk profile and should by default be placed at the highest activity level
reachable in a conversation (cf. Q5). Only the manufacturer's harness and structured output,
grounded in professional guidelines, textbooks and practice standards, can bring a function down to
the non-directive level.
What additional information would provide manufacturers with sufficient clarity and predictability
around risk assessment for informational functions while recognizing that directiveness may exist
along a continuum rather than as a binary distinction?
CDRH should make it clear that a model strictly limited to the lowest risk level (non-directive) is
achievable, but at the cost of heavy constraints and reduced functionality. Without explicit
examples, the non-directive / action-directing distinction will turn into semantic discussions that are
hard to adjudicate objectively. We ask CDRH to publish, in future publications, explicit language
examples and use cases for each level of directiveness.

Q3. CDRH seeks input on whether and when a GenAI-enabled function that results in delivery of
clinical information to patients, as opposed to HCPs, could present different or higher risks, while
also recognizing the potential benefits associated with improved patient empowerment,
engagement, and access to clinical information. What device characteristics, output features, or
safeguards might mitigate risks that could arise when a user lacks the domain knowledge to
independently evaluate an output, without unnecessarily underestimating patient capability?
In our experience, many patients already use generalist LLM platforms (Claude, ChatGPT, Gemini) to
analyze their radiology reports. This has specific impacts. Generalist LLMs usually state that they are
not healthcare professionals, but they still answer the question. As of today, they tend to be
overcautious, with many false positives, and they tend to reinforce the user's confirmation bias. We
have seen a patient come in with a 20-page AI-generated analysis of his CT scan, raising many
concerns and questions on trivial findings, which creates additional workload for the radiologist, who
has to answer them. Another problem is overconfidence: since patients generally trust AI platforms,
they also trust their outputs on radiology exams, even though these platforms are less accurate and
more error-prone, which can create false negatives. False positives ultimately put pressure on
radiologists. False negatives could theoretically be harmful, but a CT scan is always interpreted by a
radiologist, so the false negative would not be missed.
The relevant counterfactual for a patient-facing function is therefore not the absence of information,
but the unsupervised use of generalist LLM platforms, which already deliver this information with no
intended use and no oversight. The evidentiary burden of a controlled patient mode should be set
against this real-world baseline (cf. Q15), not against an idealized scenario where the patient
receives no AI output.
Our proposed mitigation is a patient mode in dedicated medical devices. Instead of leaving patients
with uncontrolled generalist LLM platforms, medical devices can offer a dedicated and controlled
patient mode. In the PACS viewer we are building, the patient mode would let patients view their
scans, read the associated reports and chat with the report. The product is prompted and cleared for
physicians, and the patient mode is a sub-prompt that still benefits from the clearance. Its
safeguards are outputs grounded on the signed report and the exam only, no new finding and no
management advice, systematic redirection to the radiologist or referring physician for any clinical
question, plain-language explanations without reassurance or alarm, and full logging of the
conversations for postmarket review.
What is at stake is ease of clearance and deployment. If the patient mode is hard to clear, it will not
reach routine use and patients will fall back on generalist LLM platforms. Our take is that clearance
of the patient mode should be granted once the general clearance (for HCPs) is obtained. The
patient mode is minimal risk, comparable to the generalist consumer LLMs patients already use, but
with guardrails implemented by the manufacturer and validation data reviewed by FDA, versus
almost nothing on the generalist platforms.

Q4. CDRH seeks input on whether and how the distinction between generalist and specialist
physicians might inform the assessment of risk for HCP-facing GenAI-enabled functions. Under
what circumstances, if any, might risk be affected when an HCP who lacks the relevant clinical
specialist knowledge receives information that falls within a specialist area of practice? What
device characteristics or safeguards might mitigate such risks?
As noted in Q3, many non-specialist HCPs (GPs, referring physicians) already use GenAI software to
analyze radiology reports. This generally has the same implications in terms of false positives and
false negatives, but with different human implications: here, false positives and false negatives
directly impact patient care, instead of just putting pressure on the radiologist. Consider a GP
reassured by a generalist LLM on an incidental nodule that actually requires follow-up.
The mitigation is the same, a dedicated non-specialist mode of the product, with a dedicated
prompt. PACS viewers often offer a "light" viewer for orthopedic surgeons or oncologists. Here, the
clearance should be included in the general clearance. More generally, a non-specialist-facing
function should carry a lower evidentiary bar than the specialist-facing function (ex: the diagnostic
PACS). False positives and false negatives have different consequences for a GP and for a radiologist,
because they sit at different stages of the diagnostic escalation pathway: the GP's output leads to a
specialist review, the radiologist's output is final. This would open a lighter pathway for general
radiology functions aimed at GPs, such as incidental finding alerts and triage.

Q5. For multi-turn conversational GenAI-enabled devices that may migrate from providing "nondirective" information to "action-directing" information over the course of an exchange, how
should risk be assessed across realistic conversational trajectories? How could the intended use of
such a device be characterized when its behavior is emergent across a conversation?
This is relevant for us, as our PACS viewer includes a chat (radiologists asking follow-up questions,
patients chatting with their report in patient mode). As discussed in Q2, drifting toward actiondirecting outputs happens by construction, so risk should be assessed at the conversation level, not
per turn.
The intended use should be defined by what the device is allowed to output, enforced by the
harness, rather than by its emergent behavior: for example, outputs grounded on the exam and its
report only, no management decision, and redirection to the radiologist or referring physician
otherwise. Without such constraints, the device should be classified at the highest activity level
reachable in a conversation. This is the default risk profile of a raw LLM output (cf. Q2). Only the
manufacturer's harness and structured output can bring the device below that level, so the harness,
not the underlying LLM, should be the object of the risk assessment and of the evaluation.
Evaluation can be done through simulated multi-turn conversations, measuring the rate of
conversations that end up with action-directing outputs, including doubtful users asking for
reassurance and users pushing outside of the intended use. For example, a patient asking whether a
nodule is dangerous, then whether the follow-up CT is really needed: the device should not end up
advising to skip it.

Q6. For GenAI-enabled care escalation functions, CDRH is considering how the evaluation may
account for both under-escalation and over-escalation. How could manufacturers characterize and
weigh these two directions of error, given that they may not be commensurable and that
acceptable trade-offs may vary by clinical context?
We see three options.
Gate: consider over-escalations and under-escalations as false positives and false negatives, and
evaluate them as is done today, with a specific threshold per disease type. In report drafting, this
means a stricter threshold on missed critical findings (pulmonary embolism, intracranial
hemorrhage) than on over-reported minor findings. This is feasible with current guidelines, but in
practice it will not keep up with the pace of frontier models.
Prevent: act on the design rather than on per-disease thresholds. Design requirements prevent most
over-escalations and under-escalations by construction (escalation wording taken from guidelines
only, systematic flagging of critical findings, no reassurance statement without a finding to support
it), formalized as process requirements in the QMS, along the lines of ISO 13485. Clearance is then
based on an overall rate of over-escalations and under-escalations across findings, below a
prespecified threshold, rather than one threshold per disease, and labeling states these rates so that
the HCP knows what to look for and remains responsible for the final decision.
Track: postmarket tracking of the escalation chain, through an evaluation function built into the
PACS. Each escalation decision is followed downstream (follow-up exam, specialist review, biopsy,
outcome) to obtain the ground truth of long-horizon diagnostic tasks and measure the accuracy of
escalation decisions in real-world use, automatically. Beyond the subject device, such a function
could serve the whole industry for the postmarket evaluation of CADe/CADx functions.

Section V: A Competency-Based Approach for Premarket Evaluation of
GenAI-Enabled Devices

Q7. Is the competency-based approach described above, i.e., device benchmarking followed by
clinical confirmation, a useful and appropriate framework for evaluating GenAI-enabled devices?
Yes. It also accounts for the future evolution of LLMs, which increasingly learn, and therefore
behave, like humans.
However, if benchmarking and clinical confirmation are both run premarket with the current
methods, the approach ends up as burdensome as the CADe/CADx paradigm, or more, especially for
a general-purpose device with tens of indications. The value of a competency-based approach lies in
changing what is benchmarked: not an exhaustive verification of each finding and each indication,
each with its own statistical power, but a general verification of competencies, sampled across the
intended use, as a board examination does. For a report drafting device, instead of one standalone
study per finding type (nodule, embolism, fracture, hemorrhage, etc.), each powered separately, a
competency benchmark samples a few hundred exams across the intended use and scores general
competencies on them, each finding being scored as described in Q12: detection of critical findings
(S.1), absence of hallucinated findings, measurement accuracy (E.3), comparison with prior exams,
scope maintenance (S.2), calibration and deferral (S.3), consistency across sites and scanners (R.1,
R.2). A radiologist is not board-certified finding by finding either: the examination samples cases and
tests the underlying competencies.
Clinical confirmation then verifies that these competencies hold in real-world use. For functions
reviewed by a specialist, it is produced in routine use after authorization (cf. Q11 and Q18).

Q8. How might the two-axis risk framework described in Section IV be considered within the
competency-based approach described above to help determine the level of evidence needed for
a premarket submission?
Combining a competency-based approach with the two-axis risk framework is already how HCPs are
trained. Radiology residents typically go through the four levels of activity. First-semester residents
discuss the exam with senior staff (Informational / Non-directive). More advanced residents draft
reports that are then validated by senior staff (Informational / Action-directing). At the end of
residency, they report on their own and only ask for help when needed (Action-taking / HCPsupervised). After graduation, they are fully autonomous. The same applies to other specialties, such
as surgery.
Our suggestion is therefore to apply the same evaluation criteria. Informational / Non-directive:
benchmarking only. Informational / Action-directing with specialist review: competency benchmark
on retrospective real-world exams, assessed per competency and per tier (cf. Q11), and clinical
confirmation as a postmarket obligation, through congruence with the signed report, i.e. the findinglevel agreement between the draft and the report signed by the radiologist. Action-taking / HCPsupervised: the same benchmark, plus premarket MRMC evaluation, and postmarket monitoring
with prespecified thresholds. Autonomous: benchmarking and prospective clinical study before
authorization. Within each level, the consequences axis sets the acceptance thresholds per tier of
findings (cf. Q1).

Q9. Would a benchmarking structure such as the one described above be likely to provide
adequate evidence of clinical knowledge, analytic capabilities, safety behavior, communication,
and generalizability to support a reasonable assurance of safety and effectiveness? Are there
elements that are missing, redundant, or inappropriately categorized? Are there externally
developed standards that could be leveraged?
The structure is sound. However, it could be extended with vertical-specific benchmarks, as is done
with the USMLE (cf. Q7 for the radiology example). Imaging exams involve specific skills, including
grounding, prioritization and differential diagnosis. For report drafting, the relevant skills are missed
and hallucinated findings, measurements, and comparison with prior exams.
Q10. CDRH recognizes that publicly available benchmarking assets may be subject to data
contamination, saturation, and limited real-world representativeness. How should a sponsor
establish that performance on a given benchmark predicts safe and effective real-world behavior
for the device's intended use? What evidence should support the construct validity of a
benchmark used to gate device evaluation, and what role should sponsor-developed benchmarks
play given potential concerns around independence and optimization to the test?
This is a very difficult question. These benchmarks are much broader than traditional ones, which are
easy to describe with statistical methods and clear metrics, so they are much harder to build.
Our take is to rely as much as possible on externally curated benchmarks, as is done today with the
USMLE. Private benchmarks remain possible, but they require specific guidelines on how to build
them, such as correlation with clinical performance, review by experts and medical societies, and
test sets from sites not used for training, to avoid contamination.
Such benchmarks are harder to build than traditional ones. They do not rely on an exhaustive, rulebased verification of every finding, but on human-judged quality, as the USMLE does. This requires
expert crafting and validation (cf. Q16), is expensive, and remains exposed to contamination. For
multi-indication devices such as a radiology foundation model, this is one more reason to open
clinical confirmation to postmarket evidence, measuring accuracy in real-world use through
congruence with the signed report and long-horizon ground truth (cf. Q6 and Q18).

Q11. CDRH is considering that clinical confirmation for a GenAI-enabled device might not require a
prospective clinical study in every case, and has described above a range of approaches of
increasing rigor and patient exposure. How might a sponsor select and justify a confirmation
approach tailored to a device's intended use and proportionate to the device's risk profile? Are
there device types or risk profiles for which one or more of these approaches would be insufficient
or inappropriate? How might the anticipated distribution of real-world inputs be taken into
consideration? Are there other methods of clinical confirmation that might help inform the
evaluation of GenAI-enabled devices?
For imaging devices whose outputs are reviewed and signed by a specialist, premarket evaluation is
a single step: the competency benchmark run on retrospective, multi-site real-world exams (cf. Q7).
The unit of measurement is the finding (cf. Q12), but the benchmark result is an overall assessment
per competency and per tier, on a rubric validated by experts, not the mechanical sum of findinglevel checks. Like an examination grade, it is human-judged in its design and in its adjudication of
ambiguous cases, and automated only in its execution. For the clinical competencies, this
assessment can be expressed relative to the two bounds, since every retrospective exam comes with
the original report signed in routine practice: the NCG (cf. Q15), 0 for routine practice, 100 for expert
consensus, is one such expression, and gives both the competency level and the clinical value. Pass
thresholds are set per competency and per tier (for the clinical competencies, NCG above 0 at
minimum, higher for critical findings). Non-clinical probes (scope maintenance, robustness,
adversarial inputs) have no routine comparator and are assessed with their own pass criteria. There
is no separate retrospective study and no premarket MRMC: team performance is measured
postmarket on the signed report (cf. Q18).
Standardized patient interactions do not apply to imaging. Prospective studies should be reserved
for autonomous functions, or functions without specialist review.
The real-world input distribution can be handled with multi-site case selection representative of the
intended use (sites, scanners, protocols, case mix), with enrichment in rare critical findings and
reweighting to the expected prevalence.

Q12. How should sponsors achieve statistically meaningful performance measurement for GenAIenabled devices?
Statistical measurement does not change in nature. Once open outputs are broken down into
measurable units (each finding of a report scored as correct, missed, hallucinated or mismeasured,
cf. RadMatch), standard methods apply: confidence intervals per competency and per tier, sample
sizes driven by the rarest tier of findings, enrichment and reweighting to prevalence (cf. Q11).
What changes is twofold. The sampling design: as in a board examination, one sampled set of exams
across the intended use, powered on the competencies and tiers, rather than one powered study
per finding type. And the nature of the result: an overall assessment per competency on an expertvalidated rubric (cf. Q11), where finding-level scores are the inputs, not the verdict.
The same unit serves the whole life cycle. Findings are scored (correct, missed, hallucinated,
mismeasured) with a weight per tier of clinical consequence (cf. Q1). The benchmark aggregates
these scores per competency (S.1, E.3, etc.) over a sample of exams across the intended use, with
pass thresholds per competency and per tier. Postmarket monitoring aggregates the same scores
per tier over all exams in routine use (cf. Q18, Q19). The USMLE analogy therefore refers to the
sampling and to the human-judged overall assessment, not to a different unit of measurement.
Where synthetically generated inputs supplement real patient data, how should sponsors account for
differences between the synthetic and real-world distributions when estimating performance, and
under what conditions, if any, is it appropriate to combine benchmarking evidence and clinical
confirmation evidence to support a single performance estimate?
In our experience in radiology, synthetic data is useful for training but cannot be used for evaluation.
Synthetic data is inherently biased, and models trained on it reproduce the biases of the models that
generated it. The use of synthetic data is still hard to frame, and the best framework remains to see
it as a training booster, not as a validation tool. A practical initial policy for regulators would be a
hard cap on the share of synthetic data allowed in training (ex: 10%), set conservatively at first and
raised progressively as research supports a larger share.
As a side note, rather than purely synthetic evaluation data, LLMs can be useful as judges or
annotators. They can enrich existing data with annotations and structure (ex: extracting findings
from a report), and even provide metrics (ex: scoring the quality of a drafted report, cf. RadMatch,
https://arxiv.org/abs/2609.01470). This is a promising field, with an extensive literature.

Q13. For which clinical domains, device functions, or subpopulations is synthetic data particularly
well-suited, or particularly inadequate, as a supplement to real-world evidence? What safeguards
would mitigate the risk that synthetic data generated by models of the same class as the device
under evaluation reproduces the very performance gaps the evaluation is intended to detect,
particularly for underrepresented subgroups?
See our answer to Q12.

Q14. For open-ended device outputs, how should performance comparators and acceptance
criteria be selected? When a panel of qualified clinicians serves as the comparator, how should the
applicable standard (for example, the standard of care versus the performance of a median
clinician in practice) be defined and justified?
For the definition, see our answer to Q9: new benchmarks in a competency-based approach should
follow how HCPs are evaluated today (ex: USMLE).
Two bounds should then be defined, on the same cases and with the same metric. The lower bound
is the performance of routine HCPs without the device, for example the original signed report of
retrospective exams. Relying only on the standard of care sets unrealistic expectations for devices
and creates a loss of chance for patients when a device could bring value; in the absence of a device,
HCPs will rely on generalist LLM platforms anyway. The upper bound is the standard of care, i.e.
expert consensus, which should remain part of both the evaluation and the marketing of the device.
This encourages devices whose performance gets close to expert level, and since AI devices scale,
they can bring expert-level skills to less experienced radiologists. A device should be accepted above
the lower bound, and aim for the upper bound (cf. Q15).
Should generalist or specialist physicians be used as a performance standard?
The comparator should follow the user targeted by the intended use. For a device intended for
radiologists, the panel is made of radiologists, and the lower bound is the routine radiologist without
the device. For a non-specialist mode (cf. Q4), the comparator is the non-specialist without the
device, and the goal is to bring their performance toward the specialist's.
When should human-AI team performance, rather than the device operating alone, serve as the
basis for evaluation?
What counts is what reaches the patient. For a function reviewed by a specialist, the signed report is
the primary measure, obtained in routine use after authorization (cf. Q18); standalone performance,
obtained at the benchmark, is the secondary one, used for re-benchmarking (cf. Q22). A premarket
MRMC study is reserved for action-taking functions (cf. Q8). For non-directive functions, standalone
performance may be sufficient on its own.

Q15. Are there ways in which performance might be assessed relative to the care, technology, or
course of action likely to occur in the absence of the device, rather than to the comparators
described in this section?
Yes. The lower bound of Q14, the routine radiologist without the device, is precisely the course of
action in the absence of the device. Measuring the device against it measures the loss of chance for
patients if the device were not available. The device can then be scored relative to the two bounds,
as a normalized clinical gain (NCG).

A device should have an NCG above 0 to be authorized, i.e. it improves on routine practice, and aim
for 100, i.e. expert level. Where an absolute ground truth exists (biopsy, long-term follow-up), the
clinician-device team can outperform the expert panel and the NCG can exceed 100; the upper
bound is then the ground truth.
This works like a ranking examination (USMLE, board certification): a single scale positions the
candidate against its peers (routine practice) and against the reference (expert level), and the same
scale serves for admission and for periodic reassessment. This is how the comparator enters the
competency-based approach. For the clinical competencies, the benchmark assessment (cf. Q11)
and the postmarket monitoring (cf. Q18, Q19) are expressed on this scale, per competency and per
tier of findings (cf. Q1), with the pass thresholds set at authorization and reused for every
subsequent model version under the PCCP (cf. Q22, Q23).
What approaches might be used to identify and justify a comparator such as unaided clinical
judgment, delayed specialist review, or no intervention?
Of the three comparators listed, unaided clinical judgment is the primary one for a function
reviewed by a specialist: the radiologist reading and reporting the exam without the draft. Delayed
specialist review also applies to report drafting, on a second dimension: by shortening reporting
time, the device reduces the delay before the signed report reaches the referring physician, so the
gain is measured in time to diagnosis as well as in accuracy. It is the primary comparator for a nonspecialist mode (cf. Q4). No intervention is the comparator for opportunistic detection functions,
where nothing would be looked for without the device.
For unaided clinical judgment, the comparator comes for free in report drafting: every retrospective
exam carries the original report signed in routine practice without the device. It is scored against the
expert consensus on the same cases and with the same metric as the device, so both bounds come
from the same case set, with no counterfactual to model and no control arm to recruit.

Q16. Should independent third parties be involved in some or all aspects of a competency-based
approach, including device benchmarking and clinical confirmation? If so, in what ways might
qualified, independent third-party participation contribute to this approach, and what
qualifications and independence criteria should apply?
Definitely. As stated in Q10, benchmark definition will be a major pillar of these new guidelines. The
issue is that, as with the USMLE today, benchmarks require human crafting and validation, not
purely rule-based definitions. This requires much more domain knowledge, and it is practically
intractable for the FDA to validate every benchmark. We recommend delegating this validation to
expert committees, each dedicated to one domain such as radiology: in practice, medical societies
(ACR, RSNA), guideline groups and academics, similar to how the USMLE works. As noted in Q10,
such benchmarks rely on human-judged quality and take time to build. In the meantime, the
practical solution is to open most of the validation to postmarket, as a high-stakes commitment:
without a successful postmarket study, the authorization can be withdrawn (cf. Q18).
Are there aspects of the assessment for which third-party involvement would be impractical or
inadvisable?
To keep things practical, medical societies cannot evaluate each device individually. Our proposal is
that they create universal benchmarks and provide a platform where devices can be benchmarked,
like the USMLE or Kaggle. They could also define guidelines for building such benchmarks. But these
benchmarks are very expensive, so making sponsors responsible for building them is not a real
solution: it would lead either to low-quality benchmarks for companies with limited budgets, or to
favoring companies with large budgets.
What safeguards or program design features would be critical to prevent third-party participation
from limiting competition or preventing innovation?
Apply the logic of 510(k) and De Novo, i.e. use existing submissions as stepping stones for new ones.
In practice, in addition to public benchmarks and medical societies, require sponsors to publish their
benchmarks if they build new ones, or to contribute publicly to existing ones. This creates a virtuous
iterative process: well-funded sponsors still benefit from being first in class, while the path opens for
smaller sponsors.

Q17. CDRH is exploring whether a competency-based approach could be applied to devices that
incorporate a variety of underlying model architectures, such as multimodal vision-language
models and generative or predictive world models. Are there aspects of the described approach
that might be ineffective or inapplicable to these devices?
In our experience, models, whether text-only or multimodal, can be split into two categories: openoutput models, including LLMs, and closed-output models, such as classification algorithms. In this
framework, closed-output models should rely on the traditional evaluation framework, since they
cannot be evaluated on benchmarks requiring open answers, like the USMLE.
What additional or distinct considerations might need to be addressed?
Agentic systems are a bridge between these two types of models. An agentic system is basically an
open-output model (the orchestrator) that uses the outputs of closed-output models to answer its
questions. We argue this will become the standard way of integrating closed-output models into
open GenAI workflows.
A specific case is a cleared GenAI device using an external closed-output model as a tool. This raises
additional questions, as it will be a very natural use case, for example a report drafting orchestrator
using a dedicated FFRCT algorithm to measure FFR on a CT scan. At first glance, two visions conflict:
either, since both devices are cleared, their combination is cleared; or, conversely, the intended use
of the orchestrator must explicitly cover the tools it uses. Our position leans toward a hybrid. The
outputs of the tools must be within the orchestrator's intended use, but the tools do not need to be
individually cleared for the orchestrator: the orchestrator can use any cleared FFRCT tool if cardiac
imaging is in its intended use and indications for use. This comes back to Q2 and Q3: the intended
use and indications for use remain the reference, and any already-cleared model used to produce
the results remains valid.

Section VI: Postmarket Monitoring for GenAI-Enabled Devices

Q18. CDRH is considering whether it may be appropriate to accept greater premarket uncertainty
regarding a GenAI-enabled device's benefit-risk profile through greater reliance on postmarket
monitoring. Under what conditions might such an approach be appropriate, and what
characteristics of a monitoring program would need to be in place to justify reduced premarket
evidence? Are there device types or risk profiles for which this approach would not be
appropriate?
We support this direction, and consider it a good solution for GenAI devices, especially multiindication devices (cf. Q7, Q11).
What it implies, in our view. For a function reviewed by a specialist, premarket evidence is limited to
the competency benchmark on retrospective real-world exams (cf. Q8, Q11), with no premarket
MRMC or prospective study. Clinical confirmation is produced in routine use: congruence between
the device output and the signed report is measured on every exam, and followed over time to the
diagnostic ground truth where it exists (cf. Q6). This monitoring is the benchmark run on every exam
in routine use, not an additional layer. Same unit of measurement (score per finding and per tier), so
premarket and postmarket results are directly comparable, and the gap between them measures the
effect of real cases, real readers and real workflows. It is also more powerful than a one-off MRMC
study (a few hundred selected cases, about ten readers): it covers all cases, all readers and the real
distribution of sites and scanners, continuously, and for multi-indication devices it is the only way to
reach statistical power on rare findings.
Conditions, in our view. The monitoring is specified at authorization, in the same document as the
PCCP (cf. Q22, Q23): metric per finding and per tier, applied to the signed report and to downstream
ground truth, thresholds, time to first report, reporting cadence; the same thresholds then serve to
validate subsequent model versions. Collection is built into the software and runs in the background,
with no additional burden for radiologists. The authorization is conditional: if the postmarket
evidence does not meet the prespecified criteria within the agreed time, it is suspended or
withdrawn. The methodology is shared and FDA-endorsed, so that results are comparable across
sponsors (cf. Q21).
This approach is not appropriate for autonomous functions, or functions without specialist review:
there is no human reader to catch errors before they reach the patient, so premarket clinical
evidence remains necessary.

Q19. Please comment on the potential approaches to postmarket performance evaluation,
including periodic re-benchmarking, sample-based clinician review, and performance degradation
monitoring. What additional approaches should CDRH consider, and how should the cadence and
triggering events for reassessment be determined?
The three approaches are complementary, and their cadence should be tied to the data collected in
routine use rather than to a fixed calendar. Continuous congruence monitoring compares the device
output to the signed report on every exam, per tier of findings (cf. Q1), with thresholds prespecified
at authorization. Periodic re-benchmarking on the premarket benchmark is run at each change of
model, prompt or harness, and at a defined cadence otherwise. Sample-based independent clinician
review focuses on discordant cases and on critical findings. Triggering events include a model or
harness update, a new site, scanner or protocol, a drop of congruence below threshold on any tier,
and a change in the case mix.

Q20. Please comment on whether the proposed approaches to postmarket monitoring can be
facilitated by machine-based supervisory agents. What considerations, including the evaluation
and reliability of the supervisory agent itself, should CDRH take into account for such an
approach?
Yes. Machine-based judges are not specific to postmarket monitoring: the same LLM-based scoring
of findings serves the premarket benchmark (cf. Q12, RadMatch, https://arxiv.org/abs/2609.01470).
But postmarket monitoring depends on them to exist at all. Scoring every exam, continuously, across
all sites, is not feasible with human adjudication; automated scoring is what makes the monitoring
permanent, exhaustive and affordable, hence viable. The conditions are that the supervisory agent is
validated against human adjudication on a reference set, independent from the model under
evaluation, and frozen in a versioned configuration; its own performance is reported and revalidated when it changes. Human review remains in the loop on a sample of cases and on all
discordant critical findings.
Q21. What roles might clinicians, healthcare institutions, professional societies, standards-setting
bodies, and other stakeholders appropriately play in postmarket monitoring, and how can these
roles be structured without diffusing manufacturer accountability?
Healthcare institutions provide access to the downstream data (follow-up exams, pathology,
outcomes) needed for long-horizon ground truth tracking, under a data agreement defined with the
manufacturer. Clinicians produce the signed report, which is the reference for congruence
monitoring, with no additional burden. Professional societies and standards bodies maintain the
benchmarks and define the postmarket evaluation methodology (cf. Q16). Accountability stays with
the manufacturer, who operates the monitoring, reports the results and remains responsible for
corrective actions.

Q22. CDRH envisions that a manufacturer's premarket competency-based assessment might serve
as a baseline against which post-deployment modifications could be re-evaluated. How might the
extent of re-benchmarking or other evidence be scaled to the nature and expected impact of a
given modification? For example, are there categories of change that might not significantly affect
safety or effectiveness of a GenAI-enabled device, or might be appropriately managed within a
sponsor's quality management system versus requiring FDA premarket review and authorization
before implementation? Are there categories of changes that might be appropriate for inclusion in
a PCCP?
The premarket benchmark is the natural baseline, provided it is designed from the start as a
regression test to be rerun. We propose the following scaling. A prompt, harness or UI change within
the same intended use: re-benchmarking, documented in the QMS. A model retraining or new
model version within the same intended use: re-benchmarking against the premarket thresholds,
under a PCCP, with postmarket monitoring continuing on the new version. A new indication, new
modality or change of activity level: premarket submission. PCCP candidates are model versions,
retraining on new data and harness updates, provided the benchmark and acceptance criteria are
unchanged.

Q23. How might PCCP concepts or other change-control approaches be adapted for GenAI-enabled
devices when the nature or scope of future modifications cannot be fully prespecified?
The PCCP should be defined by what is verified rather than by the list of modifications: the
benchmark, the acceptance criteria per tier of findings, and the postmarket monitoring thresholds.
Any modification that passes the prespecified verification is covered, whatever its nature, as long as
the intended use is unchanged. This is consistent with how a clinician remains licensed as long as
periodic reassessment is passed, without prespecifying what they will learn.

Q24. For devices built on third-party foundation models, changes to the underlying model may be
initiated by the third-party model developer rather than the device manufacturer. How can a
manufacturer detect, evaluate, and respond to such changes in a timely manner? What
mechanisms (for example, contractual, technical, or through a PCCP) could provide reasonable
assurance that third-party developer-initiated changes do not compromise the safety or
effectiveness of the device?
Our device relies on a proprietary foundation model, so this question does not apply to us directly.
Our view is that third-party model changes are compatible with the framework described in Q23. A
PCCP defined by what is verified measures the outputs of the device, not the algorithm that
produces them. A change of the underlying model is then handled like any other modification: the
new version is run through the benchmark, and is deployed only if it passes the prespecified
acceptance criteria. The practical conditions are a pinned model version with no automatic update,
contractual notification of changes by the model developer, and re-benchmarking before any version
change reaches production (cf. Q22).

Section VII: Other Topics

Q25. Would voluntary Foundation Model MAFs be practical for foundation model developers to
provide and to keep current, and what content might they include? Given that participation would
be voluntary and that model developers may have limited incentive to disclose safety-relevant
information, what would make such a program sufficiently useful for premarket review? Are there
alternative mechanisms CDRH should consider for obtaining information about underlying
foundation models?
Yes, provided the MAF process is adapted to the release cadence of foundation models. A MAF
should be available for each model release, as a systematic part of the release process, and tied to a
specific version identifier. In practice, it is a structured model card: architecture, training data
provenance, healthcare-relevant evaluations and subgroup results, known failure modes, guardrails,
update and deprecation commitments. The current MAF process, designed for materials and
components that change rarely, should become lightweight and incremental, with a new version of
the file for each release, referencing the previous one.
This is feasible. Foundation model developers are few and well resourced, and most of this content
already exists in their model cards, so formalizing it as a MAF is a limited effort for them. It is also in
their interest. Their models already reach patients through consumer platforms with no oversight
(cf. Q3). Contributing a MAF is a tangible demonstration of goodwill toward the safe medical use of
their models, and it lowers the barrier for the devices built on them.
In any case, the sponsor remains responsible for the device as deployed, with a pinned model
version, and the MAF does not replace the benchmarking of the final device (cf. Q24).

Q26. Are there additional considerations that inform the premarket and postmarket evaluation of
agentic GenAI-enabled devices, beyond those applicable to non-agentic GenAI-enabled devices?
How should the elevated risk associated with autonomous multi-step action, tool use, and
reduced opportunity for human review be reflected in acceptance criteria and oversight?
The main need is common to agentic and non-agentic devices: a widely accepted, preferably FDAendorsed, postmarket evaluation method tracking real-world performance (cf. Q18). It collects the
device output and the human edits and overrides on all or most uses, tracks the diagnostic ground
truth over long horizons with multiple decision steps, and runs in the background, with minimal user
input, so that it is neither uncomfortable nor burdensome for users.
Three considerations are specific to agentic devices: a human checkpoint before any irreversible or
high-consequence action, such as sending a report to the PACS or the EHR; full logging of tool calls
and tool outputs, so that errors can be attributed to the orchestrator or to a tool; and acceptance
criteria on the action sequences, not only on the final output (cf. Q17 on tools and intended use).
Take agentic report drafting as an example. The agent opens the exam, retrieves the clinical
indication and the prior exams, calls measurement tools (lesion segmentation, volumetry), compares
with priors and drafts the report. The checkpoint is that the draft is only displayed to the radiologist
in the viewer, and never reaches the PACS or the RIS without their review and signature. Logging
records each step with its inputs and outputs (which priors were retrieved, which tool was called on
which series, what it returned), so that a wrong measurement in the signed report can be traced to
the tool or to the orchestrator. Acceptance criteria on sequences include the following: when a prior
exists, the agent must have retrieved it before writing any comparison; a tool output outside
plausible bounds (a 40 cm nodule) must be flagged, not copied; a request outside the intended use
(a treatment recommendation) must be refused and redirected (S.2, A.1).