HL7 Caliper Accelerator (Todd Cooper)

The HL7 Caliper Accelerator appreciates the opportunity to comment on the Center for Devices and Radiological Health’s (CDRH’s) discussion paper, Considerations for the Regulation of Generative AI-Enabled Medical Devices.
Caliper is an HL7 Accelerator Project leading an international health-device interoperability standards implementation community. From this perspective, generative artificial intelligence (GenAI)-enabled medical devices should be evaluated not only for their underlying AI capabilities, but as integrated medical devices operating within broader health-information and clinical systems.
Caliper supports a risk-based, least-burdensome, total-product-lifecycle (TPLC) approach in which the medical-device manufacturer remains accountable for demonstrating and maintaining the safety and effectiveness of the finished, configured device.
Our comments focus on regulatory evidence, configuration and change control, provenance, interoperability, and lifecycle assurance. Model cards, standardized provenance, human-oversight data, third-party evaluation, and other mechanisms can support these objectives, but none should substitute for manufacturer accountability or valid scientific evidence.

Comment of the HL7 Caliper Accelerator

Docket No.: FDA-2026-N-7874

Re: Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback

The HL7 Caliper Accelerator appreciates the opportunity to comment on the Center for Devices
and Radiological Health’s (CDRH’s) discussion paper, Considerations for the Regulation of
Generative AI-Enabled Medical Devices.

Caliper is an HL7 Accelerator Project leading an international health-device interoperability
standards implementation community. From this perspective, generative artificial intelligence
(GenAI)-enabled medical devices should be evaluated not only for their underlying AI
capabilities, but as integrated medical devices operating within broader health-information and
clinical systems.

Caliper supports a risk-based, least-burdensome, total-product-lifecycle (TPLC) approach in
which the medical-device manufacturer remains accountable for demonstrating and maintaining
the safety and effectiveness of the finished, configured device.

Our comments focus on regulatory evidence, configuration and change control, provenance,
interoperability, and lifecycle assurance. Model cards, standardized provenance, humanoversight data, third-party evaluation, and other mechanisms can support these objectives, but
none should substitute for manufacturer accountability or valid scientific evidence.

The discussion paper is intended for discussion purposes and does not establish current FDA
regulatory requirements. Accordingly, the recommendations below are offered as potential
approaches for FDA’s consideration.

Considerations for the Assessment of Risk

Question 1 — Two-axis risk framework

Caliper supports the proposed framework based on device activity and the consequence of
relying on an incorrect output as a useful organizing heuristic. The distinction between actiondirecting as a higher risk function is sound, however, this depends on the impact of the action
and potential for harm, as described in the document. It may become difficult to identify where a
GenAI-enabled device will land on the chart without clear guidance.

Time sensitivity and reversibility of the action are important considerations within the framework.
The ability of the user to understand the outputs in the appropriate context is another key
consideration and recognizing that context can change at each turn should be a consideration.
These factors should inform the manufacturer’s risk analysis and the corresponding level of
regulatory evidence required.

Question 2 — Non-directive versus action-directing information
Risk should be based on the functional effect of the output, rather than terminology alone.
Manufacturers should characterize how specificity, personalization, context, and interaction
history may cause an informational function to become increasingly directive. Manufacturers
should be intentional about creating additional guard-rails and safeguards when activities of the
product move into the upper-right side of the two-axis framework. Intended-use documentation
and validation should address foreseeable movement along this continuum.

Question 3 — Patient-facing functions

Patient-facing use should be considered a contextual risk modifier rather than an automatic risk
category. Manufacturers should demonstrate that safeguards, labeling, uncertainty
communication, escalation mechanisms, and other controls are appropriate for the intended
user's ability to evaluate and act upon the output. Examples may include providing educational
resources or easy access to specialist/professionals (or specialist GenAI).

Question 4 — Generalist versus specialist users

The relevant consideration should be whether safe use depends on specialized knowledge
necessary to independently evaluate the device output. Where specialist expertise is an
important risk control, manufacturers should identify that dependency in the intended use, risk
analysis, human-factors evaluation, and labeling.

Question 5 — Multi-turn conversations

Manufacturers should evaluate realistic interaction trajectories, rather than isolated prompts and
responses. Testing should assess whether cumulative interactions cause the device to move
outside its intended use, become increasingly directive, or cross predefined safety or authority
boundaries. Ideally, this should extend to post-market surveillance as GenAI models are not
static.

Question 6 — Under-escalation and over-escalation

Under-escalation and over-escalation should be evaluated as distinct failure modes.
Manufacturers should prespecify clinically justified performance measures and acceptance
criteria for each rather than combining them into a single metric when their consequences are
not commensurable. This is an important element of planning and evaluation as trust can easily
be degraded in the healthcare ecosystem and the implications of appropriate escalation can
have serious consequences in certain circumstances.

Competency-Based Premarket Evaluation

Question 7 — Benchmarking followed by clinical confirmation

Caliper supports a competency-based framework consisting of device benchmarking followed,
where appropriate, by clinical confirmation. The regulatory unit of assurance should be the
finished device as configured for deployment, rather than the foundation model. Depending on
the device, the validated configuration may include the foundation model and version, prompts,
retrieval sources, guardrails, orchestration logic, connected tools, user interface, interoperability
components, and human-oversight mechanisms.
Question 8 — Relationship between risk and evidence

The level of evidence should increase according to the two-axis framework and the following
factors: clinical consequence; degree of independence or autonomy; difficulty detecting
incorrect outputs; irreversibility of resulting actions; and residual uncertainty after risk controls
are applied.

FDA should leave flexibility for manufacturers to justify an evidence package appropriate to the
device's intended use and risk.

Question 9 — Benchmarking structure

FDA's proposed benchmarking elements provide a strong starting point. Manufacturers should
document, as applicable; intended competencies; test datasets; prespecified evaluation
methods; scoring rubrics; acceptance criteria; adjudicator qualifications and independence;
known limitations; and the device configuration and version tested.

Caliper recommends that FDA consider a structured, versioned regulatory evidence summary,
potentially informed by model-card or system-card practices, to organize this information
consistently and in line with expectations of markets outside of the US. Such an artifact should
summarize and link to the underlying regulatory evidence; it should not replace test reports, riskmanagement documentation, clinical evidence, quality-system records, labeling, or other
applicable submission information.

Question 10 — Sponsor-developed benchmarks

Sponsor-developed benchmarks should be permitted when public or third-party benchmarks are
not fit for the device's intended purpose or if there are concerns about contamination of data.
Akin to the concept of a certified test phantom or a representative data holdout for ML validation
manufacturers should demonstrate construct validity; relevance to intended use; representative
or appropriately challenging test conditions; prespecified methods and acceptance criteria;
controls addressing contamination and optimization to the test; and appropriate independent
adjudication or confirmation.

Question 11 — Clinical confirmation

Clinical confirmation should be proportionate to device risk, novelty, and residual uncertainty.
FDA should permit a range of approaches, including retrospective evaluation, simulation,
shadow deployment, independent adjudication, real-world evidence, and prospective studies
where warranted, rather than defaulting to prospective clinical trials for all GenAI-enabled
devices.

Question 12 — Statistically meaningful performance

Manufacturers should prespecify statistically and clinically justified performance measures
appropriate to the characteristics of the device and its outputs. Synthetic and real-world
evidence should not automatically be combined unless the relationship between their
distributions and their intended inferential roles supports doing so.
Question 13 — Synthetic data

Synthetic data may be useful for rare scenarios, stress testing, edge cases, and difficult-tosample conditions. Manufacturers should demonstrate that synthetic data do not reproduce the
same biases, omissions, or performance gaps the evaluation is intended to detect. Synthetic
data should supplement, rather than automatically substitute for, representative clinical
evidence.

Question 14 — Comparators and acceptance criteria

Comparator selection should be specific to the intended use and regulatory question.
Appropriate comparators may include established reference standards; applicable clinical
guidelines; qualified expert consensus; usual care; unaided clinician performance; or human-AI
team performance.

Clinical guidelines and professional-society recommendations can provide useful grounding
where applicable but should not be universal comparators for all GenAI-enabled devices.
Consensus driven, pre-specified prompt cascades that test edge-cases and human factors, held
out from device development and initial testing, may be one consideration.

Question 15 — Comparison with care in the absence of the device

Where appropriate, manufacturers should be able to compare device performance with the
care, technology, or course of action likely to occur without the device. This may include usual
care, another technology, delayed specialist review, unaided clinical judgment, or no
intervention when that comparator provides the most meaningful assessment of benefit and risk.

Question 16 — Independent third parties

Many companies have developed domain specific benchmarks to compare performance for
hyper scaling GenAI related to different tasks, including healthcare (ex. Mercor). Independent
third parties may contribute through providing sequestered evaluation datasets; benchmark
development; testing; qualified evaluation tools; and expert adjudication. Their qualifications
should match the technical and clinical requirements of the assessment, and appropriate
independence should be demonstrated.

Third-party evaluation should supplement, not transfer, manufacturer accountability for the
validity and completeness of the evidence submitted to FDA.

Question 17 — Multimodal and other model architectures

A competency framework can apply to multimodal and imaging-enabled devices, but evaluation
should address the complete intended device function. For example, relevant evidence may
need to address steps along the pathway:

source data → model interpretation → generated output → user interaction → resulting action

rather than evaluating image-level or foundation-model performance alone.
Postmarket Monitoring

Question 18 — Premarket uncertainty and postmarket monitoring

Greater reliance on postmarket monitoring may be appropriate in some circumstances but
should not automatically justify reduced premarket evidence. The appropriateness of such an
approach should depend on the severity and reversibility of potential harm; detectability of
failure; effectiveness of human or technical safeguards; ability to intervene rapidly; and strength
and timeliness of the monitoring system. Greater caution is warranted for high-consequence or
autonomous functions.

Question 19 — Postmarket performance evaluation

Caliper supports risk-proportionate use of periodic re-benchmarking; regression testing;
performance-degradation monitoring; sampled expert review; complaint and adverse-event
analysis; and reassessment following material configuration changes. Where appropriate,
premarket competency tests should become reusable postmarket regression tests, creating
continuity between premarket evidence and lifecycle assurance. These may be described in the
PCCP. Standardized provenance can further strengthen the manufacturer's ability to reconstruct
and investigate postmarket events.

Question 20 — Machine-based supervisory agents

Supervisory AI agents may be helpful to support postmarket monitoring, but they should
themselves be appropriately evaluated for their intended monitoring function. Manufacturers
should not rely on a supervisory agent without evidence regarding its reliability, sensitivity to
relevant failures, limitations, and potential susceptibility to the same failure modes as the
monitored system. Care should be taken to ensure that the supervisory agent has distinct goals
and environments to reduce the chance of moat jumping for alternative goals to their intended
role.

Question 21 — Shared ecosystem responsibility

Healthcare institutions, clinicians, professional societies, standards-development organizations,
and other stakeholders can contribute data, signals, expertise, and infrastructure to postmarket
monitoring. However, manufacturer accountability should remain clear. The manufacturer should
remain responsible for determining whether postmarket signals affect the device's safety,
effectiveness, validated state, or regulatory status.

Clinician or patient approval, rejection, override, or modification of device outputs may provide
useful observational evidence when appropriate, but clinician agreement is not equivalent to
correctness and should not substitute for manufacturer-led performance assessment.

Question 22 — Re-evaluation following modification

The manufacturer's premarket competency assessment should provide a baseline for
evaluating subsequent modifications. The extent of re-evaluation should be proportional to the
components changed; expected behavioral impact; affected hazards and risk controls; intendeduse implications; and potential effect on safety and effectiveness.
Depending on these factors, it is reasonable that changes may appropriately be managed within
the manufacturer's QMS, within an authorized PCCP, or may reasonable require additional FDA
review.

Question 23 — PCCPs for changes that cannot be fully prespecified

Caliper recommends that FDA consider whether PCCPs for GenAI-enabled devices can define
a ‘validated change envelope’ when individual future modifications cannot be predicted
precisely. A validated change envelope could specify the permissible categories and boundaries
of change; required verification and validation methods; acceptance criteria; cumulative-change
assessment; post-implementation monitoring; and what triggers an additional FDA review. This
approach should be consistent with established PCCP principles that changes be focused,
bounded, risk-based, evidence-based, transparent, and managed from a TPLC perspective.

Question 24 — Third-party foundation-model changes

It is expected that manufacturers will build on third party foundation models due to resource
scarcity and time to market. Manufacturers incorporating third-party foundation models should
maintain sufficient contractual, technical, and regulatory controls with those parties to:

1. identify the model and version incorporated into the device;

2. detect material changes;

3. understand the safety-relevant nature of those changes;

4. assess their impact on the validated device;

5. perform appropriate regression testing or revalidation; and

6. determine whether regulatory action is required.

Relevant supplier controls may include change notification, persistent version identification,
access to relevant evaluation information, audit and investigation support, rollback capability,
and information necessary for regulatory-impact assessment. Use of a third-party foundation
model should not transfer accountability away from the medical-device manufacturer.

Other Topics

Question 25 — Foundation Model Device Master Files (MAF)

Caliper supports further exploration of Foundation Model Device Master Files. As with AI Model
Cards, such files could reduce duplicative review and provide FDA and authorized downstream
manufacturers with structured information regarding model and version identity; relevant
training-data provenance; known limitations and failure modes; healthcare-relevant evaluations;
clinically relevant subgroup performance; safety controls and guardrails; update and version
history; audit capabilities; and change-notification commitments. As noted, incentives for
foundation model developers to contribute to these efforts and maintain their accuracy may be a
challenge to create.
However, a Foundation Model Device Master File should not constitute authorization of the
foundation model for a medical-device intended use, nor should it replace the manufacturer's
obligation to demonstrate the safety and effectiveness of the finished device.

Question 26 — Agentic GenAI-enabled devices

This is a brave new world. GenAI is already embedded in many health systems and more and
more public interactions. Regulatory expectations should increase with autonomy, authority,
clinical consequence, and irreversibility of action, rather than simply with designation of a
system as “agentic.” As the opportunity for meaningful human review decreases, the
evidentiary threshold and strength of technical controls should correspondingly increase.

Cross-Cutting Recommendation: Regulatory-Grade Provenance and Interoperability

Across Questions 9, 19, 21, 22, 24, and 25, Caliper recommends that FDA consider
standardized provenance and interoperability as enabling infrastructure for lifecycle assurance.

Where GenAI-enabled devices exchange FHIR data, approaches such as the HL7 AI
Transparency on FHIR Implementation Guide may support standardized representation of:

• AI-influenced data;

• AI system and version identity;

• human and automated participants;

• provenance;

• relevant inputs and prompts; and

• linkage to model or system information.

These capabilities can support regulatory objectives involving traceability, observability,
postmarket evidence, and reconstruction of safety-relevant events.

Caliper does not recommend that FDA require FHIR for every GenAI-enabled medical device.
Applicability should depend on the device architecture, data exchange, intended environment,
and regulatory purpose. Similarly, AI Transparency on FHIR provides infrastructure for
representing AI involvement and provenance but it should not itself be considered evidence that
an AI system is safe or effective.

Conclusion

Across FDA's 26 discussion questions, Caliper recommends one consistent regulatory principle:

The medical-device manufacturer should remain accountable for demonstrating and
maintaining the safety and effectiveness of the GenAI-enabled medical device
throughout its total product lifecycle.

Competency testing, clinical confirmation, structured model or system information, standardized
provenance, human oversight, third-party evaluation, postmarket monitoring, Foundation Model
Device Master Files, and PCCPs should be viewed as tools supporting that manufacturer
assurance obligation, rather than substitutes for it.

Caliper encourages FDA to develop an approach that:

1. remains proportional to intended use and risk;

2. evaluates the finished device as configured and deployed;

3. supports fit-for-purpose, scientifically valid competency evidence;

4. uses clinically representative confirmation where warranted;

5. treats human oversight as a testable risk control;

6. enables standardized provenance and traceability where appropriate;

7. maintains clear manufacturer accountability across third-party dependencies;

8. integrates premarket evidence, postmarket monitoring, and change control; and

9. applies stronger evidence and control expectations as autonomy and potential clinical
consequence increase.

This approach would preserve FDA's risk-based and least-burdensome principles while
providing a coherent regulatory framework for devices whose models, configurations, data
dependencies, and behaviors may evolve over time.

Caliper appreciates FDA's early engagement on these issues and welcomes continued
collaboration among regulators, medical-device manufacturers, foundation-model developers,
healthcare organizations, clinicians, patients, researchers, and standards-development
organizations.

Respectfully submitted,

Todd Cooper
Technical Program Director, Caliper
Member of HL7 International
todd.cooper@caliper.hl7.org

References

1. U.S. Food and Drug Administration, Center for Devices and Radiological Health.
Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback. August 2026. Docket No. FDA-2026-N-7874.
https://www.fda.gov/media/194242/download

2. U.S. Food and Drug Administration. Marketing Submission Recommendations for a
Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software
Functions. August 2025. https://www.fda.gov/regulatory-information/search-fdaguidance-documents/marketing-submission-recommendations-predetermined-changecontrol-plan-artificial-intelligence
3. HL7 International. AI Transparency on FHIR Implementation Guide.
https://hl7.org/fhir/uv/aitransparency/

4. National Institute of Standards and Technology. HL7 AI Transparency on Fast Healthcare
Interoperability Resources (FHIR). https://www.nist.gov/programs-projects/hl7-aitransparency-fast-healthcare-interoperability-resources-fhir