SMART Health IT at Boston Children's Hospital (Ken Mandl, Dan Gottlieb, Josh Mandel)

See attached file(s)

SMART Health IT
Computational Health Informatics Program
Boston Children's Hospital

October 19, 2026

Re: Docket No. FDA-2026-N-7874, Considerations for the Regulation of Generative
AI-Enabled Medical Devices: Discussion Paper and Request for Feedback

Submitted on behalf of the SMART Health IT team (the team that originated and
maintains the SMART on FHIR ecosystem)

Dear CDRH and Digital Health Center of Excellence Leadership,

Thank you for the opportunity to comment. We share CDRH's objective of a risk-based,
least burdensome approach to generative AI (GenAI)-enabled devices, and we agree
that premarket testing alone cannot fully characterize these devices.

Our comments focus on the interfaces through which a GenAI-enabled device reads a
patient's record and, where appropriate, returns output to clinical workflows. HL7 Fast
Healthcare Interoperability Resources (FHIR) defines a standard data format for
exchanging clinical data. The SMART Health IT team at Boston Children's Hospital
originated and led the development of SMART on FHIR, the open standard for
connecting applications to electronic health records (EHRs) that federal regulation now
requires of every certified EHR, and the FHIR Bulk Data Access specification (Bulk
FHIR) for population-scale export. We also originated CDS Hooks, the HL7 specification
for invoking decision support services from within EHR workflows.

These interfaces are relevant to the paper's proposed evaluation and monitoring
approaches, which depend on retrieving health record data in a form that can be tested,
sampled, and audited. Many of these standards are required under the Health IT
Certification Program of the Office of the National Coordinator for Health Information
Technology (ONC), though performance remains uneven across vendors. We address
the discussion questions by number. The FHIR tools we cite below are publicly available
and open source.

Part 1. Premarket evaluation (Section V)

1) The interface that delivers clinical data is part of the deployment context. The
paper proposes evaluating "the final user-facing device, as configured and intended to
be deployed" (Section V.A), with benchmark datasets "representative of the deployment
contexts" (Section V.B.2). For a device that reads a patient's chart, the form in which the
chart arrives is part of that context. A device that ingests a record through SMART on
FHIR receives US Core FHIR resources, in the versions ONC adopts for certification,
with known structure, terminology bindings, and known gaps. A device that ingests a
proprietary extract, a screen capture, or pasted text receives something different. A
device tested on SMART on FHIR data may not perform the same way on a proprietary
extract, and the reverse.

Recommendation. Where a device consumes the record through a
standards-based interface, CDRH should encourage sponsors to state the
interface in the device description, with any limitations encountered, and to
construct benchmarking and clinical-confirmation inputs in a form consistent with
nationally certified EHR standards, namely US Core FHIR resources delivered
through SMART on FHIR or Bulk FHIR. Open-source tools exist for this. Our
Cumulus tools turn a Bulk FHIR export into a de-identified dataset for
population-scale analysis, and our CumulusQ tools compute FHIR-native data
quality and characterization metrics on the export, so that a sponsor can show
what a benchmarking dataset contains and where it is incomplete (JAMIA, 2024).
(Questions 9, 10, 11)

2) A certified route to population-scale real patient records exists. Several
proposals presume that a sponsor or institution can assemble the records of a large,
defined population: retrospective evaluation on real inputs, shadow deployment,
subgroup performance (element R.2), and estimation of the "anticipated distribution of
real-world inputs" (Question 11). A mechanism for this exists and is not as widely known
as it should be.

Bulk FHIR is the HL7 standard for exporting the records of a population rather than one
patient at a time. A requester names a group of patients and receives their records as
files of FHIR resources in the same US Core form that SMART on FHIR delivers for a
single patient. SMART Backend Services provides a standardized mechanism for
authorizing unattended applications to retrieve these exports. Our team developed the
specification with support from ONC, and since December 31, 2022, certified EHRs
have been required to support it for the USCDI data elements under 45 CFR
170.315(g)(10). The Centers for Medicare & Medicaid Services (CMS) uses the same
standard in three of its own APIs: the Beneficiary Claims Data API, which delivers
Medicare claims to accountable care organizations; Data at the Point of Care, which
delivers them to treating providers; and AB2D, which delivers Medicare Part A and Part
B claims to Part D plan sponsors. Claims delivered this way can be joined to the EHR
record, and the combination matters for monitoring: readmissions, emergency visits,
and prescriptions filled often appear in claims and not in any one institution's record.
Bulk FHIR is the route by which a sponsor could draw benchmarking inputs from a
health system's actual population, an institution could retrieve the records of the patients
a device touched, and a third party could deliver a sequestered dataset in a form the
device already consumes.
The capability is uneven in practice. A multi-vendor evaluation published in 2024 tested
five sites in 2023 and found that at three of the sites, exports of about a thousand
patients took 26 to 35 hours, that one attempted export of about 10,000 patients ran for
nearly two weeks without a successful download, that date filtering was supported at
some sites and not others, and that sites needed from two days to four months of setup
before a first request could be made. The paper's interest in synthetic data (Questions
12 and 13) is understandable in this light. Synthetic records are most valuable where
they cover cases real data cannot, and least valuable as a stand-in for real populations
that certified interfaces are already required to deliver. Generating synthetic records as
US Core FHIR resources lets the same metrics compare synthetic and real data directly.
Bulk FHIR has two further limits. First, an export can finish and still be missing data. Our
BulkUP tool, from the EHR Good Neighbor project, measures export speed and checks
which data are present. Second, an export taken later shows the chart as it is now, not
as it was when the device ran. To know what the device saw, its inputs have to be saved
each time it is used (comment 3).

Considerations. ONC sets certification requirements for these interfaces, so we
do not propose new technical requirements. We make three points. First, FDA's
evidence expectations already depend on ONC's certification requirements. The
data FDA would want sponsors to use includes the data ONC requires certified
EHRs to export. If the export does not work, producing the evidence becomes far
more costly. Work to make standardized extraction more performant is under
way: the CMS Health Technology Ecosystem includes a Bulk FHIR pledge, under
which EHR vendors, health systems, and payers commit to usable performance
at the scale of hundreds of thousands to millions of records and to transparent
performance measurement, and a workgroup on implementation. The framework
should say this, so that sponsors know to ask for the export and so that FDA
sees where it falls short. Monitoring also depends on capabilities that certification
does not yet require, notably a standard way for a requester to dynamically
define a cohort, such as the patients on whom a device was invoked. Second, an
FDA demand signal for this interface would matter beyond GenAI-enabled
devices. Programmatic access to EHR data at population scale is one resource
that serves many national priorities at once: AI evaluation and monitoring, quality
measurement, public health reporting, clinical research, and surveillance of drugs
and devices. Each has been pursued with its own pipeline, built and paid for
separately, when they could instead all draw on the same certified export. In
2022 we convened federal agencies, payers, providers, and vendors on exactly
this point (Multi-solving Population Data Use with SMART Bulk FHIR Access).
The finding was that the export becomes a common data source for all of these
uses once the agencies that depend on it say so. At that meeting Amy
Abernethy, formerly FDA's Principal Deputy Commissioner, described FDA as the
demand signal for high-quality datasets. CMS has sent its signal through its own
APIs and, more recently, the pledge. A statement from CDRH that its evidence
expectations for GenAI-enabled devices rest on this interface would be the signal
that reaches device sponsors, and it would strengthen every other use of the
 same data. Third, our team can brief CDRH staff on Bulk FHIR. (Questions 11,
12, 13)

Part 2. Postmarket monitoring (Section VI)

3) Postmarket monitoring needs a retrievable record of each invocation, and CDS
Hooks provides the foundation for capturing it. Shadow deployment (Section V.C.1),
sample-based clinician review, and degradation monitoring (Section VI.A) all
presuppose that an institution or sponsor can retrieve what the device was given at the
moment it ran, what it generated, and what the user did, and link that interaction to
downstream patient outcomes. Today, the interaction record usually sits in the
manufacturer's proprietary telemetry, separated from the clinical outcomes in the health
system's EHR.

HL7's CDS Hooks, originated by our team and widely implemented across EHRs,
already defines the standard structure for workflow-integrated decision support: a
structured request carrying patient context and prefetched FHIR data, a response
returning informational or actionable suggestion cards, and a feedback endpoint
recording whether the clinician accepted or overrode the suggestion with reasons. ONC
certification (45 CFR 170.315(b)(11)) also requires EHRs to export clinician feedback
data for certain decision support interventions. To serve full GenAI postmarket
monitoring, this foundation needs two extensions: persisting the exchange and user
action (including ignored outputs) as a retrievable FHIR audit record in the EHR, and
extending the pattern to GenAI interactions beyond synchronous alerts, such as
ambient documentation and multi-turn chat.

Recommendation. CDRH should encourage sponsors of workflow-triggered
GenAI devices to integrate via CDS Hooks and to retain, for each GenAI
invocation, a structured record of the inputs as received (or versioned FHIR
references resolving to the chart at that moment), the output, the model and
prompt version, and the user action, in a form the deploying institution can
retrieve through its standard FHIR interfaces.

Persisting this invocation record in FHIR, so that a Bulk FHIR export can return it
alongside the rest of the chart, addresses the retrospective gap noted in
comment 2: the invocation record preserves what the device saw at the moment
of use, while periodic Bulk FHIR exports supply downstream clinical outcomes
(Question 19) and allow data-quality tools to detect input or environment drift
(Question 22). It also lets institutions audit devices locally or run federated
analyses without relying on vendor logs, while letting manufacturers monitor
devices against standard exports the institution already supports (Question 21).
Where CDRH accepts greater premarket uncertainty in exchange for postmarket
monitoring (Question 18), it could ask sponsors to show that invocation records
can be captured, retrieved, and linked to outcomes through standard interfaces
at intended deployment sites, and coordinate with ONC to align certified
decision-support feedback exports with CDS Hooks and Bulk FHIR. (Questions
 18, 19, 21, 22)

4) For devices that act on the record through SMART on FHIR, authorization
scopes bound what the device can do through that connection. The agentic
element (A.1) asks whether an agent recognizes when an action would exceed its
safety envelope. For an application connected through SMART on FHIR, part of the
envelope is enforced at the EHR rather than by the model. SMART App Launch scopes
determine whether the application can read or write and which resource types it can
touch. An application granted read-only scopes cannot place an order through that
connection, however it reasons.

Recommendation. For EHR-integrated GenAI devices and agents, CDRH
should expect sponsors to document the scopes the device requests and why,
and should treat scope minimization as evidence relevant to the
scope-maintenance element (S.2) and the agentic element (A.1). For
clinician-supervised workflows, CDS Hooks can present suggested actions for
clinician review instead of having the agent write orders autonomously.
(Questions 1, 26)

5) Patients are connecting their records to GenAI tools through the
patient-access API, and that access changes the risk. Question 3 asks what might
mitigate risk when a patient cannot independently evaluate an output. Under the ONC
Cures Act Final Rule, certified EHRs must offer patients a standards-based API built on
SMART on FHIR (45 CFR 170.315(g)(10)), through which a patient can direct their
record to an application of their choosing, and patients are using it to connect their
records to general-purpose assistants. A function working from the patient's actual
medication list, laboratory results, and problem list is answering a different question
from one working from the patient's recollection. In the paper's insulin-dose example,
access to documentation on the current dose, recent glucose values, and renal function
changes the risk.

Recommendation. CDRH should treat authorized access to the patient's
structured record as relevant to the risk assessment of patient-facing functions,
and should be explicit that the patient-access API and an EHR's conformance to
it are not device functions, so that device oversight does not become a reason to
restrict patients' access to their data. (Questions 3, 5)

Closing

The paper asks whether CDRH should accept more premarket uncertainty in exchange
for stronger postmarket evidence (Question 18). Whether that evidence can be
produced is in large part a question of interfaces. Our comments reduce to three points:

●​ For devices that read the health record, the standards-based interface is part of
the deployment context, and test inputs should take the same form.
●​ Bulk FHIR is the certified route to population-scale real patient records. CDRH
should know where it performs and where it does not, and coordinate with ONC.
 ●​ Postmarket monitoring needs a structured, retrievable record of each invocation,
and a sponsor proposing to rely on monitoring in place of premarket evidence
should show that the record can be captured and retrieved at the intended
deployment sites.

We appreciate the opportunity to comment and would welcome continued engagement.

Respectfully submitted,

Ken Mandl, MD, MPH, Dan Gottlieb, MPA, and Josh Mandel, MD, on behalf of the
SMART Health IT Team
Computational Health Informatics Program
Boston Children's Hospital