OHDSI Medical Devices Working Group (Junhwan Kwon)

We appreciate the opportunity to comment on the discussion paper.

Please see the attached comments from contributing members of the OHDSI Medical Devices Working Group, submitted to Docket FDA-2026-N-7874 in response to Questions 10, 11, 15, 19, 21, 22, 23, and 24 of the discussion paper.

Our central recommendation is to treat three claims as distinct and linked: that a configuration was evaluated, that an implementation was available at a site, and that an implementation served a patient or request. Differences between evaluated and executed configurations should be documented, with justification for applying evaluation results to clinical practice or to a changed device.

The OMOP CDM v5.4 schema does not define a standardized representation linking an AI device’s complete controlled configuration to each patient- or request-time execution. We ask FDA to provide minimum metadata examples connecting these records to clinical data, to support benchmark validity, clinical confirmation, and postmarket evaluation. We also request GenAI-specific comparator examples that distinguish absence of device access, absence of recorded use, and use of alternative AI tools.

Thank you for considering these comments.

These comments reflect the individual views of the named contributors and do not represent the official position of the OHDSI community or their employers or affiliated institutions.

Corresponding contact: Junhwan Kwon, Ph.D., kjh0442@yuhs.ac.

October 19, 2026

Digital Health Center of Excellence
Center for Devices and Radiological Health
U.S. Food and Drug Administration
Submitted electronically via Regulations.gov

Docket No. FDA-2026-N-7874, Considerations for the Regulation of Generative AI-Enabled Medical
Devices: Discussion Paper and Request for Feedback

Dear Digital Health Center of Excellence:
We, the members of the OHDSI Medical Devices Working Group (MDWG) named below, appreciate the
opportunity to respond to the discussion paper. Our joint comments draw on the OMOP Common Data Model and
OHDSI network methods to connect device evaluation with evidence from clinical practice. The OMOP CDM
v5.4 schema does not define a standardized representation linking an AI device's complete controlled
configuration to each patient- or request-time execution; our recommendations address this linkage gap.
We respond to Questions 10, 11, 15, 19, 21, 22, 23, and 24. Our central recommendation is to treat three claims as
distinct and linked: that a configuration was evaluated, that an implementation was available at a site, and that an
implementation served a patient or request. Recording the version in use is not sufficient on its own; differences
between the evaluated and executed configurations need a documented justification before benchmark results are
carried into practice or used to support a changed device.
We ask FDA to address the following in future guidance, examples, and supporting templates:
1. Provide a minimum metadata example connecting device and configuration identifiers to benchmark,
deployment, and execution records.
2. Specify a benchmark-validity evidence package, including justification for differences between evaluated and
executed configurations and independently governed checks.
3. Define claim-appropriate capture of generated, displayed, reviewed, and used outputs, with transparent
reporting of inferred or unresolved identity.
4. Provide comparator examples for assessing care without the device that distinguish absence of device access,
absence of recorded use, and use of alternative AI tools.
5. Provide risk-based reassessment examples that distinguish research-only changes, operational device changes,
and marketing-submission decisions.
6. Define core version-identification, change-notification, timing, and audit-log information for voluntary
Foundation Model Device Master Files (MAFs) used in version-specific review.

We would welcome the opportunity to discuss these recommendations with FDA and to explore testing the
proposed metadata profile in OMOP-converted data sources.

Sincerely,

Contributing members of the OHDSI Medical Devices Working Group
MDWG edit leads: Asiyah Lin; Junhwan Kwon

Page 1 of 12
Corresponding contact:

Junhwan Kwon, Ph.D.
Research Assistant Professor
Anesthesia and Pain Research Institute, Yonsei University College of Medicine
Yonsei Institute for Digital Health, Yonsei University
50-1 Yonsei-ro, Sinchon-dong, Seodaemun-gu, Seoul 03722, Republic of Korea
E-mail: kjh0442@yuhs.ac

Contributors:
Junhwan Kwon, Yonsei University College of Medicine (kjh0442@yuhs.ac)
Asiyah Lin, OHDSI (asiyah@ohdsi.org)
Robert Nickl, Johns Hopkins University School of Medicine (rnickl@jhu.edu)
Rabia Younis, WCG Clinical (Drarain89@gmail.com)

These comments reflect the views of the contributing members of the OHDSI Medical Devices Working Group
named above. They do not represent the official position of the OHDSI community as a whole, and they
represent the individual views of the contributors rather than those of their employers or affiliated institutions.

Summary of responses
Question Topic Requested FDA action
Specify configuration-linked validity evidence and independent checks of
10 Benchmark validity
sponsor-developed benchmarks.
Provide a minimum capture checklist matched to the clinical claim and
11 Clinical confirmation
confirmation method.
Provide GenAI-specific comparator examples addressing uncertain exposure
15 Care without the device
and alternative AI access.
Define risk-based cadence, change triggers, response times, and research19 Postmarket reassessment
versus-operational distinctions.
Develop an interoperable metadata example and assign evidence-custody and
21 Stakeholder roles
signal-response roles.
Provide change-to-test examples based on affected competencies,
22 Re-benchmarking
interactions, and cumulative risk.
Clarify sufficiently bounded changes and when additional submission or
23 Change control
review is needed.
Specify minimum version, notification, timing, and log information for
24 Third-party models
voluntary MAF-based reliance.

Page 2 of 12
Linking evaluated configuration, deployment, and execution
The discussion paper frames evaluation around the final user-facing device and contemplates benchmarking
in a deployed or representative configuration. [1, Sections V.A-V.B] We propose one linked evidence chain
with three distinct claims: a configuration was evaluated; an implementation was available at a site; and an
implementation served a patient or request. Each study should identify the links needed for its clinical
question. Benchmark assets, scoring criteria, and device configurations require separate version identifiers.
Differences between the evaluated and executed configurations should have a documented justification for
carrying the assessment forward.

Minimum metadata example
The following fields provide a practical starting point for FDA examples. Capture detail should follow the
intended claim, device risk, and local data availability. [1,2] FDA examples should distinguish a small core
set of fields needed for the specified linkage and clinical claim from optional fields scaled to device risk,
with the aim of limiting additional implementation burden for smaller manufacturers and community sites
and reusing existing systems where feasible. Where request-level capture is not feasible, site-availability
records with documented limitations can still support analyses of device introduction or access (see Question
15).
Evidence layer Custodian Metadata to retain
Product and applicable device identifier; controlled configuration
Evaluated ID; model/provider version; prompt, retrieval, guardrail, and
Manufacturer
configuration routing versions as relevant; benchmark and scoring versions;
assessment reference.
Site and service instance; configuration or endpoint offered;
Institution and service
Site availability activation and retirement times; routing or fallback settings; known
provider
or uncertain effective interval.
Local encounter/request key; execution timestamp and time zone;
Patient/request Holder of execution configuration and provider-version evidence; input/output or log
execution records reference; generation, display, review, and use status where
captured.
Data source and extraction date; transformation, phenotype,
Data partner and study
Clinical analysis cohort, and analysis versions; exposure definition, comparator,
team
follow-up, outcomes; quality and missingness diagnostics.
Crosswalk among the above records; confirmed, inferred,
Relevant record unresolved, or conflicting identity; basis for assignments;
Linkage and custody
custodians notification, effective, and first-observed times; access and
retention arrangements.

We propose automatic metadata capture at source, supplemented by proportionate user verification when
required fields cannot be captured automatically. An institution-governed repository or device archive can
hold records that do not fit the EHR, with access controls and local linkage keys. OHDSI partners across
multiple countries can execute a shared, versioned protocol locally, using open-source analytic code, and
return aggregate results, data-quality diagnostics, and, where suitable, empirical calibration, without moving
patient-level data. [4-7] In OMOP CDM v5.4, DEVICE_EXPOSURE includes device, identifier, timing, and
record-source fields, although its documented conventions focus on physical devices. NOTE can hold AIassisted text incorporated into clinical notes. The optional NOTE_NLP.nlp_system field can record the name
and version of the term-extraction system; it does not by itself identify the system that generated the source
note. [11] We propose explicit links from clinical records to AI-device configuration and execution records,

Page 3 of 12
initially through the crosswalk described above and, where justified, through a future CDM representation
considered by the OHDSI community. A source-feasibility assessment should establish access to the inputs,
outputs, and interaction records required by the study. [2,11]

Illustrative case: an unannounced supplier-model update
In this hypothetical example, a regulated LLM-based device function supports patient-specific discharge
medication and follow-up recommendations for clinician review. The device is evaluated as configuration C1,
using supplier model M1, prompt P1, retrieval snapshot R1, and guardrails G1, against benchmark B1 and scoring
version S1. The supplier then changes a hosted endpoint to model M2 before the manufacturer receives notice.
Evaluated. The original report remains linked to C1/B1/S1. A C2 record identifies M2 and any other changed
components. A documented assessment links the change to affected tests, such as medication omissions,
unsupported recommendations, and preservation of clinician-approved follow-up instructions.
Available. Site and provider records identify when the endpoint was offered, which wards used it, and whether
routing or fallback differed. The visible product label alone does not establish the timing or scope of the supplier
change.
Executed. Request records identify which model served an encounter and link the generated output to its
displayed or clinician-edited version. If timing is uncertain, the last confirmed M1 state and first observed M2
state define the available bounds. Encounters lacking request-level evidence remain separately classified.
With these records, partners can examine omission and correction rates by configuration and setting, preserve the
original benchmark, and retest clinically relevant cases. Without them, a rise in corrected discharge notes remains
a signal to investigate, rather than evidence attributable to M2.

Page 4 of 12
Question 10. Predictive and construct validity of benchmarks
FDA question 10 (Section V.E; see Section V.B, Device Benchmarking, and Section V.D.2): CDRH recognizes that
publicly available benchmarking assets may be subject to data contamination, saturation, and limited real-world
representativeness. How should a sponsor establish that performance on a given benchmark predicts safe and effective
real-world behavior for the device’s intended use? What evidence should support the construct validity of a benchmark
used to gate device evaluation, and what role should sponsor-developed benchmarks play given potential concerns around
independence and optimization to the test?

Request to FDA. We ask FDA to include a benchmark-validity evidence template in future guidance: the
intended competency, benchmark and scoring versions, evaluated configuration, prespecified clinical
measures, execution linkage, and justification for configuration differences. The template should also address
the governance of sponsor-developed benchmarks.
Predictive validity. The question is whether benchmark results predict the intended real-world behavior.
Sponsors should prespecify that relationship, support it with premarket evidence, and reassess it as
postmarket evidence accumulates. Network characterization before authorization can identify gaps in case
mix, comorbidity, and care settings. Representative sampling should be distinguished from deliberate
enrichment for uncommon safety-critical scenarios. [1, Sections V.B-V.C; 2]
Network assessment. Data partners should compare competency-specific benchmark results with
corresponding clinical measures for an identified executed configuration or a justified bridge from the
evaluated configuration. Subgroup or site discrepancies should trigger investigation of configuration, input
distribution, workflow, data capture, and benchmark coverage before any revision. Preserve the original case
set, criteria, and results alongside later versions. A favorable aggregate outcome alone is insufficient to test a
competency-specific prediction.
Construct validity. Map items and scoring rules to the relevant competency elements proposed in the
discussion paper, with independent clinical review, prespecified adjudication, and evidence that scores
discriminate clinically meaningful differences rather than response fluency or format. The clinical-support
and patient-context checks in Question 11 illustrate how such mapping can be made operational. [1, Section
V.B.2; Appendix A]
Sponsor-developed assets. Specialized benchmarks can address functions and hazards poorly covered by
public assets. Their specifications, case-selection process, and scoring rules should be reviewable while heldout test cases remain protected from development, tuning, retrieval, and iterative test feedback. Network
checks should use governance, adjudication, data-access, and reporting arrangements independent of device
and benchmark development. Possible arrangements include sponsor-funded evaluations executed by a data
network or academic coordinating center that holds the protected test cases, under a protocol and analysis
plan fixed before results are available, with results reported without sponsor editorial control. We ask FDA to
indicate which arrangements it would consider sufficiently independent. Shared-protocol execution with
open-source, publicly versioned analytic code makes the methods inspectable and enables authorized data
partners to rerun analyses on locally governed data. These capabilities support independently governed
evaluation. [1, Sections V.B and V.D.2; 4-6]
Feasibility. Partners should document whether source inputs, images, outputs, and interaction histories are
available. Where only downstream outcomes or utilization are accessible, the protocol should justify their
relationship to the benchmarked behavior and identify the output properties left unobserved. Cases arising
after a configuration freeze require the same separation from development and tuning as any other evaluation
cases. [2,11]

Page 5 of 12
Question 11. Clinical confirmation and real-world inputs
FDA question 11 (Section V.E; see Section V.C, Clinical Confirmation): CDRH is considering that clinical confirmation
for a GenAI-enabled device might not require a prospective clinical study in every case, and has described above a range
of approaches of increasing rigor and patient exposure. How might a sponsor select and justify a confirmation approach
tailored to a device’s intended use and proportionate to the device’s risk profile? Are there device types or risk profiles
for which one or more of these approaches would be insufficient or inappropriate? How might the anticipated distribution
of real-world inputs be taken into consideration? Are there other methods of clinical confirmation that might help inform
the evaluation of GenAI-enabled devices?

Request to FDA. We ask FDA to provide a clinical-confirmation checklist that ties the chosen method and
claim to minimum configuration, input, execution, and user-interaction records, together with completeness
and subgroup reporting. The metadata example above can serve as a starting point.
Method and claim. Archived-input evaluation should identify the tested configuration, input selection, and
reference assessment. Shadow deployment records live outputs without showing them to users. Claims about
users acting on outputs or autonomous clinical actions need evidence that observes those interactions. The
appropriate combination of retrospective evaluation, adjudicated cases, representative interactions, and
prospective study should follow intended use and risk. [1, Section V.C.1]
Case-level clinical assessment. Research has examined documented clinical reasoning and errors caused by
overlooking patient context. [12,13] We use two concise descriptors within the discussion paper’s proposed
competency structure: soundness, whether the presented facts support the recommendation, and prudence,
whether it fits the patient’s circumstances and documented preferences within safety and intended-use
boundaries. We apply soundness principally to E.1/E.2 and prudence to S.1/S.3/E.4. [1, Appendix A] These
descriptors do not add competencies; they group existing elements to help adjudicators structure case-level
review. Qualified clinical adjudicators should assess observable outputs against clinical guidance, with
prespecified scoring, disagreement resolution, and documented independence from device and foundationmodel developers.
Input coverage and reproducibility. Use only information available at the intended decision point; keep
later outcomes separate for reference assessment. Cover relevant populations, care settings, languages, health
literacy, incomplete or contradictory inputs, and conversational length. Evaluate safety-critical consistency
across repeated runs and equivalent paraphrases where applicable. Report real and synthetic cases separately,
justify any pooling, and review synthetic cases for gaps shared with their generator. [1, Sections V.B.2-V.C.1
and Appendix A]
Network contribution. OMOP-mapped histories can support clinically reviewed case selection and
separately justified simulation scenarios. Participating sites should report differences in care pathways,
missing input/output records, and configuration linkage, including for data collected outside the United
States. Later observations can update versioned case sets while retaining prior benchmarks and the
independence safeguards in Question 10. [1, Section V.C.1; 2,11]

Page 6 of 12
Question 15. Assessment against care without the device
FDA question 15 (Section V.E; see Section V.D.1, Standards of Performance): Are there ways in which performance
might be assessed relative to the care, technology, or course of action likely to occur in the absence of the device, rather
than to the comparators described in this section? What approaches might be used to identify and justify a comparator
such as unaided clinical judgment, delayed specialist review, or no intervention?

Request to FDA. We ask FDA to provide GenAI-specific comparator examples that distinguish absence of
target-device access, absence of recorded use, and use of alternative AI tools, with explicit exposure
definitions and sensitivity analyses for uncertain or contaminated comparator care.
Define the alternative pathway. Specify what patients would likely receive at the intended decision point
without the target device: unassisted clinician assessment, another technology, delayed specialist review, or
no intervention. Preserve the timing of that care and distinguish planned from completed actions. Evaluate
the chosen comparator alongside independently justified clinical safety and performance criteria. [1, Section
V.D.1; 2]
Resolve GenAI exposure. A generated output, an opened output, and an output used in a decision are
different exposures. The study should distinguish these stages using execution and access records, the
provenance of clinical documents, and targeted record review. It should also look for plausible alternative
routes: the same model through another interface, copied AI-generated text, earlier pilot access, or advice
shared by a colleague. The comparator is care without the target device, not automatically care without any
AI.
Match the estimand to the records. When only site availability is reliable, estimate the effect of
introduction or access. When request and interaction evidence is reliable, a use-based comparison may be
possible, with confounder adjustment based on the timing and causal role of variables relative to the
prespecified exposure. Output acceptance occurring after that exposure should not automatically be treated as
a baseline confounder. Prespecify time zero, exposure windows, concurrent configuration changes, and
analyses that exclude or vary uncertain exposure assignments. Changes mediated by the device, such as
reduced specialist waiting time, should be distinguished from baseline confounders. [2]
Network implementation. Use a shared protocol for eligibility, comparator timing, outcomes, and capture
diagnostics. Interrupted time-series, difference-in-differences, or target-trial approaches should be chosen for
the adoption pattern and clinical question, with their assumptions documented rather than treated as remedies
for absent exposure evidence. [5,8-10] Clinical case review can apply the criteria in Question 11 to the
device-assisted and alternative pathways.

Page 7 of 12
Question 19. Postmarket reassessment cadence and triggers
FDA question 19 (Section VI.D; see Section VI.A, Postmarket Monitoring Approaches): Please comment on the potential
approaches to postmarket performance evaluation, including periodic re-benchmarking, sample-based clinician review,
and performance degradation monitoring. What additional approaches should CDRH consider, and how should the
cadence and triggering events for reassessment be determined?
Request to FDA. We ask FDA to provide a risk-based cadence and trigger matrix covering periodic
benchmarking, sampled clinician review, operational changes, and response timelines, with a specific
distinction between research-only and device-operational pipeline changes.
Monitoring plan. Combine periodic benchmarking, independent review of sampled real inputs and outputs,
and performance-degradation monitoring. Set frequency using harm severity and time sensitivity, autonomy,
use volume, configuration-change frequency, and data latency. Assign each trigger a responsible party,
notification interval, investigation deadline, and interim action. Immediate safety concerns require an
operational response alongside any later network analysis. [1, Section VI.A]
Change triggers. Each foundation-model or fine-tuned-model version change, including further tuning,
should initiate a documented impact assessment. Include performance-relevant prompt, retrieval, guardrail,
orchestration, routing, and interface changes, distribution shifts, and safety signals. The assessment
determines the scope of retesting under Question 22. [1, Sections II and VI.A]
Pipeline distinction. A research-only change to the extract-transform-load process, vocabulary mapping,
phenotype definition, or linkage calls for data-quality review and analytic rerunning. A change that also
alters operational inputs, retrieval context, or runtime behavior should additionally trigger a device-level
impact assessment. Preserve the earlier analysis and document the source of any changed denominator or
result. [2,5,6]

Question 21. Stakeholder roles and interoperable evidence
FDA question 21 (Section VI.D; see Section VI.B, Shared Ecosystem Responsibility): What roles might clinicians,
healthcare institutions, professional societies, standards-setting bodies, and other stakeholders appropriately play in
postmarket monitoring, and how can these roles be structured without diffusing manufacturer accountability?
Request to FDA. We ask FDA to work with healthcare institutions, data networks, and standards bodies on
an example metadata profile linking device, configuration, deployment, and execution records, distinguishing
required core fields from optional extensions, with named custodians and a defined pathway from a network
finding to a manufacturer response.
Roles. Manufacturers maintain authoritative configuration and assessment records and retain device-level
monitoring and response responsibility. Institutions and service providers govern deployment and execution
records. Data partners perform lawful linkage, run common protocols, and return results with capture
diagnostics. Clinicians and professional societies define consequential errors and appropriate reference
assessments. [1, Section VI.B; 4-6]
Candidate standards. HL7 FHIR R5 Device supports device identification and version information;
Provenance can associate records with responsible agents and recorded activities. [14] For configuration
dependencies, existing bill-of-materials formats already describe AI components: CycloneDX AI/ML-BOM
represents models, datasets, dependencies, and model-card information, and the SPDX 3.0.1 AI and Dataset
profiles describe AI packages, their training data, and related properties. [15,16] We propose testing these as
complementary building blocks. FHIR would support clinical record linkage, and a versioned BOM, in
whichever format a pilot shows to be most suitable, would describe configuration dependencies. The two
would be connected by explicit identifiers and governed event records.
Response ownership. Specify the manufacturer safety or quality contact, risk-based notification times,
access to supporting records, investigation ownership, and feedback to the reporting site. Network
governance should also protect independent interpretation and reporting of adverse findings. The minimum
profile should include source-system ownership, identity status, timing uncertainty, and retention
responsibilities.

Page 8 of 12
Question 22. Re-benchmarking after modification
FDA question 22 (Section VI.D; see Section VI.C, Considerations for Postmarket Modifications): CDRH envisions that a
manufacturer’s premarket competency-based assessment might serve as a baseline against which post-deployment
modifications could be re-evaluated. How might the extent of re-benchmarking or other evidence be scaled to the nature
and expected impact of a given modification? For example, are there categories of change that might not significantly
affect safety or effectiveness of a GenAI-enabled device, or might be appropriately managed within a sponsor’s quality
management system versus requiring FDA premarket review and authorization before implementation? Are there
categories of changes that might be appropriate for inclusion in a PCCP?

Request to FDA. We ask FDA to publish worked change-to-test examples linking the affected component to
competencies, clinical risks, and reassessment scope, while distinguishing test selection from marketingsubmission decisions.
Scale the assessment. Where the impact of a change can be bounded, use targeted competency tests and
safety regression tests that cover interactions with unchanged components. For example, a retrieval-source
revision may call for factual-support and safety-escalation testing; a routing change may require evaluation
of every model and fallback path it can select. Changes affecting intended use, autonomy, user interaction, or
broad safety behavior warrant consideration of wider benchmarking and clinical confirmation. Assess
cumulative modifications as well as each change. [1, Sections V.B.2 and VI.C]
Preserve the baseline. Link each change record to the original and revised configurations,
benchmark/scoring versions, selected tests, results, and decision rationale. Keep the clinical-analysis protocol
separately versioned and link its cohorts and execution intervals to the same configuration records. This
supports reassessment without confusing an observational protocol with a device competency benchmark. [1,
Section VI.C; 3,5]

Question 23. Change control when modifications cannot be fully prespecified
FDA question 23 (Section VI.D; see Section VI.C, Considerations for Postmarket Modifications): How might PCCP
concepts or other change-control approaches be adapted for GenAI-enabled devices when the nature or scope of future
modifications cannot be fully prespecified?

Request to FDA. We ask FDA to clarify, through examples, which clinical functions, change boundaries,
validation methods, and acceptance criteria must be specified for prospective change control, and when an
additional marketing submission and review are needed before implementation.
Bound the proposal. Distinguish unknown implementation details from the limits that can be specified:
affected functions, permitted types and extent of change, verification and validation methods, acceptance
criteria, and conditions for restricting, suspending, or reverting a change. A general promise to monitor
performance is not a substitute for defining those boundaries. For unanticipated supplier changes, use the
interim controls described under Question 24. [1, Section VI.C; 3]
Apply separate regulatory decisions. For each modification, assess whether it is consistent with an
authorized PCCP and, if not, whether the applicable device-modification requirements call for an additional
marketing submission. Some changes may be managed without an additional submission under the
applicable requirements. Document these determinations and their rationale separately from test selection.
[3]
Record changes and timing. Retain controlled identifiers and known or uncertain effective intervals for
sponsor-initiated, model-evolution, and third-party changes. Link these to deployment and execution records
so that later clinical findings can be investigated against an identifiable change history. [1, Section VI.C; 2]

Page 9 of 12
Question 24. Changes to third-party foundation models
FDA question 24 (Section VI.D; see Sections VI.C and VII.A): For devices built on third-party foundation models,
changes to the underlying model may be initiated by the third-party model developer rather than the device manufacturer.
How can a manufacturer detect, evaluate, and respond to such changes in a timely manner? What mechanisms (for
example, contractual, technical, or through a PCCP) could provide reasonable assurance that third-party developerinitiated changes do not compromise the safety or effectiveness of the device?

Request to FDA. We ask FDA to define stable model/version identifiers, change-notification commitments,
notification and effective timestamps, and audit-log access as minimum information for a voluntarily
submitted Foundation Model MAF used to support version-specific device review. The same information
should be available through a documented alternative when no MAF is used.
MAF content and access. Section VII.A discusses voluntary Foundation Model MAFs that would be held
confidentially by FDA and could be referenced by sponsors with the file holder's authorization. Footnote 24
lists update-notification commitments and audit-log availability among the possible contents. [1, Section
VII.A and footnote 24] We recommend a structured record containing the provider and served model
version; affected endpoint or service; change description and relevant safety constraints; notification
recipient and time; effective time or uncertainty; links to evaluation evidence; and arrangements for
authorized access, retention, and updates. FDA reviewers need to identify the model state described by the
referenced MAF information. Separate supplier arrangements should give manufacturers the version and
change information needed for their own device evaluation.
Detection and response. Use contractual notice, version pinning where available, request-time checks
against provider records, and sentinel testing for unexpected behavioral change. Distinguish providerreported identity from supporting evidence. A version change should trigger impact assessment without
waiting for clinical deterioration; a behavioral signal should trigger investigation even when the label is
unchanged. Prespecify response times and options for continued, restricted, fallback, or suspended operation,
including when an earlier version is unavailable. [1, Sections VI.C and VII.A; 3]
Reassessment and uncertain timing. Retest affected competencies against protected, versioned clinical
cases, with the safeguards in Questions 10 and 11. Retain the last confirmed earlier state, first observed
changed state, and any supplier-stated effective time as distinct evidence. Stratify analyses by confirmed
configuration and handle unresolved intervals explicitly. Device, provider, and institutional records together
support that reconstruction.

Page 10 of 12
Interpretive limits
These recommendations concern evidence design and implementation; this comment reports no new clinical
cohort or empirical validation of a benchmark. Soundness and prudence are descriptive checks, not a
validated scoring instrument or universal acceptance threshold, and the cited clinical reasoning studies
provide methodological motivation rather than validation of an AI assessment tool.
FHIR, BOMs, and OMOP provide complementary identification or representation functions. They do not
independently prove actual execution, clinical use, causality, or safety. Source availability, missingness,
confounding, and justified links between indirect outcomes and assessed competencies remain study-specific.
Distributed execution and collection after deployment do not by themselves establish independence or
freedom from contamination. The proposed interoperability examples are not asserted to be FDA-recognized
standards for this use.

Conclusion and next steps
Generative AI-enabled devices will change after authorization in ways that a single premarket assessment
cannot fully anticipate. The evidence chain proposed here, linking what was evaluated, what was available at
a site, and what served each patient, is intended to make those changes traceable and their clinical
consequences measurable. As next steps, the contributors would welcome the opportunity to test a minimum
metadata profile in OMOP-converted data sources, to bring a proposal for representing AI device
configuration to the OHDSI community's established process for CDM changes [11], and to work with FDA
and standards bodies to align that profile with FHIR and bill-of-materials formats. Questions about these
comments may be directed to the corresponding contact.

Page 11 of 12
References
[1] U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback. August 18, 2026. https://www.fda.gov/media/194242/download
[2] U.S. Food and Drug Administration. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical
Devices. Final guidance. December 18, 2025. https://www.fda.gov/media/190201/download
[3] U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change Control Plan
for Artificial Intelligence-Enabled Device Software Functions. Final guidance. August 18, 2025; originally issued
December 4, 2024. https://www.fda.gov/media/166704/download
[4] Hripcsak G, Duke JD, Shah NH, et al. Observational Health Data Sciences and Informatics (OHDSI): opportunities for
observational researchers. Stud Health Technol Inform. 2015;216:574-578. https://doi.org/10.3233/978-1-61499-564-7574
[5] Suchard MA, Schuemie MJ, Krumholz HM, et al. Comprehensive comparative effectiveness and safety of first-line
antihypertensive drug classes: a systematic, multinational, large-scale analysis. Lancet. 2019;394(10211):1816-1826.
https://doi.org/10.1016/S0140-6736(19)32317-7
[6] Blacketer C, Defalco FJ, Ryan PB, Rijnbeek PR. Increasing trust in real-world evidence through evaluation of
observational data quality. J Am Med Inform Assoc. 2021;28(10):2251-2257. https://doi.org/10.1093/jamia/ocab132
[7] Schuemie MJ, Hripcsak G, Ryan PB, Madigan D, Suchard MA. Empirical confidence interval calibration for populationlevel effect estimation studies in observational healthcare data. Proc Natl Acad Sci U S A. 2018;115(11):2571-2577.
https://doi.org/10.1073/pnas.1708282114
[8] Lopez Bernal J, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health
interventions: a tutorial. Int J Epidemiol. 2017;46(1):348-355. https://doi.org/10.1093/ije/dyw098
[9] Dimick JB, Ryan AM. Methods for evaluating changes in health care policy: the difference-in-differences approach.
JAMA. 2014;312(22):2401-2402. https://doi.org/10.1001/jama.2014.16153
[10] Cashin AG, Hansford HJ, Hernán MA, et al. Transparent reporting of observational studies emulating a target trial: the
TARGET statement. JAMA. 2025;334(12):1084-1093. https://doi.org/10.1001/jama.2025.13350
[11] OHDSI. OMOP Common Data Model v5.4 specification: DEVICE_EXPOSURE, NOTE, and NOTE_NLP tables and
general conventions; How to Request Changes to the CDM. Accessed October 19, 2026.
https://ohdsi.github.io/CommonDataModel/cdm54.html,
https://ohdsi.github.io/CommonDataModel/cdmRequestProcess.html
[12] Kotwal S, Klimpl D, Tackett S, Kauffman R, Wright S. Documentation of Clinical Reasoning in Admission Notes of
Hospitalists: Validation of the CRANAPL Assessment Rubric. J Hosp Med. 2019;14(12):746-753.
https://doi.org/10.12788/jhm.3233
[13] Weiner SJ, Schwartz A, Weaver F, et al. Contextual errors and failures in individualizing patient care: a multicenter
study. Ann Intern Med. 2010;153(2):69-75. https://doi.org/10.7326/0003-4819-153-2-201007200-00002
[14] HL7 International. FHIR R5 (v5.0.0): Device and Provenance resources. Accessed October 15, 2026.
https://hl7.org/fhir/R5/device.html; https://hl7.org/fhir/R5/provenance.html
[15] OWASP CycloneDX. Machine Learning Bill of Materials (AI/ML-BOM) and CycloneDX v1.7 specification. Accessed
October 15, 2026. https://cyclonedx.org/capabilities/mlbom/; https://cyclonedx.org/docs/1.7/json/
[16] The Linux Foundation. System Package Data Exchange (SPDX) Specification Version 3.0.1: AI and Dataset profiles.
Accessed October 15, 2026. https://spdx.github.io/spdx-spec/v3.0.1/

Page 12 of 12