Bold Insight Inc. (Kaitlin Stinson, Stephanie Wiltman)

Bold Insight is pleased to provide feedback regarding Generative AI-Enabled Medical Devices, as per the FDA’s request. All answers are provided alongside discussion questions in the attached document that were posed within the request document.

Docket No. FDA-2026-N-7874

Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback

Submitted on behalf of Bold Insight, Inc.

Authors: Kaitlin Stinson (kaitlin.stinson@boldinsight.com) and Stephanie Wiltman
(stephanie.wiltman@boldinsight.com)

Bold Insight is pleased to provide feedback regarding Generative AI-Enabled Medical Devices, as per
the FDA’s request. Below, all answers are provided alongside discussion questions posed within the
request document.

Each response includes a throughline assumption that is important to consider. Current HF
engineering works under the assumption that the device being evaluated (and any underlying Gen-AI)
is working as intended. In order for HF engineering to be conducted effectively, the device and its
outputs must be fully specified and articulated. In other words, explicit guardrails should be put into
place to limit and control outputs, rather than allowing for unspecified possibilities. Generative AIEnabled Medical Devices should be constrained such that all intended user inputs and outputs are
known, specified, and predetermined in order for use-related risk to be evaluated in a comprehensive
manner.

Overall, we encourage FDA to provide explicit guidance on how established HFE methods
should be adapted to address the novel characteristics of GenAI-enabled medical devices.
Although traditional HFE principles remain applicable, the variability, conversational nature,
adaptability, and potential autonomy of GenAI introduce additional considerations that may
significantly expand the scope and complexity of usability engineering activities. Given the relative
novelty of this technology, HFE practitioners may have limited experience identifying and evaluating
these emerging human-AI interaction risks and we seek to support the industry and HFE practitioners
with specific guidance.

To support consistent and effective application of HFE principles for Gen-AI enabled medical devices,
future guidance should explicitly identify the additional factors manufacturers and HFE practitioners
should consider throughout the usability engineering process, including user characterization, task
analysis, use-related risk analysis, formative evaluation, and HF validation. Specific examples of
GenAI-related user interactions, potential use-related hazards, UI risk controls, and recommended
evaluation approaches would be particularly valuable. Such guidance would help practitioners
understand how to extend established methods to evaluate GenAI-specific risks without
unnecessarily replacing or duplicating existing HFE practices.

Providing this additional direction would promote more consistent, risk-based HFE evaluations and
help ensure that GenAI-enabled medical devices are designed and evaluated to support safe and
effective use by their intended users.

Questions should be directed to the Authors above.
Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices

1. Does the two-axis risk framework, organized around device activity and the consequence of
relying on an incorrect output, appropriately capture the dimensions most relevant to the risk of
a GenAI-enabled software function?
a. If there are additional dimensions—such as the reversibility of a resulting action, the
availability of downstream safeguards, the time pressure of the deployment setting, or
the traceability of the output (i.e., to primary source materials)—that should be
represented in a risk framework, please describe and provide examples of how they
should be represented.

We propose incorporating explicit HFE directives related to this topic into guidance.
When medical devices contain Gen-AI enabled software the Human Factors Engineering
(HFE) expands beyond its typical scope of,“Can the user operate the interface?” We
recommend that use-related risk and characteristics of the human-AI interaction be explicitly
considered within the risk framework. Assessment of typical device inputs and responses may
not fully capture whether users can detect an erroneous output, opportunity for human
review/intervention, time pressure, information availability, reversibility, user expertise, or how
the UI influences reliance. Given the increased complexity of user interaction in Gen-AI
enabled medical devices and the resulting expanded scope of HFE application for these
devices, we request that guidance include a specific directive that use-related risk analysis be
explicitly referenced as a required component of the overall risk management.

2. CDRH seeks input on how to account for the spectrum of GenAI-enabled informational
functions that vary in the degree to which they direct a user to a particular action (i.e., between
“non-directive” and “action-directing”).
a. What characteristics of an output—such as its wording, specificity, personalization, or
context—could be considered as modifiers of the risk associated with an informational
function, after accounting for the device’s overall functionality and intended use?
b. What additional information would provide manufacturers with sufficient clarity and
predictability around risk assessment for informational functions while recognizing that
directiveness may exist along a continuum rather than as a binary distinction?

We propose incorporating explicit HFE directives related to this topic into guidance.
The degree of directiveness is fundamentally a user interface (UI) and content-design
consideration. Characteristics such as wording, specificity, personalization, tone, prominence,
visual hierarchy, conversational context, expressed confidence, and presentation of
alternatives can influence whether users perceive AI-generated information as informational,
advisory, or action-directing. Importantly, the inclusion of a disclaimer should not automatically
render an otherwise directive interaction non-directive, particularly when other aspects of the
 Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices

output convey authority or encourage a specific action.

From an HFE perspective, manufacturers should evaluate how representative intended users
actually perceive, interpret, and respond to AI-generated outputs within the intended context of
use, rather than relying solely on the manufacturer's intended classification of the output.
A central usability objective should be to support and evaluate appropriately calibrated reliance
on GenAI-generated information and recommendations. The goal should not simply be to
increase user trust in AI, but to ensure the user interface is designed to support users in
recognizing when outputs can reasonably be relied upon, when independent verification is
warranted, and when escalation or human review is necessary. HFE evaluations should
therefore consider use-related risks associated with automation bias, over-reliance, underreliance, perceived authority, and misunderstanding of uncertainty.

3. CDRH seeks input on whether and when a GenAI-enabled function that results in delivery of
clinical information to patients, as opposed to HCPs, could present different or higher risks,
while also recognizing the potential benefits associated with improved patient empowerment,
engagement, and access to clinical information.
a. What device characteristics, output features, or safeguards might mitigate risks that
could arise when a user lacks the domain knowledge to independently evaluate an
output, without unnecessarily underestimating patient capability?

We propose incorporating explicit HFE directives related to this topic into guidance.
Guidance should stress that HFE be focused on evaluating the product’s user interface design,
not credentialing the user. Rather than creating a generalized "AI literacy" threshold for
patients or clinicians, FDA should encourage manufacturers to define the knowledge and
capabilities reasonably expected of intended users and then demonstrate that the AI-enabled
UI supports safe and effective interaction by those users. That preserves the conventional HFE
principle of designing and evaluating the interface for the intended population rather than
testing whether users are sufficiently competent to compensate for interface deficiencies.

Additionally, avoid treating patient/HCP status as a simple proxy for ability to safely interpret AI.
Apply established HFE principles: characterize intended-user capabilities, limitations,
knowledge, literacy and experience, then determine whether the UI enables safe interpretation
and appropriate action. Safeguards could include uncertainty communication, source/context
visibility, escalation, confirmation, and boundary communication.

The context of use is captured in the Use Specification and should include any unique aspects
driven by use of a GenAI-enabled device. Specifically, interpretation of AI is not done in a
vacuum and is instead done in a space with different pressures, such as time, assumption that
device outputs are accurate or superior to one’s own expertise, etc.
 Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices

4. CDRH seeks input on whether and how the distinction between generalist and specialist
physicians might inform the assessment of risk for HCP-facing GenAI-enabled functions.
a. Under what circumstances, if any, might risk be affected when an HCP who lacks the
relevant clinical specialist knowledge receives information that falls within a specialist
area of practice? What device characteristics or safeguards might mitigate such risks?

We propose incorporating explicit HFE directives related to this topic into guidance.
HFE evaluations should consider the final user-facing device, not merely the underlying model.
Evaluations should focus on evaluation of the product interaction rather than testing the
clinician's medical expertise. The relevant question is whether the device requires particular
domain knowledge for safe use and whether the UI enables the intended HCP population to
recognize limitations, interpret outputs and determine when additional expertise is required.
The required or expected domain knowledge of intended users is captured in the Use
Specification and should be considered when evaluating use-related risk.

5. For multi-turn conversational GenAI-enabled devices that may migrate from providing “nondirective” information to “action-directing” information over the course of an exchange, how
should risk be assessed across realistic conversational trajectories?
a. How could the intended use of such a device be characterized when its behavior is
emergent across a conversation?

We propose incorporating explicit HFE directives related to this topic into guidance. We
recommend expanding conventional task analysis approaches beyond discrete PCA
sequences to interaction trajectories: user input > AI interpretation > output > user
interpretation > response > AI adaptation > eventual action. URRA should consider cumulative
interactions where individual turns appear acceptable, but the conversation migrates toward
unsafe/directive behavior. HFE evaluations should consider all potential realistic conversational
trajectories.
 Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices

Usability testing should deliberately include imperfect AI behavior. Testing only correct AI
outputs could miss some of the most safety-critical interactions. Formative and validation
approaches may need scenarios involving incorrect, incomplete, uncertain, inconsistent, overly
confident, or out-of-scope outputs to determine whether users recognize problems and
respond appropriately.

6. For GenAI-enabled care escalation functions, CDRH is considering how the evaluation may
account for both under-escalation and over-escalation.
a. How could manufacturers characterize and weigh these two directions of error, given
that they may not be commensurable and that acceptable trade-offs may vary by clinical
context?

We propose incorporating explicit HFE directives related to this topic into guidance.
URRA and HFE evaluations should not only assess whether AI generates the correct
escalation but whether the user recognizes, understands, and appropriately acts upon it. For
UI design mitigations should consider urgency, salience, clarity, alarm fatigue, trust, repeated
escalation, and consequences of over-warning.

User profiles should incorporate AI-relevant characteristics such as domain expertise, AI
familiarity, health literacy, understanding of uncertainty and ability to independently assess
outputs.
 Section IV: Considerations for the Assessment of Risk for GenAI-Enabled Devices

Section V: A Competency-Based Approach for Premarket Evaluation of GenAIEnabled Devices

7. Is the competency-based approach described above, i.e., device benchmarking followed by
clinical confirmation, a useful and appropriate framework for evaluating GenAI-enabled
devices?

We propose incorporating explicit HFE directives related to this topic into guidance.
Provide guidance on separating AI/device performance evidence from human-interaction
evidence. Benchmarking alone cannot demonstrate safe/effective use because it may not
capture interactions with users and workflows and does not establish how intended users
understand and respond to outputs.

8. How might the two-axis risk framework described in Section IV be considered within the
competency-based approach described above to help determine the level of evidence needed
for a premarket submission?

We propose incorporating explicit HFE directives related to this topic into guidance.
Recommend scaling HFE evidence based not only on clinical consequence but also on safe
and effective use of the Gen-AI device by the intended users. For example, the degree of
autonomy/directiveness and dependence on correct human interpretation/intervention is
understood by users. Higher-risk human-AI interactions may warrant more robust task
analysis/URRA analysis and extensive formative and HF validation evidence.

9. Would a benchmarking structure such as the one described above be likely to provide
adequate evidence of clinical knowledge, analytic capabilities, safety behavior, communication,
and generalizability to support a reasonable assurance of safety and effectiveness?
a. Are there elements that are missing, redundant, or inappropriately categorized? Are
there externally developed standards that could be leveraged?

We propose incorporating explicit HFE directives related to this topic into guidance. We
strongly recommend that human factors/usability be made more explicit rather than subsumed
within “communication.” Section IV, Question 4 already evaluates actual user comprehension
and automation bias. Guidance should distinguish technical communication performance from
evidence generated with representative users.
 Section V: A Competency-Based Approach for Premarket Evaluation of GenAI-Enabled Devices

10. CDRH recognizes that publicly available benchmarking assets may be subject to data
contamination, saturation, and limited real-world representativeness. How should a sponsor
establish that performance on a given benchmark predicts safe and effective real-world
behavior for the device’s intended use?
a. What evidence should support the construct validity of a benchmark used to gate device
evaluation, and what role should sponsor-developed benchmarks play given potential
concerns around independence and optimization to the test?

We propose incorporating explicit HFE directives related to this topic into guidance. A
benchmark cannot necessarily predict human-AI team performance. Real-world safe use
depends on workflow, cognitive workload, environment, trust, experience, UI presentation and
user characteristics. Representative-user evaluation provides complementary construct
validity.

11. CDRH is considering that clinical confirmation for a GenAI-enabled device might not require a
prospective clinical study in every case, and has described above a range of approaches of
increasing rigor and patient exposure. How might a sponsor select and justify a confirmation
approach tailored to a device’s intended use and proportionate to the device’s risk profile?
a. Are there device types or risk profiles for which one or more of these approaches would
be insufficient or inappropriate?
b. How might the anticipated distribution of real-world inputs be taken into consideration?
c. Are there other methods of clinical confirmation that might help inform the evaluation of
GenAI-enabled devices?

We propose incorporating explicit HFE directives related to this topic into guidance.
Guidance should explicitly recognize human factors/usability evaluation as a clinical/realisticuse confirmation method where safe performance depends upon human interaction.
Simulated-use studies with representative users can evaluate comprehension, appropriate
reliance, error detection, escalation, recovery and critical human-AI interactions without
requiring actual clinical exposure in every case.
 Section V: A Competency-Based Approach for Premarket Evaluation of GenAI-Enabled Devices

12. How should sponsors achieve statistically meaningful performance measurement for GenAIenabled devices?
a. Where synthetically generated inputs supplement real patient data, how should
sponsors account for differences between the synthetic and real-world distributions
when estimating performance, and under what conditions, if any, is it appropriate to
combine benchmarking evidence and clinical confirmation evidence to support a single
performance estimate?

We propose incorporating explicit HFE directives related to this topic into guidance.
Although this topic is mostly AI performance methodology, there is an important HFE-specific
consideration: synthetic users/inputs should not replace representative-user research when the
endpoint concerns human comprehension, behavior, trust, workload, or use error.

13. For which clinical domains, device functions, or subpopulations is synthetic data particularly
well-suited, or particularly inadequate, as a supplement to real-world evidence?
a. What safeguards would mitigate the risk that synthetic data generated by models of the
same class as the device under evaluation reproduces the very performance gaps the
evaluation is intended to detect, particularly for underrepresented subgroups?

See response to question 12.

14. For open-ended device outputs, how should performance comparators and acceptance criteria
be selected?
a. When a panel of qualified clinicians serves as the comparator, how should the
applicable standard (for example, the standard of care versus the performance of a
median clinician in practice) be defined and justified?
b. Should generalist or specialist physicians be used as a performance standard?
c. When should human-AI team performance, rather than the device operating alone,
serve as the basis for evaluation?

We propose incorporating explicit HFE directives related to this topic into guidance.
This section has one of the strongest HFE implications. For devices intended to support rather
than replace users, the meaningful safety endpoint may be performance of the human + AI
system, not AI accuracy alone. Evaluate whether AI improves/degrades decisions, introduces
automation bias, affects detection of errors, and supports appropriate reliance. Human-AI team
performance should be a core focus of evaluation in all HFE analyses and evaluations for
these systems.
 Section V: A Competency-Based Approach for Premarket Evaluation of GenAI-Enabled Devices

15. Are there ways in which performance might be assessed relative to the care, technology, or
course of action likely to occur in the absence of the device, rather than to the comparators
described in this section?
a. What approaches might be used to identify and justify a comparator such as unaided
clinical judgment, delayed specialist review, or no intervention?

We propose incorporating explicit HFE directives related to this topic into guidance.
HFE activities can contribute by comparing workflow and user performance with/without AI:
task success, error detection, workload, decision quality, time, situational awareness and
reliance behavior.

16. Should independent third parties be involved in some or all aspects of a competency-based
approach, including device benchmarking and clinical confirmation?
a. If so, in what ways might qualified, independent third-party participation contribute to
this approach, and what qualifications and independence criteria should apply?
b. Are there aspects of the assessment for which third-party involvement would be
impractical or inadvisable?
c. What safeguards or program design features would be critical to prevent third-party
participation from limiting competition or preventing innovation?

We propose incorporating explicit HFE directives related to this topic into guidance. Not
necessarily independent but HFE/usability specialists must evaluate human-AI interactions,
particularly where independence from the development team reduces confirmation bias.
Consider whether established HF validation principles may already provide a framework
without creating an entirely new certification mechanism.

17. CDRH is exploring whether a competency-based approach could be applied to devices that
incorporate a variety of underlying model architectures, such as multimodal vision-language
models and generative or predictive world models.
a. Are there aspects of the described approach that might be ineffective or inapplicable to
these devices?
b. What additional or distinct considerations might need to be addressed?

We propose incorporating explicit HFE directives related to this topic into guidance.
HFE evaluation should remain technology-agnostic: evaluate the final user-facing device and
resulting human interaction rather than model architecture alone. Human factors must be able
to evaluate all expected user interface responses from the system. These user interface
responses must adequately represent what users will actually see in real-world use rather than
genericized outputs.
Section VI: Post-market Monitoring for GenAI-Enabled Devices

18. CDRH is considering whether it may be appropriate to accept greater premarket uncertainty
regarding a GenAI-enabled device’s benefit-risk profile through greater reliance on postmarket
monitoring.
a. Under what conditions might such an approach be appropriate, and what characteristics
of a monitoring program would need to be in place to justify reduced premarket
evidence?
b. Are there device types or risk profiles for which this approach would not be appropriate?

We propose incorporating explicit HFE directives related to this topic into guidance. We
caution against using post-market monitoring to substitute for premarket evaluation of
foreseeable, safety-critical human-AI interactions. Known critical interactions should be
evaluated before release where feasible.

19. Please comment on the potential approaches to post-market performance evaluation, including
periodic re-benchmarking, sample-based clinician review, and performance degradation
monitoring.
a. What additional approaches should CDRH consider, and how should the cadence and
triggering events for reassessment be determined?

We propose incorporating explicit HFE directives related to this topic into guidance. Add
increased post-market use-related performance monitoring due to the novel nature of the
technology: emerging use problems, over-/under-reliance, misunderstanding outputs,
workarounds, ignored warnings, inappropriate prompting, unexpected interaction trajectories
and user behavior following updates. Explicitly build HFE into the post-market review process
to ensure robust post-market reporting that will adequately capture the required data needed to
inform future and ongoing risk management activities, identify emerging use problems and
use-related risks, and support the development of risk profiles for future Gen-AI devices.

20. Please comment on whether the proposed approaches to post-market monitoring can be
facilitated by machine-based supervisory agents.
a. What considerations, including the evaluation and reliability of the supervisory agent
itself, should CDRH take into account for such an approach?

We propose incorporating explicit HFE directives related to this topic into guidance.
Human review remains important when monitoring endpoints concern human comprehension
or behavior. Manufacturers should retain accountability for use-related safety.
 Section VI: Post-market Monitoring for GenAI-Enabled Devices

21. What roles might clinicians, healthcare institutions, professional societies, standards-setting
bodies, and other stakeholders appropriately play in post-market monitoring, and how can
these roles be structured without diffusing manufacturer accountability?

See response to question 20.

22. CDRH envisions that a manufacturer’s premarket competency-based assessment might serve
as a baseline against which post-deployment modifications could be re-evaluated.
a. How might the extent of re-benchmarking or other evidence be scaled to the nature and
expected impact of a given modification? For example, are there categories of change
that might not significantly affect safety or effectiveness of a GenAI-enabled device, or
might be appropriately managed within a sponsor’s quality management system versus
requiring FDA premarket review and authorization before implementation?
b. Are there categories of changes that might be appropriate for inclusion in a PCCP?

We propose incorporating explicit HFE directives related to this topic into guidance.
Recommend an HFE impact assessment for AI changes, even when GUI screens do not
change. Model/prompt/guardrail changes can alter tone, confidence, directiveness, refusal
behavior, escalation or variability and therefore alter the effectiveness of UI and the resulting
use-related risk. Model changes can affect device safety/effectiveness and should be
thoroughly evaluated from an HFE perspective.

23. How might PCCP concepts or other change-control approaches be adapted for GenAI-enabled
devices when the nature or scope of future modifications cannot be fully prespecified?

See response to question 22.

24. For devices built on third-party foundation models, changes to the underlying model may be
initiated by the third-party model developer rather than the device manufacturer. How can a
manufacturer detect, evaluate, and respond to such changes in a timely manner?
a. What mechanisms—for example, contractual, technical, or through a PCCP—could
provide reasonable assurance that third-party developer-initiated changes do not
compromise the safety or effectiveness of the device?

See response to question 22.
Section VII: Other Topics

25. Would voluntary Foundation Model MAFs be practical for foundation model developers to
provide and to keep current, and what content might they include?
a. Given that participation would be voluntary and that model developers may have limited
incentive to disclose safety-relevant information, what would make such a program
sufficiently useful for premarket review?
b. Are there alternative mechanisms CDRH should consider for obtaining information
about underlying foundation models?

We propose incorporating explicit HFE directives related to this topic into guidance.
HFE-relevant information in documentation should include known behaviors affecting human
interaction: output formatting, confidence behavior, refusal/escalation characteristics, known
limitations and changes likely to affect user mental models or reliance.

26. Are there additional considerations that inform the premarket and postmarket evaluation of
agentic GenAI-enabled devices, beyond those applicable to non-agentic GenAI-enabled
devices?
a. How should the elevated risk associated with autonomous multi-step action, tool use,
and reduced opportunity for human review be reflected in acceptance criteria and
oversight?

We propose incorporating explicit HFE directives related to this topic into guidance.
This has significant HFE implication. Sponsors should evaluate these systems
comprehensively including visibility of system status, human authorization, confirmation
checkpoints, interruption, override, reversibility, error recovery and clear allocation of
responsibility. HFE should specifically analyze and evaluate human-oversight checkpoints
before irreversible/high-consequence actions.