TUD Dresden University of Technology (Rebecca Mathias)

Human factors considerations for LLM-enabled medical
devices

Human factors considerations for LLM-enabled medical
devices
Comments on FDA Discussion Paper | Docket FDA-2026-N-7874

Submitted by: Rebecca Mathias1, Julio Cesar Chavarria Carrasco1, Akrem Dridi1, Josip Vrdoljak3,
Louis Kirstein4,5, Heinz Wiendl5,6,7, Stephen Gilbert1,2

Affiliations:
Else Kröner Fresenius Center for Digital Health, Faculty of Medicine, TUD Dresden University of
Technology, Dresden, Germany
Faculty of Business and Economics, TUD Dresden University of Technology, Dresden, Germany
Laboratory for AI and Data Science in Biomedicine, University of Split School of Medicine, Split
Faculty of Medicine, University of Münster, Münster, Germany
Prof. Valmed® – validated medical information GmbH, Langen, Germany
Department of Neurology and Neurophysiology, Medical Center – University of Freiburg, Faculty of
Medicine, University of Freiburg, Freiburg, Germany
Brain & Mind Center Freiburg, Freiburg, Germany

We submit these comments in response to the FDA's August 2026 discussion paper, Considerations for the
Regulation of Generative AI-Enabled Medical Devices with a focus on human factors in medical devices
incorporating large language models (LLMs).
We focus on how interactions with LLMs can affect user understanding, trust and subsequent actions, and
why these factors need to be considered alongside technical and clinical performance during development
and after deployment.
LLM-enabled medical devices allow open-ended inputs, variable outputs, conversational interaction and
personalization. These characteristics can offer important benefits, but they also introduce decisions that
need to be made deliberately. Flexible input, for example, may capture clinically relevant context while
making outputs less consistent. Human review may provide additional oversight while also increasing
workload, and does not necessarily mean that outputs will be meaningfully scrutinised. The appropriate
balance depends on the device, users, clinical role, context and risk.
Our comments relate to Section IV (risk assessment), Section V (premarket evaluation and human-AI team
performance), Section VI (postmarket monitoring and modifications), Section VII.B (agentic AI), and
Appendix A (benchmarking elements). They are relevant in particular to Questions 2, 3, 5, 9, 14, 19, 21, 22
and 26. Appendix A already covers scope maintenance (S.2), uncertainty communication (S.3),
communication and comprehension (E.4), robustness across interactions (R.1), and agentic capabilities
(A.1). Ideally, human factors should be considered beyond E.4. We discuss these as connected issues
rather than answering each question separately. User understanding, perceived authority, trust, verification,
workflow demands and supervision can affect safe use across several of the proposed elements.
Our earlier published practical guide to human factors for AI-enabled medical devices identified seven
interrelated risks, including misinterpretation, trust miscalibration, automation bias, deskilling, workplace
friction, indication creep, and change or mode mismatch. It described practical measures aligned with
established usability engineering, risk management, postmarket monitoring and change-control activities
(Mathias et al., doi.org/10.1056/AIpc2501297).
We acknowledge that LLM-enabled devices require more focused consideration (Table 1, column 4). Table
1 below reflects our current stage of work presenting fourteen considerations for LLM-enabled devices.
Each includes early-stage planning for premarket devices and later-stage reflection questions for devices
ready for market entry or on market, potential benefits and harms, and possible approaches for
developers/manufacturers to address the consideration. They are intended to support product-specific
decisions, not require every feature or approach in every device. The considerations can be incorporated
into established usability engineering and risk-management processes rather than treated as a separate
requirements.
Table 1. Human factors considerations for LLM-enabled medical devices
Early-stage questions support prospective design and planning, whereas late-stage questions support reflection on a more mature product. Potential benefits and harms support product-specific judgement rather than imply that more or less of a characteristic is inherently preferable.
Possible approaches are examples and reflect current guidance, practice, or plausible human-factors rationale that can be combined, adapted, or substituted according to the product context.

Possible ways to address the consideration that the
Early stage of device Late stage of device LLM features that affect this Potential benefit when well Potential harm when poorly developer can implement, specify, or support. They are
Human factors consideration
development – think about development - reflect consideration matched to purpose matched to purpose examples rather than requirements and may be
combined or used independently as appropriate.

1. Skill requirements, What does the user need to Are the skills and responsibilities LLM use may change which skills users need LLMs may appropriately reduce Some skills may need to be actively – define what the system supports, performs, or leaves to the
maintenance and role know or do to use the system, expected of users clearly defined, to apply or maintain. Because LLMs can clerical burden, standardise routine preserved to avoid a loss of confidence user
adaptation and does that change over including which, if any, need to be support summaries, recommendations, and work, support data-driven or or competence that remains necessary – role-appropriate prompts
time? preserved over time? reasoning across many steps, users may not augmented decision-making, or when the tool is unavailable, wrong, – IFU (Instructions for Use)/training suggestions for (re)training
always notice which skills are being exercised enable users to perform tasks they delayed, or out of scope. Roles and – competence or AI-literacy training programmes
less or which skills and responsibilities remain could not otherwise undertake on responsibilities may also become – design features that preserve critical skills, understanding, or
important. their own. unclear. independent judgement where needed
– define and make clear the responsibilities of the manufacturer,
deployer, and user
– monitoring human–AI agreement patterns
– competence-gated access
– retraining triggers for relevant changes

2. User interpretation and What do users need to Is the system designed so users can LLM outputs can be fluent, plausible, Summaries, simplification, Users may assume an output is – task- and risk-calibrated structure for input and output
situation awareness understand about the system appropriately interpret its outputs and summarised, selective, and probabilistic. They prioritisation, and structured outputs complete, definitive, current, or clinically – assumption markers
and situation to interpret its maintain the situation awareness may appear complete or definitive even when can help users arrive at decisions or comprehensive when it is partial, – uncertainty flagging
outputs in that context? needed for the intended use and uncertain, incomplete, inferred, or dependent make interpretations more quickly uncertain, inferred, or missing important – limitation statements
context? on the information provided. What the user and maintain awareness of the context. – evidence-strength or source-hierarchy cues
must remain aware of varies with the purpose immediate task. – prompts for missing information
of the device (a patient's clinical state, the – explicit role statements clarifying the system's role
strength and completeness of the evidence – user-controlled filtering of the information basis
presented, or the status of a task in a – withholding content where grounding is insufficient
workflow). – indicate that the user is interacting with an AI system rather
than a person
– visible indicators of automated personalisation where needed
– required fields for critical context
– configurable output length and detail
– summary-first output with expandable detail
– visible memory or context summaries
– prompts when prior context is reused

3. Scope and mode What can the system be used Is the system designed so users can LLMs can appear capable across many tasks. Clear scope and mode can help Users may unknowingly use the system – user-controlled filtering of the information basis
awareness for, and where multiple scopes understand its current scope or mode, Some systems legitimately support multiple users understand what the system is outside its intended or validated – clear scope labels
or modes exist, how will users with any switching, where present, scopes, roles, or modes, while others should intended to do and use it purpose, misunderstand what task or – visible mode indicators (for products with multiple modes)
know they are using the right intentional, visible, and safe? remain tightly constrained. Where several appropriately. Where multiple tasks role it is currently supporting, or fail to – define scope and mode separately where appropriate
one for the task? applications sit behind one product, each may or modes are supported, intentional notice when the system has switched – assess scope and mode separately where appropriate
carry its own purpose and limits and should be switching can allow one system to scope or mode. – guided task selection
defined and tested accordingly. support different needs, such as – eligibility checks
summarisation, patient education, – out-of-scope response templates
documentation, or clinician support. – user confirmation for scope or mode changes
– controlled access where scope flexibility would create risk
– role-based outputs
– competence-gated access
– separation between sessions, users, or tasks where
appropriate

4. Tone, persona, and How should the system Is the system’s tone and, where LLMs use fluent, conversational language that The right tone and perceived An inappropriate tone or persona may – explicit role statements clarifying the system's role
perceived authority communicate with users, what applicable, persona appropriate for could make them seem more human, expert, authority may improve engagement, create excessive authority, inappropriate – define tone and, where applicable, persona
tone should it have, and what the intended use, user, and role it is empathic, or authoritative than intended. These disclosure, and continuity. deference, emotional reliance, or – deliberately suppress persona where this better fits the
level of authority should it meant to convey? can cause users to attribute a particular role, confusion about the system’s role. A purpose
appear to have? level of authority, competence, or emotional tone that does not fit the user or context – use conversational styles according to the intended user and
understanding to the system. may also reduce credibility or context
engagement. – neutralise authority cues where inappropriate
– indicate that the user is interacting with an AI system rather
than a person
– empathy boundaries
 Possible ways to address the consideration that the
Early stage of device Late stage of device LLM features that affect this Potential benefit when well Potential harm when poorly developer can implement, specify, or support. They are
Human factors consideration
development – think about development - reflect consideration matched to purpose matched to purpose examples rather than requirements and may be
combined or used independently as appropriate.

5. Personalisation and How much personalisation is Is the degree of personalisation LLMs can adapt to user preferences, prior Personalisation can improve Hidden or poorly bounded – manual settings with clear user control
adaptive interaction appropriate for the intended use, designed to be appropriate for the behaviour, role, patient context, organisation, relevance, efficiency, accessibility, personalisation can reduce – visible indicators of automated personalisation where needed
and what boundaries or controls intended use, with appropriate or explicit settings. Personalisation may be engagement, and fit to clinical role or predictability, standardisation, equity, – reset, edit, or disable options
may be needed? boundaries, visibility, and user manual, automated, visible, or hidden. patient need. and auditability. It can also lead to – role-based outputs
control? intended use creep. Different users may – equity safeguards
receive different outputs without – define where personalisation is allowed and where
understanding why. standardisation by design is appropriate

6. Interaction flexibility and How flexible should user input Is the system designed to allow the Small differences in prompt wording, context Flexible interaction can support Too much flexibility can produce – task- and risk-calibrated structure for input and output
response consistency be, and how consistent do input flexibility users need while order, missing information, interaction history, natural use, exploration, and irrelevant or inconsistent outputs for – prompts for missing information
outputs need to be? keeping output variation within or context-window contents can change LLM adaptation to different cases, while similar cases. Too much standardisation – define where personalisation is allowed and where
acceptable bounds for the intended outputs. The degree of input flexibility and greater structure or guidance can can block relevant context or make the standardisation by design is appropriate
use? resulting output variation can therefore differ improve consistency where this is interaction less adaptable to real-world – choose and define the appropriate point on the fixed-to-freesubstantially across use cases. important for the intended use. use. text spectrum for a given point of interaction
– required fields for critical context
– prompt templates or libraries
– input guidance and examples
– input and interaction constraints
– context-window management
– checks for prompt-variant and clinically acceptable output
consistency
– measurement of default or non-answer rate for insufficiently
supported outputs
– output constraints where appropriate

7. Cognitive load and How should the system fit the Is the system designed to fit the LLM outputs vary in length, structure, LLMs can reduce load by Too little cognitive engagement can – task- and risk-calibrated structure for input and output
workflow integration user's setting, and where should intended workflow or use context, with relevance, timing, and review burden. summarising, drafting, prioritising, or promote shallow review, automation – comparison with the workload of the task or process otherwise
it add or reduce effort? cognitive load and interaction Additional explanation, verification, preparing handoffs. In other cases, bias, or unclear responsibility. Too much required
demands appropriate to the task and documentation, handoff, or interaction steps greater cognitive engagement may load, review steps not proportionate to – time-to-answer measurement
user? may increase or decrease workload depending be appropriate where users need to risk, poor timing, poor workflow fit, or – iterative usability testing in realistic conditions
on the task, user and use-context. notice uncertainty, verify evidence, unintended increases in review, – configurable output length and detail
document reasoning, or make a high- documentation, verification, or – summary-first output with expandable detail
stakes judgement. interaction effort can cause fatigue, – structured output
missed information, duplicated work, – intentional review pauses for high-risk decisions in proportion
workarounds, delay, or abandonment. to risk
– clear review points
– clear documentation points
– clear handoff points
– interruption management
– workflow placement that matches context of use, task load,
roles, and responsibility
– added friction for higher-risk tasks
– clear handoff and delegation boundaries

8. Trust calibration How much should users trust Does the system create the right level LLMs often produce confident, polished, Appropriate trust, if gained, can Overtrust may lead users to accept – competence or AI-literacy training programmes
the system and its outputs? of trust for the intended use that human-like text. These features can make improve uptake, efficiency, and incorrect outputs. Undertrust may lead – limitation statements
matches the reliability and limitations outputs seem more reliable, authoritative, or confidence in supported tasks. Users users to ignore valid outputs. Artificially – explicit role statements clarifying the system's role
of its outputs? clinically reasoned than they are. may benefit from a system that cautious language can also reduce – withholding content where grounding is insufficient
communicates clearly and decisively usefulness or create alert fatigue. – neutralise authority cues where inappropriate
when the evidence base supports Design alone may not be enough, since – indicate that the user is interacting with an AI system rather
that. the right level of trust also depends on than a person
the user's training and expertise. – calibrated communication cues based on testing
– source grounding
– provenance markers indicating whether content was retrieved,
user-provided, or inferred
– prompts to verify high-risk outputs
– examples showing when users should rely, verify, escalate, or
ignore outputs
– distinguish between evidence confidence and model
confidence
– define and validate confidence, uncertainty, or risk indicators, if
present
 Possible ways to address the consideration that the
Early stage of device Late stage of device LLM features that affect this Potential benefit when well Potential harm when poorly developer can implement, specify, or support. They are
Human factors consideration
development – think about development - reflect consideration matched to purpose matched to purpose examples rather than requirements and may be
combined or used independently as appropriate.

9. Explainability and What and how much should the Does the system provide the type and LLMs do not have simple fixed decision logic. Well-designed explanations can Explanation can be actively harmful if it – assumption markers
uncertainty system explain to users, and level of explanation appropriate for Explanation can take the form of source support verification, calibrated trust, is persuasive but not verifiable, – uncertainty flagging
how should it show uncertainty? the user and task, and communicate attribution, verifiable evidence or provenance, learning, accountability, and rationalises a wrong answer, creates – limitation statements
uncertainty appropriately? uncertainty language, structured output, or appropriate escalation. false confidence, or adds cognitive – evidence-strength or source-hierarchy cues
visibility of retrieved information. It may relate burden without improving decisions. – withholding content where grounding is insufficient
to an individual output, the system as a whole, – configurable output length and detail
or changes in its operation over time. – summary-first output with expandable detail
Confidence in the evidence and confidence – structured output
expressed by the model are not the same – provenance markers indicating whether content was retrieved,
thing. user-provided, or inferred
– choose explanation type deliberately
– inline citations to look-up-able sources
– source or evidence panel
– distinguish between evidence confidence and model
confidence
– withholding as an alternative to quantifying uncertainty
– define and validate confidence, uncertainty, or risk indicators, if
present

10. Information verifiability, What information or sources Is the system designed to provide the LLMs can generate plausible but unreliable Grounding, citations, source Unsupported or fabricated content may – evidence-strength or source-hierarchy cues
grounding, and provenance should support the system’s level of factual grounding and source content. Outputs may blend retrieved material, attribution, and provenance can help influence decisions. Poorly presented or – user-controlled filtering of the information basis
of outputs outputs and how should this be traceability required for the intended pre-trained knowledge, user-provided context, users verify information and improve excessive source information can also – withholding content where grounding is insufficient
made clear to users? use, with unsupported content and inference which may not always be accountability. mislead users, increase their workload, – measurement of default or non-answer rate for insufficiently
handled appropriately? distinguishable to the user. or give false reassurance. supported outputs
– source grounding
– provenance markers indicating whether content was retrieved,
user-provided, or inferred
– inline citations to look-up-able sources
– source or evidence panel
– display supporting source sections for cited claims
– verify that cited sources support the relevant claim
– red-team testing for unsupported or fabricated content
– built-in automated checks for unsupported or insufficiently
grounded content
– output constraints where appropriate
– data-quality governance over source curation, with defined
inclusion criteria
– source visibility where verification is needed
– automated verification against a validated reference or
threshold
– per-question re-grounding
– source versioning for content
– unsupported-content labels

11. Human review Where, if anywhere, is active Is the system designed so that the LLMs can make tasks quicker and easier. This Proportionate verification can Too little verification can allow incorrect – define and make clear the responsibilities of the manufacturer,
human review needed before an level and form of human review are may encourage passive acceptance of outputs, preserve the efficiencies of LLM use outputs to be accepted or acted on. Too deployer, and user
output informs a decision or next proportionate to the task and its risk? particularly when deliberate human review is while directing user attention to much mandatory verification can slow – monitoring human–AI agreement patterns
step? not built into the interaction even though the outputs or decisions that require workflows, result in superficial review, or – escalation pathways (to human or machine) when needed
task or risk may call for it. closer review or judgement. overwhelm users. – intentional review pauses for high-risk decisions in proportion
to risk
– clear review points
– prompts to verify high-risk outputs
– examples showing when users should rely, verify, escalate, or
ignore outputs
– inline citations to look-up-able sources
– display supporting source sections for cited claims
– human verification steps, adopted according to the level of risk
– source visibility where verification is needed
– added friction for higher-risk tasks
 Possible ways to address the consideration that the
Early stage of device Late stage of device LLM features that affect this Potential benefit when well Potential harm when poorly developer can implement, specify, or support. They are
Human factors consideration
development – think about development - reflect consideration matched to purpose matched to purpose examples rather than requirements and may be
combined or used independently as appropriate.

12. Agentic action and What actions should the system Where the system can autonomously Some LLM systems can plan and carry out Agentic behaviour can reduce Insufficient supervision can allow – define and make clear the responsibilities of the manufacturer,
supervision (applies where be allowed to take, and what plan and carry out a series of steps or multi-step actions, use tools, or change administrative burden, coordinate unintended or incorrect actions to occur deployer, and user
the system can controls should users have over actions, is its level of autonomy downstream systems rather than only generate complex tasks, and carry out work at or propagate before they are noticed. – escalation pathways (to human or machine) when needed
them? matched by appropriate supervision, an output. Actions may occur across several a scale or speed that would be Users may also be unclear about what – clear handoff points
autonomously carry out
control, and traceability? steps or over time, making what the system difficult for a person. It may also has been done, what remains pending, – automated verification against a validated reference or
multiple steps) has done, what remains pending, and what can reduce human workload or error or what was delegated. Excessive threshold
still be controlled less immediately visible to the where appropriate. approval steps can add workflow friction – visible action plans or status
user. and undermine the purpose of – risk- and action-based approval gates
automation. – confirmation before irreversible or high-risk actions
– controls to pause, cancel, or reverse actions
– clear handoff and delegation boundaries
– audit trail for tool calls and downstream actions
– automated checks that can trigger a pause or escalation where
appropriate

13. Longitudinal context, What should the system Is the system designed to manage LLMs may be used over long conversations, Persistent memory and longitudinal Old assumptions, errors, or irrelevant – context-window management
persistent memory, and remember, reuse, update, or longitudinal context so that what is repeated sessions, or patient/task episodes. context can improve continuity, context may carry forward and influence – visible memory or context summaries
semantic drift forget within and across retained, reused, updated, or Context may accumulate, be summarised, reduce repetition, and support later outputs. Users may not know what – controls to inspect, correct, reset, or delete remembered
interactions? forgotten remains appropriate for the truncated, or carried over through memory, and ongoing patient or clinician the system remembers, what it has context
intended use over time? prior information may continue to influence later interaction across sessions or forgotten, or how prior interactions – separation between sessions, users, or tasks where
outputs. Retained information may also take episodes. shape current output. appropriate
different forms, including persistent interaction – prompts when prior context is reused
context, source caching, and longitudinal – testing for semantic or contextual drift in extended interactions
performance-monitoring data, which serve – per-question re-grounding
different purposes and may affect the system – statelessness by design
differently. Over time, retained context or the – defined rules for context retention
system's framing may drift from what is relevant
to the intended use.

14. Change awareness and When the system changes, what Is there a defined process for LLM behaviour can change through several Targeted change communication and Insufficient communication or guardrails – in-product feedback or rating function for users to flag
revalidation triggers needs to happen internally, and managing changes, triggering routes, originating with the manufacturer, an management can help users update can leave users relying on outdated inadequate outputs
when and how should users be renewed human factors evaluation integrating platform, locally or in the underlying their mental model after material assumptions. Excessive communication – classify changes by type and origin
informed? where needed, and communicating information sources. These changes may be changes that affect behaviour, scope, or overly rigid guardrails can cause – risk-based change classification
relevant changes to users? frequent and not obvious to users. reliability, or workflow. Minor or low- fatigue, delay use, or lead users to – proportionate, targeted change notices
risk changes may require less ignore important updates. – application-level version display
prominent intervention to avoid – source versioning for content
unnecessary disruption or fatigue. – retraining triggers for relevant changes
– revalidation triggers for relevant changes
– predetermined change control plans where applicable
– controls to bound behavioural change where needed

Note. The considerations are interdependent and not all apply equally to every product. The possible approaches reflect current guidance, practice, research, or plausible human-factors rationale and are not presented as uniformly established or validated
interventions.

The table was created combining regulatory output and standards relevant to human factors and AI-enabled medical devices, together with our previous work and other research outputs in the field covering LLMs and generative AI in health care and
approaches used by developers in practice. These sources were used to identify recurring issues, define the scope of the considerations, and put together possible approaches for design and implementation by integrating current best practice and emerging
evidence. The sources are listed in the reference section below

Interactive prototype
An accompanying interactive prototype is in development to support stepwise application and revisiting of the considerations during product development and later reflection. Prototype link: https://llm-hf.vercel.app/

Assistance and Ongoing Research Disclosure
ChatGPT 5.6 (OpenAI) was used to help summarise our ongoing research work for the purposes of this comment. This work is still in progress, and the framework and accompanying tool will continue to be developed and updated. This text was reviewed
and edited by the authors listed, who remain responsible for the content. The views expressed in this comment are those of the authors and do not necessarily reflect those of their affiliated institutions.
References
1. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU)
2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828. Official Journal of the European Union L 270, 1–59 (2024).
2. U.S. Food and Drug Administration. Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations: Draft Guidance for Industry and Food and Drug Administration Staff (January 2025).
3. International Electrotechnical Commission (IEC). IEC 62366-1:2015+A1:2020 — Medical devices — Part 1: Application of usability engineering to medical devices. International Electrotechnical Commission, Geneva (2020).
4. International Electrotechnical Commission (IEC). IEC/TR 62366-2:2016 — Medical devices — Part 2: Guidance on the application of usability engineering to medical devices. International Electrotechnical Commission, Geneva (2016).
5. U.S. Food and Drug Administration. Applying Human Factors and Usability Engineering to Medical Devices: Guidance for Industry and Food and Drug Administration Staff (February 2016).
6. Chartered Institute of Ergonomics & Human Factors. Human Factors in Healthcare AI: White Paper. (CIEHF, 2024) Https://Ergonomics.Org.Uk/Resource/Human-Factors-in-Healthcare-Ai.Html.
7. Engler, M. (2024) Usability Engineering for Medical Devices using Artificial Intelligence and Machine Learning Technology. Zenodo. doi:10.5281/zenodo.14203190.
8. U.S. Food and Drug Administration. Content of human factors information in medical device marketing submissions: Draft guidance for industry and Food and Drug Administration staff. (December 2022).
9. Schönfelder, A. et al. Collaborative and Cooperative Hospital “In-House” Medical Device Development and Implementation in the AI Age: The European Responsible AI Development (EURAID) Framework Compatible With European Values. Journal of
Medical Internet Research 28, e80754 (2026).
10. Evaluation of Human Factors–Related Risks in AI-Enabled Medical Devices — A Practical Guide. NEJM AI https://ai.nejm.org/doi/abs/10.1056/AIpc2501297.
11. Freyer, O., Wiest, I. C., Kather, J. N. & Gilbert, S. A future role for health applications of large language models depends on regulators enforcing safety standards. The Lancet Digital Health 6, e662–e672 (2024).
12. National Institute of Standards and Technology (US). Artificial Intelligence Risk Management Framework : Generative Artificial Intelligence Profile. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (2024) doi:10.6028/NIST.AI.600-1.
13. The Benefits and Risks of LLMs for Facilitating Medical Decision-Making Among Laypersons | Proceedings of the 2025 ACM Designing Interactive Systems Conference. ACM Conferences https://dl.acm.org/doi/10.1145/3715336.3735779.
14. Busch, F. et al. Current applications and challenges in large language models for patient care: a systematic review. Commun Med 5, 26 (2025).
15. Ouyang, W., Han, Y., Du, H., Wang, Z. & He, Y. Understanding user acceptance of AI medical conversational agents through dual-path trust mechanisms: An empirical study based on the HSM model. International Journal of Industrial Ergonomics 115,
104041 (2026).
16. Bunning, B. J. et al. Drivers of Oncologist Preference of AI-Generated Literature Review in a Randomized Mixed-Methods Study. 2026.08.24.26361252 Preprint at https://doi.org/10.64898/2026.08.24.26361252 (2026).
17. Regina de Brito Duarte · AI-Assisted Decision-Making Framework. https://reginabduarte.github.io/framework/.
18. Foo, C. et al. The Benefits and Risks of LLMs for Facilitating Medical Decision-Making Among Laypersons. in Proceedings of the 2025 ACM Designing Interactive Systems Conference 3173–3191 (Association for Computing Machinery, New York, NY,
USA, 2025). doi:10.1145/3715336.3735779.
19. Cao, Y. et al. More than Decision Support: Exploring Patients’ Longitudinal Usage of Large Language Models in Real-World Healthcare-Seeking Journeys. in Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems 1–24
(Association for Computing Machinery, New York, NY, USA, 2026). doi:10.1145/3772318.3791946.
20. National Commission into the Regulation of AI in Healthcare: Recommendations for a future regulatory framework. GOV.UK https://www.gov.uk/government/publications/national-commission-into-the-regulation-of-ai-in-healthcare-recommendations-for-afuture-regulatory-framework.
21. Health, C. for D. and R. Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback. FDA https://www.fda.gov/medical-devices/digital-health-center-excellence/considerations-regulationgenerative-ai-enabled-medical-devices-discussion-paper-and-request (2026).
22. Good machine learning practice for medical device development: Guiding principles | International Medical Device Regulators Forum. https://www.imdrf.org/documents/good-machine-learning-practice-medical-device-development-guiding-principles
(2025).
23. Mathias, R. et al. Safe AI-enabled digital health technologies need built-in open feedback. Nat Med 31, 370–375 (2025).
24. Carl, N. et al. Enhancing clinicians’ trust in large language models via transparent source attribution: A randomized controlled evaluation in uro-oncology. European Journal of Cancer 233, 116168 (2026).
25. Furtado, V. et al. Assessing the User Experience of an LLM-Based Conversational Assistant in Diabetes Mellitus Care. J Healthc Inform Res 10, 116–153 (2025).
26. Everett, S. S. et al. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. NPJ Digit Med 9, 409 (2026).
27. Sanz Ausin, M. et al. Multicall Memory in an AI Care Agent for Chronic Care Management Among Older Adults: Retrospective Observational Study. JMIR Form Res 10, e87704 (2026).