Advanced Research Projects Agency for Health and the Johns Hopkins University Applied Physics Lab (Haider Warraich)

Dear CDRH/DHCoE Staff,

On behalf of the ARPA-H ADVOCATE Program and our Independent Verification and Validation Team - the Applied Physics Lab at Johns Hopkins University - we are submitting our comment in response to the Gen AI Discussion Paper. This comment provides our perspective on how we are approaching evaluation for performers in the ADVOCATE program. We appreciate the consideration of these comments and hope to be able to shape the very important work that is being undertaken by the agency in the evaluation and regulation of generative and agentic AI - enabled medical devices.

Regards
Haider Warraich

Assessment of Risk for GenAI-Enabled Devices:
Response from the ARPA-H ADVOCATE Program and JHU APL IV&V Teams
This document provides a joint response from the ARPA-H ADVOCATE program team and the JHU
APL independent verification and validation team, with input from Johns Hopkins Medicine
Cardiology, to the FDA Digital Health Center of Excellence discussion paper, Considerations for the
Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback.
The paper requests public comment under docket FDA-2026-N-7874. This response has two sections:
(1) cross-cutting key points and (2) direct answers and recommendations to FDA questions.
In this response, “sponsor” means the manufacturer, developer, or other entity responsible for the FDA
submission and ongoing regulatory obligations.

Section 1. Key points

We support FDA's risk-based total product lifecycle approach, device benchmarking followed by
clinical confirmation, carefully validated simulation, and strong postmarket monitoring. The points
below identify where the framework could be clarified or strengthened.
• Contextual risk and explicit modifiers. FDA's activity-by-consequence framework is a useful
starting point, but the risk of a device function can change across patients and care pathways.
Reversibility, time to harm, safeguards, traceability, repeated use, and deployment scale can raise or
lower risk. The likely alternative care pathway should be considered separately when judging net
benefit and the amount of evidence needed. Recommendation: Use the two axes as the base
framework, then require sponsors to show how each modifier changes risk and evidence
requirements for every device function and intended use context. Example: A heart failure tool that
recommends an extra diuretic dose is lower risk when it follows a narrow protocol, uses current
kidney function and electrolyte data, and requires clinician approval. The same recommendation is
higher risk when those data or timely review are unavailable.
• Configurable autonomy and clinical operating constraints. The same agent may operate in
shadow mode for one function, require clinician approval for another, and act autonomously for a
third. FDA should assess each function at its permitted level of autonomy rather than assign one
autonomy label to the entire product. Clinical operating constraints should state the actions allowed,
data required, boundaries, stop and deferral rules, and applicable site policies. Health systems may
need to select among authorized autonomy settings within those limits. Recommendation: Require
sponsors to list every in-scope clinical action, its allowed autonomy level and controls, and the
conditions that trigger escalation, deferral, or a safer mode. Changes to autonomy or operating
constraints should trigger proportionate re-evaluation. Example: The same agent might schedule
follow-up autonomously, recommend tests for clinician approval, and remain in shadow mode for
medication changes until stronger evidence is available.
• Evaluate behavior over the full trajectory. Directiveness cannot be determined from wording
alone. Evaluation should examine what users understand and do, whether the device crosses its
intended scope over multiple turns, and how risk changes across asynchronous exchanges and
repeated use. Recommendation: Measure comprehension, action and inaction, escalation and deescalation, and the complete care trajectory. Test the highest level of clinical influence permitted
within each intended workflow and verify that the device stays within its boundaries. Example:
“You may wish to seek care” can function as a directive if it reliably sends patients to the
emergency department. Reassuring language can also cause harm if it delays needed care.
• Account for deployment scale and repeated systematic errors. A failure that is rare in one
encounter may still cause substantial harm if the same system repeats it across thousands of
 encounters. This applies to both agentic and non-agentic devices. Recommendation: Use expected
interaction volume, simultaneous use, repeated use, and the potential for correlated errors to set the
rigor of stress testing, evidence, and postmarket monitoring. Example: A triage tool that overescalates only a small fraction of encounters could still generate thousands of avoidable emergency
visits when deployed across many health systems.
• Match evaluation methods to risk and use context. Benchmarking can test broad capabilities,
while clinical confirmation is needed to show performance with representative patients, users,
workflows, and case mix. The evidence package should follow the full risk profile, not a simple
informational-versus-autonomous label. Recommendation: FDA should publish an evidence table
that links each risk profile to the minimum expected combination of benchmarking, retrospective
real-patient testing, simulation, shadow deployment, independent adjudication, and prospective
evaluation. Example: Patient-facing chest pain triage may warrant shadow deployment and
prospective evaluation even without autonomous action. A bounded, easily reversible automated
function with strong safeguards may require less.
• Report the boundaries and depth of testing. Evaluation sets should cover intended functions,
patients, settings, workflows, time horizons, data quality, tools, and failure modes, including mild
illness, mimics, incomplete or conflicting information, comorbidities, and rare high-consequence
cases. A case count alone is not enough. Sponsors should state what was and was not tested and
how deeply each area was tested. Simulated patients should also be validated at both the encounter
and cohort levels against real human data. Recommendation: Require a test coverage and rigor
report that maps functions and risks to test conditions and known gaps; distinguishes basic
representation from formal scoring, stress testing, fault injection, and failure-and-recovery testing;
and documents system and model versions, data provenance, sequestered cases, adjudication,
repeated runs, human oversight, latency, concurrency, and tool or interface failures. Report
synthetic and real-data performance separately. Example: Reporting 2,000 heart failure cases is not
enough unless the sponsor shows coverage of typical heart failure, common mimics, kidney disease,
medication interactions, missing or conflicting data, longitudinal deterioration, clinician overrides,
tool outages, and high-volume use.
• Compare with realistic alternatives. A device should be compared both with an expert or clinical
reference standard and with the care likely to occur without it. Depending on intended use, the
deployed unit may be the device, a clinician-AI team, or the device within a health system.
Recommendation: Require sponsors to describe or measure the likely care pathway without the
device, including who makes the decision, when it occurs, and how it varies by setting. Use more
than one comparator when there is no single usual-care pathway. Example: A heart failure triage
agent may need comparison with patient self-triage, an RN advice line, delayed primary care or
cardiology review, and emergency department use, not only with an expert cardiologist.
• Match endpoints to the clinical function. Accuracy and timeliness may be appropriate primary
endpoints for bounded triage or diagnostic functions. Management agents that order tests or change
treatment are interventions. Process measures may be reasonable when the agent implements
guideline-based care already linked to outcomes, but novel, off-guideline, or higher-consequence
management should require stronger patient-relevant outcome evidence. Experience with
mammography computer-aided detection shows that technical performance or adoption may not
translate into better outcomes. All deployed systems should support postmarket outcome tracking.
Recommendation: FDA should publish endpoint expectations by function and risk. Example:
Viz.ai's stroke triage De Novo relied on sensitivity, specificity, and time to specialist notification. A
heart failure agent that changes medication should also be evaluated for adverse events, symptoms,
hospitalization, and function.
• Evaluate agentic workflows end to end. Human errors can also propagate, but agentic systems
can use the same mistaken premise to plan, act through multiple tools, and check their own work
 without an independent pause. Recommendation: Test complete action sequences, including
planning, tool selection and authorization, output verification, memory and state, stop and deferral
rules, recovery from failed or conflicting actions, and independent review before irreversible or
high-consequence steps. Example: Misreading kidney function could lead the same agent to
recommend, order, and communicate a diuretic increase before a clinician reviews the case.
• Treat postmarket monitoring as ongoing evidence. Premarket testing should establish a baseline,
not serve as the final assessment. Performance can change when models, prompts, retrieval,
guardrails, tools, workflows, or standards of care change. Device logs also create an opportunity to
make both AI-supported and usual care more visible and measurable. Recommendation: For higherrisk functions, require auditable records of inputs, outputs, recommendations, and actions;
representative clinician review; outcome and behavior monitoring; subgroup and site analyses; and
thresholds that trigger investigation, restriction, rollback, or FDA notification. Manufacturers
should remain accountable when health systems help with review. Example: A rise in missed
escalation among older adults after a model update should trigger targeted review and possible
rollback even if aggregate benchmark scores remain stable.
• Enable independent evaluation without creating a market barrier. Independent parties can
maintain hidden test sets, provide expert adjudication, qualify evaluation tools, and audit
manufacturer-developed benchmarks. However, cost and limited access to test infrastructure could
disproportionately burden smaller developers. Recommendation: Use transparent qualification and
conflict-of-interest rules, proportional fees, multiple qualified providers, and shared test resources
supported by the U.S. Department of Health and Human Services (HHS) or FDA. Example: A
shared, qualified virtual-patient environment and hidden case library could support independent
testing without requiring every small company to build a full evaluation platform.
Section 2. Direct answers and recommendations to FDA questions
Assessment of Risk
1. Two-axis framework adequacy
• Use the two axes as a base, not a complete risk score. Device activity and consequence are useful
organizing dimensions. Risk should also reflect patient context, intended user and care system,
reversibility, time to harm, safeguards, traceability, repeated use, and deployment scale. The likely
alternative care pathway should be assessed separately when judging net benefit and the evidence
required. Recommendation: Apply explicit modifiers to each device function and care pathway, and
require sponsors to show how each modifier changes the risk classification or evidence package.
Example: A diuretic recommendation based on current kidney function and electrolyte data with
clinician approval is lower risk than the same recommendation for a patient with kidney disease,
stale laboratory data, and no timely review.
2. Non-directive versus action-directing spectrum
• Judge behavior, not wording alone. Risk includes both steering a user toward an unsafe action
and falsely reassuring the user to avoid a beneficial action. Wording, specificity, personalization,
urgency, and context matter, but user comprehension and behavior are more meaningful.
Recommendation: FDA should publish examples across the spectrum and require testing of
comprehension, action and inaction, escalation and de-escalation. Example: In heart failure, a
response can cause harm by encouraging excessive diuretic use or by reassuring a congested patient
not to seek care.
3. Patient-facing versus healthcare-professional-facing functions
• Assess the user and task together. Patient-facing information may be higher risk when safe use
requires clinical knowledge that the patient is unlikely to have, but direct access can also improve
engagement and access. Safeguards can include a clear rationale, links to source information,
 uncertainty and limitation statements, confirmation of understanding, explicit escalation
instructions, and an easy path to clinician review. Recommendation: Test behavior and safeguards
with intended users across varied health literacy and communication needs rather than assuming all
patient-facing use is high or low risk. Example: A patient-facing medication-titration
recommendation requires stronger safeguards than a clinician-facing summary that cites the
guideline and patient data.
4. Generalist versus specialist healthcare professionals
• Treat expertise as a risk modifier. The same output may carry different risk depending on
whether the intended user can interpret it, recognize its limits, and translate it into safe action. The
relevant unit may be the clinician-AI team rather than either component alone. Recommendation:
Define the intended user and scope, use authenticated role and privilege information where
available, provide evidence links, and require referral or specialist escalation for complex or out-ofscope cases. Example: Heart failure management complicated by kidney disease may require
specialist review even when the same recommendation is appropriate in a straightforward case.
5. Multi-turn conversations
• Evaluate the full trajectory and permitted autonomy. Individual turns may appear informational
even when the cumulative interaction becomes action-directing. Risk should be assessed across
realistic conversations, asynchronous exchanges over days or longer, repeated use, and transitions
among shadow, human-approval, and autonomous modes. Recommendation: Define scope
boundaries and stop conditions, evaluate each function-autonomy mode, and test the highest level
of clinical influence permitted within the intended workflow. Example: A heart failure chatbot
limited to education is different from one that begins with education but later performs triage or
recommends medication changes.
6. Care escalation
• Measure both under-escalation and over-escalation. They are different errors with different
consequences, and usual care exhibits both at rates that vary by patient, condition, setting, and
geography. Long-term effects such as alarm fatigue and loss of trust also matter. Recommendation:
Report each error rate and apply context-specific acceptance thresholds and harm weights rather
than collapsing them into one score. Compare the device with the likely alternative care pathway as
well as expert judgment. Example: Under-escalation can delay treatment of acute decompensated
heart failure. Over-escalation can cause unnecessary emergency care, testing, anxiety, resource use,
or inappropriate medication changes.
Competency-Based Premarket Evaluation
7. Overall competency-based framework
• Useful framework, but not enough by itself. Benchmarking followed by clinical confirmation is a
reasonable structure. A benchmark exam does not replace evaluation in representative clinical use
or ongoing monitoring. Recommendation: Pair benchmarking with risk-proportionate clinical
confirmation, require a shadow and oversight period for higher-risk functions, and continue active
monitoring after deployment. Example: A triage system can pass a large case set but still fail when
real users give incomplete histories or workflow delays prevent escalation.
8. Linking risk to evidence
• Link evidence to the full risk profile. Required methods should reflect clinical function, patient
population and setting, consequence, influence over care, safeguards, reversibility, time pressure,
repeated use, and deployment scale. Expected benefit relative to realistic alternative care should
also affect the evidentiary bar. Recommendation: Publish an evidence table specifying the minimum
combination of benchmarking, retrospective testing, simulation, shadow deployment, adjudication,
 and prospective evaluation expected for each risk profile. Require sponsors to justify deviations.
Example: Patient-facing chest pain triage may require shadow deployment and prospective
evaluation even without autonomous action, while a bounded, low-consequence refill function with
strong safeguards may require less.
9. Benchmarking completeness
• Report benchmark coverage and rigor. FDA's safety, clinical proficiency, generalizability, and
agentic-capability categories are a useful core. Benchmarks should also address longitudinal and
stateful performance, human-AI team and workflow performance, and data, tool, and operational
integrity. Representative case mix, configurable autonomy, deployment scale, and correlated failure
should apply across all categories. Recommendation: Require a test coverage and rigor report that
links each intended-use function and risk to the patients, conditions and mimics, settings, time
horizons, data-quality conditions, workflows, tools, and failure modes tested. For each area, state
whether it was represented, formally scored, stress-tested, tested with injected faults, or tested
through failure and recovery. Also document system and model versions, test-data provenance,
independent or sequestered cases, adjudication, repeated-run testing, human oversight, latency,
concurrency, tool or interface failures, and known gaps. Use established clinical guidelines and
validated measures where available. Example: Reporting that a heart failure agent was tested on
2,000 cases is not enough. The sponsor should show coverage of typical heart failure, non-cardiac
dyspnea, kidney disease, medication interactions, missing or conflicting data, longitudinal
deterioration, clinician overrides, tool outages, and high-volume use, and how deeply each area was
tested.
10. Benchmark validity
• Show that benchmark scores carry over to real care. A benchmark should reflect the intended
population, condition prevalence, workflows, information quality, and interaction patterns.
Sponsors should show that benchmark performance is associated with performance in shadow or
live use. Manufacturer-developed benchmarks should receive independent review, with some cases
or scoring rules kept hidden to limit teaching to the test. Recommendation: Update or retire
benchmarks when test cases become widely known, scores saturate, clinical practice changes, or
benchmark results no longer predict real-world performance. Report realistic case-mix performance
separately from intentionally difficult stress tests. Example: A diagnostic agent may score highly on
a benchmark made mostly of confirmed disease cases but perform much worse in practice, where
many patients have mild symptoms, alternative diagnoses, missing or conflicting information, or no
target disease.
11. Choosing a clinical confirmation method
• Match clinical confirmation to risk and intended use. Retrospective testing, standardized
patients, clinician adjudication, shadow deployment, supervised live use, and prospective studies
can be used alone, in sequence, or together. Selection should reflect the device's influence on care,
consequence of failure, intended user, input distribution, and whether harm can be detected before
reaching the patient. Recommendation: Publish a decision guide showing when each method is
expected and require the confirmation sample to reflect real-world inputs and important subgroups.
Example: A medication-management function may need shadow deployment followed by
supervised prospective use, while a low-risk documentation aid may be adequately assessed with
retrospective cases and clinician review.
12. Statistical rigor
• Define exactly what is being estimated and account for model variability. Before testing,
sponsors should define the intended population and setting, unit of analysis, device version and
configuration, reference standard, endpoint, success threshold, and subgroup plan. Analyses should
 include confidence intervals, repeated runs, realistic variations in wording and information order,
and statistical methods that account for multiple outputs from the same patient, case, or template.
Recommendation: Use independent data not used for development or prompt tuning, and size
studies around clinically meaningful thresholds, rare high-consequence failures, and anticipated
deployment volume. Example: Ten responses from one synthetic patient are not ten independent
patients, and one successful run does not establish reproducible safety.
• Keep evidence types separate unless they are truly comparable. Benchmarking, synthetic
testing, clinical confirmation, process measures, and patient outcomes answer different questions.
Large synthetic studies should not overwhelm smaller but more representative clinical studies in a
single headline estimate. Recommendation: Combine results into one estimate only when
populations, endpoints, sampling, adjudication, and weighting are comparable and prespecified.
Continue postmarket outcome tracking when premarket authorization relies on process measures.
Example: A device that is 95 percent correct on 10,000 synthetic cases and 82 percent correct on
500 real cases should not be reported as 94 percent correct unless the studies measure the same
endpoint and are weighted to the intended-use population.
13. Synthetic data suitability
• Use synthetic data to expand coverage and stress-test, not replace real evidence. Synthetic data
are useful for controlled perturbations, rare scenarios, regression testing, and well-understood
guideline-based decisions. They are weaker as the sole evidence for real-world communication,
workflow, effectiveness, or subgroup performance. Recommendation: Report provenance,
generation methods, coverage, limitations, and differences from real data. Retain independent realworld holdouts and clinical confirmation. Example: Synthetic cases can test the same heart failure
scenario with a missing laboratory value, different wording, or a rare contraindication, but cannot
by themselves show how patients act on advice.
• Validate both individual patients and the simulated population. A plausible simulated patient
does not prove that a large synthetic cohort reflects the amount and pattern of human variation.
Using similar model families to generate cases, define answers, and judge performance can also
create shared blind spots. Recommendation: Compare interaction-level realism and cohort-level
variation with real human data, compare subgroup performance on real and synthetic cases, and
account for cases that share the same source patient, template, or generator. Example: One hundred
thousand nearly identical cooperative patients may miss the communication, disclosure, literacy,
and ambiguity challenges that determine real-world performance.
14. Comparators and acceptance criteria
• Use both ideal and real-world comparators, and test the deployed unit. Open-ended outputs
should be compared with an expert or clinical reference standard and with the realistic usual-care
alternative. If intended use is clinician plus AI, test the team; if it is autonomous, test the device
alone. Test both when the device can operate in more than one mode. Recommendation: Prespecify
minimum safety requirements that must be met independently and cannot be offset by stronger
performance on other measures. Then assess improvement over the realistic baseline using patientrelevant outcomes or validated process measures appropriate to the function. Example: A heart
failure management agent may be compared with specialist consensus, local usual care, and the
clinician-agent team while still meeting a non-negotiable limit on contraindication and monitoring
errors.
15. Counterfactual comparators
• Compare with the likely pathway without the device. This comparison is central to estimating
real-world benefit and risk. Sponsors can use documented workflows, retrospective or prospective
data, shadow deployment, and site-specific observation to identify what is likely to happen without
the device. The comparator may differ by setting and patient group. Recommendation: Require
 sponsors to state the expected counterfactual pathway, who makes each decision and when, and
how uncertainty and site variation will be handled. Use multiple counterfactuals when no single
alternative dominates. Example: For ambulatory heart failure triage, alternatives may include selfmanagement, an RN line, delayed primary care or cardiology review, or emergency department
presentation.
16. Third-party involvement
• Use independent review where it adds credibility, without creating a market barrier.
Independent parties can maintain hidden datasets, audit benchmarking, provide expert adjudication,
qualify tools, and develop consensus methods. They should have relevant expertise, disclosed
conflicts, structural independence, reproducible methods, secure data practices, and auditable
decisions. Recommendation: Use transparent qualification criteria, proportional fees, multiple
qualified providers, and shared HHS- or FDA-supported test resources so smaller developers are
not priced out. Example: A qualified third party could maintain a hidden patient-simulation library
and scoring rubric that manufacturers cannot optimize against directly.
17. Other model architectures
• Extend the framework, but tailor the tests to the architecture. Multimodal systems need
evaluation of each input type and of how inputs are combined, including missing, conflicting,
corrupted, or misaligned data and overreliance on one modality. Cross-modal reference data may be
scarce, so synthetic data may supplement but should not replace real aligned cases. Generative or
predictive world models also require testing of temporal consistency, uncertainty propagation, and
degradation over longer horizons. Recommendation: Require component-level and end-to-end
testing that maps modalities, tools, and model components to clinical functions, failure modes, and
acceptance criteria. Test the deployed configuration during clinical confirmation. Example: A
multimodal model should not ignore a subtle but critical imaging finding because a larger volume
of text suggests a benign diagnosis.
Postmarket Monitoring
18. Premarket and postmarket balance
• Allow greater premarket uncertainty only under clear limits. Strong postmarket monitoring
may justify some premarket uncertainty when actions are reversible, monitoring is timely and
representative, harm can be detected before it spreads, and the manufacturer can quickly restrict,
roll back, or suspend the function. Recommendation: Do not substitute postmarket monitoring for
robust premarket evidence when a function is autonomous, irreversible, novel, or high consequence.
Define minimum monitoring capabilities and response times by risk profile. Example: A clinicianconfirmed, guideline-based recommendation may rely more on process measures and postmarket
outcomes than an autonomous thrombolysis action, which needs stronger premarket evidence.
19. Monitoring approaches
• Use several complementary monitoring methods. Periodic re-benchmarking, representative
clinician review, drift monitoring, real-world behavior and outcome tracking, and review of
complaints, near misses, overrides, and escalation patterns answer different questions.
Recommendation: Set the review cadence according to decision influence, consequence, use
volume, and time to harm. Trigger reassessment after changes to models, prompts, retrieval,
guardrails, tools, workflow, intended use, data distribution, or standards of care. Example: A
medication-management system should be monitored both for decision quality and for downstream
kidney injury, electrolyte abnormalities, symptoms, hospitalization, and treatment burden.
20. Machine-based supervisory agents
• Use supervisory agents to screen and prioritize, not as the only safety control. Machine-based
supervisors can review more interactions than humans and flag patterns for clinical review, but they
are themselves safety-critical and may share the same blind spots as the system they monitor.
Recommendation: Evaluate the supervisor independently for detection of important failures, false
alarms, subgroup performance, drift, and shared dependencies. Require versioning, audit logs,
human sampling, escalation thresholds, and a fallback when the supervisor is uncertain or
unavailable. Monitor both the clinical agent and its supervisor. Example: A supervisor built on the
same foundation-model family may miss the same unsafe medication interaction, so targeted human
review and independent test cases remain necessary.
22. Re-benchmarking after changes
• Scale re-evaluation to expected clinical impact. The amount of code changed is a poor proxy for
risk. Changes to the foundation model, prompts, retrieval, guardrails, tools, interface, workflow,
autonomy, or intended use can all alter clinical behavior. Recommendation: Define change tiers in
advance. Low-impact changes may be handled within the quality system, moderate changes may
require targeted re-benchmarking and shadow testing, and high-impact changes may require
broader clinical confirmation or FDA review. Example: A small prompt change that weakens
emergency escalation language may require more re-evaluation than a large back-end change that
does not affect clinical output.
23. Adapting Predetermined Change Control Plans
• Use Predetermined Change Control Plans for bounded changes and monitoring for
everything else. These plans are most useful when future changes have defined limits, test
methods, acceptance criteria, and rollback procedures. They are less useful for open-ended changes
that cannot be meaningfully prespecified. Recommendation: Require clear approved change
boundaries, version traceability, automated regression testing, staged deployment, rollback
capability, and triggers for additional review or a new submission when changes exceed those
boundaries. Example: Adjusting a dosing threshold within a prespecified range may fit a plan;
adding autonomous medication ordering to an educational tool should not.
24. Third-party foundation-model changes
• Use both contractual notice and behavior testing. Model identifiers and release dates may not
reveal silent changes to an underlying third-party model. Manufacturers therefore need a small set
of safety-critical regression cases and a baseline for expected behavior. For deterministic models,
fixed cases can be compared output by output; for stochastic models, repeated-run distributions
should be compared with baseline. Recommendation: Require version pinning where feasible,
advance notice and change logs from model providers, repeated-run regression and distributionshift testing, staged release, rollback, and escalation when behavior changes without a declared
version change. Example: If the model name stays the same but a triage prompt begins producing a
different pattern of responses, the manufacturer should detect the shift before broad deployment.
Other Topics
25. Foundation Model Device Master Files
• Potentially useful only if current, version-specific, and actionable. A voluntary Foundation
Model Device Master File could reduce duplicate review and provide a consistent source of
information about widely reused models. At minimum, it should cover model and version identity;
training-data provenance at an appropriate level; supported uses and known limitations; healthcare
and subgroup evaluation; safety controls; change history and notice commitments; logging; and
incident response. Recommendation: Define a standard core and update cadence, allow confidential
reference by device sponsors, and require each sponsor to verify the exact model version it deploys.
 When model developers do not participate, allow contractual disclosures, independent attestations,
regulator-accessible system cards, and sponsor-run change detection. Example: Several device
manufacturers using the same foundation model could reference one current file while remaining
responsible for their own use, configuration, and clinical evidence.
26. Agentic AI systems
• Evaluate agentic systems end to end. Human errors can also propagate, but an agentic system may
plan, act through multiple tools, and check its own work using the same mistaken premise, with
little independent review between steps. Each clinical function should have a defined autonomy
mode and operating constraints. Recommendation: Test planning and sequencing, tool selection and
authorization, verification of tool outputs, memory and state, boundary adherence, error recovery,
safe stopping and deferral, and resistance to prompt injection. Require auditable action traces and
human review before irreversible or high-consequence actions, particularly early in deployment.
Postmarket monitoring should capture complete action sequences, near misses, overrides, tool
failures, and failures to stop. Example: In heart failure care, a misread symptom, vital sign,
laboratory value, or image could lead to an unsafe patient message and medication order. The agent
should recognize missing safety data and stop before the action is executed.