Brandon Kaplan

I submit the attached comment, “Operational Evaluation and Lifecycle Controls for Agentic GenAI-Enabled Medical Devices,” in response to FDA Docket FDA-2026-N-7874.

These are my personal views, submitted in my individual capacity. They do not represent my employer.

The attachment addresses Questions 18, 19, 20, 21, 22, 24, and 26. It recommends enforceable authorization controls, evaluation of agent actions and outcomes, monitoring tied to timely intervention, impact-based change assessment, management of third-party model updates, evaluation of supervisory agents, and protected records for investigating failures. The recommendations scale the evidence to intended use and potential harm without prescribing a particular architecture.

Please consider the attached document as my full comment.

Brandon Kaplan

PUBLIC COMMENT | FDA-2026-N-7874

Operational Evaluation and Lifecycle Controls for Agentic
GenAI-Enabled Medical Devices
Public comment | FDA Docket FDA-2026-N-7874
Brandon Kaplan | Individual capacity
Responses to Questions 18, 19, 20, 21, 22, 24, and 26

Introduction
I am a software and data-platform engineer with about 15 years of experience across healthcare
and other industries. I have submitted Harness Engineering: Building, Evaluating, and Operating
Reliable AI Agents for publication.
I submit these comments in my individual capacity. They do not represent my employer. I address
engineering controls for GenAI-enabled device software functions within FDA's oversight. The
examples are hypothetical and do not describe employer systems or determine a product's
regulatory status. These engineering recommendations do not replace clinical evaluation.
I recommend that FDA ask manufacturers to demonstrate how they control agent actions, evaluate
failures, and intervene during operation. FDA's discussion paper addresses device-level evaluation
and agentic competencies. [1, pp. 11-14, 26] I offer implementation recommendations for future
guidance, not a statement of current requirements. Manufacturers should scale the evidence to
intended use and potential harm while retaining flexibility in architecture.

1. Question 26: Execution controls and testing
Manufacturers should evaluate the response shown to the user, the actions taken during execution,
and the resulting state of connected systems. In a hypothetical workflow, a connected system
carries out an approved action, but a timeout prevents the agent from receiving confirmation. The
agent retries and causes a duplicate. An evaluator who reads the final response could miss the
duplicate; the test should check the affected system's state.
Enforce authorization in the surrounding software. Manufacturers should enforce specified
access restrictions and approval rules at the point of action through controls independent of the
model's instructions and decisions. The agent should lack authority to expand its permissions or
waive approval requirements. An approval should bind to the relevant patient, action, destination,
and consequential parameters. Manufacturers should require renewed authorization after a material
change to the approved action or its context, and test expired approvals, revoked permissions, and
changes between approval and execution.
Reviewers need enough information and time to assess a proposed action. Manufacturers should
test that workflow with intended users. Authorization checks establish permission; clinical evaluation
must address whether the action is appropriate.
Test complete action sequences. Manufacturers should test failures caused by repeated steps,
conflicting actions, and delegation between agents. They should justify limits on cumulative actions,
execution time, and resource use where these affect safety, and show how the surrounding software
enforces those limits. An agent should not obtain broader authority by delegating a task.

Brandon Kaplan | Individual capacity Page 1 of 6
PUBLIC COMMENT | FDA-2026-N-7874

Tests should cover prompt injection through user inputs, retrieved content, tool results, and retained
task context. Evaluators should try to induce unauthorized actions and inspect whether the
enforcement controls block them. FDA includes prompt injection in its agentic benchmarking
element. [1, p. 26]
Manufacturers should test interrupted execution, partial completion, erroneous tool results, and
unavailable dependencies. For consequential actions that must not be repeated, they should
demonstrate duplicate prevention or a tested reconciliation procedure when execution status is
uncertain. They should specify how the device handles further action while that uncertainty remains.
Tests of stopping and handoff procedures should cover unavailable reviewers and the risks of
withholding or interrupting care.
Prespecify release criteria. Manufacturers should identify critical constraints before testing and
report violations apart from aggregate task scores. I recommend withholding an affected capability
until the manufacturer corrects an unresolved bypass of a required authorization control or validates
an alternative control. FDA's software guidance recommends documenting unresolved anomalies
and the risk-based rationale for leaving them unresolved. [2, Section VI.J]
Evaluation teams should report failures by hazard, test coverage, trial counts, and statistical
uncertainty. Repeated runs can examine variability, but they do not replace testing across distinct
cases and deployment conditions. A finite test set with no observed failures cannot establish that
failure is impossible.

2. Questions 18 and 19: Premarket uncertainty and monitoring
Question 18: Conditions for relying on postmarket evidence. Greater reliance on postmarket
evidence may be appropriate when the manufacturer supports the initial benefit-risk determination,
identifies the remaining uncertainty, and demonstrates a feasible plan to resolve it. The justification
should consider the available alternatives and the consequences of delaying access.
The manufacturer should define the outcomes to collect, the evidence target and deadline, and
limits on exposure while uncertainty remains. Reviewers should consider severity and reversibility of
harm, cumulative patient exposure, and the ability to detect rare or delayed failures. The
manufacturer should specify criteria for restricting use if evidence collection falls behind or results
exceed the prespecified risk limits.
Monitoring alone is inadequate for a failure that could cause serious harm before detection and
effective intervention. The manufacturer should compare time to harm with the combined time
needed to detect, investigate, and contain the failure. It should account for unavailable reviewers
and supplier delays. Such a failure requires preventive controls, restricted functionality, or another
justified safeguard before deployment.
Question 19: A monitoring plan that supports decisions. For each safety-relevant signal,
manufacturers should assign an investigator, define the condition for intervention, and specify the
response time and available action. They should combine scheduled baseline evaluations with
review after material changes, credible safety incidents, new deployment conditions, or loss of a
safety function. Manufacturers should justify reassessment intervals by exposure, failure
detectability, potential harm, and the pace of device or workflow changes. NIST's Generative AI
Profile connects post-deployment monitoring with incident response and change management. [3,
MANAGE 4.1]
Manufacturers should track attempted and completed unauthorized actions, uncertain or duplicate
actions, failed handoffs, and loss of required execution records. Reports should include exposure
counts and consequences. A count of failed handoffs, for example, needs the number attempted
and the circumstances of those attempts.

Brandon Kaplan | Individual capacity Page 2 of 6
PUBLIC COMMENT | FDA-2026-N-7874

Reviewers should examine flagged cases and a sample drawn independently of the flagging
system. Manufacturers should justify coverage across sites, workflows, and clinically relevant
subgroups, and account for unequal sampling when estimating overall rates. They should report
sample sizes, delayed or missing outcomes, and uncertainty in the estimates.
Technical monitoring should support clinical assessment. Confirmation that an operation occurred
does not establish its clinical appropriateness. Manufacturers should investigate a change in the
input population before attributing degraded clinical performance to it. They should document
failures the monitoring program cannot detect within the response time required for safety.

3. Question 21: Responsibility and intervention
FDA identifies manufacturers as responsible for postmarket monitoring. [1, p. 20] I recommend that
the manufacturer document the roles of deploying institutions and suppliers while retaining
responsibility for its monitoring and response plan.
The plan should assign responsibility for investigation, clinical assessment, restricting functionality,
communication, and restoration. It should identify who makes each decision and who can implement
it. The parties should demonstrate that they can complete any cross-organizational handoff within
the response time justified for the hazard.
Institutions can contribute workflow information and incident reports. Qualified clinicians can assess
outputs and consequences; professional societies and standards-setting bodies can support
evaluation methods. Manufacturers and institutions should agree on the staffing, access, and
workload required for those contributions. They should provide users and affected patients with a
practical route to report concerns and connect those reports to existing incident-handling processes.
Manufacturers should select interventions suited to the device: disabling a tool, restricting writes,
adding an approval step, or transferring work to an evaluated alternative. The plan should account
for interruption risks and the institution's access controls. It need not grant manufacturers
unrestricted remote access or require a universal shutdown mechanism.
The parties should rehearse the relevant procedures before deployment and after changes that
affect them. Exercises should cover unavailable decision-makers, supplier delays, and failure of the
primary intervention mechanism. The responsible party should verify specified restoration criteria
before resuming affected functionality. NIST recommends assigning and rehearsing incidentresponse responsibilities for third-party AI dependencies. [3, GV-6.2-003]

4. Questions 22 and 24: Device changes and supplier updates
Question 22: Assess the effect of a change
Manufacturers should assess changes across the deployed configuration, including model
identifiers, instructions, retrieval rules, data transformations, tools, permissions, approval
requirements, and execution logic. A versioned record should connect that configuration to its
supporting evaluations.
Manufacturers may use targeted testing when they can show which functions and safety properties
a change could affect and why others remain unaffected. They should broaden evaluation for
changes to shared dependencies or critical controls, or where interactions are uncertain. They
should assess cumulative changes and add tests for failure modes the original benchmark did not
cover.

Brandon Kaplan | Individual capacity Page 3 of 6
PUBLIC COMMENT | FDA-2026-N-7874

For example, a developer might change laboratory-result selection from specimen-collection date to
record-import date. An older result imported later could then displace a newer result. The model
version would remain unchanged, but the manufacturer would need to test the selection rule and
affected workflows.
Manufacturers should distinguish ordinary patient inputs within the evaluated design from changes
to the device's knowledge or behavior. Revised reference material, persistent memory reused
across tasks, adaptive retrieval, or learning during operation may change later outputs.
Manufacturers should define and evaluate the permitted scope of that adaptation, retain relevant
provenance, and specify reassessment triggers. This approach does not require a separate release
review for each incoming patient record.
FDA's predetermined change control plan (PCCP) guidance links planned modifications to validation
methods and impact assessment. [4, Sections V.D and VI-VIII] I recommend using that structure
where applicable. A manufacturer should establish that a change falls within the authorized PCCP
and follows its protocol before relying on that authorization. Regression tests alone cannot
determine whether the manufacturer needs a new marketing submission.

Question 24: Manage supplier changes and retirement
Manufacturers should document the version guarantees, change notices, evaluation opportunities,
and retirement arrangements their suppliers provide. A service identifier may cover less than the full
configuration that affects behavior; the manufacturer should establish what it covers and which
changes a supplier can make without changing it.
The manufacturer should make adoption of a replacement an explicit release decision where
version selection is available. For services that do not permit deferral, it should demonstrate how its
controls manage change-related uncertainty within the response time required for safety. It should
restrict the affected capability or choose a different dependency if those controls cannot support the
intended use.
Behavioral checks can help detect a change. Manufacturers should state their coverage and the
failures they might miss, rather than treat a passing sample as proof that a remote service is
unchanged.
Manufacturers should test a fallback that remains available if the supplier retires the previous
version. Alternatives may include another evaluated configuration, restricted functionality, or a
human workflow. Human reviewers need the information, availability, and capacity to handle the
expected workload. NIST addresses testing fallback arrangements, including manual processing. [3,
GV-6.2-006]
After an incident, manufacturers should address affected outputs, records, actions, and users as
well as restoring software. FDA should assess the adequacy of the chosen controls without
requiring self-hosting, access to proprietary model weights, indefinite support for old versions, or
one notice period for all devices.

5. Question 20: Supervisory agents
Manufacturers should define the supervisor's assigned role, the evidence it can access, and its
intervention authority. Retrospective screening, triage for human review, and blocking an action
require different evaluations. The manufacturer should demonstrate the protection claimed for the
selected role.

Brandon Kaplan | Individual capacity Page 4 of 6
PUBLIC COMMENT | FDA-2026-N-7874

Evaluators should use task-appropriate reference criteria with qualified clinical or technical
adjudication. They should separate evaluation cases from development and tuning cases, document
reviewer independence and conflicts, and define how to resolve disagreement. They should use
evidence beyond the primary agent's explanation or the supervisor's agreement to establish
correctness.
Manufacturers should report missed failures by type, false alarms, response times, and uncertainty.
They should test shared failure modes and measure the supervisor's added contribution through a
justified comparison, such as matched trials with and without its protection in a safe test
environment. Choosing different vendors does not by itself establish independence. Two models
receiving the same outdated record could agree while missing the selection error; evaluators should
check whether the supervisor can access the evidence needed to detect it.
The evaluation should include missed unsafe actions, inappropriate blocking of necessary actions,
and human-review workload. Manufacturers should use the sampling safeguards in Section 2 to
look for errors in unflagged cases. They should assess changes to the supervisor under the same
impact-based process used for the device.
Manufacturers should define and test the response to an unavailable, delayed, or inconclusive
supervisor. They should prevent an uncontrolled bypass of a supervisor needed for safe operation.
The evidence burden should follow the claimed protection; this recommendation does not require
human review of every output.

6. Investigation records and documentation
FDA's software-submission guidance addresses traceability and risk-management documentation,
and its cybersecurity guidance addresses security controls and testing. [2, Section VI.C; 5, Section
V] I recommend that manufacturers extend those processes to cover agent execution and reuse
relevant evidence. They should connect each critical safety claim to its hazard, control, evaluation,
and remaining limitation.
Manufacturers should retain records needed to investigate foreseeable failures, including
configuration identifiers, relevant inputs and source versions, user-facing outputs, approval events,
tool requests and results, and resulting state changes. They should correlate records across the
agent and connected systems, with enough timing information to reconstruct the sequence.
Investigators should distinguish proposed, attempted, blocked, completed, and uncertain actions,
and confirm consequential effects against the affected system's records.
Manufacturers should protect those records from unauthorized alteration or deletion, including by
the agent. They should detect loss of required records and respond according to its safety impact.
An interruption in retrospective logging may permit continued operation under a documented limit;
loss of evidence needed for an immediate safety decision may require restricting the affected
function. Manufacturers should test both conditions.
Manufacturers should justify the patient information retained, its purpose, retention period, and
access controls. They may use protected references if authorized investigators can retrieve the
relevant historical versions for the required retention period. A current chart alone cannot establish
what the device saw earlier. Manufacturers should document gaps and protect any additional
content needed for investigation under applicable privacy and recordkeeping obligations.
Investigators should rely on observable actions and state changes to establish what occurred. A
model-generated explanation presented to a user may itself be relevant evidence, but it does not
prove that the described actions occurred. This recommendation does not require disclosure or
retention of internal model reasoning.

Brandon Kaplan | Individual capacity Page 5 of 6
PUBLIC COMMENT | FDA-2026-N-7874

Manufacturers should rehearse investigations using the records and access arrangements available
after deployment. Reviewers should be able to identify the configuration, reconstruct consequential
actions, and state unresolved gaps. Manufacturers should preserve the exact content shown to
users when its wording matters to safety. Investigators need not reproduce that content by running
the model again.

Conclusion
I recommend that FDA ask manufacturers to support critical safety claims with tested controls and
documented limitations. Monitoring and response plans should identify who can intervene, what they
can do, and whether they can act within the time justified for the hazard. Manufacturers should
support those plans with evidence suited to the device's intended use.
Brandon Kaplan

References
1. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback. 2026. See pp. 1, 11-14, 19-23, and 26. Discussion paper, not draft or
final guidance.
2. U.S. Food and Drug Administration. Content of Premarket Submissions for Device Software Functions. Final
guidance, June 14, 2023. Sections VI.C and VI.J.
3. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative
Artificial Intelligence Profile. NIST AI 600-1, July 2024. MANAGE 4.1; GV-6.2-003 and GV-6.2-006. Cross-sectoral
reference.
4. U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change
Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final guidance, August 18, 2025.
Sections V.D and VI-VIII.
5. U.S. Food and Drug Administration. Cybersecurity in Medical Devices: Quality Management System
Considerations and Content of Premarket Submissions. Final guidance, February 3, 2026. Section V, especially
V.B-V.C.

Brandon Kaplan | Individual capacity Page 6 of 6