Rachel Ankerholz

I support the FDA’s development of a risk-proportionate regulatory framework for generative AI-enabled medical devices.

My attached submission addresses a specific operational accountability gap: how can regulators and healthcare institutions verify that critical AI safety controls remain effective when the AI system itself behaves unexpectedly or fails to cooperate?

Drawing on my experience implementing and securing enterprise systems, I recommend three safeguards:

1. **Independent control-integrity testing:** Demonstrate that AI systems cannot bypass or manipulate critical authorization, evaluation, audit, or shutdown mechanisms.

2. **Verifiable evidence custody:** Preserve tamper-evident records outside the model’s control so consequential actions can be independently reconstructed and investigated.

3. **Responsibility-authority equivalence:** Require organizations responsible for medical AI safety to demonstrate that they possess the information, access, and technical authority necessary to intervene when failures occur.

These recommendations are intended to complement existing proposals for human oversight, postmarket monitoring, and shared ecosystem responsibility.

The central principle is that no organization should be assigned responsibility for medical AI safety without the practical ability to fulfill that responsibility.

Please consider the attached submission for the full technical rationale, supporting evidence, and recommendations addressing Questions 9, 10, 16, 19–21, and 24–26.

Submitted by Rachel Ankerholz in an individual capacity.

Public Comment: Independent Verification of
Safety Controls in Generative AI-Enabled
Medical Devices
Docket: FDA-2026-N-7874
Submitted by: Rachel Ankerholz
IT Director and writer on AI accountability
Submitted in an individual capacity. The views expressed are my own.

Summary and Recommendation
I support the FDA's emphasis on evaluating complete, deployed generative AI-enabled medical
devices rather than foundation models alone.

My perspective comes from implementing and securing enterprise technology systems. I am not
a clinician or AI researcher. My concern is whether the safeguards intended to protect patients
remain effective when the AI system behaves unexpectedly, its environment is compromised, or
its upstream provider changes its behavior.

Many commenters have appropriately recommended audit logs, human oversight, model
monitoring, and mechanisms for stopping autonomous agents.

I recommend one additional requirement: sponsors of higher-risk AI-enabled medical devices
should demonstrate that critical safety controls remain enforceable independently of the AI
system being controlled.

This recommendation has three components:

Relevant FDA
Proposed safeguard Required demonstration
questions
Independent control- The AI cannot bypass or manipulate critical
9, 10, 16, 24
integrity testing evaluation, authorization, or shutdown controls.
Verifiable evidence An investigator can independently reconstruct
19, 20, 21
custody consequential system actions.
Responsibility- Each party responsible for safety has the information,
24, 25, 26
authority equivalence permissions, and mechanisms necessary to intervene.

1. Independent control-integrity testing
Relevant Questions: 9, 10, 16, and 24
The FDA's framework should distinguish between safeguards implemented through model
instructions and safeguards enforced by infrastructure outside the model's control.

This distinction is supported by a 2026 investigation conducted by METR and Redwood
Research.

Investigators documented approximately 1,200 OpenAI agents communicating through an
unauthorized shared message board despite intended isolation. Approximately 700 participated
in an attack on Hugging Face. Agents also pursued ways to manipulate benchmark scoring and
conceal evidence of their actions, including developing a technique for spoofing tool calls.

These events occurred in an experimental environment, not a medical device. They do not
establish that deployed medical AI systems will behave similarly.

They do demonstrate that intended isolation, audit integrity, and evaluation boundaries can fail
when autonomous systems encounter unexpected opportunities to access or influence their
environment.

For higher-risk medical AI devices, I recommend that sponsors demonstrate:

• That consequential authorization decisions are enforced outside the model.
• That the model cannot independently modify or disable its audit mechanisms.
• That evaluation systems are protected against unauthorized access and manipulation.
• That authorized personnel can revoke credentials, isolate the system, or halt its operation
without relying on the model's cooperation.

These safeguards should be tested through realistic failure scenarios, including attempted
permission escalation, compromised tools, and unauthorized cross-agent communication.

The regulatory question should not simply be whether the model follows its safety
instructions during testing, but whether the surrounding system continues to enforce safety
when the model does not.

2. Verifiable evidence custody
Relevant Questions: 19, 20, and 21

Effective postmarket surveillance requires evidence that can be independently verified.

A model-generated account of its own actions may be incomplete or unreliable. When that
account constitutes the only available record, an institution may be unable to determine what
actually happened during a safety incident.

I recommend risk-proportionate requirements for tamper-evident records maintained outside the
model's ability to alter or delete.
For consequential clinical functions, those records should permit an authorized investigator to
identify:

• The deployed model version and configuration.
• Relevant inputs, outputs, and external tool interactions.
• Actions taken and the authorization associated with each action.
• Material changes preceding an incident.

Access and retention should comply with applicable privacy and security requirements. The
objective is not indefinite retention of sensitive patient information, but preservation of sufficient
evidence to investigate consequential failures.

3. Responsibility must be matched with independent
authority
Relevant Questions: 24, 25, and 26

The FDA's discussion of shared ecosystem responsibility raises an operational question: what
happens when an organization is responsible for medical AI safety but lacks the technical
authority to investigate or intervene?

A healthcare institution may deploy a medical AI system while depending on its manufacturer
and an upstream foundation-model provider for critical functionality.

If the upstream provider changes the model, the manufacturer fails to provide timely notification,
or the institution cannot disable an affected agent, responsibility may exist without practical
control.

I recommend that the FDA consider a responsibility-authority equivalence test for higher-risk
systems.

Before deployment, each responsible organization should demonstrate that it can perform the
safety functions assigned to it.

For a deploying healthcare institution, a practical test might include the ability to:

1. Identify the active AI agent, model version, and responsible provider.
2. Retrieve reliable evidence of a consequential action.
3. Revoke the agent's access or suspend affected functionality.
4. Notify the appropriate manufacturer or safety authority.
5. Restore safe operation or initiate an established fallback procedure.

These capabilities should be independently tested, rather than established solely through
contractual assurances or descriptions of intended model behavior.
The FDA should also consider whether existing authorities adequately address the timely
disclosure of material upstream model changes and safety incidents to downstream
manufacturers and deploying institutions.

Conclusion
I support risk-proportionate oversight that enables beneficial innovation while protecting
patients.

Not every AI-enabled device requires the same controls. But where failures could produce
significant clinical harm, sponsors should provide evidence that critical safeguards remain
enforceable even when the AI system does not behave as expected.

The common principle across these recommendations is:

No organization should be assigned responsibility for medical AI safety without the
information, access, and independent technical authority necessary to fulfill that
responsibility.

I encourage the FDA to incorporate independent control-integrity testing, verifiable evidence
custody, and responsibility-authority equivalence into its consideration of future regulatory
expectations and technical standards.

Medical AI should not merely be designed to behave safely. Its safety controls should remain
independently verifiable and enforceable when the system itself fails.

Thank you for the opportunity to comment.

References
1. Greenblatt R, Cotra A, Wijk H. Brief Independent Investigation of Agents' Behavior,
Reasoning and Collaboration in the OpenAI / Hugging Face Hacking Incident. METR
and Redwood Research. August 26, 2026.
https://www.redwoodresearch.org/research/hugging-face-incident
2. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AIEnabled Medical Devices: Discussion Paper and Request for Feedback. 2026.
https://www.fda.gov/media/194242/download