Operation Citizen (Thom Barrett)

I am submitting the attached comment on the discussion paper
"Considerations for the Regulation of Generative AI-Enabled Medical
Devices."

The comment argues that FDA should treat generative and agentic medical
devices as control systems, not only as models subject to performance
testing. Accuracy matters, but most failures in deployed healthcare
systems do not begin with an incorrect output. They begin when a system
is used for a purpose different from the one tested, when a reviewer
lacks a meaningful ability to disagree, when a configuration changes
after testing, when a clinical statement loses its source, or when a
promised remedy cannot be delivered. Benchmark performance does not
reveal those failures.

Drawing on the AI Assertion and Chain Integrity Framework (AACIF) and
three public cases — the nH Predict post-acute-care algorithm, ambient
clinical documentation, and a non-AI failure of recourse at Prospect
Medical Holdings — the comment sets out four families of controls, each
tied to the paper’s own questions:

I. Purpose, authority, incentives and reversibility (Questions 1-6 and
26): state the objective that actually drives deployment, treat financial
incentives as part of the control environment, test whether human review
functions, and measure autonomy by what can be reversed.

II. Competency and versioned evidence (Questions 7-17 and 25): the
deployed system, not the model alone, is the device; evidence should stay
tied to the configuration it tested; a system can change clinically even
when the model does not.

III. Persistent chain identity and reconstructable action (Questions
18-24 and 26): clinically consequential AI outputs should remain
traceable after they enter records and downstream systems, and higher-risk
agentic systems should keep records sufficient to reconstruct what they
did.

IV. Effective recourse and closed-loop assurance (Questions 18-24): a
declared safeguard and the capacity to perform it are separate claims;
override, appeal and reversal data belong in postmarket monitoring; and
external review needs owners, deadlines and verified closure, as GAO-26108499 found of FDA’s own Third Party Review Program.

Each recommendation is written so that a reviewer could inspect it: a
defined trigger, a required response, and evidence that the control
operated. All twelve are to be applied in proportion to autonomy,
clinical consequence, irreversibility, complexity, and the opportunity
for effective human review.

A companion demonstrator maps all 26 questions in the discussion paper to
the controls that answer them, with every figure tied to a listed source:
https://aacif-fda-demonstrator.netlify.app/

Thom Barrett, Founder, Operation Citizen; author of the AI Assertion and
Chain Integrity Framework (AACIF). This comment was prepared with the
assistance of Claude, an AI assistant made by Anthropic.

Where to Send the Letter
September 15, 2026 · four channels, ranked by whether anyone is obliged to answer.

Short answer
Yes — but a committee is the weakest of the four channels available, and sending it there
first would waste the strongest one.
Rank by obligation to respond, not by prestige.

1. The open comment docket — the only channel with a legal duty attached
In notice-and-comment rulemaking the agency must consider comments and respond to
significant ones in the preamble of the final rule. That is a legal obligation. Nothing else on
this list carries one.
S7-2026-31, the pay-to-play rescission, is open until November 9, 2026. A comment
there becomes part of the administrative record, must be addressed if significant, and is
citable afterward — including in litigation brought by someone else.
MLC-01 and MLC-02 are directly on point for that docket: the Commission is proposing to
remove a record-keeping requirement governing how advisers approach public pension
boards, while the instrument meant to measure those advisers’ leverage has never
operated. That is a comment, not a letter.

2. Inspectors General — they accept public complaints and publish
Unlike the Government Accountability Office, agency IGs take submissions from anyone.
They have jurisdiction over exactly the kind of finding the letter contains, and their reports
are public.
• Interior OIG — MLC-03 (the ledger that does not reconcile), MLC-06 (a
discretionary channel with no floor), MLC-08 (an acknowledged record withheld).
• FHFA OIG — MLC-07 (ten years, no qualifying purchase).
• SEC OIG — MLC-02, on a control adopted and never operated.
An IG referral is the closest thing to an actual audit anyone outside government can trigger.
It is the highest-value channel after the docket.

3. Congressional committees — send for the record, expect nothing
Worth knowing before you spend effort here: GAO cannot be asked to do work by a
member of the public. Requests come from committees and members. A citizen who
identifies a control failure has no route to the government’s own auditor except through a
legislator.
That limitation belongs in the series. It is MLC-09 in a different costume — the aggregation
function exists and is reachable only by the people whose inaction produced the finding.
Send anyway, to four committees: Senate Banking and House Financial Services for the
disclosure and housing-finance comments; Senate Energy and Natural Resources and
House Natural Resources for the land comments. Send to the committee office rather than
to individual members, and ask one question with a date attached: does the committee
intend to request GAO review of any of these items, and by when?
Expect no reply. The absence is usable — and it is the same finding you already reached
about abdication, now with a letter and a date supporting it.

4. The parties themselves — already covered
The right-of-reply letters are the fastest channel and the most likely to produce an actual
answer, because a company or agency press office responds to a publication deadline in a
way a committee does not.

Sequencing
1. Now — file on S7-2026-31. It closes November 9 and it is the only door with a legal
duty behind it.
2. At publication of Part 0 — send the seven right-of-reply letters, ten business days.
3. With the management letter — IG referrals and committee letters go the same day
the letter publishes, so the piece can state that they were sent and to whom.
4. Later, separately — the FOIA requests for the NEC parcel count and the Yosemite
exchange file. They run long and become their own piece.

Two things to handle
Disclose it. The series would be reporting on proceedings you are a participant in. One line
in the standing paragraph: the author filed a comment in this proceeding and sent these
findings to the named offices on this date. Disclosed, it is unremarkable. Undisclosed, it is
what someone else writes about you.
Do not send the letter to the committees before publication. Advance notice to an
oversight committee is advance notice to the industry that talks to it. Publish, then send.

Public Comment — Generative and Agentic AI-Enabled
Medical Devices
Docket: FDA-2026-N-7874

Re: Considerations for the Regulation of Generative AI-Enabled Medical Devices —
Discussion Paper and Request for Feedback

Submitted by: Thom Barrett, Founder, Operation Citizen; author of the AI
Assertion and Chain Integrity Framework (AACIF)

Date: October 12, 2026

Companion demonstrator: https://aacif-fda-demonstrator.netlify.app/

I welcome the agency's decision to ask these questions before generative and agentic systems become deeply
embedded in clinical practice. It is important to establish sound controls now, because weak controls will be harder
to correct later. These systems are not merely more capable versions of earlier AI devices. They can reason across
several steps, call external tools, rely on foundation models their developers do not control, and act within clinical
workflows with fewer opportunities for human intervention. My comments are directed to the agency's discussion
paper. I do not treat that paper as proposed policy, and I do not assume that FDA's present authority reaches every
recommendation offered here.

FDA should treat generative and agentic medical devices as control systems, not only as models subject to
performance testing. Accuracy remains important, but it is not sufficient. Most failures I have studied in deployed
healthcare systems did not begin with an incorrect model output. They began when a system was used for a purpose
different from the one tested, a reviewer lacked a meaningful ability to disagree, a configuration changed after
testing, a clinical statement lost its source, or a promised remedy could not be delivered. Benchmark performance
will not reveal those failures.

I developed the AI Assertion and Chain Integrity Framework (AACIF) to examine the full path from an AI output to
institutional action. The framework begins with the data and model, then follows the instructions, tools, human
review, downstream systems, audit, and recovery. I applied AACIF to public cases involving the nH Predict postacute-care algorithm, ambient clinical documentation, and a non-AI failure of recourse at Prospect Medical. That
work supports four families of controls responsive to the paper's questions:

1. Purpose, authority, incentives and reversibility (responding to Questions 1–6
and 26).
2. Competency and versioned evidence (Questions 7–17 and 25).
3. Persistent chain identity and reconstructable action (Questions 18–24 and
26).
4. Effective recourse and closed-loop assurance (Questions 18–24).
The companion demonstrator maps all 26 questions in the discussion paper to responsive controls. It applies the four
control families to two public cases and shows how a clinical system can acquire new authority while its underlying
model remains unchanged. Every figure links to a listed source.
These controls should be applied in proportion to risk. A tool that drafts text for clinician review does not require the
same controls as an agent that can place orders, communicate with patients, or affect access to care. Controls should
become more demanding as autonomy, clinical consequence, irreversibility, and institutional reliance increase.

I. Purpose, authority, incentives and reversibility
A. State the goal that drives real-world use.
Intended use describes what a device is said to do. It does not necessarily show what an institution seeks to gain
from using it. The nH Predict case illustrates that distinction. The algorithm estimated how long Medicare
Advantage patients would require post-acute care, and those estimates informed coverage decisions. Court records
in Estate of Lokken v. UnitedHealth and an October 2024 Senate report raised a question that accuracy testing alone
could not answer: Was a tool presented as supporting care also being used to reduce care and cost? I refer to this as
objective substitution. A system may be suitable for one stated purpose while the incentives surrounding it drive a
different one.

For higher-risk systems, the record should state the operative objective at the outset of deployment. AACIF calls this
Layer 0. The record should identify the clinical purpose, any financial or operational goals that may influence use,
the measures applied to the system and its reviewers, potential conflicts with the patient's interest, and the controls
used to address those conflicts. This is especially important when an output can affect denial of care, treatment
duration, referrals, authorization, discharge, scheduling, or eligibility. FDA should not evaluate a system as purely
clinical when the institution rewards a different result.

B. Treat financial incentives as part of the control system.
Medical AI operates within an economic system. An organization may benefit financially when care is denied,
shortened, delayed, or redirected. A reviewer may be evaluated on speed or agreement with the algorithm. A vendor
may be compensated from claimed savings. These pressures can alter how an output is interpreted, challenged,
overridden, or enforced. The same model output may therefore produce a different result for the patient.

High-consequence uses should include an incentive and beneficiary record. It should identify who bears the cost of a
false positive or false negative, who benefits from each major outcome, whether compensation depends on those
outcomes, whether an override harms a reviewer's performance rating, and whether the system is tuned for cost,
speed, or lower utilization. That information is relevant to real-world safety in the same way benchmark accuracy is
relevant.

C. Test whether human review works.
Human review can exist formally and still fail in practice. A reviewer may lack necessary facts, work under
production pressure, be evaluated on agreement with the algorithm, or face a penalty for disagreement. The person
remains "in the loop" in name while exercising little independent judgment. A signature box does not establish
meaningful oversight.

For high-consequence systems, FDA should require evidence that human review is effective. Reviewers need
authority to override the system, sufficient information and time to evaluate the output, freedom from penalties for
disagreement, and a clear escalation path. Overrides and reversals should be recorded and monitored. Most
important, the downstream action must change when a reviewer intervenes.

D. Measure autonomy with reversibility.
Agentic systems present additional risk because they can take several steps and use tools with fewer points for
human review. Two systems built on the same model may have materially different risk profiles. A system that
drafts a note for clinician review does not exercise the same authority as one that sends an order, schedules care,
changes treatment, communicates with a patient, or controls another device.

A high-impact agentic system should have an authority and reversibility profile. It should identify the actions and
tools the system may use, approval requirements, human checkpoints, escalation triggers, stop conditions, and which
actions can be reversed, by whom, and within what period. It should also describe safeguards for actions that cannot
be reversed. Requirements should follow this risk profile rather than the model name alone.

II. Competency and versioned evidence
A. The full deployed system is the device.
A generative or agentic medical device is more than a model. It may combine a foundation model with fine-tuning,
instructions, retrieval sources, memory, external tools, safety layers, control logic, a user interface, permissions,
human review, and connections to other systems. Any one of these components can alter behavior even when the
model remains unchanged. Saying that a product "uses Model X" identifies the engine, not the complete vehicle.

A July 2026 AACIF public comment to the Federal Trade Commission made this point directly. For an agentic
system, the Layer 0 record should function as a bill of materials for the complete system, not merely as a model
card. FDA should tie the evidence to a versioned deployment envelope that identifies the exact configuration tested.

B. Evidence must stay tied to the setup it tested.
A benchmark result should not become a permanent certificate attached to a product name. If the model,
instructions, sources, tools, or authority change, the earlier evidence may no longer apply. Evidence should follow
the configuration. Each safety or effectiveness claim should identify the configuration that supports it. After a
material change, the record should state whether the earlier evidence remains applicable. Not every change requires
full retesting, but the decision should be documented rather than assumed. This is especially important when a
device depends on a third-party foundation model that can change outside the developer's control.

C. A system can change even when the model does not.
A system can change in a clinically significant way even when the model remains the same. Risk may change when
a tool is added, retrieval sources expand, system instructions change, memory is enabled, a threshold shifts, approval
becomes automatic, escalation rules change, the target population expands, or an output enters a new workflow.
Change control should distinguish model identity from system state because patients experience the complete
system.

D. State where the system can and cannot be trusted.
Competency should be expressed as a boundary. The record should identify the populations and clinical settings in
which the system has performed, settings that are excluded or weakly supported, foreseeable failure modes, how the
system handles uncertainty, how it detects inputs outside its tested range, and what it does next. "Clinically
validated" is not an adequate description when testing covered only a narrow population, workflow, data source, or
configuration. Users need the boundary.

III. Persistent chain identity and reconstructable action
A. A doctor's signature does not preserve the source of AI text.
AACIF's review of ambient clinical documentation used DAX Copilot as a public example. The concern was not
that the product had been shown to be defective. The concern was that AI-generated language could enter the
medical record as ordinary text while essential facts disappeared: which model and version produced it, which
configuration governed it, which sources were used, which language came from the AI, which language the clinician
changed, and which downstream systems later acted on it. Once that identity is lost, the event may be impossible to
reconstruct. Retention creates a separate gap. Each deployment may retain session audio, transcripts, and summaries
for a different period, yet that retention setting does not travel with the note. Months later, a clinician or auditor may
be unable to determine whether the source recording still exists.

Stress tests using several AI systems showed an important distinction. A clinician's signature can establish sign-off
without establishing independent validation. The reviewer may not have enough information to evaluate AIgenerated text as AI-generated text. The signature shows approval. It does not show that the reviewer knew which
portions required additional scrutiny.

B. Important outputs must stay traceable across systems.
Clinically significant AI outputs should retain enough metadata to remain traceable after they move into other
systems. The record may include the encounter ID; the time context was retrieved; source record IDs; the model,
version, adapter, prompt, and system configuration; the age of the knowledge used; output time; labels
distinguishing AI and human text; clinician edits and approval; downstream transmission; and checks for later
alteration. FDA need not prescribe a single format. The essential requirement is that a clinically significant AI
statement should not become untraceable after crossing a system boundary.

C. Agentic systems need records that can rebuild their actions.
When a system acts rather than merely advises, its record should be more complete. A high-risk agentic system
should preserve enough information to identify the configuration in use, the patient or workflow context, tools
called, key intermediate states, attempted and completed actions, warnings and errors, human approvals, overrides,
escalations, downstream effects, and any rollback or correction. This does not require storing every internal
calculation or private model reasoning. It requires enough operational evidence to reconstruct a consequential
action. The record should become more complete as patient risk, autonomy, irreversibility, the number of affected
systems, and the difficulty of repair increase. A drafting tool may require only configuration identity. A system
directing clinical action requires substantially more.

IV. Effective recourse and closed-loop assurance
A. A promised safeguard is not proof that it can work.
Some of the clearest lessons arise outside AI. My work on Prospect Medical Holdings examined its malpractice selfinsurance and captive-insurance structure. That case showed a basic control failure: a safeguard may exist on paper
while the money or authority needed to honor it is insufficient, unavailable, disputed, or held by an entity unable to
act. The reserve amounts in that matter require careful classification, so I do not rely on a single figure here. The
broader lesson is that a promised protection and the capacity to provide it are separate claims. Both require evidence.

The same principle applies to AI. An organization may state that it has human escalation, incident response,
rollback, corrective action, appeals, vendor support, and patient remedies. FDA should ask whether those protections
will function when needed. For high-consequence systems, each safeguard should identify the responsible party, the
person with authority to act, available resources, expected response time, contingency plans if a vendor fails, an
enforceable path for patient or clinician review, and the capacity to correct harm at scale. A rollback plan that cannot
be executed is not a control. An appeal right that cannot be meaningfully used offers little protection.
B. Track challenges and reversals after deployment.
Appeals and overrides also create valuable evidence. Postmarket monitoring should track appeal, override, and
reversal rates; time to resolution; harm during the waiting period; patterns by population, reviewer, and facility; and
reasons for reversal. A high reversal rate may reveal a problem that ordinary performance monitoring misses. A low
appeal rate should not automatically be treated as reassuring. It may reflect procedural friction, inadequate notice,
patient limitations, cost, or pressure not to challenge the system.

C. Outside review needs controls too.
GAO's September 21, 2026 report on FDA's Third Party Review Program, GAO-26-108499, shows that external
review also requires controls. GAO reported that FDA conducted 25 audits of third-party review organizations from
2000 through 2026. Several recent audits with known deficiencies took more than six months to resolve because
FDA had no deadline for completing the audit and communicating the result. GAO made one recommendation with
three parts: establish selection criteria, update post-audit practices, and set target deadlines. HHS agreed. If FDA
relies on external testing, third-party validation, conformity review, or foundation-model master files, the
arrangement should define reviewer qualifications and independence, scope and timing, reporting duties,
responsibility and deadlines for remediation, verification of closure, and consequences if remediation fails.

D. Assurance must close the loop.
Assurance is a closed loop: detect, assign, remediate, verify, close, and monitor for recurrence. A known issue that
remains open is not a completed control. A promised correction is not evidence of repair. A review that identifies a
problem without an owner or deadline creates information, not assurance.

V. A method for connecting each risk to an inspectable control
In July 2026, AACIF was used in a public comment to the Federal Trade Commission on AI accuracy and hidden
steering. A public demonstrator showed how the framework operates. That work produced a method I call the
preventive fork. For each failure, it asks: What failed? Where did it fail? What control should have existed? What
should that control have done? How could that action have changed the result? What evidence would show that the
control worked? The companion demonstrator applies the same method to this discussion paper.

The method converts broad AI risks into inspectable requirements. If an agent can call external tools, it needs a
defined permission boundary. An attempt to cross that boundary should trigger a block, escalation, or human
approval. The action record should show whether the gate functioned as intended. A reviewer can inspect that
evidence. General claims that a system is "safe," "transparent," or "human-centered" cannot be evaluated in the same
way.

VI. A clinical workflow that shows why the four families are distinct
A clinical system may initially summarize a patient's chart and suggest next steps without authority to act. The
health system may later allow it to retrieve additional records, flag a possible medication issue, send patient
instructions, schedule follow-up, prepare an order for approval, and transmit data to another clinical system. The
underlying model may remain unchanged while the system's authority and risk materially increase.

Each control family addresses a different part of that change. The purpose and authority record identifies what the
system may do, when clinician approval is required, whether cost or speed influences use, and how its actions can be
stopped. The competency and evidence record identifies the configuration tested and whether new tools or
workflows require further review. The identity and action record shows what the system retrieved, which tools it
used, what it attempted, what the clinician changed, and what moved downstream. The recourse record shows how
an error is detected, reversed, escalated, reviewed, corrected, and closed. Model accuracy cannot substitute for any
of these controls.

VII. Summary of recommendations
FDA should apply these principles in proportion to risk:

1. State the clinical purpose and any financial or operating goals that may shape use.
2. Treat financial incentives as part of the control system and disclose them when they can affect high-impact
decisions.
3. Set the system's allowed actions, tools, approval points, and escalation rules before giving it autonomy.
4. Test whether human reviewers have the facts, authority, time, independence, and power to change the
outcome.
5. Tie validation evidence to the exact deployed setup, including the model, instructions, sources, tools,
control logic, and workflow.
6. After a major change, document whether the old evidence still applies.
7. Keep important AI outputs traceable after they enter medical records and other systems.
8. Require action records that can rebuild what higher-risk agentic systems did.
9. Test whether appeals, rollback, incident response, escalation, vendor support, and patient remedies can
work when needed.
10. Track override, appeal, and reversal data after deployment.
11. Require outside assurance to include findings, owners, deadlines, verified closure, and escalation.
12. Scale every control to autonomy, patient impact, irreversibility, system complexity, and the chance for
effective human review.

VIII. Conclusion
Generative and agentic AI can improve medical care. Model accuracy alone cannot make these systems safe. The
cases I examined through AACIF show the gaps. An accurate system may still serve the wrong objective. A
reviewer may be present but unable to exercise independent judgment. The model may remain unchanged while the
surrounding system changes. A clinician may sign a note without knowing the origin of its language. An external
reviewer may identify a problem that no one resolves. A remedy may exist on paper without the means to provide it.
An agent may take consequential clinical actions that no one can later reconstruct.

FDA should ask more than whether the model performs well. It should ask what objective governs the system, what
the system is authorized to do, which configuration was tested, what has changed, whether reviewers can exercise
independent judgment, whether consequential actions can be reconstructed, and whether promised remedies are
operational. These are present problems in healthcare, not future abstractions. Establishing sound controls now will
be easier than repairing weak systems after they become more autonomous.

Respectfully submitted,

Thom Barrett

Founder, Operation Citizen

Author, AI Assertion and Chain Integrity Framework (AACIF)

This comment was prepared with the assistance of Claude, an AI assistant made by
Anthropic.