Clinical Excellence Research Center, Stanford University (Aditya Narayan)

Please see the attached file.

PUBLIC COMMENT | FDA DOCKET NO. FDA-2026-N-7874
Making Clinical AI Evaluability a Design Requirement
Response to: Considerations for the Regulation of Generative AI-Enabled Medical Devices

October 19, 2026
To the FDA Digital Health Center of Excellence:
We appreciate FDA’s invitation to comment on the regulation of generative AI-enabled medical
devices. The discussion paper recognizes a central problem: the space of inputs and outputs for these
systems is too large to evaluate by exhaustively sampling clinical encounters. Yet much of clinical AI
development still treats safety as a matter of obtaining a high score on enough encounters. This creates
an important mismatch between what developers measure and what regulators need to know.
Our recommendation is that for consequential clinical actions, evaluability should be a property of the
system’s design. A developer should be able to show which clinical facts the system established,
which physician-approved rules governed the decision, which rules were actually applied in decisionmaking, and which actions occurred. The inputs and actions should be narrow enough to test
independently. The decision logic should be explicit enough to inspect and, where appropriate,
formally verify. The resulting record should allow an independent reviewer to reconstruct why a
patient received a particular recommendation, prescription, test, diagnosis, or referral.
A central unit of evaluation should be the clinical ‘primitive:’ a narrowly defined, independently
checkable task or finding used in a larger clinical workflow. For example, extracting the potassium
value, units, specimen date, and patient demographics from a laboratory report is a set of primitives;
deciding whether a medication can safely be renewed is not. A higher level decision like that depends
on several such findings and clinical rules. Rather than treating an entire medical encounter as one
black-box prediction, evaluation should begin with these small operations and how they are
assembled. The sections below explain why current evaluation approaches are insufficient, how such a
system works in practice, and what evidence FDA could request. The comments principally address
Questions 7–14, 16, 19, 22–24, and 26.

1. The central weakness of today’s evaluation: optimizing the test is not the same as controlling
the clinical decision
Benchmarks, clinical vignettes, and end-to-end encounters ask a natural question: given a patient
presentation, does the AI give an acceptable answer? This is how we assess trainees and practicing
clinicians, and it is useful for identifying systems that lack basic clinical competence. For software,
however, the same approach leaves a much more important question unanswered: what decisions led
to the answer, and will the software reliably obey the same safety constraints the next time it
encounters a slightly different patient?
Consider a medication-renewal benchmark. An AI model is shown a chart with an abnormal
laboratory result and is asked whether to renew a medication. Its final response says no, which
happens to be the right answer, and the model is awarded full credit. But that result is compatible with
very different underlying behaviors: the model may have recognized the abnormal test, noticed that
the laboratory result was out of date, reacted to a familiar phrase in the benchmark, or simply learned
that caution earns a higher score. Rubrics do not establish which clinical condition the system
recognized or whether the condition actually governed its decision.
This problem becomes more acute when benchmark performance is used as the target of
reinforcement learning, prompt iteration, or other optimization. Repeated feedback teaches a model to
produce outputs that the evaluator rewards. If the evaluator sees only a polished final recommendation
or an AI-generated explanation, a rising score may reflect better clinical decisions, or better
production of a reassuring-looking answer. The system may learn to mention a required safety check
without actually performing it. A reduction in visible errors is not proof that the underlying failure
modes have been eliminated; some may simply become less visible to the test.
A large system prompt does not solve the problem. Developers may write extensive instructions
covering red flags, contraindications, escalations, scope limits, and clinician approval. Those
instructions express intended behavior, but they are not evidence of execution. A transcript that says
“all contraindications were checked” does not reveal whether the model actually checked each
relevant fact against a recorded source. Adding more examples to the prompt, or training longer
against an encounter-level reward, still does not give an auditor a reliable account of which safety rule
determined a specific action.
FDA should therefore treat scenario benchmarks as a minimum screen for gross deficiencies,
communication, and ordinary-case performance, not as the primary proof that a consequential clinical
workflow is controlled. The crucial shift is from scoring whole answers to verifying the operations
that produce the answer. This is especially important for end-to-end agents whose intermediate
choices are not visible in a final response. (Questions 7, 9, 10, and 14.)

2. Clinical simulators cannot supply the missing counterfactual ground truth
Patient simulators and synthetic encounters are sometimes presented as the answer to the limits of
benchmark coverage: generate many more synthetic patients, let the AI treat them, and observe how
the virtual patients respond. The difficulty is not a lack of variety in the simulated cases. It is that we
often cannot know whether the simulator’s clinical responses to alternative decisions are true.
Routine clinical data record what clinicians chose to do and what subsequently happened. They
generally do not show what would have happened to that same patient had the clinician taken a
different action. This missing counterfactual matters enormously. The patients who receive
medications may systematically differ from those who do not. Clinicians also avoid many treatments
in patients with known contraindications. A simulator trained on those records sees extensive
examples of selected, usually permissible actions, and few examples of the actions that clinicians
correctly chose not to take. Even very large datasets do not automatically identify how those
unobserved interventions would have affected patients.
Imagine a simulated patient with an important contraindication to a commonly prescribed medication.
Because clinicians ordinarily avoid prescribing it in such patients, the training data may contain few
comparable instances in which the medication was given. A simulator can still generate a confident
and plausible clinical trajectory after the AI prescribes it. But the trajectory is produced by an inferred
model of treatment effects, not by observed ground truth for that patient. The simulator might report
an uneventful course after a decision that a clinical review panel would reject. A model evaluated
against that simulator may then be rewarded for following the simulator’s invented version of
medicine.
This is a basic causal-identification problem that can’t be fixed merely by increasing the number of
synthetic patients. If two different models of treatment response agree with the observed clinical
records but disagree on the outcomes of rarely taken actions, the records alone cannot tell us which
simulated counterfactual is right. Clinical simulation can appear precise while remaining least reliable
exactly where safety evaluation needs it most: unusual, contraindicated, or high-consequence actions.
There is a narrower use for simulated encounters. A synthetic patient can help inspect whether an
agent communicates clearly, asks understandable questions, avoids coercive language, or follows a
specified conversational script. An independent reviewer or communication rubric may assess those
visible features. That kind of test should be explicitly described as a communication or interface test.
The simulator’s apparent medical improvement, deterioration, satisfaction, or “survival” should not be
treated as evidence that the AI’s clinical intervention was correct unless the relevant causal response
has itself been established. (Questions 11–13 and 17.)

3. A concrete alternative: separate interpretation, clinical decision, and action
A clinical AI system can distinguish three activities: interpretation, clinical decision-making, and
action. Separating them is what allows the clinical decision-making step to be codified as an explicit,
executable care pathway instead of being left inside a model’s unconstrained response. Consider a
patient asking an AI assistant, “Can you renew my blood-pressure medicine?” Safely processing that
request may require reading laboratory reports, reconciling medications, checking contraindications,
following a physician-approved pathway, obtaining authorization, and confirming that the prescription
reached the correct destination. An opaque agent attempts these in one interaction while an auditable
design separates them.

First: interpretation translates messy records into narrow clinical facts
The AI may read a patient message, a medication list, a signed laboratory report, and a clinician note.
Rather than asking the model to infer the broad conclusion “this refill is safe,” the system asks it to
establish smaller facts: Which medication and dose is requested? What is the measured potassium
value, in what units, and on what specimen date? Does the patient report a new concerning symptom?
Is the required laboratory result actually available, or has the patient merely said that “my labs were
fine”?
These narrow tasks are the clinical primitives defined above. A good primitive can be checked against
its source without resolving the entire clinical decision: “This patient’s signed report records a
potassium of 4.3 mmol/L on September 15” is independently testable. “This patient is safe for
automatic renewal” combines multiple judgments and cannot serve as a low-level primitive.
Because each primitive has a clear reference, it is feasible to measure its error rate on real records and
to challenge it with targeted cases: two patients in the same document, a family member’s laboratory
value, a corrected result, an old result copied into a new note, an unfamiliar unit, or a phrase such as “I
never had my potassium rechecked.” Given a primitive’s narrow specification, optimization on cases
like these are unlikely to lead to proxy failures.
Second: decision-making applies an explicit, physician-approved care pathway
Once those facts have been established, the clinical decision can be governed by a readable decision
tree or comparable executable specification. A clinician panel defines the pathway for the intended
population and setting: which findings are required, which findings disqualify automatic renewal,
when new information must be obtained, which circumstances require escalation, and when a clinician
must authorize the action. The software then evaluates those conditions directly, rather than asking an
LLM to remember a policy buried in a long prompt.

Consider a clinical pathway for evaluating suspected pulmonary embolism. The appropriate course of
action depends on the combination of multiple clinical findings: presenting symptoms, vital signs,
thromboembolic risk factors, anticoagulant use, pretest probability, available laboratory results, and
contraindications to particular investigations. A clinician-approved decision model can explicitly
define how these findings interact to produce different sequences of care. For example, a patient with
concerning symptoms and hemodynamic instability may require immediate emergency evaluation,
whereas a stable patient with low pretest probability may undergo a structured rule-out pathway. A
patient with higher pretest probability may require diagnostic imaging, with the choice and timing of
investigation further determined by renal function, pregnancy status, or other contraindications. Each
pathway can specify not only the initial decision but the subsequent actions: obtaining additional
information, ordering an appropriate test, reviewing its result, reassessing the patient's clinical state,
and arranging treatment or follow-up. Codifying this logic makes it possible to formally verify
properties across the combinations of clinical states represented in the system: that patients meeting
urgent criteria are always escalated, that contraindications appropriately modify investigation
pathways, that unresolved findings cannot be silently treated as negative, and that a sequence of care
cannot be marked complete before its required steps occur. Rather than testing a handful of
representative cases, verification can systematically establish whether the implemented decision logic
satisfies the clinically defined requirements across its specified state space.

A finite decision tree allows a reviewer to inspect the complete set of decision branches and, for
suitable properties, formally verify that no represented state can bypass a required authorization or
turn “unknown” into “normal.” This provides a guarantee of logical consistency from case to case -
significantly better than what any empirical result from a benchmark can do.

Third: action turns the decision into a recorded, verifiable event
The system divides its possible outputs into narrowly specified action primitives, each with defined
inputs and a recorded result. Most actions are programmatic tool calls, such as creating a prescription,
placing a referral, ordering a test, or updating the patient’s record. Communication is divided into
similarly specific actions: request both numbers of a blood-pressure reading, explain a particular
medication using its supplied guidance, or ask the patient to explain a documented care plan in their
own words. Each communication action specifies what information to request or convey; the language
model performs the smaller task of adapting that content to the conversation, without independently
supplying medical advice from its own knowledge. This discretizes the conceptual output space: a
broad instruction such as “manage this patient” becomes a controlled selection among explicit,
inspectable actions. The record captures which action was invoked, its inputs, and the resulting
message or tool response, allowing reviewers to verify both what was selected and what actually
occurred.
4. Why low-level tasks are easier to validate, and harder to optimize around
One value of small primitives is that they reduce the number of ways a component can appear
successful while concealing an error. “Provide a safe and comprehensive medication plan” is a loosely
defined task: an evaluator must judge medical accuracy, completeness, interpretation, tone,
uncertainty, and execution at once. A model may earn a favorable overall score while failing a
particular contraindication check. “Extract the latest signed potassium result for this patient, including
value, units, and specimen date” has a comparatively unambiguous answer.
The same applies to action. “Manage the patient’s refill” is difficult to assess from prose. “Submit this
authorized prescription and return the pharmacy, transaction identifier, and accepted/rejected status”
has an external receipt. A model cannot pass by merely sounding as though an order succeeded. Error
analysis becomes granular: wrong-patient attribution, incorrect units, missed negation, missing result,
incorrect branch selection, authorization bypass, or failed tool execution.
This approach also changes what reinforcement learning should optimize. A model may still be
trained to interpret notes or communicate effectively. But it should not receive credit for producing a
plausible high-level medical decision if the decision itself belongs to a separately specified, clinicianapproved control process. The most safety-critical requirements become properties of the executed
software, rather than preferences that a reward model hopes the generative model will internalize.

5. Codified decision logic makes audit and change control practical at scale
Separating interpretation, decision, and action enables codified decision logic, which in turn makes
audit and change control practical. For every deployed decision, an auditor can identify which clinical
conditions were evaluated and which rule determined the disposition. A prompt-heavy system may
describe the same requirements in pages of instructions, but changing those instructions does not
guarantee that the model will apply the revision consistently. In a codified system, the governing
branch is explicit, versioned, and executable.
Suppose new clinical guidance changes the circumstances under which a refill requires additional
laboratory review. In an opaque end-to-end agent, the developer might edit the prompt, optimize with
reinforcement learning, perform another large batch of encounters, and infer from the overall score
that the change worked. In a codified pathway, the team can update the specific rule, formally check
its required properties, inspect the affected branches, and replay historical cases to see exactly which
dispositions change. The difference is between hoping that a policy update influences behavior and
knowing which clinical decisions it alters.
The same principle applies to postmarket audits. An auditor investigating an incorrect refill could
reconstruct: the source laboratory report; the value the model extracted; whether the result was current
and associated with the patient; the decision branches evaluated; the policy and software versions; and
the transaction receipt. If the extraction was wrong, the error is localized to interpretation. If the
extracted findings were correct but the wrong clinical branch fired, the decision logic is responsible. If
the decision was correct but the prescription never reached the pharmacy, the defect lies in execution.
All three cases may appear as one “incorrect answer” in an end-to-end evaluation, but they require
very different remedies.
Crucially, this evidence can be generated as part of routine operation. A regulator need not manually
reread every transcript to know which rule was applied. Aggregate monitoring can identify the
frequency of particular branches, unresolved states, overrides, and failed actions, while independent
reviewers can replay any selected patient decision against its retained record. This makes independent
assurance more scalable as the number of clinical functions, sites, updates, and patients grows.
(Questions 19, 22–24, and 26.)

6. Physician consensus defines the care pathway; formal verification checks that it is enforced
Once a clinical pathway is explicit and executable, clinician consensus can define what that pathway
must require, and verification can establish whether the software enforces those requirements.
Clinician agreement is indispensable. In an end-to-end test, however, a physician usually sees a
scenario and judges the final recommendation. That establishes something about the sampled answer;
it does not establish that the system reliably follows the underlying clinical principles across other
combinations of facts.
A more powerful role for physicians is to specify those principles directly. For a defined function, a
qualified panel can establish the accepted clinical criteria, required evidence, contraindications,
urgency rules, permissible alternatives, escalation points, and authorization boundaries. Where
practice legitimately varies, the care pathway can encode the acceptable alternatives and the
conditions under which each is appropriate. The regulator can then review both the clinical legitimacy
of that specification and evidence that the software follows it.
Formal verification has a specific meaning here: given an executable representation of a care pathway,
it checks whether stated properties hold across the states covered by that representation. For example,
the reviewer can establish that a required clinician-approval gate cannot be bypassed, or that an
unresolved safety-critical result cannot authorize a prescription. Coupled with validated interpretation
of the relevant patient facts, this provides a strong, affirmative form of evidence for clinical
correctness within the pathway’s defined scope. A favorable test score cannot make the same claim.
This also preserves the right comparison for Question 14. The medical reference should be physicianadjudicated, evidence-based care—not a model’s ability to impersonate an average physician on an
exam. Clinicians establish what the system ought to do. Verification establishes whether the specified
decision logic actually does it. Clinical evidence then establishes how the complete device performs
with real patients and workflows.

7. What FDA should ask sponsors to submit
For consequential clinical functions, we recommend an evaluation package built around two
reviewable artifacts. The depth of each should be proportionate to the autonomy of the device and the
consequence of an incorrect decision.

Part A. A register of low-level interpretation and action tasks
The sponsor should identify the clinically material primitives used by the system and the actions it can
execute. For each interpretation primitive, the register should provide the precise specification,
allowable outputs (including unresolved states), the evidence source, the applicable patient and time
context, and its independently assessable reference. It should report performance using real labeled
source material, with special attention for cases where the primitive’s value is None of the Above
(NoTA) or can’t be assessed due to a lack of information. . For each action primitive, the sponsor
should specify what authorizes the action, how its completion is confirmed, and what the system does
if execution fails.
FDA should review whether these are genuinely small, bounded tasks or high-level clinical
conclusions labeled as primitives. A task such as “identify the potassium value and specimen date
from this patient’s signed report” has a limited evaluation surface. “Determine whether all relevant
clinical considerations have been addressed” does not. Likewise, a prescription action is not complete
merely because an LLM states that it sent an order; its completion should be supported by an actual
receipt from the receiving system.

Part B. The physician-approved, executable clinical decision specification
The sponsor should provide the care pathway that consumes those primitives and determines which
actions are permissible. This can be a decision tree, decision table, state machine, or another
representation with the same ability to show the applicable conditions and resulting actions.
Independent clinical reviewers should confirm that the encoded pathway reflects an appropriate
standard of care for the intended use, including contraindications, exceptions, unresolved information,
urgent escalation, and authorization boundaries. Independent technical reviewers should then examine
evidence that the deployed implementation follows that pathway.
For suitable decision logic, FDA should expect systematic verification of the core safety properties
that clinicians have identified. Examples include: an unknown contraindication cannot become a
negative finding; a laboratory value from the wrong patient cannot qualify a prescription; an urgent
finding cannot be overridden by a routine missing-data request; a clinician-only decision cannot be
executed without the required authorization; and a failed electronic order cannot be recorded as
successfully completed. The review should identify the states covered by each proof and retain an
executable or replayable account of which rules ran in actual cases.
These two artifacts can sit alongside the clinical confirmation and patient-outcome evidence
contemplated in FDA’s paper. Their purpose is different: clinical studies estimate performance and
patient effects; a primitive register and verified decision specification reveal what the software can be
relied upon to do, how failures are located, and how changes are controlled. An independent evaluator
can then test the small model-dependent components, verify the clinical control logic, and audit the
deployed decisions without relying on an AI-generated account of its own behavior. (Questions 7–10,
14, 16, and 26.)

Conclusion
FDA has correctly recognized that generative AI medical devices cannot be adequately evaluated by
testing every possible encounter. But the answer should not be an ever-larger supply of benchmarks,
virtual patients, or clinician-scored conversations. Those methods sample what a system appears to do.
For safety-critical clinical decisions, regulators should also demand evidence of what the software is
built to do and what it actually did.
The design standard should be an explicit, clinician-approved care pathway operating on small,
independently validated clinical findings and executing verifiable actions. Such systems can look
more complicated on paper because their decision points are visible. That visibility is their advantage:
rules can be inspected before deployment, checked systematically, reconstructed afterward, and
changed with confidence. Building clinical AI this way can make meaningful regulatory oversight
more rigorous and more scalable as software assumes greater responsibility for patient care.

Respectfully submitted,

Aditya Narayan

Raghav Samvedam

Nirav R. Shah, MD, MPH

Clinical Excellence Research Center, Stanford University
Selected sources: FDA. Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback. 2026. Docket FDA-2026-N-7874.
https://www.fda.gov/media/194242/download
Hernán MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC; 2020.