Our Response to the FDA’s Request for Comments on Generative AI

In August I wrote about what the FDA’s discussion paper reveals about an emerging framework for Clinical GenAI. The paper closed with 26 questions and an invitation to answer them. Comments closed yesterday. We filed our response last week.

Our response to the discussion paper’s questions is structured around a framework built on four core principles:

1. Judge a device by the decisions it makes independent of a licensed clinician.
2. Raise the evidentiary bar as the potential for harm rises.
3. Evaluate the complete device: the LLM, the clinical harness, the controls and guardrails, and the data.
4. Differentiate clinical evidence by the context in which a device makes its decisions.

Each comes from problems we have had to solve while developing a device that makes defined clinical decisions independently during post-operative recovery: clinical reference standards, prospective evaluation, challenging edge cases, LLM changes, and ongoing performance monitoring.

1. Judge a device by the decisions it makes independent of a licensed clinician.

The FDA’s proposed two-axis plot puts device activity on one axis, running from information a device provides (non-directive, then action-directing) to actions a device takes itself (under a clinician’s supervision, then fully autonomously). That axis combines two distinct things: whether a device informs or acts, and how much supervision it acts under. Neither measures how many kinds of clinical decisions a device is allowed to make. A device that decides on its own within one narrow, validated protocol and a device that decides on its own across many conditions can sit at the same point on that axis, yet they fail in different ways and need different evidence of compliance and patient safety.

Our comment distinguishes three levels of clinical autonomy, defined by what bounds a device’s independent decisions:

Clinician-directed AI produces output that a clinician reviews and approves before it reaches the patient. Protocol-bounded AI decides on its own within a validated clinical protocol and escalates to a clinician when a case falls outside it. Broadly autonomous AI decides across many conditions without a defined protocol. Each level calls for more clinical evidence than the one before it, because each places more of the clinical decision with the device.

A post-operative recovery device is protocol-bounded. It may assess symptoms, provide recovery guidance, and decide when to direct the patient to a clinician, and it does not diagnose or manage unrelated conditions. Evaluating it as a broadly autonomous device would ask for evidence about decisions it never makes. The large language model (LLM) underneath it may be the same one a broadly autonomous product uses; the protocol, the escalation rules, and the validated scope built around the LLM are what set its level of clinical autonomy.

That is why we argue the unit of evaluation should be the decision a device makes independent of a licensed clinician. A clinician checking each output is one safeguard among several. For a decision the device has been validated to make on its own, within its protocol, that review need not remain a permanent requirement, because the clinical evidence behind the decision takes its place. The next section sets out what that evidence should be.

2. Raise the evidentiary bar as the potential for harm rises

In regulatory terms, evidence is what a sponsor must show before authorization that a device’s decisions are safe for its intended use. The FDA’s two-axis framework says that bar should rise as a device acts more independently and as its errors can do more harm. We proposed three kinds of clinical evidence that should rise with it:

Performance against an independent clinical reference. The device’s decisions are compared with the judgments of clinicians who did not see its output, in prospective use that resembles real deployment, and the comparison is reported for each decision the device makes on its own.

Escalation bias. When a device is uncertain, it should err toward sending the patient to a clinician. Under-escalation and over-escalation are measured separately because they carry different harms: one misses a complication, and the other sends a recovering patient to the emergency department without need.

Transparency and auditability. A clinician or reviewer should be able to see what the device was told, what it decided, and why, for every clinical decision, so that errors can be traced.

Postmarket monitoring cannot substitute for that clinical evidence when a device directs what a patient does next. If a device tells a patient with emergency symptoms that their recovery is on track, the patient acts on that direction, and the harm follows from it. The evidence that a device rarely gives that kind of direction wrongly has to exist before the device reaches patients.

3. Evaluate the complete device

A manufacturer may build on an LLM developed by someone else, without control over its weights, training data, or future updates. The manufacturer is responsible for everything around it: the clinical harness that turns the LLM’s output into a defined clinical decision, the controls that set which decisions, if any, the device may make on its own, the guardrails that keep it within its validated scope, the clinical data it decides from, the testing of its decisions against clinical evidence, and the records of what it decided. A patient relies on that complete system, and it is what should be evaluated.

LLMs change often, and when one does, the question is whether the device still performs. Because an LLM developer may retire earlier versions, a manufacturer cannot always roll back, so the regulatory focus should be on demonstrating continued clinical competency. If a replacement LLM cannot demonstrate acceptable clinical performance, manufacturers need a defined fallback that keeps patients safe. That can be an open-weight model the manufacturer controls and has validated for exactly this situation, or diverting decisions to appropriate human oversight until competency is re-established.

4. Differentiate clinical evidence by the context in which a device makes its decisions

Two devices with similar autonomy and similar consequences of error may decide from very different information. A patient’s description of swelling is different evidence from a photograph, a video, or a longitudinal record with lab results and wearable data. Rather than add a third axis to the FDA’s framework, we suggest that the context a device decides from should shape the competency it is expected to demonstrate. More information does not guarantee a correct decision, but denser and multimodal data raise the resolution of what a device can see: a device that reads a wound photograph alongside the patient’s own description and recovery history has more to decide from than one that reads text alone. So the clinical evidence for a device should come from the same kinds of input it will receive in use. A device validated on text conversations has shown nothing yet about how it decides from photographs, and a device that decides from photographs, records, and wearable data should show that the added information makes its decisions more accurate.

Our answers to the FDA’s proposed approaches

Our full comment answers all 26 of the FDA’s questions. On the questions below, the paper put forward an approach and asked whether it was right, and our answers add to or differ from what it proposed.

Patient-facing delivery alone does not warrant a higher risk tier. The paper suggests patient-facing functions might sit higher on its consequences axis, because patients may not be able to recognize an incorrect output, while also noting the benefits of patient empowerment. We argued that what matters is the consequence of an incorrect output, the degree of independent clinical authority, and the safeguards around the function. Bounded intended use, validated escalation, and human-factors evaluation can support patient-facing autonomous functions without underestimating patients.

A device should not grade itself. The paper asks whether machine-based supervisory agents could help with postmarket monitoring. They can, if the monitor has itself been tested against an independent clinical reference. Postmarket evaluation needs a mechanism separate from the model making the clinical decisions. That need not be an outside institution, but an algorithmic monitor that shares the device’s failure modes may repeat its errors.

AI-generated test cases need independent answers. The paper suggests synthetic data could supplement evaluation where real patient data is limited, and asks how to keep synthetic data from reproducing the gaps the evaluation is meant to find. Synthetic cases are useful for rare, complex, and high-consequence situations. Our preferred approach starts from real, clinician-adjudicated patient cases and introduces variation from there. An AI that generates a case should not be the authority on its correct answer.

Foundation Model Master Files should stay voluntary. The paper proposes voluntary Master Files that foundation model developers could file confidentially with the FDA for device sponsors to reference. They could reduce duplicative review, but a sponsor that can independently establish the safety and effectiveness of its finished device should not be disadvantaged when its model developer does not participate.

The thesis behind our comments

Our thesis is that patient-facing clinical AI, once authorized to make bounded clinical decisions independently, will create clinical capacity that is not limited by clinician labor. Today nearly all clinical work scales with clinician hours: every symptom a recovering patient reports from home waits for a nurse, a physician assistant, or a surgeon to hear it and decide what it means. A device authorized to decide on its own whether a post-operative symptom is part of normal recovery takes that work off the clinician’s queue. Symptoms that need a clinician still go to one.

That only works if the device is held to a standard. Clinicians earn the authority to decide independently through training, examination, supervised practice, and continued assessment, and a device should earn it through clinical evidence of the same seriousness. Our four principles scope that evidence to the decisions a device makes without a licensed clinician, the harm those decisions can cause, the complete device built around the LLM, and the context it decides from. When the evidence holds, the time a nurse or surgeon no longer spends confirming normal recovery goes to the patients whose recovery is not normal.

Our full comment, with all 26 answers, is on our site. The FDA posts filings to its public docket as it reviews them, and ours will appear there. To see what the public is telling the FDA, follow our FDA GenAI Response Tracker.