Yubin Kim

Please see the attached public comment, "Testing whether oversight can stop unsafe actions by generative AI medical devices," responding to selected questions in FDA-2026-N-7874. I offer this comment in my personal capacity. The attachment includes my affiliations, disclosure of coauthored references, and supporting sources.

Yubin Kim | Public comment | FDA-2026-N-7874

Testing whether oversight can stop unsafe actions by generative AI
medical devices
Docket FDA-2026-N-7874
Yubin Kim, PhD Candidate, Massachusetts Institute of Technology
Comment in a personal capacity
I recommend adding a compact action-level evidence record to the assessment of agentic devices:
what the system was allowed to do, what it actually changed, when a problem became observable,
and whether an authorised person or safeguard could intervene in time. This would help connect
competency testing to practical decisions about oversight and postmarket monitoring.
Recent work already evaluates executed EHR workflows and tool-using patient conversations [2,3].
A useful next step is to test the effectiveness of the safeguards around those actions. Detecting
an error after a completed episode provides different evidence from preventing a consequential
action during use.
My research includes medical-agent evaluation in MDAgents [4] and controlled comparisons of
agent architectures [5]. I am an incoming Research Scientist at Google DeepMind and previously
conducted research at Google Research and Google DeepMind. These are my personal views, not
those of MIT, Google or my collaborators. I am a coauthor of references [4–6]. The recommendations below respond to the discussion paper [1]; they are proposed evaluation practices, not
current FDA requirements or evidence of clinical safety.
1. Define the action and the opportunity to intervene
Questions 1 and 5
The framework could describe permitted actions, approval points and when approval expires.
For conversations, assess the path from general information to a personalised recommendation or
action, including prohibited transitions. Relevant modifiers include reversibility, time until harm,
downstream exposure, and the time and authority available to intervene. For persistent systems,
specify what information may carry into later encounters and when it must be checked again.
For example, a clinician may approve a proposed action, after which the selected patient or the
underlying medication record changes. Evaluation should test whether the system detects that
change and obtains any required new approval before proceeding. A record of an earlier approval
alone would not demonstrate that the eventual action remained authorised. This is an illustrative
test case; the clinical consequences and acceptable response would need to be established for the
particular intended use.
2. Add an action-level evidence record to competency testing
Questions 9 and 26 and Appendix A element A.1
For each safety-relevant test, I suggest recording the initial state, permitted action, applicable
safety condition, required checkpoint, observed action, resulting state and adjudicated outcome.
A useful report would distinguish a successful task, a safely interrupted task, an unsafe action, and
an incomplete or unassessable run. A correct final response should not erase an earlier violation.
The test set should cover ordinary cases as well as conflicting or stale records, interrupted tool
calls, duplicate writes after retries, inappropriate information reuse across patients, and instructions embedded in retrieved content. Where feasible, check actions against the resulting environment state rather than relying on the agent’s account of what it did. PhysicianBench offers a
relevant example of execution-grounded checks [2]. Clinical meaning still requires appropriately
qualified adjudication.
For the approval-change example above, an illustrative record could read: approval applies to
record version A; the record changes to version B before execution; fresh approval is required;
Yubin Kim | Public comment | FDA-2026-N-7874

the safeguard blocks execution; an independent state check confirms that the prohibited action
was not committed. It should also record when the change became observable, the last safe
intervention point, and when the block took effect. A warning generated after execution would
count as detection, not prevention. These are proposed record contents, not observed results.
For each high-consequence action, prespecify the conditions under which execution must be
blocked, the evidence needed to release the block, and the maximum acceptable intervention
delay. A tested bypass of a mandatory checkpoint should trigger investigation and remediation
rather than be averaged away by successes on unrelated tasks.
A risk-based set of tests would connect each identified hazard to a tested safeguard without
requiring private internal reasoning. Sponsors could reuse existing test logs rather than create a
separate reporting system. The scope should follow the intended use and hazard: controlled state
checks may suffice for an automatic block, while a claim that clinical staff can intervene requires
evidence about that human workflow.
3. Measure effective intervention and the work it requires
Questions 6, 14, 16 and 20
Monitoring studies should distinguish detecting a hazard, notifying the responsible person, and
successfully interrupting or correcting the action. Compare candidate monitors at the same
decision point, with matched model and inference budgets where applicable and information
available at that time; a retrospective evaluator must not receive credit for information that
arrived after the opportunity to intervene. Include straightforward rule checks where the safety
condition can be specified, alongside model-based and human review. Freeze the agent, tools and
case definitions for the comparison, calibrate alert thresholds on separate cases, and evaluate on
held-out cases. If claiming prevention, run the safeguard online and verify the resulting state;
retrospective detection alone cannot establish that an action was stopped.
I suggest reporting consequential violations per initiated episode, hazards detected before the
intervention deadline, late or missed alerts, and unnecessary alerts in hazard-free episodes. Report
under-escalation and over-escalation separately. Include reviewer minutes per 100 episodes, queue
delay and time spent resolving false alerts. Prespecify the available review capacity, and evaluate
the actual human–device workflow when human response is part of the safeguard. These measures
can reveal a monitor whose alert queue prevents timely action.
Clinical adjudicators should use prespecified rubrics and record disagreements, relevant expertise,
independence and conflicts. Automated scoring should be checked against clinical review on the
consequential errors it is meant to detect. Different model providers or role labels alone do not
establish independent verification. Test shared failure modes, including a primary agent and
its supervisor relying on the same stale source or incomplete tool result. TeamBench motivates
scrutiny of nominal verifier roles, but its non-clinical findings do not quantify medical risk [6].
4. Keep the bridge to clinical confirmation explicit
Questions 7, 10, 11 and 12
I support benchmarking followed by risk-proportionate clinical confirmation. Each benchmark
should identify the intended-use hazard it tests, the relevant patient and workflow conditions it
covers, and what remains outside its coverage. Sponsors should test the relationship between
benchmark results and independently adjudicated performance in representative clinical cases,
including the users and intervention process where these affect safety.
Synthetic and deliberately fault-injected cases can test whether a safeguard works under a known
challenge. They do not estimate how often that challenge occurs in practice. Report them separately from naturally occurring errors and real-case performance unless a justified statistical model
supports combining them. Preserve timeouts, missing traces and failed runs in the accounting;
do not drop them merely because a score cannot be assigned. Repeated runs and multiple check2
Yubin Kim | Public comment | FDA-2026-N-7874

points from the same patient should not be treated as independent patients. Report uncertainty
at the appropriate sampling unit, including for rare hazards and relevant subgroups.
Retrospective or shadow studies may be suitable for some questions. They cannot by themselves establish whether people will notice, trust or act on an alert displayed during care. The
confirmation design should address whichever part of that interaction supports the safety claim.
5. Reassess changes against the safeguard they could affect
Questions 19, 22 and 24
A minimal, privacy-preserving event record should link actions and resulting state changes to the
device configuration, model version where available, source identifiers, tool permissions, memory
policy and approval events. Records should permit reconstruction without collecting private
internal reasoning or unnecessary patient information.
For a change to a model, tool, retrieval source, interface or memory policy, identify which safety
conditions or intervention opportunities could change and justify the regression tests selected.
Where a third-party provider does not expose reliable version identifiers, document that uncertainty, use stable probe cases to look for behavioural changes, and define a fallback or suspension
procedure. Passing those probes would not prove that an undisclosed change was harmless. Regulatory review and change-control obligations should remain separate from a benchmark pass
decision.
6. Make reliance on postmarket monitoring conditional on demonstrated control
Questions 18 and 21
Greater reliance on postmarket evidence is more defensible where hazards are observable before
consequential harm, the responsible actor can respond within the available time, exposure can
be bounded, and interruption or recovery has been tested. Missing telemetry, an unavailable
reviewer or an untested rollback weakens that case. Monitoring is particularly limited for rapidly
harmful or irreversible actions.
Manufacturers should specify who receives each signal, who has authority to stop the system, and
how that decision is confirmed. Hospitals, clinicians and independent evaluators can contribute
evidence without diffusing manufacturer accountability. A useful public output would be a worked
example of the record and a monitoring report that includes failed interventions, resource burden
and unresolved evidence gaps.
References
[1] FDA. Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback. Sections IV–VII and Appendix A. https://www.fda.gov/media/
194242/download
[2] Liu R, et al. PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments.
arXiv:2605.02240v1, 2026. Preprint. https://arxiv.org/abs/2605.02240
[3] Vatanparvar K, et al. PatientAgentBench: A Benchmark Framework for Evaluating PatientFacing Health AI Agents. arXiv:2607.25485v1, 2026. Preprint. https://arxiv.org/abs/2607.25485
[4] Kim Y, et al. MDAgents: An Adaptive Collaboration of LLMs for Medical DecisionMaking. NeurIPS 2024. https:// proceedings.neurips.cc/ paper_files/ paper/ 2024/ hash/
90d1fc07f46e31387978b88e7e057a31-Abstract-Conference.html
[5] Kim Y, Gu K, Park C, et al. Capable language models can outgrow the benefits of collaboration. Nature Machine Intelligence 8, 1157–1172, 2026. https://doi.org/10.1038/s42256-02601268-y
[6] Kim Y, et al. TeamBench: Evaluating Agent Coordination under Enforced Role Separation.
arXiv:2605.07073v1, 2026. Preprint. https://arxiv.org/abs/2605.07073