Elizaveta Kutis

As a physician working with digital health data and clinical interpretation, I believe regulation of generative AI-enabled medical devices should recognize a fundamental reality: neither human intelligence nor artificial intelligence is error-free. The goal should not be an unrealistic expectation of zero error, but the design of clinical systems that prevent an individual error from propagating into patient harm.
Human clinicians and AI systems fail through different mechanisms. Clinicians may make errors related to fatigue, incomplete information, cognitive bias, workload, or fragmented medical records. GenAI systems may produce incorrect, incomplete, poorly calibrated, or unsupported outputs. These differences matter, but in both cases safety depends heavily on the architecture surrounding the decision.
Healthcare already uses mechanisms to reduce human error: defined professional roles, supervision, multidisciplinary review, standardized workflows, escalation pathways, quality assurance, and independent confirmation of high-risk decisions. Similar principles should be incorporated into the evaluation of GenAI-enabled medical devices.
For this reason, I support evaluating not only the AI system in isolation, but also the human-AI clinical decision architecture in which the system operates. Two systems using the same foundation model may carry very different levels of clinical risk depending on whether an output is directly acted upon, independently verified by a clinician, compared with other data sources, or escalated to a specialist before a consequential decision is made.
I therefore suggest that risk assessment include an explicit consideration of a system’s error-containment and verification architecture: how easily can an incorrect AI output reach a patient or trigger an irreversible clinical action, and what independent safeguards exist before that occurs?
I also believe GenAI can reduce an important source of medical error that receives less attention: incomplete visibility into a patient’s health between clinical encounters. Traditional care is often based on episodic office visits, self-report, laboratory testing, and limited snapshots of physiology. With appropriate validation, authorization, privacy protections, and data provenance, longitudinal digital health information may provide valuable additional context.
Data from wearable devices and sensor-based digital health technologies may contribute information about sleep, activity, physiologic patterns, and other measurable behaviors. Patient-reported outcomes, medication information, pharmacy dispensing data, laboratory results from major laboratories, imaging, and electronic medical records may also be integrated into a more complete longitudinal picture.
GenAI-enabled systems and agents may be particularly useful for organizing these fragmented data streams before clinical review. Their value may lie not only in generating recommendations, but in data reconciliation, identification of inconsistencies, detection of missing information, longitudinal summarization, and prioritization of findings for clinicians.
However, clinically meaningful AI-generated conclusions should remain traceable to the underlying source data. A clinician should be able to determine which laboratory result, sensor measurement, medication record, questionnaire response, or other patient information materially contributed to an output. Source provenance, timestamps, units, reference ranges, missing-data handling, and distinctions such as prescribed versus dispensed versus actually taken medication are important components of safe interpretation.
For clinical confirmation, I encourage FDA to evaluate human-AI team performance in addition to AI performance alone when the intended use involves clinician oversight. In many real-world settings, the relevant question is not whether AI performs better than a physician in isolation, but whether a properly designed human-AI system reduces errors, improves recognition of clinically important information, shortens time to appropriate evaluation, and supports more consistent decision-making.
Finally, postmarket monitoring should assess not only model performance drift, but also whether the overall clinical workflow continues to function as intended after model updates, changes in data sources, or changes in deployment conditions.
The regulatory objective should therefore extend beyond controlling model error. It should promote clinical architectures in which AI, clinicians, validated data sources, quality-control mechanisms, and escalation pathways work together to make healthcare safer, more transparent, and more informed.