Chavez Manuel

I submit these comments as a scientist with 14 years of experience in cGMP pharmaceutical and biologics manufacturing and quality control. I am not a medical-device specialist; I offer a cross-domain systems-assurance perspective on risk assessment, human oversight, and agentic AI.

**1. Input integrity belongs in risk assessment. (Questions 1, 3, 4, 9, 11, 19)**

The proposed framework considers what a GenAI-enabled device does and the consequence of an incorrect output. Another consideration deserves explicit attention: the integrity of the information on which the device acts.

A GenAI system can reason correctly from incorrect information and still reach the wrong conclusion. This matters when performance depends on sensors, wearables, user-entered values, or conversationally reported information. A mistyped value, missing unit, questionable measurement, incomplete context, or inconsistent information can propagate through an otherwise capable system, while fluent language can make the result appear more trustworthy than its inputs warrant.

FDA already recognizes aspects of this problem in proposed competency testing, including ambiguous inputs, implausible values, transcription errors, and unit confusion. I encourage FDA to carry that concept into risk assessment and expected system behavior, rather than treating input failures only as benchmark cases.

A useful principle is: do not silently reason through a material uncertainty simply because the system can continue. Flag the gap. Depending on risk, the system may identify the questionable input, request confirmation, constrain its output, or defer when an adequate basis cannot be established.

Human review should not automatically be treated as a control. Oversight mitigates a failure only when the intended reviewer can detect it. If the reviewer cannot assess whether the underlying input was sound, review of a fluent output may provide less protection than the architecture implies.

**2. Evaluate the configured system, including its boundaries. (Questions 7, 9, 10, 11, 14)**

I support evaluating the final device as configured for use, including prompts, retrieval, guardrails, orchestration, and user interaction, rather than the foundation model alone.

My practitioner shorthand is: validate the code; test the instructions; qualify the system. Deterministic functions can be verified against defined requirements. Instructions, retrieved context, and guardrails require systematic challenge because their effects are probabilistic and contextual. The assembled system must then demonstrate adequate performance for its intended use.

Benchmarks should test the guardrails, not merely successful completion of the nominal task. Boundary transitions deserve attention: sufficient to insufficient information, trustworthy to questionable input, in-scope to out-of-scope, and ordinary to safety-critical conditions. Both failure to intervene and unnecessary intervention matter.

Where human oversight is claimed as a mitigation, clinical confirmation should consider whether the intended user can recognize failures review is intended to catch. “Human in the loop” describes an architecture; it does not by itself demonstrate an effective control.

**3. Separate generative capability from decision authority where feasible. (Questions 20, 26)**

One useful architecture is a deterministic core with a generative surface. Authoritative state, permissions, required checkpoints, prohibited actions, and safety-critical transitions can, where appropriate, be enforced deterministically. The generative component can interpret information, communicate, request missing information, or propose actions within those boundaries.

This does not eliminate the need to evaluate the model. It makes some safety boundaries independently enforceable rather than relying exclusively on a probabilistic model to obey them. Agentic evaluation should examine not only task success, but whether the system remains within its defined authority when faced with conflicting information, erroneous tool output, missing prerequisites, or attempts to circumvent controls.

The same concern applies to machine-based supervisory agents. A second model is not automatically an independent control. If a supervisor shares important failure modes with the system it monitors, agreement may represent correlated failure rather than independent assurance.

These are not device-specific regulatory prescriptions. They reflect a broader assurance principle: trustworthy performance depends not only on model capability, but on trustworthy inputs, tested boundaries, meaningful oversight, and clear control over what the model is allowed to do.