SomaSoft (Mark Nafe)

Comment of SomaSoft on Docket FDA-2026-N-7874: testing whether a device’s checks actually ran

SomaSoft tests whether AI systems fail loudly when they cannot answer. We comment on one failure mode we
believe the proposed benchmarking elements and postmarket approaches do not yet reach, and offer a
concrete, inexpensive test for it.

The failure mode. A GenAI-enabled device usually works inside an architecture that includes
retrieval, drug and lab lookups, de-identification, rules engines and external data feeds. When one of
those components cannot run (it times out, a field is renamed, a code is misspelled, or a parameter is
out of range), the device can return an ordinary-looking result: "no interactions found", "no
identifiers detected", "no findings". The output is fluent and well formed, and it states no
uncertainty, because the model was never told that the check failed. We call this a silent failure:
"could not evaluate" rendered as "nothing found".

This is now measured in the literature. Sethi et al. (arXiv:2609.14758, September 2026) found that
language-model agents almost never misreported an outcome when a tool returned an explicit error (0.0%),
but did so 45.3% of the time when the tool reported success while returning empty, stale or truncated
data. Requiring an explicit OK/FAILED status reduced the rate from 14.10% to 0.87%. The error lives in
the interface between the model and its tools, not in the model’s clinical knowledge.

Q9 (missing elements). Element S.3 evaluates whether the device "accurately represents the confidence
and limits of its outputs". As described, it tests the model’s expressed uncertainty about clinical
content. It does not test whether the device knows that one of its own components failed to run. A device
can pass S.3 and still state "no interactions found" when the interaction lookup never executed. We
suggest either extending S.3 or adding a safety element, execution integrity: whether failures of the
device’s own components surface as an explicit, named failure state in the output rather than as a
clean negative result. It can be tested by fault injection: deliberately making each dependency fail
(timeout, empty-but-successful response, malformed data, renamed field, out-of-range parameter,
misspelled input) and recording whether the output says so. A summary metric is straightforward: of
the cases where the device could not evaluate, the proportion that surfaced an explicit failure. Unlike
most benchmarks, this one is hard to contaminate or saturate (cf. the concerns in Section V), because it
is specific to the device’s own architecture.

Q19 (postmarket approaches). Performance-degradation monitoring is unlikely to detect this failure
mode on its own. A silent failure produces outputs that look normal, so output-distribution and drift
dashboards stay green. In our own operational systems we have found this pattern repeatedly: in one case
a check reported a clean negative result every day for 67 days because an upstream data field had been
renamed. Nothing about the output looked wrong. We suggest that periodic re-benchmarking include the
fault-injection suite, and that it be triggered by any change to a dependency (a data feed, a
terminology version, a third-party model or API), not only by changes to the model. This is cheap: it
needs no clinical adjudication, because the correct answer for every probe is "this could not be
evaluated".

Q20 (machine-based supervisory agents). Caution is warranted where the supervisory agent is itself a
language model judging outputs. Recent work reports that LLM judges perform poorly at detecting false
reports of success (Advani, arXiv:2606.09863, June 2026: no LLM judge exceeded an AUROC of 0.65). A supervisor that judges outputs
cannot see a failure that left no trace in the output. Supervisory monitoring should therefore observe
component status signals directly, and should itself be tested with injected faults before it is relied
on, so the supervisor is held to the same standard as the device.

Q16 (independent third parties). Fault-injection testing is well suited to independent third parties.
It needs access to the device’s interfaces but not to proprietary training data, it produces a
reproducible pass/fail table, and its probes can be published as an open test suite, so that results
are comparable across manufacturers.

Thank you for the opportunity to comment.

Mark Nafe
SomaSoft