LabInTrace (Alexander Openstone)
Themes it raises
FDA questions it names
Q1 · The two-axis risk frameworkQ10 · Benchmark contamination and saturationQ16 · Independent third partiesQ18 · Trading premarket certainty for postmarket monitoringQ20 · Machine-based supervisory agentsQ24 · Third-party foundation model changesQ26 · Agentic devices
The comment as filed
LabInTrace LLC submits the attached comments on questions 1, 10, 16, 18, 20, 24 and 26. See attached.
Attachment
Comment on Docket No. FDA-2026-N-7874
Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback
October 19, 2026
Submitted by Alexander Openstone, Founder, LabInTrace LLC (Wilmington, DE).
LabInTrace builds evaluation and governance for AI decisions in regulated laboratories
(CLIA/CAP, GMP). Our background is laboratory automation and instrument software, and our
current work is benchmarking AI agents on pre-analytical specimen decisions grounded in CLSI
and CAP requirements. We welcome the risk framework that scales evidence with directiveness and
consequence. Our comments address four points.
1. Outcome accuracy alone cannot show that a decision was grounded (Q10,
benchmark construct validity). In our benchmark work, agents that reached correct answers
by fitting patterns in the test data passed alongside agents that reasoned from the published clinical
standards. Outcome scores could not tell them apart. We have also seen agents pass by transcribing
rules disclosed in the task rather than deriving them. For GenAI devices, valid benchmark evidence
should therefore include (a) held-out cases that differ from the visible ones in the specific condition
the device is supposed to reason about, and (b) control solvers that use shortcuts and must fail. A
benchmark that a shortcut solver passes does not measure the claimed competency.
2. Traceability should cover the decision, not only the output (Q1, Q26). We support
adding traceability to the risk framework, as Q1 suggests. For agentic devices (Q26), traceability to
primary sources is not enough on its own, and neither are reasoning traces or logs, which are the
system describing its own behaviour. They show what the agent did, not whether the action
followed the applicable rule given the evidence available at the time. We suggest that, for agentic
functions above the lowest risk positions, FDA ask for decision records that bind each
consequential action to (a) the inputs it relied on and (b) the version of the rule or policy in force, so
that a party other than the device can check the decision without trusting the device’s own account.
This complements the human-oversight checkpoints before irreversible actions described for
agentic competencies, and lets the human reviewer see whether an action was justified, not only
that it was proposed.
3. Independent adjudication should be independent in method, not only in
organisation (Q16). The paper asks for adjudicators that are structurally independent of the
sponsor and the model developer, including when the adjudicator is an LLM. We agree, and add
that an LLM adjudicating another LLM’s outputs is likely to share its failure modes, especially
where the reference answer must be derived from published standards rather than recognised from
patterns. Where the correct result can be computed from the authoritative logic (for example, a
laboratory rule, a measure definition or a labeled threshold), the adjudicator should compute
expected truth from that logic and should never consume the device’s output when doing so. LLM
adjudication may be appropriate for open-ended outputs, but its agreement with clinicians should
itself be measured on cases where the two disagree.
4. Supervisory agents for postmarket monitoring need the same independence (Q18,
Q20, Q24). We support accepting some premarket uncertainty in exchange for strong postmarket
monitoring, provided the monitoring does not rely on the device’s own reports. A machine-based
supervisory agent is useful when it (a) is separate from the monitored device and its foundation
model, (b) checks decisions against the versioned rule and the recorded inputs rather than rereading the device’s explanation, and (c) produces results that can be reproduced later from the
same records. Monitoring that only summarises device logs inherits the device’s blind spots.
Foundation model updates should trigger re-running the same checks on a fixed reference set, so
that a change in behaviour shows up as a measured difference.
We would be glad to share our benchmark designs and findings with the Digital Health Center of
Excellence. Background paper: Openstone, A. Credibility Over Auditability, Zenodo,
https://doi.org/10.5281/zenodo.20209229.
Alexander Openstone LabInTrace LLC alex.openstone@labintrace.com