EG14Y Systems Inc

Docket FDA-2026-N-7874 | EG14Y Systems Inc. | GenAI-Enabled Medical Devices Discussion Paper

SUMMARY
EG14Y builds version-pinned, reproducible evidence tooling for healthcare interoperability and AI change assurance. We comment on Q1, 9, 10, 16, 19, 20, 22-26, for GenAI-enabled functions within FDA’s device scope, as risk-proportionate options. Central recommendation: preserve the premarket benchmark as a re-executable evidence record (specification, configuration, scoring criteria, results) so postmarket reassessment (Section VI.C) is interpretable. Exact re-execution may be impossible; the record still enables like-for-like comparison.

1. PRESERVED BENCHMARK EVIDENCE (Q9, 22, 23)
Premarket packages could include hash-identified fixtures; rubrics and acceptance criteria fixed before testing; the configuration under test (model version, prompts, retrieval, guardrails, orchestration); and tool and environment versions. For nondeterministic systems, also preserve repeated-run distributions and predefined materiality criteria. Benchmarking supports, not replaces, clinical confirmation (Q9).
Q22: Re-benchmark results could be triaged as pass, fail, different (behavior changed, safety impact undetermined) or unresolved. "Different" is a triage status, not a regulatory classification; separating it from demonstrated deterioration supports risk-proportionate change assessment under applicable quality-system, PCCP and premarket requirements.
Q23: For modifications that cannot be prespecified, preserve a stable set of safety-critical invariants while letting the specific tests evolve. A change-impact record would identify affected capabilities, authorization boundaries and benchmark elements, keeping the original assessment beside updated tests; historical comparison and reassessment against current clinical knowledge may both be needed.

2. VALIDITY AND INDEPENDENCE (Q10, 16)
Contamination and saturation are better managed by process than by excluding sponsor benchmarks: pre-register test plans and, for higher-risk functions, use sequestered holdouts and reproduction by a party independent of benchmark authorship and environment configuration. Record independence as a profile (organization, operator, tool, environment, data source), not a binary label; a same-team rerun shows repeatability only. Reproduction is not attribution: a finding may reproduce yet remain unattributed (device, model or fixture).

3. THIRD-PARTY MODEL CHANGES (Q24, 25)
Q24: combine contractual version pinning and notification where obtainable, configuration records, and canary fixtures run on a set cadence and on any provider-side signal. Canaries may miss hidden updates; document residual uncertainty.
Q25: We support exploring voluntary Foundation Model MAFs as reusable model evidence (versions, known limitations, safety-relevant behavior, update policies, audit logs). A MAF cannot substitute for evaluating the configured device; absent provider participation, combine available contractual assurances, configuration records and independent canary testing.

4. SUPERVISORY AGENTS (Q19, 20)
Evaluate supervisory agents for sensitivity, false-positive rate, calibration against independently adjudicated cases, performance under distribution shift, and timely human escalation. Deterministic assertions should govern machine-verifiable invariants; model-based adjudication may supplement them for contextual judgments. Trigger reassessment on material changes to models, tools, authorization policies or observed performance, not only fixed intervals.

5. AGENTIC DEVICES (Q1, 26; Appendix A.1)
A.1 covers oversight checkpoints and safety-envelope recognition. We suggest adding enforcement verification: showing a control prevents the consequential action at the tool or action layer, not merely that the agent states compliance. One approach: matched scenarios under authorized and restricted conditions, with an explicit authorization contract and independently observed downstream effects, testing whether a prohibited action committed (e.g., via queued operations, delegated workers or alternative tools). Results would distinguish verified enforcement, confirmed violation and insufficient evidence. For Q1, a downstream safeguard should earn risk-reduction credit in proportion to evidence of its effectiveness under operational and adversarial conditions.

TRADE-OFFS
These practices add burden; scale them to risk (Q8). Hashing and configuration capture are cheap; holdouts and independent reproduction fit higher-risk functions. Fixed regression cases preserve comparability; refreshed ones limit overfitting.

DISCLOSURE
EG14Y is a commercial company developing assurance tooling in this area; recommendations are intended to be vendor-neutral. The matched-scenario agent evaluation above is a proposed research protocol, not a clinically validated method or FDA-qualified tool.

Version-Pinned Benchmark Evidence: Method Summary
Attachment to EG14Y Systems Inc. comment, Docket FDA-2026-N-7874 (Considerations for the Regulation of Generative
AI-Enabled Medical Devices). Status: method specification. No empirical results are reported in this document.

1. Purpose
Supports the comment's recommendation that a premarket benchmark be preserved as a re-executable evidence artifact, so that
later re-benchmarking (Discussion Paper Section VI.C) compares like with like. The method below is tool-neutral and was designed
for healthcare interoperability testing (FHIR implementation guide upgrades); we propose it as a pattern for GenAI-enabled device
evidence, not as a validated FDA submission format.
2. Core cycle
Step What is recorded
Baseline Fixtures (identified by content hash), assertions, scoring rules and acceptance criteria fixed before execution; exact
versions of every component under test and of the tools used.
Change The modification under evaluation (new version of a model, implementation guide, prompt, or configuration).
Compare Same fixtures re-run against the changed system; per-fixture result compared with baseline.
Explain Cause assigned from a fixed list: specification, dependency, terminology, testing tool, implementation, configuration,
fixture, or insufficient evidence.
Reproduce A second party re-executes from the evidence package, ideally without author coaching (blind mode hides the expected
cause).
Disposition Final classification and any follow-up action recorded with the evidence.

3. Result classification
Class Meaning
PASS / FAIL Assertion met / not met against pre-specified criteria.
DIFFERENT Behavior changed between versions; effect on safety or conformance not yet determined. Distinct from "worse".
UNRESOLVED Evidence insufficient to classify or attribute. A finding can be reproduced yet unresolved: reproduction is not
attribution.

4. Evidence and independence controls
Control Description
Independence profile Independence recorded across organization, operator, tool, infrastructure, package or dependency resolver,
terminology service, and configuration, rather than labeled with a single "independent" flag. A same-team rerun
shows repeatability only.
Deterministic first Deterministic assertions are authoritative. Any AI-assisted analysis is optional, disclosed in result metadata, and
requires a human review step.
Synthetic data only Fixtures are synthetic; no PHI or production credentials are used.
Named results Tool- or vendor-level results are published only with participant agreement; aggregate and methodology findings
may be shared.
Scope limits Not a certification, compliance determination, or product ranking.

5. Mapping to Discussion Paper topics
Paper reference Relevance of the method
VI.C / Q22-23 (re-benchmarking, PCCP) Frozen, hash-identified baseline enables like-for-like re-run after a change; DIFFERENT
class supports change-scaled review.
Q10, Q16 (validity, independence) Pre-specified criteria, sequestered items maintained apart from the sponsor, independence
profile for reproducers.
Q19-20 (monitoring, supervisory agents) Cadenced re-runs and canary fixtures; deterministic assertions remain authoritative over
model-generated judgments.
Q24-25 (third-party model changes, MAF) Retained execution evidence lets a behavior change be attributed to the model, device
configuration, or the test.

6. Current application and limits
The method is being applied to a pilot comparing a published implementation guide release (Da Vinci Prior Authorization Support
2.1.0) with a commit-pinned later build (2.2.1 continuous-build snapshot), using synthetic fixtures. The pilot is a testing scenario
within the HL7 Da Vinci Burden Reduction Connectathon track; no findings are claimed here. Transfer to GenAI-enabled devices
(open-ended outputs, clinical adjudication, third-party models) is a proposal and would need its own validation. Questions or
protocol requests: evgupta@eg14ysystems.com.