What role should independent third parties play?

R6. Hold benchmarking to the standards CDRH already applies to bench testing (Questions 9 through 13 and 16) This follows from S4 and S6. The paper says that "test methods and acceptance criteria would be prespecified prior to testing," consistent with expectations for non-clinical bench performance testing. I strongly support this and would extend the parallel. Four points: Non-determinism. A GenAI device produces a distribution of outputs, not an output. Performance should be reported as a distribution: repeated runs on identical inputs, disclosed decoding parameters (temperature, sampling settings, seed handling), and reported variance. The paper's position in R.1 that variation in safety-critical behavior — escalation, refusal, diagnostic conclusion — is a failure rather than acceptable noise should be adopted as a firm acceptance criterion. Synthetic data lineage. Synthetic inputs generated by a model of the same class as the device under evaluation share its blind spots. They may supplement real data for stress-testing and for rare presentations, but they should never be the sole basis for a claim about subgroup performance, and the generator should be of demonstrably different lineage from the device. Sponsors should be expected to report the synthetic share of each evaluation set and to show that performance on synthetic and real inputs is concordant before combining them into a single estimate (Question 12). LLM adjudicators. Where an LLM serves as adjudicator, it should be from a different model family than the device, validated against human adjudicators on a held-out sample with reported agreement, and disclosed in the submission. Correlated error between device and judge is the obvious failure mode and it is invisible without this check. Docket No. FDA-2026-N-7874 — Individual comment — Page 6 Sequestered assets. The proposal for independent third parties to maintain sequestered evaluation datasets (Question 16) is the strongest available answer to contamination and optimization-to-the-test, and I support it. The essential safeguard is that the sponsor never sees the held-out set and cannot iterate against it. Sponsor-developed benchmarks are appropriate for demonstrating coverage of the intended use; they are not appropriate as the sole gate for authorization. On competition concerns, the ASCA model — multiple accredited bodies, published methods, FDA-recognized standards — is a reasonable template that avoids a single gatekeeper.

Response to Question 16 Should independent third parties be involved in some or all aspects of a competency-based approach, including device benchmarking and clinical confirmation? What qualifications and independence criteria should apply? What safeguards would prevent third-party participation from limiting competition or innovation? I believe independent third-party involvement is not merely useful but structurally necessary for one specific category of function within the competency-based approach: safety-critical recognition, escalation, and calibration (Appendix A, elements S.1 to S.3). These are the elements most vulnerable to a conflict of interest that self-assessment cannot resolve, because the entity best positioned to know where its own device is weak is also the entity with the strongest incentive not to surface that weakness in a benchmarking result it authors. On independence criteria, I’d propose the standard be defined at two levels, not one: 1. Organizational independence. The third party has no financial relationship with the device sponsor tied to benchmarking outcomes, and no equity, licensing, or revenue-sharing arrangement with the foundation model developer whose model underlies the device under review. 2. Evaluator independence. Where the adjudication method itself uses an AI system (an “LLM-as-judge” pattern, which the paper rightly flags as still requiring the same independence scrutiny), that evaluator model should not be the same model family, or a fine-tuned variant of the same model family, as the device under test. A GPT- based device benchmarked by a GPT-based judge is not independent in any way that matters, even if the organizations administering the test are unrelated. On the competition concern, I’d note that a credentialing or accreditation structure (similar to ASCA, which the paper references) mitigates the innovation-limiting risk better than a closed panel of pre-approved evaluators would. CDRH setting qualification criteria that any structurally independent entity can meet, rather than naming or licensing a fixed set of evaluators, preserves competitive entry for third parties while still enforcing the independence standard.

Question 16 — Role of independent third parties The most useful third-party role is not certifying devices but qualifying and maintaining evaluation assets, since contamination and saturation are problems no individual sponsor can solve and that worsen over time. FDA already has a fit-for-purpose mechanism: the MDDT program qualifies a tool for a specified context of use, which is exactly what a benchmark is. Recommendation. Use MDDT to qualify benchmark suites for defined contexts of use, with an explicit re-qualification interval to address saturation and contamination drift, and require qualified benchmark holders to be structurally independent from both device sponsors and foundation model developers. This is lighter than third-party device certification, avoids the competition concerns raised in the question, and creates a shared public good.

Question 16: Independent third-party participation Qualified independent third parties should have a role in protocol review, benchmark governance, clinical adjudication, and review of serious or disputed errors for moderate- and high-consequence functions. Independence is especially important when outputs are open-ended, sponsor-developed test assets are used, or scoring requires clinical judgment. Clinical adjudicators should have training and recent experience matched to the device's function, intended population, and care setting. For specialty functions, relevant board certification or equivalent Page 3 PUBLIC COMMENT | FDA-2026-N-7874 specialty expertise should generally be expected. Panels should include more than one practice environment when care patterns or resource availability may affect the standard. Independence criteria should address financial relationships, participation in device development, access to sponsor-selected information, and the sponsor's ability to exclude unfavorable cases or adjudicators. Third-party participation should remain proportionate to risk. FDA could support modular methods, reusable evaluation protocols, and transparent qualification standards so that independent assessment does not become a barrier available only to large sponsors. The goal should be credible separation between product development and clinical judgment, not an exclusive certification market.

Question 16: Independent third parties Independent third parties can: Keep hidden test data. run benchmark tests. provide clinical review. confirm sponsor methods. perform multi-site studies. audit monitoring programs. The third party must have suitable clinical, statistical, technical, and human- factors knowledge. The third party must disclose conflicts. Payment must not depend on a positive result. FDA should permit multiple qualified organizations. One organization must not control market access. The manufacturer must remain accountable for the device. Newton’s Tree Inc Considerations for the Regulation of Generative AI-Enabled Medical Devices FDA Docket No. FDA-2026-N-7874 Layer 3: Operational assurance

Question 10: Benchmark construct validity. Sponsor-developed benchmarks are valuable for intended-use specificity and should be permitted, but they should be complemented by at least one sequestered test set held by a party structurally independent of both the sponsor and the foundation model developer, to guard against optimization to the test, a role naturally suited to the third-party mechanisms discussed in Question 16. Question 11: Selecting and justifying clinical confirmation approaches. For real-world data and data collected outside the United States, I recommend requiring a formal transportability assessment: characterization of case-mix, practice-pattern, coding, and data-capture differences between source and target settings, with quantitative adjustment (for example, standardization or reweighting) where differences are material. This is consistent with FDA's RWE program and would be a natural application of modern causal-inference and transportability methods; OUS evidence should be neither privileged nor discounted but transported transparently. Questions 12-13: Statistical measurement and synthetic data. Combining benchmarking and clinical-confirmation evidence into a single pooled performance estimate should generally be avoided unless the sampling frames are demonstrably exchangeable; presenting them as complementary, separately quantified lines of evidence is more defensible and more informative for review. Questions 14-15: Comparators and acceptance criteria. I also support comparators reflecting the care likely to occur in the device's absence (unaided clinical judgment, delayed specialist review, or no intervention), which is the comparator most relevant to public health impact and to least-burdensome evaluation of access-extending devices. Usual-care characterization from RWD and, where randomization is infeasible, target-trial-emulation designs applied to observational data are established, principled ways to construct and justify such counterfactual comparators. Section VI: Postmarket Monitoring Question 18: Accepting greater premarket uncertainty. A pre-/post-market rebalancing is ill-suited for autonomous functions that take actions with severe consequences; furthermore, for functions where users cannot detect errors, post-market signals would only become apparent after harm has occurred, a situation that pre-market evidence is specifically intended to prevent. Incorporating these conditions into the FDA’s existing guidance on benefit-risk uncertainty would maintain a doctrinally consistent approach rather than an exceptional one. Question 19: Postmarket performance evaluation approaches. I encourage CDRH to add a fourth approach where feasible: outcome-linked surveillance, linking device exposure to downstream utilization and clinical outcomes in administrative, claims, and EHR data; behavioral metrics alone can miss harms that manifest downstream (for example, delayed care after under-escalation). Existing infrastructure, including distributed data networks of the kind FDA already uses for medical product surveillance, and coordinated-registry and NEST-type approaches, could be leveraged; this is also where real-world performance monitoring expectations in the January 2025 draft guidance connect naturally to this paper. Questions 22-23: Modifications, re-benchmarking, and PCCPs. Where the nature of future modifications cannot be fully prespecified, the normal condition for GenAI systems; PCCP concepts could be adapted from prespecifying the modifications to prespecifying the verification protocol: a “change-envelope” PCCP that fixes the competency baseline, the re-benchmarking protocol, the acceptance criteria, and the boundaries of the authorized intended use, within which classes of change may be implemented upon passing verification, with results documented and auditable. This preserves the PCCP's core logic (FDA reviews the method once; the method governs many changes) while fitting the continuous-update reality of these systems. Section VII: Other Topics Question 26: Agentic AI systems. Acceptance criteria should tighten as autonomy increases and the opportunity for human review declines, consistent with the activity axis of the risk framework, and an agentic system whose action sequences result in control of another medical device should be evaluated against the risk profile of the controlled device, not merely its own. Conclusion The discussion paper reflects a careful, risk-proportionate, least-burdensome orientation that I support. My principal recommendations are: (i) keep the risk framework two-dimensional and operationalize the additional dimensions, including error detectability by the user, as documented modifiers on the consequences axis; (ii) publish an illustrative mapping from risk position to expected evidence, and an anchored directiveness rubric applied to the empirical distribution of device outputs; (iii) require predictive-validity evidence for gating benc

Question 16 - Role of independent third parties Independent third parties can add value in benchmark custody, sequestered test-set maintenance, conformity testing, expert adjudication, and qualification of reusable regulatory-science tools. Their role should improve independence without diffusing sponsor responsibility. FDA should avoid structures that inadvertently create a small mandatory certification market that limits competition. Participation should be based on transparent qualification criteria, conflict-of-interest controls, reproducible methods, and auditability. The sponsor should remain accountable for the safety and effectiveness of its final device configuration even when third-party evidence is used.

Question 16 — Should independent third parties participate? Use independent evaluators to add capacity — not to create a new bottleneck or a private monopoly. A network of qualified, FDA-overseen third parties — building on the existing ASCA and MDDT precedents — can maintain sequestered datasets, adjudicate cases, and audit postmarket performance. That network needs multiple organizations, transparent accreditation criteria, conflict-of-interest requirements, and appeal mechanisms, so the ecosystem never depends on a single gatekeeper.

Question 16. Independent third parties can strengthen evaluation, provided independence is transparent and access remains competitive. Third parties are particularly useful for sequestered test assets, adversarial evaluation, expert adjudication, cybersecurity assessment, and verification of configuration/evidence integrity. FDA should avoid exclusive certification structures. Qualifications, conflicts, methods, error rates, and financial relationships should be disclosed; sponsors should retain the ability to use multiple qualified pathways. III. Postmarket Monitoring and Change

This interacts with Question 16. Sponsor-developed adversarial assets are less likely to be exposed but carry an evident independence problem, since the same party selects both the attacks and the acceptance criteria. One available balance is to publish the attack category taxonomy and scoring methodology while retaining the specific probe payloads, which preserves reviewability of the construct without publishing a reproduction recipe. Sequestered assets held by a qualified independent party would address both concerns more completely, at the cost of the program design considerations raised in Question 16. Recommendation: For the S-series elements, a sponsor's construct validity argument should address exposure specifically, including whether assets post-date the model's training data and whether they appear in public alignment datasets. Where sponsor-developed adversarial assets are used, the taxonomy of attack classes and the scoring rubric should be prespecified and disclosed even where individual payloads are not. 8. Agentic systems: tie acceptance criteria to the action surface (Question 26) Element A.1 appropriately includes resistance to prompt injection through user inputs, retrieved content, and tool outputs, which reflects the established finding that indirect injection through retrieved content is a distinct attack surface from direct user input (Greshake et al., AISec 2023). I would add one consideration about how the elements interact for agentic devices. A boundary failure under S.2 in a non-agentic informational device produces an inappropriate output that a user may or may not rely upon. The same failure in a device with tool access produces an action. The consequences axis in Figure 1 captures the severity of relying on an incorrect output, but for agentic systems the relevant quantity is closer to the severity of an action taken without any opportunity for reliance to be withheld. Recommendation: For devices in the action-taking columns, acceptance criteria should account not only for the likelihood of unsafe generation but also for the action surface exposed to the model, the reversibility of available actions, the authorization scope granted to the device, the availability of independent confirmation before high-consequence or irreversible actions, and the capability to halt or roll back an in-progress action sequence. Where meaningful autonomous action is possible, S.2 and A.1 should be evaluated as jointly interacting controls rather than as independent checklist items, since the human review that moderates risk elsewhere in the framework is absent by construction. Closing The discussion paper is correct that the range of possible inputs to a GenAI-enabled device may be too large for exhaustive testing to be practical, and the competency-based structure is a sound response to that constraint. My comments concern the durability of that structure across the device lifecycle: that the artifact evaluated at Public Comment, Docket FDA-2026-N-7874 | Polina Moshenets, SichGate Page 7 premarket is the artifact that reaches the patient, that changes to it are enumerated in a way that prompts characterization, that the safety elements are re-measured actively rather than inferred from observational use, and that results are compared at a resolution fine enough to make a regression visible. I would be glad to provide further detail on any of the above if it would be useful to the Center. Respectfully submitted, Polina Moshenets Founder, SichGate Polina.Moshenets@sichgate.com Public Comment, Docket FDA-2026-N-7874 | Polina Moshenets, SichGate Page 8

Question 16 — Should independent third parties participate in competency-based evaluation? Response Yes—particularly for high-impact systems. Manufacturer self-evaluation is necessary, but it should not be the only evidence available for systems whose failures can materially affect patient safety. Qualified independent third parties can contribute sequestered benchmarks, independent clinical adjudication, reproducibility testing, and separation between development and evaluation. FDA itself identifies these as potential roles for third parties in the discussion paper. [1] Independence should be substantive rather than nominal. At minimum, FDA should consider conflict-of- interest disclosure, transparent methods, appropriate clinical expertise, freedom from compensation structures tied to favorable findings, protection against manufacturer cherry-picking of only favorable evaluations, and safeguards against third-party frameworks becoming barriers to competition or innovation. V. Premarket Uncertainty and Real-World Monitoring

Question 16 — Independent third parties Third-party involvement is well suited to benchmark administration, escrow of held-out evaluation sets, and clinical adjudication — activities where independence from the sponsor is the substantive point. The design risks are real and familiar from other accreditation regimes. I recommend: • Published recognition criteria for qualifying bodies, so recognition is not discretionary and entry is contestable. 10 of 19 Docket No. FDA-2026-N-7874 • Strict separation of advisory and assessment roles. An entity that advised a sponsor on device development, benchmark design, or submission strategy must not assess that device. This is elementary quality-system independence and should be stated as a disqualification, not a disclosure. • Public disclosure of financial relationships between assessing bodies and sponsors, including aggregate revenue concentration — a body deriving most of its revenue from a small number of sponsors is not independent in any meaningful sense regardless of per-engagement firewalls. • A preserved first-party pathway. Third-party assessment should be an option that reduces review friction, never a mandatory gate. Making it mandatory converts recognized bodies into rent-collecting chokepoints and disproportionately burdens smaller developers, which is a competition harm and, over time, a safety harm through reduced diversity of approaches. I would also flag that third-party assessment capacity for generative medical devices does not currently exist at scale in any form I am aware of. If CDRH intends third parties to play a significant role, the paper should acknowledge the capacity-building timeline, because a framework that depends on institutions that do not yet exist will not be operable on the timeline the technology is moving.

Question 16: independent third parties. Sequestered benchmark datasets and independent adjudication panels are the right roles for qualified third parties, and ASCA and MDDT are reasonable scaffolds. One caution: fee structures and access terms must scale with sponsor size, or third-party certification becomes a toll gate that only incumbents can pay, with the competitive consequences the paper itself warns against. CDRH's instinct to evaluate these systems the way medicine evaluates clinicians, through demonstrated competency, enforced boundaries, and supervised confirmation rather than exhaustive enumeration of inputs, is the right one. I appreciate the agency's early engagement on this topic and would welcome the opportunity to contribute further, including on evaluation scenarios specific to disorders of gut-brain interaction. Respectfully submitted, Richard Pescatore, D.O. Founder and Chief Executive Officer, BellyMD, Inc. Board-Certified Emergency Physician Philadelphia, Pennsylvania

FDA Question 16 - Independent third parties Trace ID. TR-Q16 | FDA Q16; Sec. V.E; App. B; pp. 18-19 / 28-29 BCR response. Use qualified independent third parties for sequestered datasets, red-team testing, adjudication, and benchmark maintenance, with transparent qualification/conflict rules, multiple eligible providers, rotation, and retest/appeal mechanisms. BCR rule basis. BCR-R11,R13,R15,R17 Solution-stack link. S5,S7,S12 Closure evidence. Independence/conflict criteria, multiple eligible providers, rotation/retest/appeal, sequestered evidence Page 18 BCR Realization Audit - FDA GenAI Medical Devices - REV4 Pass / re-open. Independence demonstrated without single unreviewable gatekeeper Re-open when: Conflict, vendor, dataset, or qualification status changes.

Response to Question 16: Third-Party Testing — Essential, But With Structural Safeguards WHM strongly supports the use of independent third-party organizations in benchmarking and clinical confirmation. Independent adjudication provides credibility, reduces sponsorship bias, and improves the reliability of performance evidence. However, if FDA is not deliberate about the structure of third-party testing, it risks creating a duopoly of approved testing organizations — a small number of well-resourced labs with high barriers to entry that impose costs which smaller manufacturers cannot sustain. FDA should expand its Accreditation Scheme for Conformity Assessment (ASCA) program to explicitly include qualified AI benchmarking organizations, with published, standardized fee structures and published qualification criteria. Multiple qualified testing organizations should be actively encouraged, including academic medical centers that have the domain expertise to conduct specialty-relevant evaluation. The goal is a competitive, pluralistic testing ecosystem — not a regulatory gatekeeping bottleneck.