Collective Good Dx, Inc. (Sean Sylvia)

Please see the attached comments submitted by Collective Good Dx, Inc. regarding “Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback,” Docket No. FDA-2026-N-7874.

Collective Good Dx, Inc.
101 Glen Lennox Drive, Suite 300
Chapel Hill, NC 27517

October 19, 2026
Center for Devices and Radiological Health
Food and Drug Administration
Comment on Considerations for the Regulation of Generative AI Enabled Medical
Devices
Docket No. FDA-2026-N-7874
Dear CDRH colleagues:
Collective Good Dx, Inc. submits these comments as a company developing methods for clinical AI
evaluation. Our work focuses on eliciting specialty-matched clinical judgment, preserving meaningful
disagreement, and documenting evidence against a defined protocol. This comment represents the views
of Collective Good Dx, Inc. and does not represent the views of the University of North Carolina at
Chapel Hill (UNC–Chapel Hill).
We ask CDRH to describe, through guidance or pre-submission feedback, how independent clinicianpanel evidence can support benchmarking, clinical confirmation within a competency-based assessment,
and postmarket sample review. The criteria should address specialty matching, independent initial
judgments, clinically meaningful disagreement, transparent adjudication, and documentation in an
auditable evidence package. The suitability and rigor of a panel should depend on the device's intended
use, risk, and evaluation question. Clinician participation or consensus alone does not establish validity.
We understand that the discussion paper seeks early input and does not represent draft or final guidance.
[1] Our recommendations concern the properties of evidence, rather than endorsement of a named
commercial product or a replacement for FDA review or institutional governance.

Question 14 Comparators and clinician panels
For open-ended outputs with multiple clinically acceptable responses, a single reference label may not
adequately describe performance. A structured panel can preserve the range of acceptable judgments and
identify consequential disagreement. Where a well-supported clinical reference standard exists, it can
remain appropriate. Comparator selection should be justified for the intended use. [1, pp. 16, 18]
Acceptance criteria should rest on a prespecified protocol: who is qualified, how reviewers are matched to
the task, what they are asked, how judgments and disagreements are recorded, and what adequate
performance means. A majority label can be useful for some purposes, but it should not conceal material
differences among qualified reviewers. In a hypothetical example, most reviewers accept an AI-generated
clinical summary, but one identifies an omitted medication allergy. The concern should be recorded,
assessed for clinical significance, and reflected in the safety determination even if overall agreement is
high.
TRIPOD-LLM emphasizes transparent, task-specific reporting. [2] Building on those principles, we
recommend prespecifying the unit of analysis and denominator for each measure, the rubric and version,
and the handling of missing information, abstentions and multiple acceptable responses. Reports should
distinguish clinically important omissions from added or altered information, show item-level agreement
and uncertainty, and justify agreement measures for the rating scale and prevalence. Agreement should be
reported separately from accuracy against a justified clinical reference standard.

FDA-2026-N-7874 | 1
The applicable standard of care and the performance of a median clinician in practice are not
interchangeable. A standard-of-care comparator asks whether an output meets a justified clinical standard.
A median-clinician comparator describes typical observed performance, which need not meet that
standard in every case. Sponsors should identify and justify the comparator they use.
Consensus should not require reviewers to eliminate meaningful dissent. Disagreement that persists after
independent review of the same case under the same rubric may provide evidence about uncertainty in the
task. It should be characterized before aggregation, with an account of whether it reflects multiple
acceptable options, missing information, reviewer error, or a potential safety issue. We recommend
preserving individual judgments alongside any adjudicated conclusion. The protocol should prespecify
how unresolved safety-related disagreement affects acceptance criteria, triggers additional evaluation, or
informs corrective action. Aggregate scores and holdout metrics should be interpreted with this record
rather than assumed to capture every clinically important difference.
Automated scoring should be validated for the intended task and should not be assumed to reproduce the
distribution of qualified clinical judgments.
Reviewer qualifications should match the clinical content and complexity of the outputs. For specialistcontent tasks, generalist-only review may miss errors or distinctions that require specialist knowledge.
The protocol should justify the balance of task-specific expertise, relevant cross-specialty perspectives,
and intended-user representation.
Whether to evaluate a human-AI team, the device alone, or both should follow intended use. Clinicians
participating in a human-AI workflow do not automatically provide an independent comparator. Sponsors'
own clinical reviews can inform development and quality management, but should be distinguished from
evidence generated by independent adjudicators.

Question 16 Independent third parties
Qualified, independent third parties should contribute to clinician adjudication in benchmarking and
clinical confirmation when that approach is appropriate to the device and task. The discussion paper
considers roles for third parties and documented adjudicator independence. [1, pp. 14-17] One workable
model is a panel of independent clinicians, selected for specialty and experience relevant to the task, who
score cases separately under a prespecified rubric, with individual judgments and disagreement preserved
before adjudication.
We recommend documenting independence from both the device manufacturer and the foundation-model
developer, where applicable. The protocol should also address the independence of those who select
cases, match reviewer expertise, administer the evaluation, and maintain the judgment record. Financial
and organizational relationships, payment arrangements, and control over case selection, scoring, and
reporting should be disclosed and managed.
A hospital evaluating a tool it intends to deploy may provide valuable local evidence, but its deployment
interests should be distinguished from independent adjudication. Sponsor selection or funding of
reviewers neither establishes nor necessarily precludes independence. Safeguards should include
compensation unrelated to findings, protection of individual judgments from sponsor influence, and
retention of unfavorable findings in the evidence package.
The useful deliverable is a documented evidence package containing the evaluated device configuration
and version, protocol and rubric versions, the intended-use sampling frame and rationale, reviewer
qualifications and conflicts, individual initial judgments, adjudication decisions, and reasons for
unresolved disagreement. It should report reviewer agreement, distinguish initial judgments from
subsequent adjudication, and explain how disagreements were resolved and any aggregate conclusion was

FDA-2026-N-7874 | 2
reached. A sponsor could include it in a submission, and a health system could use it in local evaluation.
Its evidentiary value would depend on the method and results; it would not replace FDA review or
authorize deployment. Panel findings should be interpreted within the tasks and settings evaluated;
additional evidence may be needed to establish workflow performance or patient benefit.
Transparent protocols may reduce duplicative evaluation across sponsors and health systems. Reuse of
particular results should be justified for the device and model version, intended use, population, setting,
and evaluation question.
Qualifications should include clinical alignment with the outputs under review, a method suited to the
task, documented independence, and sampling relevant to intended use. An LLM-based adjudicator
should also be assessed for dependence on the model under evaluation and for the validity of its
judgments. Automation should not be presumed independent simply because a different tool name is
used.
The criteria should remain functional and open to any qualified provider. FDA's discussion of the
Accreditation Scheme for Conformity Assessment (ASCA) offers a possible analogy for demonstrated
competence under defined methods. [1, p. 17] Safeguards should prevent control of outcomes by sponsors
while allowing proportionate approaches when specialist availability, cost, or task characteristics make a
particular panel design impractical. Manufacturers remain responsible for the validity of the evidence they
submit.

Question 19 Postmarket monitoring
We support periodic benchmarking, sample-based clinician review, and longitudinal performancedegradation monitoring as complementary approaches. [1, pp. 19-21] Independent clinician review can
examine clinical appropriateness and disagreement that automated scores may not capture. Conversely,
automated monitoring can help identify patterns or changes that warrant targeted clinical review.
When sample-based clinician review is used, we recommend carrying forward the relevant safeguards
from premarket clinical confirmation: prospective criteria, reviewer qualifications, documented
independence, and preservation of clinically meaningful disagreement. Manufacturer-run scoring should
be described accurately and distinguished from independent review.
Review should occur on a prespecified cadence proportionate to risk and exposure, with additional
assessment after relevant triggering events. Triggers should include a material foundation-model or
orchestration change, a material shift in the patient population or deployment setting, a prespecified
change in drift metrics, and safety signals from users. The protocol should identify who evaluates the
signal, when review is escalated, and how unresolved concerns are handled. We do not propose one
universal interval for all devices.
Samples should cover the intended-use distribution, relevant specialties and clinically important
deployment settings. The protocol should justify site coverage, any pooling across sites, and targeted
sampling of important failure modes, with sample sizes and review intensity proportionate to risk and
exposure. Premarket assessments should support comparable evaluations over time after relevant changes.
Repeat evaluations should use qualified reviewers with documented calibration proportionate to the task.
Protocols should record changes in reviewers, rubrics, case mix, and clinical context. These records can
help distinguish device-related changes from other sources of variation.

Question 20 Machine-based supervisory agents
Machine-based supervisory agents may support monitoring and triage. In a clinical-summary study,
Croxford and colleagues reported strong agreement between aggregated LLM and physician ratings under

FDA-2026-N-7874 | 3
the study rubric. [3] A controlled clinical-note preprint found weaker discrimination of omissions than of
added or altered content for the judge designs tested, with improved omission detection after redesign. [4]
These task-specific findings support validation by clinically important error type rather than reliance on
aggregate agreement alone. Sponsors should document the supervisor’s data, model and evaluation
methods, potential dependencies on the device, and performance under relevant changes. Where
independent clinician adjudication is needed for the intended use and risk, automated monitoring should
complement that assessment.

Question 21 Roles and manufacturer accountability
Manufacturer accountability for the device should remain with the manufacturer. Other parties can
contribute evidence and monitoring capacity without obscuring who is responsible for obtaining,
evaluating, and acting on that evidence.
Clinicians can provide specialty-matched judgment under a defined protocol. Their work should be
recognized as skilled adjudication, with appropriate qualifications and compensation. Clinicians who use
a device in care can also provide safety signals; their participation alone does not establish an independent
confirmation panel.
Healthcare institutions should retain responsibility for local deployment decisions. Independent sample
review may supplement specialty coverage that an institution cannot maintain internally for every model,
but should not replace its governance processes or transfer the manufacturer's postmarket responsibilities
to the hospital.
Professional societies can help define standards of care, reviewer qualifications, and task-specific
methods. If they also generate device-specific evidence, standards-setting and evaluation roles should be
distinguished and relevant conflicts disclosed and managed. The same competence and independence
criteria should apply regardless of provider type. Standards-setting bodies can also help define sampling,
disagreement reporting, and documentation of independence.
Roles should be documented in advance, including responsibility for receiving findings, assessing safety
concerns, and taking corrective action. Contributions from clinicians, institutions, or third parties should
strengthen manufacturer accountability rather than provide a reason to defer it. [1, pp. 20-21]

Related questions
Question 4. When a generalist receives specialist-content information, a fluent response may contain
errors the user cannot readily detect. We recommend specialist input in the evaluation of such outputs
even when the intended user is a generalist. The evaluation should also consider the safeguards and
clinical context that may mitigate this risk.
Question 10. Public and sponsor-developed benchmarks each have limitations. Independent test
administration and qualified review can help address concerns about optimization to the test, but do not
by themselves establish that a benchmark predicts safe real-world behavior. Sponsors should justify
representativeness, scoring methods, and the relationship to intended use, and distinguish internal review
from independent evidence.
Question 18. We recommend that any increased reliance on postmarket monitoring be supported by a
credible, risk-proportionate plan that includes independent, specialty-matched clinician review where
clinically necessary and preserves meaningful disagreement. Such monitoring should have defined
triggers, response responsibilities, and limits. It should not be assumed to compensate for inadequate
premarket evidence where errors could cause serious harm before monitoring can detect and address
them.

FDA-2026-N-7874 | 4
Limitations that undermine clinician panel evidence
We ask CDRH to assess whether clinician-panel evidence has the following limitations before relying on
it for clinical confirmation or postmarket sample review:
1. Material disagreement is hidden by a forced majority label without a record of the underlying
judgments.
2. Sponsor-controlled review is presented as independent without adequate documentation and
safeguards.
3. Reviewer expertise does not match the clinical content and complexity of the task.
4. Sampling is not representative of the intended use, or acceptance criteria are not prespecified.
5. Automated scores are treated as equivalent to qualified clinical adjudication without task-specific
evidence supporting that use.
6. Reviewer performance, including appropriate agreement and calibration measures, is not documented,
limiting interpretation and comparison across evaluations.
Our recommendation is to preserve independence, appropriate expertise, and meaningful disagreement as
assessable properties of the evidence. It does not ask the Agency to require a named product or endorse
Collective Good Dx, Inc.
We appreciate CDRH's invitation for early input.
Respectfully submitted,

Sean Sylvia, PhD
CEO, Collective Good Dx, Inc.
Associate Professor, Department of Health Policy and Management
UNC–Chapel Hill
sean@collectivegood.io
Drew Narayan, MS, MBA
Chief Operating Officer, Collective Good Dx, Inc.

References
[1] U.S. Food and Drug Administration, Center for Devices and Radiological Health. Considerations for the Regulation
of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback. Page references use the printed
page numbers. https://www.fda.gov/media/194242/download
[2] Gallifant J, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine.
2025;31:60–69. https://doi.org/10.1038/s41591-024-03425-5
[3] Croxford E, et al. Evaluating clinical AI summaries with large language models as judges. npj Digital Medicine.
2025;8:640. https://doi.org/10.1038/s41746-025-02005-2
[4] Fox S, Markham L, Lail R, Karotsieris M. LLM Judges Verify Presence, Not Absence: Omission Blindness in AI
Clinical Notes and What Recovers It. arXiv. 2026. Preprint. https://arxiv.org/abs/2608.31016

FDA-2026-N-7874 | 5