Abdurrahman Gumus

Comment of Abdurrahman Gumus, Ph.D. (Associate Professor of Computer Engineering, Isparta University of Applied Sciences, Turkiye), submitted in an individual academic capacity. The attached PDF (3 pages) responds to Questions 7, 9, 10, 12, 16, 19, 20, 21 and 26 of the discussion paper. In summary, it recommends prespecified, criterion-level scoring rubrics with penalty criteria; validation of LLM graders against independent multi-clinician consensus, reporting Cohen’s kappa with Gwet’s AC1 and agreement between clinicians first; repeated runs with pass^k and clustered, paired confidence intervals; disclosure obligations for sequestered third-party benchmarks; treating each grader version as a rater whose change triggers re-validation in postmarket monitoring; and trajectory-level audits of tool use for agentic functions. Full comment attached as PDF.

Docket No. FDA-2026-N-7874 · Abdurrahman Gumus, Ph.D., Isparta University of Applied Sciences · October 2026

Re: Docket No. FDA-2026-N-7874, “Considerations for the Regulation of
Generative AI-Enabled Medical Devices: Discussion Paper and Request for
Feedback”
Comment of Abdurrahman Gumus, Ph.D., Associate Professor, Department of Computer
Engineering, Isparta University of Applied Sciences, Turkiye. Submitted in October 2026 in my
individual academic capacity; the views expressed are my own and not those of my university.

I respond below to Questions 7, 9, 10, 12, 16, 19, 20, 21 and 26. My doctorate (Electrical and
Computer Engineering, Cornell University) was in biomedical microsystems, followed by
postdoctoral research on wearable and diagnostic devices at Cornell and KAUST. My current
research covers deep learning for medical image analysis and LLM-based reasoning and multiagent systems, and includes the design of a benchmark in which tool-using agents answer
clinical evidence questions by searching PubMed and are graded, criterion by criterion, against
the conclusions and GRADE certainty ratings of recent Cochrane systematic reviews. The
recommendations below draw on that design work and on the published evaluation literature,
not on results of my own.
Questions 7 and 9: the benchmarking structure. I support the proposed two-step approach,
device benchmarking followed by clinical confirmation, provided that competence is defined by
individual, gradeable criteria rather than a single score. The paper envisions that sponsors
“describe and justify scoring rubrics” (p. 13). I recommend a prespecified rubric of binary criteria
for every item, stating what an output must contain (positive points) and must not contain
(penalties, e.g., a fabricated citation, certainty stronger than the evidence, an individualized
recommendation outside the device’s scope, an unjustified refusal), with disclosure of who wrote
or reviewed each criterion. Scores should be reported per element, not only as a composite,
since similar composites can conceal different failures.
S.2 already counts over-refusal, “declining requests that fall within the device’s intended use” (p.
24), as a failure; it should be measured with benign but sensitive control items, with under- and
over-refusal reported separately. In OR-Bench, models’ rejection rates for safe and for toxic
prompts had a Spearman rank correlation of 0.89 [1], so a benchmark scoring only harm
avoidance may reward withholding. On Question 9, S.3 should be retained: the paper treats
presenting uncertain information with false confidence as “a safety failure” (p. 24), yet a
systematic review of 519 health-care LLM studies found that “calibration and uncertainty (1.2%)
were infrequently measured” [2]. For evidence summaries, the GRADE certainty rating offers an
external scoring standard. One element seems missing: source traceability, which Question 1
lists as a possible risk dimension (p. 9). For functions that cite sources, every cited source
should exist, have actually been retrieved or provided, and support its statement.
Questions 10 and 12: contamination and statistics. Against “data contamination, saturation,
and lack of representativeness of real-world conditions” (p. 12), two practices would help. First,
items whose reference answers were published after each model’s training cut-off, with dates
disclosed; these also test whether a device “acknowledges uncertainty for evidence not
reflected in its training” (p. 25). Second, construct-validity evidence: the pool’s difficulty
distribution, including the share of items current frontier models fail (items every system passes
cannot discriminate), item provenance and the last refresh date; Esposito and Zhang argue for
“protected live item pools” and “transparent procedures” [3]. For statistical meaning, each item
should be run at least three times, with the probability that all runs pass (pass^k) reported

Page 1 of 3
Docket No. FDA-2026-N-7874 · Abdurrahman Gumus, Ph.D., Isparta University of Applied Sciences · October 2026

beside the mean, consistent with R.1’s “reproducibility across repeated runs” (p. 26).
Confidence intervals should account for clustered items and paired model comparisons, and
power analysis should be used to prespecify the smallest detectable difference [4], informing
how “endpoints and sample sizes might be determined” (p. 16).
Question 16: third parties and LLM graders. I support the principle that adjudicator
independence and qualifications “would still be applicable when the expert adjudicator is itself
an LLM” (p. 14). Any LLM grader should be validated against multi-clinician consensus with
documented independence, as the paper describes for expert clinical adjudication (p. 15), on a
sample spanning the quality range. Cohen’s κ should be reported with Gwet’s AC1, which is
“less affected by prevalence and marginal probability” [5]: in one evaluation, two physicians
scoring the same model’s test plans agreed on 113 of 132 cases (86%), yet linear-weighted κ
was 0.28, which the authors attribute to “severe class imbalance” [6]. Agreement between
clinicians should be reported first, as the reference for grader agreement; in the HealthBench
meta-evaluation, the GPT-4.1 grader’s macro-F1 was 0.709, while theme-level physician-tophysician averages were 0.569 to 0.730 [7]. Third parties maintaining “sequestered evaluation
datasets and benchmarks” (p. 17) should make the disclosures above and, as Esposito and
Zhang propose, report evaluator credentials and compensation [3]; publishing procedures and
validation statistics would let several parties run comparable benchmarks, one safeguard for
competition.
Questions 19 to 21: postmarket monitoring. As Zhang and Esposito put it, “a judge version is
a rater” [8]: a change of grader model, prompt or rubric should count among the “defined
triggering events” (p. 19) and require re-validation before scores are compared over time. A
fixed overlap set graded by every human rater and grader version lets rater effects and drift be
estimated [8]. An LLM grader used in monitoring can be seen as one of the “machine-based
supervisory agents” on which the paper seeks feedback (p. 20), and its reliability should be
checked against periodic “sample-based clinician review” (p. 19). On Question 21: clinicians can
be specialty-matched adjudicators for that review; healthcare institutions and independent
evaluation networks can maintain sequestered item pools and publish validation statistics;
professional societies can translate guidelines into must and must-not criteria, including
acceptable escalation (S.1); standards-setting bodies can maintain a common reporting
template, for which TRIPOD-LLM [9] is a starting point. These parties would supply
measurement instruments, while the manufacturer remains accountable for acceptance criteria,
monitoring results and corrective action.
Question 26: agentic functions. Benchmarking should grade the trajectory as well as the final
output: log every tool call and result, and audit retrieval adequacy, citation traceability, scope
maintenance (S.2) and escalation (S.1). Date-restricted tool access can help separate retrieval
from recall of memorized conclusions. Building on A.1’s “accurate tool use and recognition of
erroneous tool outputs” (p. 26), acceptance criteria could include trajectory-level must-not
criteria, such as citing a source that was never retrieved.
Thank you for the opportunity to comment.

Abdurrahman Gumus, Ph.D.
Associate Professor, Department of Computer Engineering
Isparta University of Applied Sciences, Turkiye
abdurrahmangumus@isparta.edu.tr

Page 2 of 3
Docket No. FDA-2026-N-7874 · Abdurrahman Gumus, Ph.D., Isparta University of Applied Sciences · October 2026

References
[1] Cui J, Chiang W-L, Stoica I, Hsieh C-J. OR-Bench: An Over-Refusal Benchmark for Large Language Models.
arXiv:2405.20947v5 (2025). https://arxiv.org/abs/2405.20947
[2] Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applications of Large Language
Models: A Systematic Review. JAMA. 2025;333(4):319-328. doi:10.1001/jama.2024.21700
[3] Esposito M, Zhang L. The Benchmark Ceiling: Human Judgment, Evaluation Scarcity, and the Political
Economy of AI Capability Measurement. Working paper, arXiv:2607.01254v2 (2026). https://arxiv.org/
abs/2607.01254
[4] Miller E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.
arXiv:2411.00640 (2024). https://arxiv.org/abs/2411.00640
[5] Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1
when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC
Med Res Methodol. 2013;13:61. doi:10.1186/1471-2288-13-61
[6] Brodeur PG, Buckley TA, Kanjee Z, et al. Superhuman performance of a large language model on the
reasoning tasks of a physician. arXiv:2412.10849v3 (2025). https://arxiv.org/abs/2412.10849
[7] Arora RK, Wei J, Soskin Hicks R, et al. HealthBench: Evaluating Large Language Models Towards Improved
Human Health. arXiv:2505.08775 (2025). https://arxiv.org/abs/2505.08775
[8] Zhang L, Esposito M. Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program.
Working paper, arXiv:2610.07003 (2026). https://arxiv.org/abs/2610.07003
[9] Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies using large language
models. Nat Med. 2025;31(1):60-69. doi:10.1038/s41591-024-03425-5

Page 3 of 3