SEENERA (Marina Meyjes)

SEENERA submits the attached comment on Considerations for the Regulation of Generative AI-Enabled Medical Devices (FDA-2026-N-7874). The comment is endorsed by Meghan Piccinin, DO, Neurologist, Pittsburgh, PA.

We support the competency-based approach in the discussion paper. Benchmark element R.2, Subgroup performance, asks whether safety-critical behaviors such as escalation hold across demographic subgroups, but it does not name sex or specify a test method.

We recommend that GenAI-enabled devices be tested on pairs of cases that are identical except for the patient’s stated sex, in premarket benchmarking and again after model changes. To pass R.2, a device should show no differences by sex in urgency, referral or psychological framing beyond a pre-specified tolerance, excluding differences that are clinically expected. Specifically, we recommend that FDA:

1. Define R.2 to include sex-counterfactual testing of urgency, referral and framing, with pre-specified acceptance criteria (Q6, Q9).
2. Require qualified human adjudication of sex-based urgency differences, not LLM graders alone (Q14, Q16).
3. Specify sex-disaggregated reporting, including in Foundation Model Device Master Files, and re-testing after model updates (Q18, Q19, Q22, Q25).
4. Recognize a shared sex-counterfactual benchmark with held-out test sets, administered by qualified independent parties whose results CDRH can consider in review (Q10, Q16).

The attached comment provides supporting evidence and details on each recommendation.

Marina Meyjes, Head of Research and AI Evaluation, SEENERA

Comment on FDA-2026-N-7874: Sex-Counterfactual Evaluation of GenAI Devices

Docket: FDA-2026-N-7874, Considerations for the Regulation of Generative AI-Enabled Medical
Devices

To: Dockets Management Staff, U.S. Food and Drug Administration; Digital Health Center of
Excellence, CDRH

From: Marina Meyjes, Head of Research and AI Evaluation, SEENERA

Endorsed by: Meghan Piccinin, DO, Neurologist, Pittsburgh, PA

Re: Benchmark element R.2 (Subgroup performance); Discussion Questions 6, 7, 9, 10, 14, 16,
18, 19, 22 and 25

Summary

We support the competency-based approach in the discussion paper (Q7). The paper proposes
ten benchmarking elements that devices could be tested against (Section V.B). One of them,
R.2, Subgroup performance, asks whether safety-critical behaviors such as escalation hold
across demographic subgroups, dialects and literacy levels (Appendix A, p. 26). It does not
name sex or specify a test method.

Our recommendation is that GenAI devices be tested on pairs of cases that are identical except
for the patient's stated sex, in premarket benchmarking and again after model changes. To pass
R.2, a device should show no differences by sex in urgency, referral or psychological framing
beyond a pre-specified tolerance, measured across all case pairs and excluding differences that
are clinically expected. This comment addresses patient sex; the same method can be applied
to other characteristics covered by R.2. Specifically, we recommend that FDA:

1.​ Define R.2 to include sex-counterfactual testing of urgency, referral and framing, with
pre-specified acceptance criteria (Q6, Q9).
2.​ Require qualified human adjudication of sex-based urgency differences, not LLM graders
alone (Q14, Q16).
3.​ Specify sex-disaggregated reporting, including in Foundation Model Device Master Files,
and re-testing after model updates (Q18, Q19, Q22, Q25).
4.​ Recognize a shared sex-counterfactual benchmark with held-out test sets, administered
by qualified independent parties whose results CDRH can consider in review (Q10,
Q16).

Evidence

Studies of LLMs, including preprints, report sex-based differences in model outputs for
otherwise identical cases:

Study Setting Finding
Zack et al., Assessing Clinical vignettes GPT-4 produced differential
the potential of GPT-4 and differential diagnoses and plans that varied
to perpetuate racial diagnosis with patient race and gender;
and gender biases in the authors report that this
health care, Lancet variation reflected demographic
Digital Health (2024), stereotypes.
peer-reviewed

Wong, Identical Younger women received lower
Gender-dependent neurological emergency referral rates than
diagnostic symptoms, three age-matched men (0–6.7% vs
substitution in LLM frontier model 23.3–96.7% across models).
medical triage (2026), families The author attributes this
preprint mainly to models diagnosing
younger women with idiopathic
intracranial hypertension. The
difference disappeared at age
65.

Pan, Evaluating Patient-facing Responses downplayed
differences in GPT-4 health advice urgency for women and
treatment by gender escalated it for men; women
(2025), MIT S.M. were more often directed to
thesis supportive resources and men
to emergency care. LLM
graders diverged from human
annotators on relative urgency.

Luisto et al., Retrieval-augme Differences in treatment
Gendered biases in a nted assistant urgency and symptom
Finnish healthcare AI developed for associations. Female variants
assistant (2026), Finnish public framed around childcare and
preprint wellbeing reproduction regardless of
services; 36 relevance. Bias arose in both
queries in male, generation and retrieval; some
female and patterns recurred across runs
gender-neutral and others appeared
variants inconsistently.

Rickman, Evaluating Summaries of Gemma downplayed women's
gender bias in LLMs gender-swapped needs more often than men's;
in long-term care, real care records Llama 3 showed no
BMC Medical for 617 older gender-based differences.
Informatics and people
 Decision Making
(2025).
Points relevant to the discussion paper:

●​ In the Finnish case, differences arose in the retrieval component as well as in
generation, which testing the underlying model alone would not capture.
●​ The differences concern urgency and framing. A device could meet an aggregate
accuracy threshold while assigning different urgency by sex.
●​ In the diagnostic-substitution study, models rated symptom severity similarly for women
and men, but more often diagnosed younger women with a condition more common in
women of childbearing age and referred them to outpatient care rather than the
emergency room. Sex-specific prevalence can reorder a differential diagnosis; we
recommend that it not lower the urgency indicated by red-flag symptoms.
●​ In a randomized trial in which 50 physicians reviewed GPT-4 advice on chest-pain video
vignettes featuring a white male or Black female patient, accuracy improved with no
increase in race or gender bias (Goh et al., 2025). Patient-facing and autonomous
functions do not include that physician review.

Recommendation 1: Define R.2 to include sex-counterfactual testing (Q6, Q9)

For action-directing and patient-facing devices (Section IV), the same case should be run with
only the patient's stated sex or gender changed, and outputs compared on:

●​ Urgency: triage level or time-to-care.
●​ Referral: emergency, specialist, primary care, or self-care.
●​ Framing: attribution of symptoms to stress, anxiety or psychological causes.
●​ Diagnostic substitution: a sex-linked, lower-urgency diagnosis displacing a
higher-urgency one.

Case sets should include conditions where under-triage of women has been reported, and
symptom descriptions that vary in tone, including emotional or hesitant language.
Outputs need not be identical, since sex-specific epidemiology can change a differential
diagnosis. Sponsors should pre-specify clinically expected differences, with supporting
evidence. A device should pass R.2 only if all other differences in urgency, referral or framing
stay within a pre-specified tolerance. Because some differences appear only in some runs
(Luisto et al.), each case pair should be run repeatedly, with differences reported as rates with
confidence intervals.
This specifies a test method for R.2 rather than adding a new element (Q9). Under- and
over-escalation rates should each be reported by sex as well as in aggregate (Q6).

Recommendation 2: Require qualified human adjudication (Q14, Q16)

The discussion paper states that its independence and qualification expectations for
adjudicators would apply when the adjudicator is an LLM. For sex-based urgency differences,
an LLM should not be the sole adjudicator, because it may show the same differences it is
assessing. In the MIT evaluation, LLM annotations diverged from human assessments on
relative urgency. We recommend:

●​ A clinician panel that includes expertise in the conditions tested.
●​ Each output graded on its own, with patient sex masked where possible, before outputs
are paired.
●​ Patient input into the rubric for the framing measure.
●​ LLM graders allowed for screening, with human review of all flagged pairs and a random
sample of unflagged pairs.

Recommendation 3: Specify sex-disaggregated reporting and re-testing (Q18, Q19, Q22, Q25)

●​ FDA's suggested Master File content includes performance across clinically meaningful
subgroups (footnote 24). It should name sex as one of those subgroups rather than
leaving the choice to the developer. Q25 notes that developers may have limited
incentive to disclose safety-relevant information.
●​ Sex-counterfactual results should be part of the baseline a modified device is
re-benchmarked against, including after foundation model updates, prompt revisions or
new retrieval sources.
●​ If FDA relies more on postmarket monitoring (Q18), real-world escalation and referral
rates should be monitored by sex where sex is recorded, and through sample-based
clinician review (Q19) where it is not.

Recommendation 4: Recognize a shared benchmark with acceptance criteria (Q10, Q16)

A sponsor that builds its own test set also chooses which conditions and phrasings to test, and
public sets are subject to contamination and optimization to the test (Q10). For
sex-counterfactual testing under R.2, we recommend:

●​ A recognized shared benchmark, qualified through the Medical Device Development
Tool (MDDT) program or developed as a consensus standard.
●​ A public set for development and a held-out set used only for regulatory evaluation.
●​ Held-out sets maintained and periodically refreshed by parties structurally independent
of the device sponsor and model developer, the standard the discussion paper applies to
adjudicators.
●​ Cases developed from patient and clinician interviews as well as textbook presentations.
●​ Statistically defined acceptance criteria, such as a maximum urgency gap between
counterfactual pairs, that a device must meet.

Section V.D.2 lists, as a possible third-party role, conducting part of the competency-based
assessment and issuing a determination or certificate that CDRH could consider in its review.
We recommend that sex-counterfactual testing under R.2 be eligible for that role, with more than
one qualified party able to perform it (Q16).
Respectfully submitted,

Marina Meyjes, Head of Research and AI Evaluation, SEENERA, marina@seencertified.org
Endorsed by: Meghan Piccinin, DO, Neurologist, Pittsburgh, PA
References

●​ U.S. FDA, Considerations for the Regulation of Generative AI-Enabled Medical Devices,
discussion paper, docket FDA-2026-N-7874 (Aug. 18, 2026). Full text (PDF).
●​ Wong, Q.H. Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same
Symptoms, Unequal Urgency. arXiv:2606.03641 (2026). Preprint.
●​ Pan, E. Evaluating Differences in GPT-4 Treatment by Gender in Healthcare
Applications. S.M. thesis, Massachusetts Institute of Technology (2025).
●​ Luisto, R., Snell, K., Vartiainen, V., Sanmark, E. & Äyrämö, S. A case report on gendered
biases in a Finnish healthcare AI assistant. medRxiv (2026). Preprint.
●​ Rickman, S. Evaluating gender bias in large language models in long-term care. BMC
Medical Informatics and Decision Making 25(1) (2025).
●​ Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in
health care: a model evaluation study. Lancet Digital Health 6(1), e12–e22 (2024).
●​ Goh, E. et al. Physician clinical decision modification and bias assessment in a
randomized controlled trial of AI assistance. Communications Medicine 5, 59 (2025).