Sihem Khelifa
“A pathologist cannot meaningfully verify an output when the model’s training data, internal behavior, limitations, or relevant changes are not sufficiently transparent.”
What they argued
Pathologist accountability: manufacturers must own foundation-model change detection and re-benchmarking, not labs; no position on autonomy, evidence tiers, or PCCP.
Themes it raises
FDA questions it names
Q19 · Postmarket performance evaluationQ21 · Clinicians, institutions and societiesQ22 · Re-benchmarking after a modificationQ24 · Third-party foundation model changes
Across the five cross-cutting questions
High-consequence work: Not stated
The comment as filed
FDA should clarify whether pathologists are expected to assume diagnostic responsibility for outputs generated by foundation-model-enabled devices that cannot be independently reviewed, explained, or quality controlled at the laboratory level. Accountability for model validity, model drift, and foundation-model changes should remain with the parties that develop, control, and maintain the model.
FDA should also clearly distinguish manufacturer obligations for re-benchmarking, performance monitoring, and evaluation of foundation-model changes from laboratory quality-control obligations, to avoid transferring a continuous surveillance burden to pathology practices.
Attachment
Pathologist Accountability and Quality Control of
Black-Box Foundation Models
Comments on FDA, Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback
1. Responsibility for Results Generated by Black-Box Models
CONCERN
Pathologists may be expected to sign out and assume professional responsibility for clinically
significant results generated by foundation-model-enabled devices even when the basis for the output
cannot be independently reviewed, explained, or quality-controlled at the laboratory level. A
pathologist cannot meaningfully verify an output when the model's training data, internal behavior,
limitations, or relevant changes are not sufficiently transparent.
RECOMMENDATION
FDA should clearly define accountability for the final reported result. The device manufacturer should
remain accountable for the validity, reliability, traceability, declared limitations, and change control of
the finished device output. Pathologists should remain responsible for appropriate clinical
interpretation and use, but should not be required to assume sole responsibility for an output that
cannot be meaningfully reviewed or quality-controlled. Sufficient evidence, traceability, uncertainty
information, and safeguards should be available before a pathologist is expected to sign out such a
result.
2. Ongoing Quality Control and Laboratory Workload
CONCERN
Periodic re-benchmarking, sample-based clinician review, performance-degradation monitoring, and
reassessment after changes to a third-party foundation model could shift substantial surveillance,
documentation, and verification work to pathology laboratories. Without clear boundaries, these
obligations may increase pathologist workload rather than deliver the intended efficiency benefit.
RECOMMENDATION
FDA should clearly distinguish manufacturer-level model surveillance from laboratory-level
operational quality control. Manufacturers should be responsible for detecting foundation-model
changes, evaluating their impact, re-benchmarking the finished device, monitoring performance
degradation, and communicating required actions to laboratories. Laboratory quality control should
be risk-based, feasible, and limited to confirming continued performance within the local pathology
workflow. The expected staffing, documentation, review, and sign-out burden should be evaluated
when assessing the real-world benefits and risks of these devices.
Relevant FDA discussion questions: Questions 19, 21, 22, and 24.
Comments on FDA GenAI-Enabled Medical Devices Discussion Paper Page 1