Kaitlin Sundling

As a pathologist, physician scientist, and researcher with expertise in machine learning, I oppose the use of currently commercially available generative AI (gen AI) and large language models (LLM) platforms in medical devices. Gen AI and LLMs carry unpredictable risks, and current models are known to contain potentially harmful hallucinations and preoccupations. Please see the attached document for additional information and references.

FDA Docket: FDA-2026-N-7874

October 19, 2026

As a pathologist, physician scientist, and researcher with expertise in machine learning, 1–3 I
oppose the use of currently commercially available generative AI (gen AI) and large
language models (LLM) platforms in medical devices. Gen AI and LLMs carry unpredictable
risks, and current models are known to contain potentially harmful hallucinations and
preoccupations.4,5

Unlike purpose-built medical machine learning algorithms that can be trained on highquality medical data, most commercially available gen AI and LLM platforms today are
based on internet-scraped training data, which is now well-known to contain information
that is false, harmful, and biased.6,7

Without reliable training data and reliable performance from gen AI algorithms, it is
currently not possible to establish meaningful validation standards to guide clinical
implementation. Existing guidelines for AI implementation are insu icient, as they do not
encompass the unpredictable quality, quantity, and type of errors that may occur with gen
AI systems. If validation standards are at some point established, re-validation for
implementation of updated gen AI models will almost certainly be necessary.

Another issue that must be addressed in gen AI integrated systems is automation bias. In
the context of laboratory diagnostics, the College of American Pathologists has identified
the need for pathologist oversight of AI algorithms that impact laboratory diagnosis. 8
However, it is unclear how pathologist vigilance will be maintained at a systems level amid
the significant risks of automation bias.9 Gen AI algorithms may incorporate unexpected
biases that would be di icult for human oversight to detect with routine use.

Finally, gen AI-integrated medical devices post significant risks to both patients and
physicians. It is unclear who will be accountable when gen AI systems cause errors, and
many gen AI systems already do not maintain documentation to determine how an error
occurred. Physicians already do not understand that they are accountable for errors made
by gen AI tools integrated into electronic health records and in other medical contexts. In
pathology, we need only look back to the 1980s to see evidence of what harm can be done
to patients and to patient trust in the healthcare systems when there is a lack of
accountability and poor-quality practices are allowed to proceed unchecked. 10

Please protect patients from preventable harm and stop the use of generative AI in medical
devices until long-term, reliable performance standards and validation guidelines are
established by experts. At that point, regulations that incorporate input from all
communities of interest should be established to protect patient safety and public health.
A discussion paper is a reasonable starting place; however, given the significant impacts
and errors we are already seeing with gen AI use in healthcare, 11 I strongly recommend that
the FDA facilitate open and transparent discussion among experts with opportunities for
questions and comments from the public on gen AI-integrated medical devices.

Kaitlin Sundling, MD, PhD

References

1. Rydzewski NR, Shi Y, Li C, et al. A platform-independent AI tumor lineage and site
(ATLAS) classifier. Commun Biol. 2024;7(1):314. doi:10.1038/s42003-024-05981-5

2. Sundling RJ, Sundling KE. Implications of Generative Artificial Intelligence for
Cytopathology Education. The Communicator, American Society of Cytopathology,
Cytopathology Program Directors Committee Newsletter. February 2024.
https://cdn.ymaws.com/cytopathology.org/resource/resmgr/fellowship_program/the_c
ommunicator/2024_comm/the_communicator_february_20.pdf

3. Kim D, Sundling KE, Virk R, et al. Digital Cytology Part 2: Artificial Intelligence in
Cytology A Concept Paper with Review and Recommendations from the American
Society of Cytopathology Digital Cytology Task Force. J Am Soc Cytopathol. Published
online December 2023:S2213294523002466. doi:10.1016/j.jasc.2023.11.005

4. Where the goblins came from. OpenAI. April 23, 2026. Accessed May 4, 2026.
https://openai.com/index/where-the-goblins-came-from/

5. Wile R. OpenAI blames “nerdy personality” for ChatGPT’s obsession with goblins. NBC
News. April 30, 2026. Accessed May 4, 2026. https://www.nbcnews.com/tech/technews/openai-chatgpt-goblin-nerdy-personality-rcna342855

6. Niemeyer K. Reddit comments are “foundational” to training AI models, COO says.
Business Insider. Accessed October 19, 2026.
https://www.businessinsider.com/reddit-comments-ai-training-models-googleopenai-jen-wong-hu man-2025-1

7. Stackpole B. Bringing transparency to the data used to train artificial intelligence | MIT
Sloan. March 3, 2025. Accessed October 19, 2026. https://mitsloan.mit.edu/ideasmade-to-matter/bringing-transparency-to-data-used-to-train-artificial-intelligence

8. College of American Pathologists. Statement for the Record U.S. House of
Representatives Committee on Education and the Workforce Subcommittee on
Health, Employment, Labor, and Pensions; Building an AI-Ready America: Adopting AI
at Work. Published online February 3, 2026.
9. Hanna MG, Pantanowitz L, Jackson B, et al. Ethical and Bias Considerations in Artificial
Intelligence/Machine Learning. Mod Pathol. 2025;38(3).
doi:10.1016/j.modpat.2024.100686

10. Bogdanich W. Lax Laboratories: The Pap Test Misses Much Cervical Cancer Through
Labs’ Errors --- Cut-Rate “Pap Mills” Process Slides Using Screeners With Incentives to
Rush --- Misplaced Sense of Security? Wall Str J. November 2, 1987. Accessed
October 19, 2018.
http://search.proquest.com.ezproxy.library.wisc.edu/docview/135230319?accountid=
11. Burke G, Schellmann H, Phan K, Wenig S. Researchers say an AI-powered transcription
tool used in hospitals invents things no one said. The Associated Press. November 1,
2024. Accessed October 19, 2026. https://www.ap.org/news-highlights/best-of-theweek/honorable-mention/2024/researchers-say-an-ai-powered-transcription-toolused-in-hospitals-invents-things-no-one-said/