Humza Nusrat
Themes it raises
The comment as filed
I am a clinical medical physicist at Henry Ford Health, and part of my work involves AI agents for radiation therapy planning and plan review. These comments are my own and do not represent my institution.
I support FDA’s move toward competency-based evaluation of generative and agentic devices. I would ask the agency to address one problem directly: agreement between two AI systems is not the same as independent verification.
More and more, these devices are being checked by other models, such as AI second checks, LLM-based graders, and test suites written by AI. When the device and its checker share training data, assumptions, or a model family, they tend to miss the same cases. The result is high agreement that looks like validation but hides the errors that matter most. Radiation oncology learned this lesson years ago, which is why an independent check of the treatment dose is standard practice (AAPM TG-219).
For generative and agentic devices, I recommend that submissions:
1. Explain how verification was kept independent of the device, including what does the checking, whether the checker shares data or models with the device, and whether it sees only the final output or the device’s reasoning as well.
2. Report performance on cases designed to expose shortcuts and attempts to game the evaluation, not only agreement with clinicians or reference labels. This matters most for systems trained or tuned against an automated score.
3. Not count a device’s own reasoning or self-reported success as evidence that it behaved correctly, unless that reasoning has been tested against independent ground truth.
4. Disclose any AI-generated software and explain how its tests were written independently of the model that wrote the code.
5. Apply the same standard to postmarket monitoring, so that steady agreement between a device and its monitor is not mistaken for safety.
Where physical ground truth exists, such as absolute dose in radiation therapy, it should take priority over checks based on other models or on consensus.
Thank you for considering these comments.
Humza Nusrat, PhD, MCCPM