Oluwamodupe Akintan
The comment as filed
Thank you for the opportunity to provide feedback on the discussion paper. I am submitting these comments in my individual capacity. The views expressed are my own and do not represent my employer or any organization with which I am affiliated.
A few points stood out to me from a privacy and AI engineering perspective:
1. Consider data sensitivity as part of the risk assessment.
The proposed framework focuses on what the device does and the consequence of an incorrect output. I think the sensitivity of the data being accessed or processed should also be considered as a risk modifier. A device can produce a clinically correct output while still creating harm through unnecessary access, use, or disclosure of sensitive patient information.
2. Include data-access boundaries when evaluating agentic AI.
For agentic systems, testing should consider not only whether the system completes a task correctly, but whether it accesses only the data and tools needed for that task. This could include testing permission boundaries, attempts to access unrelated patient information, human approval before sensitive actions, and auditability of actions taken.
3. Build privacy safeguards into postmarket monitoring.
Postmarket monitoring may involve collecting and reviewing large amounts of real-world patient interaction data. I suggest explicitly considering data minimization, retention limits, access controls, and de-identification where appropriate. Monitoring should collect enough information to identify safety and performance issues without automatically retaining every patient interaction.
4. Include privacy-related changes in oversight of third-party foundation models.
For devices that rely on third-party foundation models, material changes should not be limited to model performance. Changes to data retention, logging, training or fine-tuning practices, tool permissions, or other data-handling behavior could also change the risk profile of the device. These types of changes could be included in change-notification expectations and, where appropriate, Foundation Model Master Files.
Thank you for considering these comments.