Tatiana de A. Furquim (Correia)

See attached file(s)

My background is software quality and cybersecurity risk management across several
industries, working with medical devices for the last 10 years.
I support a risk-based, total product lifecycle direction. What I couldn’t find in the paper
is how the competency-based approach relates to what manufacturers already do. We
have IEC 62304 processes, ISO 14971 risk files, Secure Development Practices,
SBOM and vulnerability monitoring, V&V, config management, change control. Which of
these carry over unchanged for GenAI? Which need rework? And what’s actually new?
My guess is the new parts are mostly data, output variability, model behavior, and
dependence on a third-party model vendor. A mapping table would go a long way.
Reviewers would benefit as much as manufacturers.
The two-axis framework is fine as a starting point. My problem with it is that it treats
autonomy as a proxy for risk. I’ve seen tightly bounded automatic functions that were far
safer than “advisory” features people stopped double-checking after a week. What
matters is whether someone can catch the error, how long they have to react, whether
the action can be reversed, and whether the safeguards have been shown to work (not
just designed).
Same issue with human oversight. Calling the clinician the final decision-maker doesn’t
prove much. Can that clinician, with the screen in front of them and the time they really
have on a shift, tell the output is wrong? If human review is listed as a risk control, it
needs to be tested and monitored like a risk control.
Evaluating the deployed configuration is useful but benchmarks and clinical confirmation
need to connect back to verification of specific safety requirements to be meaningful. I
believe acceptance criteria need to be set per clinically significant failure mode and the
practices of keeping traceability from risk to test should apply here as well.
The reference standard question deserves its own section. Open-ended outputs often
don’t have one right answer. Domain experts (real ones, matched to the specialty)
should define what’s acceptable. Ultimately, two models agreeing is not evidence of
anything clinical.
Cybersecurity needs to be part of the safety evaluation, not a parallel workstream.
Prompt injection, poisoned retrieval content, a compromised tool response: any of these
can change clinical behavior in a model that benchmarks well. Cybersecurity specific
requirements need to be defined and verified prior to final pen testing. For agentic
functions tests should include tool permissions, action sequences, interrupt handling,
and recovery from partial failure. If the agent drives another device, follow it all the way
to what that device physically does to the patient.
I’d push back on trading premarket evidence for postmarket monitoring. That only works
if a failure gets caught before people are hurt. Quarterly review can surface a problem
after hundreds of patients have been exposed. For anything that can cause immediate,
irreversible harm, monitoring is not a substitute. Supervisory agents: if the supervisor
shares the model or the same bad inputs, it will likely make the same mistake.
Change control. The “device” is the model plus prompts, retrieval sources, guardrails,
tools, orchestration. Change any one and safety can shift while the model hash stays
identical. For third-party models the manufacturer has to know which version is live, get
notice before it changes, assess the impact, and block anything unassessed from
reaching patients. If the vendor can’t support that, I’d argue that alone weighs against
using the model.
PCCPs could work within defined boundaries. My concern is that rerunning the original
benchmarks won’t catch a failure mode those benchmarks were never built to find. FDA
should say how a manufacturer decides when the old evidence still applies and when it
doesn’t.