Persistence Analytics Group (Neil P. Osnato)

Persistence Analytics Group LLC submits the attached supplemental comment in Docket FDA-2026-N-7874 regarding the regulation of generative AI-enabled medical devices.
This submission supplements PAG’s August 24, 2026 comment and focuses on Questions 19, 22, 24, and 26.
PAG recommends that FDA distinguish among:
Baseline variability → Change detection → Decision materiality → Revalidation
Repeat-run variability should be measured for safety-critical behaviors where stochasticity can affect interpretation of benchmark or postmarket results. However, a measured reproducibility floor should not itself become an acceptable-performance threshold.
A reproducibility floor can establish what change an evaluation is capable of detecting. It does not establish what level of instability is acceptable for continued reliance.
PAG also recommends that third-party foundation-model changes be treated as changes to a critical dependency requiring a defined evidence-revalidation process where material, rather than allowing prior evidence to carry forward automatically.
For agentic GenAI-enabled medical devices, FDA should evaluate consequential actions at the action level, including intended scope, applicable human-oversight conditions, tool execution, dependency state, and whether material changes should have required renewed review before the action proceeded.
PAG is not proposing a universal numerical threshold for acceptable variability, nor suggesting that every model update or deviation requires renewed FDA authorization. The narrower recommendation is that evidence supporting continued reliance be matched to the consequence of the decision and revalidated when material conditions change.
Premarket authorization establishes an initial reliance boundary. Continued reliance should depend on whether the evidence supporting that boundary remains sufficient after material change.
Please see the attached supplemental comment for the complete discussion and references.
Neil P. Osnato
Founder
Persistence Analytics Group LLC

Persistence Analytics Group LLC | Supplemental Comment | FDA-2026-N-7874
U.S. FOOD AND DRUG ADMINISTRATION
Docket No. FDA-2026-N-7874
SUPPLEMENTAL COMMENT OF
PERSISTENCE ANALYTICS GROUP LLC
Baseline Variability, Material Change, and Revalidation
of Generative AI-Enabled Medical Devices
Submitted October 8, 2026
____________________________________________________________________________________________

Persistence Analytics Group LLC (PAG) submits this supplemental comment in response to the Food and Drug
Administration's August 2026 discussion paper on generative AI-enabled medical devices. PAG previously
submitted a comment on August 24, 2026 (FDA-2026-N-7874-0029) addressing initial authorization, continued
reliance, third-party dependencies, agentic action, and material-change revalidation.
This supplemental comment focuses on Questions 19, 22, 24, and 26. It addresses a narrower issue that has
become clearer as empirical work on clinical language-model agents has developed: repeat-run variability
should be measured and incorporated into postmarket evaluation, but a measured variability floor should not
itself become an acceptable-performance threshold.
FDA's discussion paper already recognizes reproducibility as part of generalizability, contemplates periodic rebenchmarking and performance-degradation monitoring, asks how reassessment should be triggered after
modifications, and identifies additional concerns for third-party foundation models and agentic systems.[1]
PAG recommends that FDA connect those concepts through an explicit distinction between baseline variability,
change detection, decision materiality, and revalidation.
I. A REPRODUCIBILITY FLOOR IS A MEASUREMENT BASELINE, NOT AN
ACCEPTABILITY THRESHOLD
Recent preprint research using the FairMedAgent evaluation harness illustrates why the distinction matters.
Holding a clinical scenario and descriptor constant and repeating the same condition ten times produced action
changes in 8.7% of replicate comparisons for one tested model, with action-specific instability ranging from
2.2% to 17.9%. A second model also exhibited a measurable pooled floor. The authors appropriately caution
that a counterfactual difference cannot be interpreted without first measuring how often the agent changes its
action when nothing has changed.[2]
PAG does not suggest that these particular rates are generalizable to other devices, models, clinical tasks, or
deployment settings. The regulatory significance is methodological: a sponsor evaluating a variable GenAIenabled system should know the repeat-run variability of the safety-critical behavior being measured before
using an observed difference as evidence of drift, disparity, degradation, or material change.
A reproducibility floor can establish what change an evaluation is capable of detecting. It does
not establish what level of instability is acceptable for continued reliance.
The first concept is measurement capability. The second is a safety-and-effectiveness judgment. Conflating
them could turn inherent variability into an implicit safe harbor even where the affected action is clinically
consequential.
II. FDA SHOULD SEPARATE BASELINE VARIABILITY, CHANGE DETECTION, AND
DECISION MATERIALITY
For Questions 19 and 22, PAG recommends that postmarket reassessment distinguish three separate inquiries:
 Baseline variability: what variation occurs when the relevant inputs and operating conditions are
intentionally held constant?
 Change detection: has observed behavior moved beyond the expected baseline or otherwise changed in a
way the monitoring method can reliably detect?
 Decision materiality: is the detected change relevant to the device's safety, effectiveness, intended use,
reliance conditions, or authorized action scope?
 Persistence Analytics Group LLC | Supplemental Comment | FDA-2026-N-7874
Those inquiries can be organized as a lifecycle:
Baseline Evidence -> Expected Variability -> Dependency -> Material Change -> Observed
Deviation -> Decision Materiality -> Revalidation
A change should not require the same response merely because it is detectable. Conversely, a change should not
be disregarded merely because some repeat-run variability was present at baseline. The appropriate regulatory
or quality-system response should depend on the nature of the changed condition, the affected clinical behavior,
the device's risk, and the consequence of continued reliance.
For higher-consequence GenAI-enabled functions, sponsors should pre-specify categories of events that require
reassessment. Depending on the device, relevant triggers may include a foundation-model version change;
changes to prompts, retrieval sources, tool interfaces, orchestration logic, guardrails, or human-oversight
checkpoints; changes in the deployment population or setting; or observed degradation in a safety-critical
action. The trigger should initiate a defined evidentiary response rather than merely create a log entry.
III. THIRD-PARTY FOUNDATION-MODEL CHANGES SHOULD TRIGGER EVIDENCE
REVALIDATION
Question 24 asks how manufacturers can detect, evaluate, and respond when a third-party foundation-model
developer changes an underlying model. PAG recommends treating the third-party model as a critical
dependency whose version, update state, and relevant safety behavior are linked to the evidence supporting
continued reliance.
Where an underlying model changes, the manufacturer should be able to determine whether the evidence that
supported the prior device state remains sufficient. That may include re-running action-specific tests, reestablishing the relevant variability baseline, checking tool-use and refusal behavior, and confirming that
previously validated safety-critical behaviors remain within the device's reliance boundary.
This approach need not imply that every upstream update requires a new FDA submission. The narrower point
is that an upstream change should not be treated as operationally immaterial merely because it was initiated
outside the device manufacturer's control. The manufacturer remains responsible for determining what evidence
is necessary for continued safe and effective use.
IV. AGENTIC DEVICES REQUIRE ACTION-LEVEL ASSURANCE, NOT ONLY OUTPUTLEVEL PERFORMANCE
Question 26 appropriately asks how autonomous multi-step action, tool use, and reduced opportunity for human
review should affect acceptance criteria and oversight. For agentic devices, PAG recommends evaluating the
specific action state in addition to overall model performance.
Capability is not authority. Successful execution is not, by itself, evidence that the action was
authorized within the intended scope and under the conditions then present.
For a device that can cause or initiate a consequential action, evaluation should ask whether the action was
within the device's intended use and current action scope; whether required human checkpoints were satisfied;
whether the tool or downstream system actually executed the intended action; and whether any material change
in the model, tool, policy, data source, user context, or other dependency should have required renewed review
before the action proceeded.
This is especially important where a benchmark can report a satisfactory overall result while materially
different actions occur across repeated runs. Postmarket monitoring should therefore preserve enough versionlinked, action-level evidence to reconstruct what the system attempted, what actually occurred, and which
authorization or oversight condition applied at the time.
V. RECOMMENDED PRINCIPLES
PAG recommends that FDA consider the following principles as it develops its approach:
 Characterize repeat-run variability for safety-critical behaviors when stochasticity can affect the
interpretation of benchmark or postmarket results.
 Treat the measured variability floor as a detection baseline, not as an automatic threshold of acceptable
clinical instability.
 Persistence Analytics Group LLC | Supplemental Comment | FDA-2026-N-7874
 Combine periodic reassessment with event-triggered reassessment tied to pre-specified categories of
material change.
 For third-party foundation models, link upstream model changes to a defined evidence-revalidation process
rather than relying on the prior device state by default.
 For agentic devices, evaluate consequential actions at the action level, including scope, human-oversight
conditions, tool execution, and relevant dependency state.
 Keep manufacturer accountability intact even where monitoring, benchmarking, adjudication, or other
evidence functions involve independent third parties.
PAG is not proposing a universal numerical threshold for acceptable variability, nor suggesting that every
model update or observed deviation requires renewed FDA authorization. PAG also does not disclose in this
public comment its proprietary scoring, weighting, materiality thresholds, durability analytics, or determination
methods. The narrower recommendation is that the evidence supporting continued reliance be matched to the
consequence of the decision and be revalidated when material conditions change.
CONCLUSION
FDA's discussion paper correctly treats GenAI-enabled medical devices as a total-product-lifecycle problem.
The next step is to make explicit that repeat-run variability, change detection, and acceptable continued reliance
are different questions.
Premarket authorization establishes an initial reliance boundary. Continued reliance should
depend on whether the evidence supporting that boundary remains sufficient after material
change.
A durable framework should therefore ask not only whether a GenAI-enabled device performed adequately
when first evaluated, but also whether a later observed change is distinguishable from expected variability,
whether the change is decision-material, and what evidence is required before reliance continues.
REFERENCES
[1] U.S. Food and Drug Administration, Considerations for the Regulation of Generative AI-Enabled Medical Devices:
Discussion Paper and Request for Feedback (Aug. 2026), including Questions 19, 22, 24, and 26.
[2] Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar & Rahul Joshi, Counterfactual Fairness Audits of MultiStep Clinical LLM Agents Require a Measured Per-Action Instability Floor, arXiv:2609.03221 (2026).
[3] U.S. Food and Drug Administration, Methods and Tools for Effective Postmarket Monitoring of Artificial Intelligence
(AI)-Enabled Medical Devices (FDA regulatory science program).

Respectfully submitted,
Neil P. Osnato | Founder, Persistence Analytics Group LLC
Decision Assurance™ | Infrastructure Assumption Verification™
Evidence before reliance. Revalidation after material change.
neil@persistenceanalyticsgroup.com | 609-464-9055 | persistenceanalyticsgroup.com