VarunaForge AI (Tarun Bhutani)

VarunaForge AI Limited (Dubai International Financial Centre) develops independent post-market performance monitoring for AI-enabled medical devices, with an initial focus on pathology and laboratory diagnostics. Our comments draw on that work and on discussions with manufacturers and laboratories running AI devices in routine care. Although the paper addresses generative AI, the monitoring principles below apply to AI-enabled devices generally. We comment on Questions 18, 19, 21, 16 and 22.

Question 18 (premarket uncertainty in exchange for postmarket monitoring)
This trade is defensible only when three conditions hold: a usable reference standard exists in routine care or can be obtained through sampled adjudication; volume at each deployment site is sufficient to detect a clinically meaningful change within an acceptable time; and the monitoring plan, including thresholds and actions, is prespecified before deployment. It is unsuitable where volume is low and harm can accrue before a signal emerges, or where the reference arrives long after the output, as with prognostic outputs. We recommend that sponsors state, per site, the smallest performance change their monitoring can detect and the expected time to detect it.

Question 19 (postmarket approaches, cadence and triggers)

Monitor at the level where drift occurs. Performance should be tracked by site, instrument or scanner, software version and user group. Pooled averages can mask a single degrading site.
Prespecify statistical rules with a known false-alarm rate that remain valid under repeated looks. Monitoring that alarms too often is ignored; monitoring that never alarms provides no assurance.
Report the minimum detectable change for each review period. The absence of a signal is meaningful only relative to what the data could have detected.
Combine reference-based metrics with earlier, reference-free signals: the distribution of device outputs against baseline frequencies, and agreement between the device output and the final clinical determination. Comparing these helps separate a change in case mix from a change in the device.
Use event-triggered reassessment alongside a calendar. A change log of the deployment environment (reagents and pre-analytical steps, equipment maintenance, software and model versions, workflow changes) should trigger targeted re-benchmarking, using known-answer control material where feasible.
Keep quality-control material separate from clinical cases in the analysis.
Set cadence in proportion to volume and risk.

Question 21 (roles of clinicians, institutions, societies and standards bodies)
Deploying institutions hold the reference data and know about local changes. Their quality systems, for example in accredited laboratories, can route signals through existing corrective-action and change-control processes. The manufacturer remains accountable for the device and the institution for its use. Independent analysis adds a consistent cross-site view that neither party can produce alone: a manufacturer sees its own device, and an institution sees its own site. Standards bodies could define a minimal common dataset for post-market performance reporting (device and version, site, instrument, output, reference result, timestamp) so that results are comparable across deployments.

Question 16 (independent third parties)
Independent third parties are as valuable after deployment as before it. Suggested criteria: no financial interest in the device’s commercial performance; analysis plans locked before data are reviewed; a traceable audit trail from case-level data to each reported result; and data minimisation, since performance monitoring does not require images or direct identifiers. Decisions and corrective actions remain with the manufacturer.

Question 22 (re-benchmarking scaled to modifications)
Any modification, including updates to third-party foundation models, should re-baseline post-market monitoring. Baselines should be versioned, and monitoring should segment results by model version so that post-change performance is compared directly with pre-change performance.

We would welcome the opportunity to provide further detail.

Tarun Bhutani
Founder & CEO, VarunaForge AI Limited