Binesh Sadanandan

I submit this comment in my individual capacity, drawing on my doctoral research at the University of New Haven and papers coauthored with colleagues. We studied medical vision-language models, which take an image and a written question and generate an answer. Our experiments used chest X-ray datasets and did not test patient outcomes in clinical care.

I support the proposed approach outlined by the Food and Drug Administration: benchmark the final device, then follow with clinical confirmation. I recommend making dependence on patient-specific evidence an explicit part of this evaluation.

Questions 9 and 17: Use of clinical evidence
Appendix A, E.2 and R.1, printed pp. 25-26

R.1 already includes equivalent paraphrases and missing inputs. These tests should also establish whether the relevant clinical input affects the response. A model can give stable answers to reworded questions while relying more on question text.

In the later dissertation evaluation, additional training reduced answer changes across equivalent wording from 8.5% to a mean of 3.5% across five training runs. The test contained 238 questions and 991 pairs from 236 patients, with no patient or image overlap with the additional training. A separate audit of the same trained versions used 241 questions. Agreement between answers with and without the image rose from 53.5% to a mean of 76.8%. The model still produced a response when the image was removed. These findings did not show improved image dependence.

An unchanged answer alone cannot establish how the image was processed. I recommend clinically reviewed controls that remove necessary inputs or change relevant evidence, with expected responses specified in advance. Missing evidence may call for a request for more information or deferral. Changes to irrelevant image content should not change the clinical answer. These checks can use the device interface without requiring access to model weights.

Questions 10 and 12: Benchmark validity and sampling
Section V.B.2, printed pp. 13-14

Our benchmark study included clinical review of whether reworded questions retained their meaning. Negation or a request to exclude a diagnosis can change the task. Sponsors should document this review, test previously unused wording, and distinguish rewording from repeated runs of an identical prompt.

Repeated questions from one patient are not independent patient observations. Report the number of patients and cases, with uncertainty estimates that account for repeated measurements. Keep results from deliberately difficult test sets separate from estimates for the intended clinical population unless pooling is justified. Generated variants do not add independent patient evidence.

Questions 9, 11 and 14: Deferral
Appendix A, S.3, printed p. 24; Section V.C, pp. 14-16

Report how often the device answers, the errors among those answers, and what happens to deferred cases. Our research on uncertainty screening supports evaluating error rates together with the fraction of cases answered. A retrospective rule may produce high accuracy on retained cases yet depend on expert annotations unavailable during use.

Clinical confirmation should test the actual handoff to a clinician, including delays and review workload. Report clinically serious errors and whether some patient groups face more deferrals. Added model calls and human review have costs. A limited set of offline checks followed by review of failures may reduce this burden.

Questions 18, 19, 22 and 24: Changes and monitoring
Section VI, printed pp. 19-22

Retesting should cover the clinical behavior affected by changes to the model, prompts, retrieved information, image processing, or deferral rules. Preserve test cases for comparisons across versions and add previously unused cases. An improvement in consistency should not excuse worse use of clinical evidence.

Monitoring may support greater premarket uncertainty when harm is limited or reversible, failures can be detected promptly, and a tested response is available. It offers insufficient protection where a serious error can cause harm before detection. The scope of testing should follow intended use and risk, with acceptance criteria justified for that device.

Attachment1 contains the full comment, study limits, and citations to our papers on question wording, consistency training, image dependence, and uncertainty screening. Attachment2 contains the cited dissertation pages, including Tables 5.6 and 5.8 and Section 7.2. The numerical results above come from the later dissertation evaluation.

Binesh Sadanandan | FDA-2026-N-7874

Public comment on evaluation of generative AI medical devices
Docket FDA-2026-N-7874
Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and
Request for Feedback
Submitted by Binesh Sadanandan | October 2026

I support the proposed approach of benchmarking the final device followed by clinical confirmation.
I ask FDA to make dependence on patient-specific evidence an explicit part of that evaluation. A
model can become more consistent under rewording without making better use of the clinical input.
Tests of consistency should therefore be accompanied by tests of whether the relevant evidence
affects the response.
This comment draws on my doctoral research at the University of New Haven on medical visionlanguage models [2]. These models take an image and a written question as input and generate an
answer. The study tested yes-or-no questions about findings in chest X-rays. It examined whether
answers changed when a question was reworded, and whether they depended on the image supplied.
The experiments used existing datasets and did not test patient outcomes in clinical care. I submit this
comment in my individual capacity.
References to FDA page numbers below use the printed pages of the discussion paper [1]. Selected
dissertation pages are supplied in Attachment2.

Dependence on the clinical input
Questions 9 and 17; Appendix A, E.2, p. 25, and R.1, pp. 25-26
R.1 already includes repeated runs, equivalent paraphrases, and missing or contradictory inputs. I
recommend making the relationship between these tests explicit for devices that combine text with
images or other patient data. The evaluation should ask whether the response remains stable when the
clinical meaning is unchanged, and whether it responds appropriately when the relevant evidence
changes.
Chapter 5 compared MedGemma-4B, a medical image-and-text model, before and after additional
training. The model could already answer questions about images. The added training aimed to
reduce answer changes across equivalent versions of a question while retaining answer accuracy. It
changed only a small part of the model. We described this training approach in an earlier paper [3].
The figures below come from the later dissertation evaluation, which excluded patients and images
used in the additional training.
The wording test contained 238 questions and 991 pairs of equivalent questions from 236 patients. A
separate image audit used the same trained versions and test split, with 241 questions. Its three
additional questions had no evaluable pair of equivalent questions. The results show why both tests
are needed [2, Tables 5.6 and 5.8, pp. 145-146 and 151-153].
 Binesh Sadanandan | FDA-2026-N-7874

Before additional After additional
Measure and test set
training training

Different answers within a paraphrase pair
8.5% 3.5% ± 0.6
991 pairs from 238 questions

Same answer after image removal
53.5% 76.8% ± 3.1
241 questions

Answer changes after image replacement
22.0% 19.9% ± 1.2
241 questions
The additional training was repeated five times with different random seeds. Results after training are means ±
standard deviations across those runs; deviations are in percentage points. They describe variation between training
runs, not uncertainty in population performance.

In our earlier work, we assessed consistency and image dependence together [4]. The later
dissertation audit found that additional training reduced answer changes under equivalent wording,
but did not show improved image dependence. The text-only test compares the model response with
and without the image. A response remains available in the text-only condition, and the test records
whether it changes. Agreement on one yes-or-no answer cannot establish how the image was
processed. In this audit, replacement also used another test image rather than a matched image with
an opposite finding. The results support further controlled testing, not a claim that every unchanged
response ignores the image.
For device evaluation, I recommend using clinically reviewed controls with known expected
responses. These could include removal of a necessary input, matched changes in the finding under
study, and changes to irrelevant image regions. A clinically meaningful change should produce the
appropriate response; an irrelevant change should not. When necessary evidence is missing,
requesting that evidence or deferring may be the correct response. Such tests can be conducted
through the device interface without requiring access to model weights. This approach also addresses
the broader problem of models learning shortcuts that succeed on a benchmark and fail under
changed conditions [5].
A possible addition to E.2 is: "For devices whose intended use depends on patient-specific input,
evaluation should include controlled tests of whether the device uses the relevant input appropriately.
Sponsors should justify the expected response to removal or clinically meaningful alteration of that
input, and report these results alongside performance under equivalent wording."

Benchmark design and the unit of analysis
Questions 10 and 12; Section V.B.2, pp. 13-14
In our benchmark study, we compared answers to reworded questions about the same image and
checked question equivalence with clinical reviewers [6]. Benchmark construction should preserve
the clinical meaning of each question. Negation, uncertainty, and requests to exclude a diagnosis can
change the task. A change in answer is not necessarily an error when the question changes in one of
these respects. Sponsors should document how equivalence was assessed and retain the original case,
its variants, the expected response, and the clinical adjudication. A test set should include wording
that was not used to train the model or choose its prompts. Repeated runs of an identical prompt
should be reported separately from variation caused by rewording.
 Binesh Sadanandan | FDA-2026-N-7874

The number of generated variants should not be treated as the number of independent clinical
observations. In the primary wording test, 991 pairs came from 238 questions and 236 patients. FDA
should ask sponsors to report each denominator and estimate uncertainty at a level that respects this
dependence, ordinarily the patient or another justified sampling unit. Variation between repeated
training runs answers a different question. The reported standard deviations in my dissertation do not
provide patient-level confidence intervals.
Balanced or deliberately difficult test sets can expose failures that a prevalence-weighted average
may conceal. Clinical confirmation must also represent the intended population and workflow. I
recommend reporting these estimates separately unless the sponsor justifies a common target
population and an appropriate weighting method. Synthetic cases may extend a stress test, but
generating more variants does not add independent patient evidence. A fixed regression set can
support comparisons across versions; a separate set of previously unused cases is needed to assess
repeated optimization to that test.

Deferral and the cases that remain
Questions 9, 11, and 14; Appendix A, S.3, p. 24, and Section V.C, pp. 14-16
S.3 appropriately includes both failure to defer and excessive deferral. Evaluation should report how
often the device provides an answer, the errors among those answers, and what happens to the
deferred cases. Where a device supplies numerical confidence, that confidence should be checked
against observed correctness. Earlier work on neural-network calibration shows why a confidence
score cannot be assumed to be calibrated [7]. A verbal statement of certainty likewise needs
evaluation in the intended clinical task.
In a related study, we used uncertainty in the probabilities assigned to possible answers to screen for
both errors and answer changes under rewording [8]. This score can be computed from a single
model run. Its use as a screening measure still needs validation for the intended clinical task.
Chapter 7 illustrates the trade-off. It tested a rule that kept a prediction only when answers agreed
across reworded questions and passed a check for image dependence. In an audit of 861 questions
from the PadChest chest X-ray dataset, this rule retained 33.0% of predictions from a model that had
received additional training. Accuracy was 96.8% among retained predictions, compared with 91.5%
across all predictions [2, Section 7.2 and Table 7.5, pp. 214-217]. This experiment used an earlier
trained version and a separate evaluation cohort. Its results should be read separately from the
Chapter 5 comparison above. The rule also used image regions marked by radiologists for 637
questions, information that would not generally be available for an undiagnosed patient. Removing
the region-based check reduced the retained fraction to 26.9%. The same trained version showed no
accuracy gain from the rule on a second chest X-ray dataset.
These findings support reporting the fraction of cases answered alongside their error rate, with
clinically serious errors distinguished from minor errors. Any threshold should be fixed before
evaluation on an independent test set. Performance should also be examined in groups likely to
receive different rates of deferral. A favorable average among answered cases can coexist with
delayed care or added workload for everyone else.
Clinical confirmation should test the actual handoff to a clinician, including whether the clinician
receives enough information, whether review occurs in time, and how the combined workflow
compares with care without the device. Additional model calls and human review have costs. A
limited set of offline checks followed by clinical review of failures may keep the burden manageable,
 Binesh Sadanandan | FDA-2026-N-7874

but a retrospective rule that uses expert annotations requires a separate evaluation before it can
support runtime deferral.

Testing after changes and the role of monitoring
Questions 18, 19, 22, and 24; Section VI, pp. 19-22
The Chapter 5 results show that an update can improve consistency while leaving image dependence
unimproved. Post-update testing should therefore cover the affected clinical behavior, even when the
model architecture is unchanged. For the final device configuration, the evaluation record should
identify the model version and relevant changes to prompts, retrieved information, image processing,
decoding, and deferral rules. A third-party model update should be traceable to the device versions
and evaluations that used it.
I recommend retaining paired clinical cases and their controls for regression testing, supplemented by
previously unused cases. The amount of retesting can follow the plausible effect of the change. A
change in image processing calls for evidence on image interpretation; a change in prompting or
retrieval may affect the use of evidence and the decision to defer. Sponsors should explain the scope
of retesting and any competencies they consider unaffected. A small code change alone does not
justify a small clinical evaluation.
Monitoring can add evidence about changing inputs, unusual cases, and failure patterns that are
difficult to estimate before use. I would support accepting more premarket uncertainty only where
harm is limited or reversible, failures can be detected promptly, and a tested response such as
rollback or clinical review is available. For errors that can cause severe harm before anyone can
detect them, postmarket monitoring provides insufficient protection. The need for stronger premarket
evidence follows from the consequences and detectability of error.
The proposed framework can accommodate these checks without imposing one test or numerical
cutoff on every generative AI device. The sponsor should justify the controls, endpoints, and
acceptance criteria for the intended clinical use. My request is that a favorable consistency score be
accompanied by evidence that the device responds appropriately to the patient information on which
its use depends.

References
[1] U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical
Devices: Discussion Paper and Request for Feedback. August 2026. Docket FDA-2026-N-7874, document
FDA-2026-N-7874-0001. https://www.regulations.gov/document/FDA-2026-N-7874-0001
[2] Sadanandan B. Paraphrase Sensitivity in Medical Vision-Language Models: Measurement, Mechanisms,
Mitigation, and Deployment Safety. Doctoral dissertation, University of New Haven, 2026. Revision
supplied 22 September 2026. Tables 5.6 and 5.8; Section 7.2 and Table 7.5. Cited pages and accompanying
methods are reproduced in Attachment2.
[3] Sadanandan B, Behzadan V. Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical
Vision-Language Models. Proceedings of the 7th Conference on Health, Inference, and Learning.
Proceedings of Machine Learning Research. 2026;333:703-720.
https://proceedings.mlr.press/v333/sadanandan26a.html
[4] Sadanandan B, Behzadan V. Consistent but Dangerous: Per-Sample Safety Classification Reveals False
Reliability in Medical Vision-Language Models. Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition Workshops. 2026:6961-6968. https://arxiv.org/abs/2603.20985
 Binesh Sadanandan | FDA-2026-N-7874

[5] Geirhos R, Jacobsen J-H, Michaelis C, Zemel R, Brendel W, Bethge M, Wichmann FA. Shortcut learning
in deep neural networks. Nature Machine Intelligence. 2020;2:665-673. https://doi.org/10.1038/s42256020-00257-z
[6] Sadanandan B, Behzadan V, Jayan L, Kurup AG. PSF-Med: A Clinician-Audited Benchmark for
Paraphrase Sensitivity in Medical Vision-Language Models. Multimodal Foundation Models for
Biomedicine Workshop, Conference on Computer Vision and Pattern Recognition. 2026.
https://bineshkumar.me/phd-thesis/assets/papers/psf-med.pdf
[7] Guo C, Pleiss G, Sun Y, Weinberger KQ. On Calibration of Modern Neural Networks. Proceedings of the
34th International Conference on Machine Learning. Proceedings of Machine Learning Research.
2017;70:1321-1330. https://proceedings.mlr.press/v70/guo17a.html
[8] Sadanandan B, Behzadan V. Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in
Medical Vision-Language Models. UNSURE Workshop, International Conference on Medical Image
Computing and Computer Assisted Intervention. 2026.
https://papers.miccai.org/miccai-2026-sat/paper/UNSURE2026_059.pdf

Dissertation evidence supporting FDA public comment
Attachment2
Binesh Sadanandan
Docket FDA-2026-N-7874

Source
Paraphrase Sensitivity in Medical Vision-Language Models: Measurement, Mechanisms, Mitigation,
and Deployment Safety. Doctoral dissertation, University of New Haven, 2026. Revision supplied 22
September 2026.

This attachment reproduces selected pages from the 344-page dissertation PDF supplied with this
work. The following pages retain the original dissertation text, tables, and printed page numbers.
They support the empirical claims in Attachment1 and include the relevant methods and limits of
those claims.

Contents
Printed pages 144-146 (attachment pages 2-4). Patient- and image-disjoint evaluation, Table 5.6, and
the distinction between training variability and uncertainty in the test population.

Printed pages 151-153 (attachment pages 5-7). Image-reliance re-audit of the same adapted models,
Table 5.8, and explanation of the 238-question and 241-question endpoints.

Printed pages 214-217 (attachment pages 8-11). Retrospective admission rule, its dependence on
expert annotations, and Table 7.5. This experiment used earlier adapters and a separate cohort.

The dissertation evaluates research models on retrospective chest-radiograph question-answering
tasks. These excerpts do not establish clinical benefit or a validated runtime deferral procedure.
Third, we use symmetric KL rather than alternatives like margin MSE or JS divergence.

Symmetric KL provides a smooth gradient signal and naturally handles the asymmetry between over-condent and under-condent predictions.

5.7 Results
5.7.1 Main Results: MIMIC-CXR

We repeated the experiment after correcting three problems in the original evaluation: the

training pool contained operator-changing paraphrases (Can X be ruled out?) that teach

the model the wrong same-polarity answer, the original evaluation artifact was not saved in

a form that reproduced the reported counts, and the test set was disjoint from training by

question identity only. The rerun repartitions the MIMIC pool at the subject level, so that no

patient and no image is shared across the train/test boundary; trains on operator-preserving

paraphrases from the disjoint 656-subject training split (4,059 same-polarity binary pairs, two

epochs, ve seeds); and evaluates base and adapted models on the identical paraphrases of the

held-out test, saving every prediction to a per-question manifest. Following the convention

that a detection question is the clinically meaningful endpoint, presence-of-nding is the

primary endpoint (238 questions, 991 pairs, drawn from 236 held-out subjects and 238

held-out images); the full binary set (373 questions, 1,445 pairs, which also contains viewidentication and abnormality questions) is reported as a secondary.

Unit of analysis and sources of variability. Three nested units appear in the numbers

that follow, and they are not interchangeable. The pairwise ip rate takes the paraphrase pair

as its denominator (991 pairs on the primary endpoint), pairs are nested within questions

(238), and questions are nested within images (238) and within subjects (236). The querylevel ip rate takes the question as its denominator and counts a question once regardless

of how many paraphrases it carries, which is why the two rates dier in magnitude and

why only the query-level rate answers the clinician-facing question of whether an answer
Table 5.6: Main results on the patient- and image-disjoint MIMIC-CXR held-out test.
Primary endpoint: presence-of-nding (n = 238 questions, 991 same-polarity paraphrase pairs, spanning 236 held-out subjects and 238 held-out images); the all-binary
set (n = 373, spanning 241 subjects and 243 images) is a secondary. For reference, the
held-out split contains 245 subjects and 247 images in total; the dierence is questions of
other types or without an evaluable same-polarity paraphrase. Zero subject overlap and
zero image overlap with training, asserted in code before any metric is reported. Provenance: the split is generated by scripts/training/make_patient_disjoint_splits.py
(seed 42, subject-level partition of the 979-subject pool: 656 train / 78 val /
245 test subjects); adapters are trained on the 656-subject split (4,059 samepolarity binary pairs, two epochs; ve seeds for Targeted and three for Full LoRA)
by fleet_launch_patient_disjoint.sh; and eval_patient_disjoint_lora.py
asserts the zero-overlap condition and writes a per-question manifest to
results/lora_fleet_patient_disjoint/eval_{targeted,full}_s*.json. Values
are mean ± std across seeds. On the primary endpoint Targeted LoRA cuts the pairwise
ip rate from 8.5% to 3.5% ± 0.6 (about 59%) with no observed accuracy reduction (84.5%
to 84.7% ± 1.0; paired dierence +0.25 pp, 95% t interval [−1.00, +1.51]), and every seed
−7 −3
is signicant by paired McNemar (p from 2.5 × 10 to 1.8 × 10 ). Full LoRA reaches a
slightly lower pairwise ip rate (2.9% ± 0.3) but is statistically indistinguishable at the query
level (8.0 versus 8.1; Welch t = 0.08 on the per-seed means, dierence +0.08 percentage
points, 95% CI [−2.4, +2.6]) and gives up about 1.2 points of accuracy (83.3% ± 0.2).
Secondary results, namely the earlier question-identity-disjoint rerun and the training-size
sweep, are reported in Section 5.7 and Appendix A.8.

Model Pairwise ip Query-level ip Accuracy
Primary endpoint: presence-of-nding (n = 238 questions, 991 pairs)
Base MedGemma-4B 8.5% 19.8% 84.5%
+ Targeted LoRA (5 seeds) 3.5% ± 0.6 8.1% ± 1.8 84.7% ± 1.0
+ Full LoRA (3 seeds) 2.9% ± 0.3 8.0% ± 1.1 83.3% ± 0.2

Secondary: all binary questions (n = 373 questions, 1,445 pairs)
Base MedGemma-4B 8.1% 18.5% 84.5%
+ Targeted LoRA (5 seeds) 2.9% ± 0.7 6.2% ± 1.6 85.5% ± 1.1
+ Full LoRA (3 seeds) 2.2% ± 0.1 5.6% ± 0.7 84.8% ± 0.3
changes at all. Nesting is close to degenerate above the question on this split, since the 238

presence questions come from 238 held-out images and 236 held-out subjects, so the units

where dependence actually accumulates are the pair within the question and the question

within the image, not the patient. Accuracy, text-only agreement, and swap sensitivity are

all question-level quantities. Separately, every ± attached to a ip rate, an accuracy, or an

agreement gure in this chapter is a standard deviation across training seeds : it describes how

much the training procedure varies from one seed to the next, and it is not an interval for the

test cohort. The two quantities answer dierent questions and are not pooled into one gure.

The intervals reported here, the [−1.00, +1.51] percentage-point accuracy interval and the

[−2.4, +2.6] Welch interval, are likewise computed over per-seed means and are therefore also
training-variability statements. Clustered intervals at the pair, question, and image levels

are not computed in this chapter; the resampling helper written for that recomputation is

scripts/analysis/cluster_bootstrap.py, and no number reported here uses it.
Across ve seeds (Table 5.6), the targeted LoRA cuts the presence-only pairwise ip rate

from 8.5% to 3.5% ± 0.6 (about a 59% reduction, the ± being a standard deviation across

the ve training seeds) and the query-level rate from 19.8% to 8.1% ± 1.8, with no observed

accuracy reduction (84.5% to 84.7% ± 1.0; paired per-question dierence +0.25 percentage

points, 95% t interval [−1.00, +1.51] across the ve seeds). We report the interval rather

than claiming equivalence: the data are consistent with anything from a one-point loss to

a 1.5-point gain, so they exclude a large accuracy cost but do not establish exact parity.

−7 −3
Every seed is signicant by paired McNemar (p from 2.5 × 10 to 1.8 × 10 ; 37 to 41

questions ip for the base model only, against 5 to 14 for the adapted model only). On

the secondary all-binary set the pairwise rate falls from 8.1% to 2.9% ± 0.7 and accuracy is

unchanged (84.5% to 85.5% ± 1.1). The base pairwise rate (8.5%) is lower than the 14.6%

reported before the operator-preserving cleanup, because the earlier gure was inated by

operator-changing pairs whose ips were correct behavior.

The split denes the scope of the generalization claim. The test set is disjoint from
On these original-pipeline adapters, and on this n=98 ip bank alone, the additional 26.1M

parameters (from 4.38M to 30.5M) reduce the ip rate at the expense of image reliance. The

image-swap results provide the stronger evidence for this interpretation.

This comparison motivates the central argument of Chapter 6: ip rate alone is an

inadequate safety metric. This ordering, however, is specic to the original-pipeline adapters

and does not replicate on the clean, patient-disjoint adapters, where the two are tied on image

reliance and both are markedly more text-reliant than the base model (Section 5.7.5); the

targeted adapter is the better intervention on accuracy and cost, not on grounding. On

those original adapters, then, Full LoRA had the better ip-rate number and the worse

image-reliance prole; Section 5.7.5 shows that this particular contrast does not survive a

clean audit.

5.7.5 Safety Re-Audit of the Clean Adapters

The comparison just given, and the image-reliance audit of Chapter 6, both rest on adapters

trained by the original pipeline. Because the mitigation result above is measured on the clean,

patient-disjoint adapters, we re-ran the image-reliance audit on those exact adapters, using

the same protocol as Chapter 6 and the same patient- and image-disjoint held-out set. The

audit reports a slightly larger population than the ip-rate table above: text-only agreement

and swap sensitivity are dened on a single question, whereas a ip rate requires at least one

paraphrase to compare against. The presence endpoint therefore carries 241 questions here

against 238 there, and the all-binary endpoint 382 against 373; the three and nine additional

questions are those the equivalence audit left without a surviving paraphrase. The two tables

are computed on the same split and the same adapters, and the small dierence in n is this

and nothing else.

The earlier ordering does not replicate. On the presence-only endpoint the targeted and

full adapters are statistically indistinguishable on both image-reliance measures: text-only

agreement is 76.8% ± 3.1 versus 76.6% ± 1.7, and swap sensitivity is 19.9% ± 1.2 versus
Table 5.8: Image-reliance re-audit of the clean, patient-disjoint adapters, using the same
protocol as Chapter 6: the text-only condition removes the image token from the prompt
entirely, and the swap condition re-pairs each question with another test image. All measurements are on the patient- and image-disjoint held-out set (zero subject and zero image
overlap with training). Higher text-only agreement means more text-reliant; higher swap
sensitivity means more image-reliant. Two results stand out. First, the targeted and full
adapters are statistically indistinguishable on both image-reliance measures, so the ordering
reported in Chapter 6 for the original-pipeline adapters (targeted 66.3% text-only agreement
and 32.7% swap sensitivity against full 80.6% and 14.3%) does not replicate here. Second,
both adapters raise text-only agreement sharply relative to the base model (53.5% to about
77% on the presence-only endpoint) while slightly lowering swap sensitivity, so both buy a
substantial part of their consistency from the question text rather than from more stable use
of the image.

Model Text-only agree ↑ risk Swap sens. ↑ Accuracy
Primary endpoint: presence-of-nding (n = 241)
Base MedGemma-4B 53.5% 22.0% 84.7%
+ Targeted LoRA (5 seeds) 76.8% ± 3.1 19.9% ± 1.2 84.3% ± 1.0
+ Full LoRA (3 seeds) 76.6% ± 1.7 20.9% ± 1.3 82.4% ± 0.5

Secondary: all binary questions (n = 382)
Base MedGemma-4B 67.0% 18.1% 84.5%
+ Targeted LoRA (5 seeds) 69.8% ± 2.7 18.5% ± 0.4 85.2% ± 1.1
+ Full LoRA (3 seeds) 69.4% ± 1.8 19.7% ± 1.3 84.5% ± 0.4
20.9% ± 1.3, with the full adapter marginally the more swap-sensitive of the two; each ±
here is a standard deviation across training seeds, ve for the targeted adapter and three

for the full adapter, and not an interval for the test cohort. The contrast reported for the

original-pipeline adapters (targeted 66.3% text-only agreement and 32.7% swap sensitivity

against full 80.6% and 14.3%) is therefore specic to those adapters and that evaluation

set. On the clean adapters, the claim that targeted adaptation preserves image dependence

relative to full adaptation is not supported.

The more important observation concerns how both adapters compare with the base

model. Text-only agreement rises from 53.5% to about 77% for both, a 23-point increase,

while swap sensitivity falls slightly (22.0% to 19.9% and 20.9%). Both interventions therefore

achieve a substantial part of their consistency through greater reliance on question text. The

result also applies to our mitigation: a low ip rate does not establish grounding, even when

mechanistic analysis guides the intervention. Together with Thrust 4, these results show

that the targeted adapter reduces paraphrase sensitivity without improving grounding, and

partly at the expense of it.

The audit changes why we prefer the targeted adapter. The targeted adapter remains

preferable on accuracy and parameter cost: at statistically indistinguishable query-level

consistency (8.1% ± 1.8 versus 8.0% ± 1.1; (Welch t = 0.08 on the per-seed means, dierence

+0.08 percentage points, 95% CI [−2.4, +2.6])) and tied image reliance, it preserves accuracy
(84.3% ± 1.0 versus 82.4% ± 0.5, against a base of 84.7% on this n=241 audit set; the
base moves with the evaluation set, so the primary n=238 endpoint of Table 5.6 has a
base of 84.5% and a paired dierence of +0.25 points) while modifying 0.1% of parameters

rather than 0.72%. It reaches the same consistency at lower parameter cost and with higher

accuracy, while oering no grounding advantage. Any deployment of either adapter must

therefore be paired with the image-reliance checks of Chapter 6, not justied by the ip-rate

reduction alone.
LoRA to 92.7% and Base to 92.8% at severity 5, both still above the 90% level, while Full

LoRA stays nearly at (94.1%). Calibrating conformal sets at this conservative target keeps

actual coverage above 90% under shift for the two models whose predictions respond to

corruption. Retrospectively, that makes the 95% target the setting to test in a prospective

evaluation of Targeted LoRA, because it is the only target tested here that held actual

coverage above 90% at every corruption severity.

7.2 Deployment Gate: An Oine Readiness Audit
The evaluation results suggest a natural deployment protocol: admit predictions only

when they pass both a consistency and an image-reliance test. What we evaluate here

is that protocol run retrospectively, on a cohort with radiologist annotations, and it is

best read as a deployment-readiness audit rather than as an online gate; the reasons

are set out after the rule. To avoid ambiguity we state the rule exactly as implemented

(scripts/miccai/run_deployment_gate.py). A prediction is admitted if and only if

agree(p1 , p2 ) ∧ swap_changed ∨ roi_matters ,


| {z } | {z }
consistency image reliance

where agree(p1 , p2 ) requires the model to give the original answer on the rst two paraphrases, swap_changed is true when the answer changes under the swapped image, and
roi_matters is true when the region-only and background-only crops yield dierent answers. The two image-reliance terms are a disjunction, not a conjunction, and roi_matters

is available only on PadChest (637 of 861 records); on MIMIC the rule therefore reduces to

agree(p1 , p2 ) ∧ swap_changed. The gate does not use the text-only condition.1 Predictions
1 The implementation contains a third image-reliance disjunct, question_type_uses_image, which res
when the per-nding swap rate exceeds 0.05. It is inert on every record analysed here, and the reason is
a property of the stored data rather than of the rule as written. The nding label exists upstream: it is
populated on all 4,251 rows of the PadChest ip bank. It is dropped when run_safety_eval.py serialises
the per-question records, so no stored record carries a finding_label eld; the helper that builds the per-
nding rates skips empty labels, and the lookup therefore always returns the 0.0 default. The two-term rule
failing either criterion are agged for human review. The evaluation sets are the curated

MIMIC (n = 98) and PadChest (n = 861) ip banks.

7.2.1 Why This Is an Audit and Not an Online Gate

Two properties of the rule limit the interpretation of its results. The swap partner is an

arbitrary image from the same cohort, taken in rotation, not a matched opposite-nding

image, so swap_changed measures sensitivity to image replacement in general rather than

a response to nding-specic evidence. And the roi_matters term is available only on the

637 of 861 PadChest records with radiologist bounding boxes, where on its own it admits

between 16.5% and 21.2% of cases depending on the model, so gated accuracy on PadChest

is partly inuenced by annotation availability, and the MIMIC evaluation rests on the swap

term alone.

Two of the three inputs are available at inference time for a new patient, and one is not.

Paraphrase agreement needs only the question, and swap_changed needs only some other

image: the implementation supplies it by rotation, giving question i the image of question

i + 1, so it never consults the true label and could be computed in deployment against any
unrelated study. roi_matters is the exception. It compares a region-only crop against

a background-only crop, which presupposes a radiologist bounding box for the nding in

question, and that is exactly what a deployed system does not have for an undiagnosed

patient. It is available for 637 of the 861 PadChest records and for none of MIMIC.

Because the image-reliance terms are a disjunction, the size of this eect scales with how

often roi_matters is the only reason a prediction is admitted. Recomputing from the stored

records: of Base's 358 admitted PadChest predictions, 76 (21.2%) are admitted solely on

the region test; for Targeted LoRA 52 of 284 (18.3%), and for Full LoRA 38 of 230 (16.5%).

above is the rule that actually ran, which we conrmed by reproducing the admission rates in Table 7.5
exactly from the stored records. This implementation detail changes the evaluated rule: restoring the label
to the serialised records activates the third disjunct and raises Base PadChest admission from 41.6% (358 of
861) to 64.6% (556 of 861), because 71 of the 92 nding labels clear the 0.05 threshold. Every gate number
in this chapter is the two-term rule's and must not be mixed with numbers recomputed after such a change,
which would be measuring a three-term rule.
Dropping the region term, which is what an online gate would have to do, lowers PadChest

admission from 41.6% to 32.8% for Base, from 33.0% to 26.9% for Targeted LoRA, and from

26.7% to 22.3% for Full LoRA. The MIMIC conguration, where the rule already reduces to

agree(p1 , p2 ) ∧ swap_changed, is deployable as written.
The two datasets support dierent uses of the rule. The MIMIC numbers describe a rule

a deployment could run. The PadChest numbers describe a readiness audit: they answer

on a cohort where we know where the nding is, what would this gate have admitted?,

which concerns pre-deployment evaluation rather than runtime triage. An online variant

would either drop the region term at the admission cost quantied above, or replace it with

an annotation-free saliency proxy, which Chapter 6 gives reason to distrust: attention is a

coarse localizer whose patch ranking does not track causal importance.

Because admission already forces the rst two paraphrases to agree, the residual ip

rate reported for admitted predictions is necessarily carried by paraphrases the gate never

inspected (the third and later). We therefore report it as ip rate on uninspected paraphrases,

to test whether a two-paraphrase gate ensures stability on uninspected rephrasings. The

results show that it does not. Requiring agreement on all available paraphrases instead

of the rst two drives that residual to 0.0% in every cell, at a lower admission rate: Base

admits 13/98 (MIMIC) and 97/861 (PadChest), Targeted LoRA 23/98 and 272/861, and

Full LoRA 14/98 and 206/861. The cost of certifying stability is therefore admitting far

fewer predictions, which is the substantive trade-o a deployment would face.

On PadChest, the gate admits 41.6% of Base predictions at 96.9% accuracy (up from

78.6% unltered). For Targeted LoRA, the gate admits 33.0% at 96.8% accuracy (up from

91.5%). A majority of predictions are still rejected for failing at least one safety criterion,

but the admitted subset is far more accurate than the population.

The accuracy benet varies by model and dataset. On MIMIC the gate improves accuracy

for two of the three models and provides no gain for the third: Base admits 22.4% at

95.5% accuracy against 83.7% unltered, and Full LoRA admits 14.3% at 92.9% against
Table 7.5: Deployment gate results. The gate admits a prediction i the model gives the
original answer on the rst two paraphrases and shows image reliance (the answer changes
under the swapped image, or, on PadChest only, the region-only and background-only crops
disagree); it does not use the text-only condition (Section 7.2). Acc (all): unltered accuracy.
Acc (gated): accuracy on admitted predictions. Flip (uninspected paras): because admission forces the rst two paraphrases to agree, this residual is carried entirely by the third
and later paraphrases, which the gate never checked; it measures whether a two-paraphrase
gate certies stability on unseen rephrasings. Requiring agreement on all paraphrases drives
it to 0.0% in every cell at a lower admission rate (Base 13/98 and 97/861; Targeted LoRA
23/98 and 272/861; Full LoRA 14/98 and 206/861). The low admission rates reect the
severity of safety failures across models.

Model Dataset Admitted Acc (gated) Acc (all) Flip (uninsp. paras)

Base MIMIC 22.4% 95.5% 83.7% 40.9%
PadChest 41.6% 96.9% 78.6% 72.9%

Targeted LoRA MIMIC 30.6% 76.7% 77.6% 23.3%
PadChest 33.0% 96.8% 91.5% 4.2%

Full LoRA MIMIC 14.3% 92.9% 81.6% 0.0%
PadChest 26.7% 83.9% 66.0% 10.4%

81.6%, while Targeted LoRA admits 30.6% at 76.7% against 77.6% unltered, which is no

improvement at all. A rule that helps the two weaker models and does nothing for the

strongest one is consistent with the reading developed in Section 7.3: what the rule removes

is largely cases the weaker models answer from the text prior, and Targeted LoRA has fewer

of those to remove.

A practical limitation is that the gate requires additional forward passes: the paraphrases,

the swapped image, and, on PadChest, the region-only and background-only crops. It does

not require a text-only pass, because the text-only term is not part of the implemented rule.

The entropy-based bridge from Section 7.1.6 oers a cheaper alternative: a single-forwardpass entropy threshold that approximates the multi-pass rule. The entropy threshold admits

more predictions at a lower admitted-case accuracy.