HealthRadar / Twenty Poms, LLC (Edward A. Rose)

Please see the attached technical comment from HealthRadar / Twenty Poms, LLC on measuring clinically consequential failure and evaluator reliability in generative-AI-enabled medical devices. The comment includes results and limitations from the publicly available HealthRadar Yardstick and Yardstick 2.0 benchmark studies.

HEALTHRADAR | FDA TECHNICAL COMMENT

Clinical failure measurement and evaluator
reliability
Technical comment on generative-AI laboratory interpretation
Docket FDA-2026-N-7874 | 11 October 2026
Submitted by Edward A. Rose, M.D., M.S.A., on behalf of HealthRadar / Twenty Poms, LLC, and
Steven E. Rose, Ph.D.
Key point for FDA: Clinically consequential failure should be measured separately from
surface-level answer quality, and any automated evaluator used to judge clinical AI should
itself be evaluated as a measurement instrument.

Scope and central proposal
This comment addresses measurement methods for patient-facing, text-based interpretation of
laboratory results. It concentrates on questions 9, 10, 12, 14, 16, and 20, with limited
methodological responses to questions 6, 13, 19, and 24. It does not take a position on the level of
regulation, mandatory third-party review, or the regulatory classification of any HealthRadar
function. The FDA paper is a discussion document, not draft or final guidance. [1,2]
The central measurement proposal is to evaluate consequential clinical meaning separately from
the amount of factually correct or well-written text. A second proposal is to evaluate the
reliability of the scoring process itself. A fluent report is not necessarily clinically adequate, and a
completed automated assessment is not necessarily a valid reference judgment.

Basis and limitations of this contribution
The original HealthRadar Yardstick study used 15 purposively constructed synthetic laboratory
cases, three generator families, and three repetitions per family, yielding 135 reports. Initial
assessment was AI-assisted, followed by selective physician adjudication of 91 reports; 44
retained their locked primary assessments. This was not independent physician scoring of every
report, and interphysician reliability was not measured. [3]
All nine reports classified as potentially unsafe in the original cohort arose from a single complex
case. Separately, 83 of 405 scheduled automated judge evaluations failed procedurally, and the
subset with valid evaluations from all three judges excluded every report classified as potentially
unsafe. These observations motivate methodological questions; they do not establish clinical
error prevalence or patient harm. [3]
HealthRadar was not evaluated as an intervention in that original cohort. Yardstick 2.0 is a
separate prospective paired synthetic benchmark of Raw versus Guided model output. Its public
results are summarized below. The study is not peer reviewed, used 15 synthetic cases and one
HealthRadar-associated physician scorer, and did not measure patient outcomes or real-world
effectiveness. [4]

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

1. Measure consequential clinical meaning
Question 9; related methodological point for question 6
A report-level endpoint can capture a material defect that sentence-level accuracy or a mean
quality score may obscure. In the original HealthRadar manuscript, material clinical failure
combines the overall categories materially deficient and potentially unsafe. Those categories
remain distinct: a material interpretive defect is not automatically a demonstrated harm event.
[3]

A proposed reporting structure
For each report, retain the overall category and the specific reason for it: incorrect principal
finding, inadequate urgency, omitted next step, unsupported diagnosis or treatment direction,
inappropriate reassurance, or failure to respect an information boundary. Assess these elements
against the supplied facts and a case-specific clinical reference rather than a preferred prose
template.
Clinical completeness and clarity can be recorded separately. A safety-relevant omission is not
offset by additional accurate explanations of secondary findings. An illustrative stress case might
contain several minor abnormalities and one time-sensitive finding; the test is whether the action
message remains appropriate, not merely whether every abnormal value is mentioned.

A balanced escalation assessment
Report under-escalation and over-escalation separately, with denominators tied to cases in which
each error is assessable. State the clinical scenario and plausible consequence rather than
assigning a universal numerical exchange rate between missed urgency and unnecessary
referral. The two error types should not cancel each other in an aggregate score.
Distinguish timing from venue. A response that preserves an appropriate action window through
a clinically defensible alternative setting is not necessarily incorrect. Conversely, an unsupported
emergency diagnosis or unnecessary insistence on admission is a separate claim requiring
support. The standard should accommodate defensible options without losing the meaning of
urgency.

Missing information is part of the test
Facts not supplied remain unknown. Absence of a symptom history does not establish that the
patient has no symptoms. A reference can specify what is inferable, what remains uncertain, and
which action can still be justified despite uncertainty. Useful uncertainty handling is not
equivalent to adding a generic disclaimer or deferring every difficult question.
These are proposed operational definitions, not evidence that a particular numerical threshold
predicts safety. Acceptance criteria require an intended use, a target population and workflow,
an explicit harm model, and adequate precision. The cited studies do not establish a universal
passing score.

Prospective example: Yardstick 2.0
HealthRadar recently completed a prospective paired synthetic benchmark examining whether
structured clinical context altered material clinical failures in patient-facing laboratory
interpretations generated by three frontier-model families. Fifteen prospectively defined

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

synthetic cases were evaluated across three model families, three repetitions, and two paired
arms, yielding 270 reports. The Raw arm received frozen case facts and common generation
instructions. The Guided arm received the same material plus structured clinical context
specifying principal and supporting findings, routing information, safety drivers, uncertainty,
and capability boundaries. [4]
Material clinical failure occurred in 18 of 135 Raw reports (13.33%) and 2 of 135 Guided
reports (1.48%). The case-balanced Guided-minus-Raw difference was -11.85 percentage points,
with a 95% paired case-cluster percentile interval of -25.19 to 0.00 percentage points. The
favorable difference was concentrated in three safety-sensitive cases involving hyponatremia,
severe neutropenia, and severe hypertriglyceridemia; 12 of 15 case-level effects were zero. [4]
Importantly, the number of reports classified as Excellent was nearly identical between arms (71
Raw and 70 Guided). Eighteen of the 20 material failures across both arms involved timing or
disposition boundaries. Thus, in this benchmark the principal observed difference did not reflect
a generalized shift toward more polished output, but rather fewer prespecified material clinical
failures involving clinically consequential action boundaries. [4]
These findings require cautious interpretation. The study used 15 purposively constructed
synthetic cases, the favorable effect was concentrated in three cases, the confidence interval
reached zero, clinical scoring was AI-assisted and physician-confirmed by one HealthRadarassociated physician, and no patient outcomes or real-world clinical performance were
measured. Independent external replication remains necessary. The study therefore illustrates a
measurement problem; it does not establish general product effectiveness or population-level
error rates. [4]
The example supports distinguishing clinically consequential failure from overall fluency or
apparent answer quality and preserving case-level structure when estimating uncertainty.

2. Define a defensible benchmark and reference
Questions 10 and 14; related methodological point for question 13
An open-ended answer can be clinically appropriate in more than one form. A proposed
reference is therefore a documented range of acceptable clinical responses, not a single model
answer. It can identify required conclusions, acceptable alternatives, action timing, uncertainty,
prohibited unsupported claims, and the evidence underlying each judgment.

Protect the clinical reference from the intervention
Agreement with a software engine is not a clinical correctness criterion when that engine is the
intervention being tested. Reference construction needs evidence distinct from the intervention’s
own output. The roles of case designer, reference author, scorer, adjudicator, and software
developer should be reported, including roles held by the same person.
For a prospective study, lock case facts, references, scoring rules, and analysis decisions before
examining study outputs, with a visible amendment process. This is a proposed prospective
safeguard, not a retrospective description of every pilot step. In the original HealthRadar study,
standards were frozen before scoring, and the statistical analysis plan was frozen after
adjudication but before masked statistical execution. [3]

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

A reproducible reference record includes the source and version of clinical evidence, its
applicable population, decision rationale, and acceptable disagreement. Masking generator
identity reduces one opportunity for bias; it does not establish independent reference
construction or interrater reliability.

Use synthetic cases for a stated purpose
A deliberately enriched synthetic challenge set can expose a predefined failure mode, create
controlled contrasts, and protect patient privacy. It cannot by itself estimate error frequency in
the population using a deployed product. The case-selection rationale and omissions should be
explicit; rare dangerous cases may be deliberately overrepresented.
When generative tools assist case design, independently check numerical coherence, units,
reference intervals, supplied history, and the clinical rationale. Do not establish a case’s validity
solely by obtaining agreement from other language models. Keep evaluation cases and keys
separate from tuning, prompt refinement, and judge calibration.
A proposed construct-validation program would check whether the rubric captures clinically
meaningful defects and distinguishes appropriate from deliberately flawed outputs on held-out
material. External validity would then require representative real-input evaluation in the
intended workflow, examining whether benchmark-detected weaknesses recur and whether
users understand and act appropriately on outputs. Until that relationship is established,
challenge-set performance remains distinct from demonstrated performance in clinical use.
Separate evidence is needed for tasks outside the benchmark, including PDF ingestion, optical
character recognition, longitudinal integration, patient comprehension, and clinical workflow
effects. A text-only benchmark does not validate those stages. Real-input and user-interaction
studies answer different questions and should be reported as separate evidence layers.
IMDRF’s 2025 good machine learning practice principles provide relevant foundations for testdata independence, appropriate reference standards, and assessment in the intended human-AI
workflow. They do not supply a ready-made pass threshold for this laboratory-interpretation
task. [5]

3. Make denominators and evaluators auditable
Questions 12 and 20

Respect the experimental unit
Fifteen cases evaluated repeatedly do not become 135 independent clinical presentations. Report
the number of distinct cases, generator configurations, repetitions, and scored outputs separately.
A paired comparison can preserve all arms and repetitions within each case during case-level
resampling. Precision remains limited when the number of distinct cases is small.
Predefine the quantity being estimated. A report-level failure proportion, the proportion of cases
with any failure, and disagreement in action timing across repetitions measure different things.
The probability of observing at least one failure also depends on how many repetitions are
attempted. Stable wording is unnecessary; stable clinical meaning is the relevant repeatability
construct.

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

Synthetic and real-world observations should not be pooled into an apparently populationrepresentative percentage without a justified target distribution and weighting or transport
assumptions. Even when a combined estimate is defensible, show the component results and
sensitivity to those assumptions. A challenge-set result alone is not a prevalence estimate.

Evaluate the evaluator as a measurement instrument
An automated judge or supervisory agent needs its own assessment against an appropriately
constituted human reference, on held-out material not used to tune that evaluator. Candidate
measures include sensitivity to consequential failures, false-positive review burden,
disagreement by failure type, and the proportion of attempted assessments that produce usable
judgments.
Availability is not accuracy. Track requested generations, delivered candidate content, usable
candidate reports, attempted judge assessments, valid judge assessments, and final humanreviewed decisions as distinct stages. Retain content and metadata so that a refusal, truncation,
parse failure, and clinically deficient answer can be distinguished rather than silently conflated.
The original HealthRadar judge analysis illustrates why this matters: 322 of 405 planned judge
cells were valid; 83 failed procedurally. Restricting analysis to the 61 reports with complete
judgments from all three judges excluded the potentially unsafe reference reports. Missing
evaluations therefore changed which reports were represented. This finding does not prove why
the judge calls failed or establish the accuracy of a fully independent clinical reference. [3]
A proposed analysis reports missingness by case and risk category, retains the original
denominator, and makes the handling of unavailable evaluations explicit. A missing judge result
is not evidence that the candidate is safe, and it is not automatically a clinical error by the
candidate. Human review or an unresolved-status pathway can prevent procedural success from
being mistaken for clinical assurance.

4. Specify what independent assessment means
Question 16
Potentially separable assessor functions include custody of held-out test cases, protocol and
provenance review, masked clinical adjudication, analysis verification, and version-specific
reporting. Each function requires a defined scope and competence. Expertise in laboratory
interpretation does not establish competence in behavioral-health crises, autonomous
prescribing, imaging, or every other clinical domain.

Disclose independence rather than assume it
A proposed assessor record distinguishes independence of personnel, reference construction, test
data, analysis, and financial relationships. A vendor-funded assessment is not made independent
merely by masking its name. A developer-led research study can be informative, but it is not
equivalent to an external clinical audit of that developer’s own product.
Practical safeguards include a prespecified scope, documented scoring criteria, payment terms
unrelated to a passing result, traceable disagreement resolution, retained source outputs, and
disclosure of competing commercial interests. A report can state which safeguards were present
and which were not rather than using an unqualified independence label.

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

The assessment output would identify the tested system configuration, intended use, evaluation
date, case coverage, uncertainty, unavailable observations, and limits on extrapolation. It would
not translate a favorable benchmark into a claim of universal safety, regulatory authorization, or
validation of untested workflows.

Costs and access are measurement-design considerations
Independent review can add expertise and separation from development, while also adding
expense, turnaround time, and questions about access to proprietary information. Shared
reporting formats and bounded review scopes may make evidence easier to compare. These are
operational options, not an assertion that a particular mandatory review model is warranted or
that HealthRadar is a recognized assessor.

HealthRadar’s own position
HealthRadar develops clinical interpretation and evaluation methods and therefore has a
potential commercial interest in this area. The evidence described here is developer-led and
includes a single physician scorer/adjudicator. This comment does not assert that HealthRadar is
a qualified independent certification organization, that FDA endorses HealthRadar, or that either
Yardstick benchmark has established external clinical validity.
Any claim of broader evaluator validity would require evidence beyond developer-led
benchmark work. The scope of an individual study and the scope of an assessor’s public claim
should therefore be stated separately.

5. Preserve the meaning of an assessment over time
Limited methodological responses to questions 19 and 24
A proposed version record identifies the model and observable version information, generation
settings, prompt, input transformations, retrieval components, safety controls, output processing,
interface, and clinical reference version. Unavailable vendor information remains unavailable; a
product name alone does not uniquely identify the tested configuration.
For an update, paired regression testing can compare old and new configurations on the same
preserved cases and references. A retained safety suite supports continuity; separate held-out
cases reduce dependence on a repeatedly used test set. Changes to clinical evidence and reference
standards should be recorded separately from changes to the software under test.
Offline regression testing cannot establish deployment safety by itself. Any proposed operational
review pathway needs a named owner for flagged outputs, a way to resolve unavailable
assessments, and a traceable disposition. HealthRadar has not demonstrated a deployed
postmarket monitoring service, and this comment proposes no universal monitoring interval or
regulatory change threshold.

Conclusion
Generative clinical output can be fluent while losing the clinically important conclusion, action
window, or boundary of inference. Conversely, an evaluator can complete a scoring task while
failing to provide a dependable measurement of those same properties. Evaluation frameworks

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC
 HEALTHRADAR | FDA TECHNICAL COMMENT

should therefore preserve clinically consequential endpoints, auditable denominators, case-level
dependence, reference provenance, and the performance limitations of the evaluator itself.
The HealthRadar benchmarks are limited synthetic studies, not evidence of population-level
safety or real-world effectiveness. Their methodological value is narrower: they provide worked
examples of why clinically consequential failure and evaluator reliability should be measured
explicitly rather than inferred from prose quality, aggregate agreement, or procedural
completion.

Disclosures
Commercial-interest disclosure. HealthRadar develops clinical interpretation and evaluation
methods and may seek commercial applications for scientific review and clinical-assurance
services. The authors therefore have a potential commercial interest in this area. The studies
cited here were developer-led; no claim of independent certification or FDA endorsement is
made.
AI assistance. Artificial intelligence tools were used to assist with literature and regulatorysource review, organization of source material, drafting, editing, and preparation of this
comment. The authors independently reviewed the cited sources and the substantive clinical,
scientific, methodological, and regulatory statements in the final submission. All interpretations,
conclusions, and recommendations presented in this comment were reviewed and approved by
the human authors, who accept responsibility for the submitted content.
Author authorization. Both authors authorized public release of the study findings and
limitations summarized in this comment and approved citation to the public Yardstick
manuscripts.

References
1. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback. 2026. https://www.fda.gov/media/194242/download
2. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AI-Enabled Medical Devices: official
landing page and submission instructions. Docket FDA-2026-N-7874; comments due October 19, 2026.
https://www.fda.gov/medical-devices/digital-health-center-excellence/considerations-regulation-generative-ai-enabledmedical-devices-discussion-paper-and-request
3. Rose EA, Rose SE. Clinical quality and repeatability of large language model interpretations of synthetic laboratory reports.
HealthRadar Yardstick original-cohort public manuscript. 2026. https://healthradarmvp.com/research/yardstick/
4. Rose EA, Rose SE. HealthRadar Yardstick 2.0: Effect of structured clinical context on material failures in large-language-model
laboratory interpretations. Public manuscript, Version 1.0, October 5, 2026; published October 8, 2026. Not peer reviewed.
https://healthradarmvp.com/research/yardstick-2/
5. International Medical Device Regulators Forum. Good Machine Learning Practice for Medical Device Development: Guiding
Principles. IMDRF/AIML WG/N88 FINAL:2025.
https://www.imdrf.org/sites/default/files/2025-02/IMDRF_AIML%20WG_GMLP_N88%20Final.pdf

Docket FDA-2026-N-7874 | HealthRadar / Twenty Poms, LLC