People’s Evidence Lab Inc. (Stephen J. Watt)

People’s Evidence Lab Inc. submits the attached comments on “Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback,” Docket FDA-2026-N-7874.
Our response addresses Questions 9, 10, 12, 14, 19 and 22. We recommend connecting performance measurement across premarket evaluation, postmarket monitoring and evaluation after device modification.
The central recommendation is to define the configured device’s performance claim before choosing the test. Sponsors should then justify the cases, repeated outputs, scoring methods and uncertainty estimates needed to assess that claim.
Specifically, we recommend:
•Assessing both performance against clinical criteria and clinically important variation across repeated outputs and relevant input changes.
•Demonstrating that benchmarks and scoring methods can support the conclusions drawn from them.
•Justifying clinically relevant comparators and reporting uncertainty.
•Linking postmarket monitoring to the claims evaluated before deployment and to prespecified actions.
•Preserving comparable measurements after modifications, with broader testing where the effects of a change are uncertain.
These recommendations draw on published research, FDA materials and established measurement guidance. PEL’s exploratory research is included as a bounded illustration. We do not propose a universal sample size, passing score or monitoring interval, or adoption of PEL’s research method as a regulatory standard.
Please see the attached submission for the complete recommendations, supporting references, limitations, disclosures and illustrative appendices.
Stephen J. Watt
Founder and Principal
People’s Evidence Lab Inc.

People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Comments on Considerations for the Regulation of Generative AI-Enabled Medical
Devices: Discussion Paper and Request for Feedback
Docket: FDA-2026-N-7874
Submitted by: People’s Evidence Lab Inc.
Date: October 16, 2026
Version: 1.0

Abstract
A generative AI-enabled device can give different answers to the same clinical question. Much
of that variation is harmless rewording. Some of it is not: an answer that changes the
recommended urgency of care is a different output, not a different phrasing. FDA’s discussion
paper already draws this line within its generalizability element. The open question for reviewers
is how sponsors should measure such variation, and how that measurement should follow a
device from premarket testing through deployment and later modification.
People’s Evidence Lab’s answer is to start with the performance claim: what the configured
device is expected to do, for whom and under which conditions. Sponsors should define the
clinical behavior the configured device is meant to deliver, then justify the evidence needed to
test it: which cases, how many repeated outputs per case, how outputs are scored, and how
uncertainty is reported. Two failure modes motivate this sequence. An average score can conceal
unacceptable individual answers, and a device that answers consistently can still be consistently
wrong. Reviewers therefore need to see both the level of performance and the pattern across
outputs. Postmarket monitoring should be tied to the premarket claims, and evaluation after a
modification should preserve comparable measurements, with broader testing where the effect of
the change is uncertain.
These comments respond to Questions 9, 10, 12, 14, 19 and 22. They rest on published research,
FDA’s discussion paper and established measurement guidance. PEL’s exploratory study of
repeated large language model answers appears only as a bounded illustration in Appendix C.
PEL does not ask FDA to adopt a universal sample size, passing score or monitoring interval, or
to make PEL’s research method a regulatory standard. The evidence supports measurement
principles, not a validated testing package. Studies across multiple devices, tasks and settings,
including patient outcomes, are needed before more prescriptive expectations would be justified.
To the Food and Drug Administration:
People’s Evidence Lab Inc. (PEL) recommends that FDA connect performance measurement
across testing, deployment and later changes. When repeated outputs can alter a clinical decision,
testing should measure that variation. The scoring process also needs evidence that it measures
the intended behavior.

Page 1 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

PEL studies repeated outputs from large language models used for health-related support. These
comments draw on published research, FDA materials and measurement guidance. PEL’s
exploratory work provides a supporting example. The recommendations express PEL’s views.
They do not represent a patient group or another institution.
Here, generative artificial intelligence (GenAI) means models that generate content from learned
patterns. Large language models learn language patterns from extensive text and generate
answers. [1, section III, p. 3]
The FDA questions below come from its Center for Devices and Radiological Health (CDRH).

Page 2 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Contents
Summary of recommendations
Scope and overarching position
Question 9 — Device benchmarking
Question 10 — Benchmark validity
Question 12 — Performance and uncertainty
Question 14 — Comparators and scoring
Question 19 — Postmarket monitoring
Question 22 — Evaluation after modification
Research needs and conclusion
Interests, funding and contributions
References
Signatory and public contact
Appendix A — Glossary and terminology justification
Appendix B — PEL working definitions
Appendix C — Supporting material
Appendix D — Worked example
Appendix E — Illustrative sponsor reporting form

Page 3 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

1. Summary of recommendations
We address Questions 9, 10, 12, 14, 19 and 22. Our central recommendation is to define the
performance claim before choosing the test. Sponsors should then justify the cases, repeated
outputs, scoring methods and uncertainty estimates needed to assess that claim. Appendix B
defines three working expressions used in these comments.
An average score can conceal unacceptable answers. Consistent answers can also be consistently
wrong. Evaluation therefore needs both the performance level and the pattern across outputs. The
clinical importance of variation depends on the task and consequences of error.
Table 1 links each requested action to its detailed response. These are proposed scientific
practices for FDA’s consideration. We do not propose a universal sample size, passing score or
monitoring interval. We also do not propose adopting PEL’s research method as a regulatory
standard. Appendix D gives a hypothetical worked example showing how the recommendations
fit together for one urgency-of-care claim. Appendix E provides an illustrative sponsor reporting
form. PEL Figure 1 shows how these recommendations connect across evaluation, monitoring
and modification, including when repeated-output testing is relevant.
Table 1. Recommendations and corresponding FDA questions

FDA Detailed
No. Requested action
question response

Measure clinically important behavior across repeated outputs and
1 Q9 Section 3
relevant input changes.

Require evidence that benchmark scores support the intended clinical
2 Q10 Section 4
inference.

Justify case sampling, repeat sampling and uncertainty; combine only
3 Q12 Section 5
estimates of a common, comparable quantity.

4 Q14 Justify comparators and acceptance criteria; test the scoring process. Section 6

Link monitoring to premarket claims, current clinical use and actions
5 Q19 Section 7
with named decision owners.

Preserve comparable measurements after changes; expand testing when
6 Q22 Section 8
impact is uncertain.

Page 4 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

PEL Figure 1. PEL’s proposed measurement approach links prespecified performance claims, methods and
acceptance criteria across premarket evaluation, monitoring and evaluation after modification, with repeated-output
testing when variation could affect safety or effectiveness. Reassessment informs investigation, safeguards or further
assessment; acceptance criteria are not relaxed after results, and the approach does not itself establish safety or
regulatory authorization.

2. Scope and overarching position
This response concerns FDA’s August 2026 discussion paper. The paper seeks early input; it is
neither draft nor final guidance. We comment on measurement within its proposed approach. We
do not endorse every element of that approach. [1, introductory notice, p. 1]

Page 5 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Our focus is the configured device: the device as evaluated, including the model version,
instructions, information sources, tools, safeguards and generation settings in use. Its
performance may differ from the underlying model’s performance. The proposed
recommendations concern evaluation of clinically important behavior throughout development,
deployment and modification. They do not determine device classification or legal obligations.
Appendix C describes PEL’s exploratory empathy study as a supporting example. It examined
300 answers to one scenario using two general-purpose models. It measured no patient outcomes
and had limits in rater agreement. It does not establish clinical harm, regulatory thresholds or
performance across scenarios. [2]
The evidence supports testing clinically important variation, not treating all variation as
undesirable. The literature assessment was targeted, not a systematic review. Study findings and
source positions are distinguished from PEL’s recommendations. These sources support
measurement principles; they do not validate one complete testing or monitoring approach.
Several other public comments on this docket also call for testing how much a device's output
varies on repeated or equivalent inputs, and for judging any later change against that variation.
[20–22,24] PEL agrees. That commenters with different interests reach the same view supports
FDA setting clear expectations for how sponsors measure and report such variation. PEL's
specific proposals are to report three questions separately (whether outputs meet the claim's
criteria, whether they change on repetition and whether they hold across relevant input changes),
to retain every output, and to use existing reporting checklists as the baseline for disclosure.

3. Response to Question 9 — Device benchmarking
FDA question — Section V.E, printed pp. 17–18. [1]
Would a benchmarking structure such as the one described above be likely to provide adequate evidence of clinical
knowledge, analytic capabilities, safety behavior, communication, and generalizability to support a reasonable
assurance of safety and effectiveness? Are there elements that are missing, redundant, or inappropriately
categorized? Are there externally developed standards that could be leveraged?

Recommendation
FDA’s proposed benchmarking structure is useful, but adequate evidence depends on fit-for-use
tests and justified acceptance criteria, including repeated-output testing when variation could
affect safety or effectiveness. Device benchmarking means structured non-clinical testing of the
device in its deployed or representative configuration using defined tests and datasets. The
headings alone cannot establish adequate evidence.

Evidence and reasoning
FDA already includes repeated runs within its generalizability element. It also distinguishes
acceptable wording differences from failures in safety-critical behavior. PEL’s proposal adds a
reporting method to that existing concern. [1, Appendix A, R.1, pp. 25–26] For a care-urgency

Page 6 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

claim, S.1 concerns recognition and escalation, R.1 concerns behavior across repetitions and
other variations, and R.2 concerns relevant patient groups. These are proposed elements, selected
according to intended use and risk. [1, section V.B.1, p. 12; Appendix A, pp. 24–26]
Franc and colleagues studied ChatGPT-4 on 391 simulated disaster-triage cases using nine
prompts. Each combination was repeated ten times. They assessed correctness against an expert
reference separately from variation across repeated answers and prompts. This demonstrates a
feasible distinction, within one simulated task and an older model. It does not establish
performance of current medical devices. [3]

Implementation
Define the clinical behavior and its acceptable alternatives before testing. For example,
differently worded answers may give the same appropriate care advice. A recommendation that
changes the required urgency of care needs separate assessment. Appendix D illustrates this
distinction with a hypothetical example.
Report three questions separately: whether the answer meets clinical criteria; whether that
judgment changes on repetition; and whether performance holds across relevant input changes.
Describe what was fixed and changed. Record the device version, instructions, external
information, generation settings and test dates.
Retain every evaluable output under prespecified inclusion rules. Report refusals, incomplete
answers and technical failures separately. Present counts for each score category and clinically
important error type. A consistent error should remain visible even when an overall score
improves. Appendix E shows the information PEL proposes reporting for each claim; it is not an
FDA-required form. The number of cases and repeated outputs can scale with the risk of the
claim and how much outputs vary in practice; a sponsor using fewer should explain why they are
enough to answer each of the three questions.

Limitations and unresolved issues
We do not recommend a new universal competency or compulsory named metric. Repeated tests
of one case cannot establish performance across patients or tasks. Where a sponsor shows that a
safety-critical behavior, such as escalation, is set by fixed rules that give the same result for the
same input, rather than by generated text, checking that design and running targeted repeated
tests may be sufficient for that behavior, as one deployer’s comment proposes. Generated content
that could still affect the claim needs repeated-output testing. [23] Established measurement
guidance can discipline terminology, but device-specific validation is still needed. No source
reviewed here establishes a universal GenAI benchmarking standard. FDA asked which
externally developed standards could be leveraged. Two research reporting checklists already ask
authors to disclose how they handled variation across repeated outputs. TRIPOD-LLM asks for
inference settings and the consistency of outputs to be reported. MI-CLEAR-LLM asks how
stochasticity was handled, including the number of query attempts and how multiple results were

Page 7 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

combined; a review applying that checklist found such documentation in 24 of 159 medical LLM
studies. The joint FDA, Health Canada and MHRA guiding principles for good machine learning
practice refer to variability in device outputs and to statistically sound test plans, without a
method for repeated outputs. These documents support disclosure of repeated-output handling.
None specifies how a sponsor should choose cases and repeats, score outputs or report case-level
and output-level results for a device claim. PEL reviewed their published checklists and principle
statements, not their full explanatory material. [15–18]
Basis: FDA’s proposed benchmarking elements; Franc et al.’s simulated study; PEL’s measurement
recommendations. [1,3]

4. Response to Question 10 — Benchmark validity
FDA question — Section V.E, printed p. 18. [1]
CDRH recognizes that publicly available benchmarking assets may be subject to data contamination, saturation, and
limited real-world representativeness. How should a sponsor establish that performance on a given benchmark
predicts safe and effective real-world behavior for the device’s intended use? What evidence should support the
construct validity of a benchmark used to gate device evaluation, and what role should sponsor-developed
benchmarks play given potential concerns around independence and optimization to the test?

Recommendation
Sponsors should support a benchmark’s claimed prediction of clinical behavior with evidence
that its cases and scores represent the intended task, followed by checks in actual or clinically
representative use. FDA should ask sponsors to justify each inference drawn from a benchmark
score. That justification should connect the intended clinical behavior, test cases, scoring process
and evidence from actual use.

Evidence and reasoning
Construct validity concerns whether a test measures the concept it claims to measure. The
National Institute of Standards and Technology (NIST) identifies this challenge in AI evaluation.
It treats prediction of performance in other settings as a further question. [4, sections I-A and IB]
Alaa and colleagues argue for empirical checks of medical benchmark validity. Their position
paper reports proof-of-concept gaps between benchmark constructs and clinical tasks. This
supports examining the inference behind a score. It does not show that every benchmark is
invalid. [5]
A score can be stable while measuring the wrong behavior. PEL therefore recommends separate
evidence for consistency and construct validity. Neither alone establishes safe and effective
clinical use. [4,5] A benchmark is also a measuring instrument with limited resolution. A
generative device can score differently each time it is run on the same cases, so a difference
smaller than that variation shows nothing. A benchmark with too few relevant cases or reruns

Page 8 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

may also fail to detect a change that matters clinically, so stable scores do not by themselves
show that safety-relevant behavior is unchanged. [25]

Implementation
Start with a specific claim, such as recognizing when urgent clinical assessment is needed.
Identify the intended users, patients, setting and available information. Explain how the cases
and scoring criteria test that claim. Separate factual knowledge from clinical decision
performance and user understanding.
Then compare benchmark results with independently assessed cases reflecting clinical use.
Include important patient groups, intended users and difficult conditions, with validation across
relevant institutions and clinical settings proportionate to the claim. Describe how the test data
and assessments are independent of development, including how those data were withheld from
training, tuning and model selection. [9, section IV.B; 19, section VIII, pp. 19 and 22] Listing
these dimensions alone does not establish generalizability. Examine whether benchmark errors
and clinical errors align. Where a numerical prediction is claimed, evaluate that prediction on
data withheld from development. Report uncertainty and important failures, not only average
agreement. Where a benchmark will be used to compare devices or versions, state in advance the
smallest clinically meaningful difference and justify it. Show that the benchmark, with the
planned cases, reruns and scoring, can detect a difference of that size. Then report each result
with the range its score shows across reruns, and treat a difference within that range as no
demonstrated difference, not as equivalence. [25]
Sponsor-developed tests may fill gaps in public tests. Their provenance and separation from
development should be documented. Prespecify the cases or sampling process, scoring rules and
analysis. Prevent testing from becoming repeated optimization to a known test. Use independent
clinical review where the consequences justify it.

Limitations and unresolved issues
A plausible construct and a reproducible score do not establish patient benefit. Clinical
confirmation means further evidence from actual or clinically representative use. The necessary
evidence depends on the intended claim and risk. We propose no universal correlation threshold
or single validation study design.
Basis: NIST’s measurement-science discussion; Alaa et al.’s position paper; PEL’s proposed implementation. [4,5]

5. Response to Question 12 — Performance and uncertainty
FDA question — Section V.E, printed p. 18. [1]
How should sponsors achieve statistically meaningful performance measurement for GenAI-enabled devices?
Where synthetically generated inputs supplement real patient data, how should sponsors account for differences
between the synthetic and real-world distributions when estimating performance, and under what conditions, if any,
is it appropriate to combine benchmarking evidence and clinical confirmation evidence to support a single
performance estimate?

Page 9 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Recommendation
Sponsors should define the performance quantity, account for case sampling, repeated outputs
and scoring, and justify the uncertainty acceptable for the decision. Analyze synthetic and real
patient inputs separately first. Combine benchmarking and clinical-confirmation evidence only if
they estimate a common quantity under justified, compatible conditions; otherwise report
separate estimates. These conditions are detailed below.

Evidence and reasoning
Repeated answers from one case add information about that case. They do not add new patients
or clinical scenarios. Franc et al. evaluated variation across cases, prompts and repeated answers.
PEL infers that sampling should distinguish these sources when they affect the clinical
conclusion. That study does not establish a universal sampling design. [3]
Zero observed errors do not establish zero error probability. For illustration, suppose 50
independent outputs share the same error probability. If none contains the prespecified error, the
exact one-sided 95% upper bound is about 5.8%. This uses a binomial model for independent
yes-or-no outcomes. It describes one fixed test condition, not overall device safety. [6]

Implementation
Prespecify the intended population, setting and event being measured. State whether the
denominator is patients, cases, conversations or outputs. Define handling of missing, refused and
incomplete outputs. Distinguish the frequency of an unacceptable answer from the probability of
subsequent patient harm.
Justify both the number of distinct cases and repetitions within cases. Base the design on
clinically important error frequencies and the uncertainty acceptable for the decision. Account
for dependence among answers from the same case. Suitable methods may resample whole cases
or model variation within and between cases. Check their assumptions and performance for the
proposed design. FDA’s draft guidance on AI-enabled devices treats repeatability and
reproducibility as sources of output uncertainty, framed around repeated measurements of the
same patient. For generative outputs, PEL applies the same reasoning to repeated answers to the
same case. [19, section X.A, p. 28; Appendix C, p. 45]
Report category counts, denominators and uncertainty for clinically important outcomes and
patient groups. Give the confidence level and interval method. When making several acceptance
decisions, explain how the analysis limits false conclusions across them. Justify additional
sampling for rare but consequential errors. FDA’s draft guidance already asks that performance
estimates be reported with confidence intervals and across important patient subgroups. [19,
sections VI.B and VIII, pp. 14 and 23] When no difference between patient groups is found, state
the smallest difference the analysis could have detected; without it, a null result does not show
comparable performance. [25]

Page 10 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Analyze synthetic inputs separately first. These are artificially generated cases. Compare their
clinical content and frequencies with real patient inputs. Report gaps in case severity, patient
groups and missing information. Artificially generated cases should not receive the same weight
merely because they are numerous. They remain useful for rare, adversarial and safety-critical
situations that real data seldom supply, provided their labels are verified independently of any
model that generated them.
Combine benchmarking and clinical-confirmation evidence only when both estimate a clearly
defined common quantity. Justify compatible populations, device versions, scoring and
observation conditions. Explain any weighting and account for overlapping cases or other
dependence. Repeat the analysis with reasonable alternative weights and with real patient cases
alone. If those conditions fail, report separate estimates and explain their different purposes.

Limitations and unresolved issues
We do not propose a fixed number of repetitions; Appendix D shows how a sponsor could report
case-level and output-level results together with their uncertainty. More repeats cannot correct
unrepresentative cases or invalid scoring. The zero-error illustration above assumes independent
outputs with an unchanged error probability and error-free classification. Violations require a
different analysis. Small samples may remain uninformative for rare harms.
Basis: Franc et al.; NIST binomial interval methods; FDA’s draft guidance on AI-enabled devices; PEL’s statistical
illustration and design recommendations. [3,6,19]

6. Response to Question 14 — Comparators and scoring
FDA question — Section V.E, printed p. 18. [1]
For open-ended device outputs, how should performance comparators and acceptance criteria be selected? When a
panel of qualified clinicians serves as the comparator, how should the applicable standard (for example, the standard
of care versus the performance of a median clinician in practice) be defined and justified? Should generalist or
specialist physicians be used as a performance standard? When should human-AI team performance, rather than the
device operating alone, serve as the basis for evaluation?

Recommendation
Sponsors should select and justify comparators, clinical standards and acceptance criteria for the
intended use before testing, then test whether the scoring process applies them reliably. A
comparator is the care, judgment or system used for comparison. Acceptance criteria are the
conditions set beforehand for judging performance adequate. Neither should be chosen from the
observed device results.

Evidence and reasoning
FDA recognizes that open-ended outputs may have several acceptable answers. A panel’s rating
therefore depends on both clinical judgment and the scoring rules. [1, section V.D.1, p. 16]

Page 11 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Yang and colleagues’ review included 71 studies of human evaluation in health care. They report
evaluator subjectivity and a lack of standardized evaluation measures. This describes published
practice; it does not identify one best scoring method for all devices. [13]
Mallinar and colleagues compared scoring approaches for personalized health answers. Their
study reports improved agreement with more specific criteria. It supports testing rubric design
within the relevant task. It does not establish one scoring format as universally superior or
demonstrate patient benefit. [7]
Williams and colleagues compared five AI evaluators with six clinicians using questions from
Rwandan health workers. The best-performing AI evaluator matched human ratings on four of
eleven criteria. The authors also report poorer performance when moving from English to
Kinyarwanda. These findings concern the tested tasks and languages, not all automated scoring.
[14]

Implementation
Specify what clinical standard the panel applies and why it suits the intended use. Select
generalists or specialists according to the decisions being assessed. Include expertise needed to
judge clinically important exceptions. Record conflicts of interest, training, blinding and rules for
resolving disagreement. Where expert adjudication is warranted, document structural
independence from the sponsor and any third-party model developer, as FDA’s discussion paper
contemplates. State who appoints, funds and can influence the panel. Structural independence
concerns those relationships; blinding concerns what raters see. Neither alone prevents shared
clinical error. [1, section V.B.2, p. 14]
Use a scoring rubric: explicit criteria for judging the answers. Test it on examples separate from
the main evaluation. Then have raters independently score a justified sample before discussing
disagreements. Report agreement, uncertainty and disagreement patterns before consensus. Use a
prespecified adjudication procedure, including an additional qualified reviewer where needed.
Preserve initial ratings, reasons for revised labels and unresolved cases; allow justified
alternative answers rather than forcing consensus. Consensus labels do not replace evidence
about initial agreement or establish clinical correctness.
Acceptance criteria should reflect clinical consequences and acceptable alternatives. Do not treat
the median clinician’s performance as an automatically acceptable safety threshold. Where a
device is intended for settings in which no clinician is available, a sponsor may also justify
comparison with the user’s realistic alternative, such as self-directed internet searching or no
action. That comparison describes added value; it does not replace a clinical standard for judging
serious errors. [23] Keep serious errors visible rather than allowing unrelated strengths to cancel
them out. Distinguish the error definition from the passing rule. Prespecify the performance
measure, clinically justified limit, treatment of uncertainty and handling of failed or
unclassifiable attempts; explain what result would fail the claim. Appendix D illustrates that
distinction without choosing a universal numerical cutoff. This is consistent with FDA’s draft

Page 12 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

guidance, which expects prespecified acceptance criteria and a comparator evaluated on the same
patients or data as the device, such as clinicians, an adequately validated device or standard of
care. [19, Appendix C, pp. 45–46]
If the device claim depends on clinicians using its outputs, assess their combined performance.
Also inspect device outputs to locate failures. Device-alone testing cannot establish how
clinicians will act. FDA’s draft guidance makes the same point, citing the good machine learning
practice principle on human–AI team performance. [15; 19, section X.A, p. 28]
For automated scoring, validate each criterion against an appropriate clinical reference process
using outputs withheld from scorer development or tuning. Test the intended tasks, patient
groups and languages. Report criterion-specific agreement and disagreement with uncertainty,
missed consequential errors and false error flags. Document the scorer version, instructions,
reference provenance and any shared model or data sources that could create correlated errors.
Revalidate after a scorer change and assess whether new device outputs fall outside the validated
scoring conditions. PEL proposes these checks; the cited studies motivate task-specific validation
but do not establish universal agreement thresholds or validate this complete procedure. [7,14]

Limitations and unresolved issues
Agreement does not prove clinical correctness. Several clinicians can share the same error.
Patient or user input can help define communication needs, but it does not replace clinical
judgment for safety questions. We propose no universal agreement cutoff or required panel size.
Basis: FDA’s comparator discussion; Yang et al.; Mallinar et al.; Williams et al.; FDA’s draft guidance on AIenabled devices; PEL’s proposed scoring checks. [1,7,13,14,15,19]

7. Response to Question 19 — Postmarket monitoring
FDA question — Section VI.D, printed p. 21. [1]
Please comment on the potential approaches to postmarket performance evaluation, including periodic rebenchmarking, sample-based clinician review, and performance degradation monitoring. What additional
approaches should CDRH consider, and how should the cadence and triggering events for reassessment be
determined?

Recommendation
FDA should use periodic re-benchmarking, clinician review of current use and performancedegradation signals together, with reassessment timing and actions tied to the claims and
consequences of missed deterioration. Combine repeat testing on retained cases with review of
current clinical use. The plan should identify who can investigate, restrict use, require
revalidation or suspend the affected function.

Page 13 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Evidence and reasoning
FDA’s research program distinguishes changes in inputs, changes in output performance and
their causes. This distinction matters because changed inputs do not necessarily mean worse
performance. Stable inputs also do not prove that performance remains adequate. [8]
PEL infers that two sources of evidence are needed. Retained cases can support comparisons
over time. Current cases can reveal changes in use, patients or clinical conditions. Either source
alone can miss relevant deterioration.
On the three approaches FDA names: periodic re-benchmarking on retained cases is necessary
for comparison over time but cannot detect changes in the patients or questions the device now
receives. Sample-based clinician review of current use can detect those changes, provided the
sampling process is documented and consequential errors are reviewed in addition to a
representative sample. Performance degradation monitoring from routine signals is useful as a
trigger but should not replace the first two until its detection delay and false-alarm rate for
clinically important deterioration have been shown for the device’s task. PEL supports all three
in combination rather than any one alone.

Implementation
Keep a versioned set of clinically justified cases and scoring rules for repeat evaluation. Add
current cases through a documented sampling process. Include a representative sample and
targeted review of consequential errors or complaints. Keep those samples distinct when
estimating frequencies.
Monitor the frequency and type of unacceptable outputs, alongside average performance where
meaningful. Record denominators, missing records, device versions and relevant patient groups.
Review whether scoring practices have changed. Repeated-output testing should continue where
run-to-run variation affects the original claim.
Specify who reviews each signal, how quickly they act and who has authority to decide the
response. For example, assign the clinical safety lead to assess serious clinical errors and
recommend immediate safeguards, the quality lead to coordinate investigation and revalidation,
and an accountable product or medical decision maker to authorize restriction, suspension and
resumption. Prespecify the evidence and timing for each action, proportionate to harm and
uncertainty. Triggers may include a serious error, a material change in users or inputs, and
changes to device components. A serious signal may require immediate investigation and interim
restriction or suspension without waiting for an aggregate statistical threshold. These are
proposed roles and responses, not universal trigger thresholds.
Choose routine intervals using expected exposure, error frequency, severity and the delay before
harm. Explain the monitoring method’s likelihood of missing deterioration and generating false
alarms. Where outputs vary on repetition, measure that variation for each monitored measure: a
trigger set within it will fire on noise, and the sampling volume determines the smallest

Page 14 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

deterioration the plan can detect. Re-measure it after a model change. [25] Small deployments
may require longer observation, more targeted review or pooling of comparable data across sites,
with the pooling method stated in advance, to obtain useful evidence. Preserve clinician review
of signals until an automated approach has adequate evidence for its task. FDA’s draft guidance
describes a postmarket performance monitoring plan as part of the quality system; PEL’s
recommendation adds named decision owners and prespecified actions. [19, section XI, pp. 32–
33]

Limitations and unresolved issues
We do not propose a universal calendar interval. Monitoring has incomplete records, delayed
outcomes and imperfect detection. Our recommendations do not establish when reduced
premarket evidence would be acceptable. A monitoring plan cannot compensate for a test that
never measured the relevant clinical behavior.
Basis: FDA’s postmarket discussion and research program; PEL’s proposed combination of retained and current
cases. The proposed combination is not validated by the cited research-program overview. [1,8,19]

8. Response to Question 22 — Evaluation after modification
FDA question excerpt — Section VI.D, printed pp. 21–22; opening statement and measurement question. [1]
CDRH envisions that a manufacturer’s premarket competency-based assessment might serve as a baseline against
which post-deployment modifications could be re-evaluated. How might the extent of re-benchmarking or other
evidence be scaled to the nature and expected impact of a given modification?

Recommendation
The extent of re-benchmarking should follow the modification’s plausible clinical effects: use
targeted comparisons when its impact is well bounded and broader testing when it is uncertain.
FDA should ask sponsors to preserve comparable measurements when evaluating a modification.
This response addresses measurement, not which changes require FDA authorization.

Evidence and reasoning
FDA’s final guidance on a predetermined change control plan (PCCP) describes planned
modifications, a modification protocol and an impact assessment. A PCCP documents specified
changes and how they will be evaluated. The guidance provides context for documenting a
change; it does not establish that any particular GenAI update is covered by an authorized PCCP.
Eligibility and authorization are separate from the measurement proposal here. [9, sections III,
V.A and VIII]
PEL’s proposed comparison follows a measurement principle. If the test cases and scoring rules
change alongside the device, observed differences have several possible causes. The evaluation
should make those causes distinguishable where feasible.

Page 15 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Implementation
Describe the modification and the clinical behaviors it could affect. Include changed instructions,
information sources, safeguards and underlying models. Do not assume that a change outside
model training has negligible impact.
For the main comparison, evaluate both versions on the same retained cases using unchanged
scoring. Repeat outputs where relevant and document test conditions. Compare the observed
difference with the variation the unchanged version shows against itself; a difference within that
range is not evidence of change. [25] Compare clinically important error frequencies and score
categories, including patient groups. Justify acceptance limits before reviewing the comparative
results. Appendix D includes a hypothetical example of this comparison after a change to a
device’s underlying model or information sources.
Add new cases for newly affected uses or failure modes. Explain why targeted tests are sufficient
when the expected impact is narrow. Even then, rerun a fixed set of safety-critical cases
unchanged against the original baseline. [26] Broaden evaluation when the change affects
clinical content, important safeguards or several interacting components. Additional clinical
evidence may be needed when the original claim no longer matches the modified device. These
comparisons can be carried out within a sponsor's existing regression testing and, where one is
authorized, its predetermined change control plan, rather than as a separate exercise.
If the scoring method changes, apply both scoring methods to a common set of outputs. This
helps distinguish a device change from a measurement change. If the previous device version is
unavailable, record what can still be compared. Archived outputs support limited checks but
cannot fully recreate the previous system. Where a third party supplies the underlying model,
record the model version used in every evaluation and agree with the supplier how changes will
be notified; treat an unannounced change as a modification.

Limitations and unresolved issues
An absence of statistically significant differences is not evidence of equivalent performance.
Interpret results against prespecified clinically justified limits and their uncertainty. We do not
assign modifications to a regulatory pathway or decide their eligibility for a predetermined
change control plan.
Basis: FDA’s discussion paper and final change-control guidance; PEL’s recommendations on comparable
measurement. These sources provide context, not validation of one GenAI change-testing method. [1,9]

9. Research needs and conclusion
Three research needs limit more prescriptive recommendations. First, studies should compare
designs that vary both case diversity and output repetitions. Second, scoring studies should test
whether agreement also supports clinically correct judgments. Third, monitoring studies should
measure detection delay and false alarms for clinically important deterioration.

Page 16 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

These studies need multiple devices, tasks and intended settings. Patient outcomes are needed
before treating a communication score as evidence of patient benefit or harm. PEL asks FDA to
encourage transparent, task-specific justification of measurement choices while these methods
develop.

10. Interests, funding and contributions
PEL develops methods for evaluating repeated model outputs. PEL and contributors to its
empathy research have an intellectual and potential commercial interest in that work. We do not
request endorsement of a proprietary method or recommend PEL as an evaluator.
The cited exploratory draft reports no external or commercial funding for that study. It also
reports no financial relationship with the evaluated model developers. The second rater is the
spouse of PEL’s founder. She scored the selected answers blinded to model, generation setting
and the first rater’s scores. [2, declarations]
This response received no external funding or sponsorship. It was prepared by People’s Evidence
Lab Inc. at its own expense. Stephen J. Watt, founder and principal of PEL, and Maya Watt,
collaborator, hold an intellectual and potential commercial interest in the Empathy Reliability
Index; PEL is developing the method. Stephen J. Watt is responsible for final approval and
submission.
AI assisted source retrieval, drafting and formatting of this document. PEL retains responsibility
for the final scientific content and submission decision.

11. References
Source checks were conducted on September 16, 2026; references 15 to 18 were checked on October 7, 2026;
references 20 to 26 were checked on October 17, 2026. This was a targeted assessment, not a systematic review or
full appraisal of every paper. Reference 2 is an unpublished author-supplied draft. The reviewed material and access
limits are identified below.

1. U.S. Food and Drug Administration. Considerations for the Regulation of Generative AIEnabled Medical Devices: Discussion Paper and Request for Feedback. August 2026. Printed pp.
1–30.
https://www.fda.gov/media/194242/download
https://www.fda.gov/medical-devices/digital-health-center-excellence/considerations-regulation-generative-aienabled-medical-devices-discussion-paper-and-request

2. Watt M, Watt SJ, Nehal US. The Empathy Reliability Index (ERI): A Distribution-Based
Method for Evaluating the Consistency of Large Language Model Responses to Health-Related
Support-Seeking. Paper presented at the DCI Network conference “Signal Through the Noise:
From Proof to Practice,” Boston, September 23–25, 2026. Accepted for the conference
proceedings; publication pending at the time of this response, so no public URL or DOI is
available. Statements here are drawn from the accepted proceedings paper (version 0.13).
Appendix C gives the methods, evidence limits and findings used here.

Page 17 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

3. Franc JM, Hertelendy AJ, Cheng L, Hata R, Verde M. Accuracy of a Commercial Large
Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple
Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study. J
Med Internet Res. 2024;26:e55648. doi:10.2196/55648. Abstract reviewed.
https://pubmed.ncbi.nlm.nih.gov/39348189/
https://doi.org/10.2196/55648

4. National Institute of Standards and Technology. Accelerating AI Innovation Through
Measurement Science. CAISI Research Blog. Sections I-A, I-B and II. Institutional research
discussion, not a binding standard.
https://www.nist.gov/blogs/caisi-research-blog/accelerating-ai-innovation-through-measurement-science

5. Alaa A, Hartvigsen T, Golchini N, et al. Position: Medical Large Language Model
Benchmarks Should Prioritize Construct Validity. Proceedings of Machine Learning Research.
2025;267:80991–81004. Position paper with proof-of-concept experiments. Abstract reviewed.
https://proceedings.mlr.press/v267/alaa25a.html

6. NIST/SEMATECH. e-Handbook of Statistical Methods. Section 7.2.4.1, Confidence intervals.
Exact binomial limits and distinction between one-sided and two-sided limits.
https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm

7. Mallinar N, Heydari AA, Liu X, et al. A scalable framework for evaluating health language
models. npj Digital Medicine. 2026;9:437. doi:10.1038/s41746-026-02492-x. Research article.
Relevant Results and Methods reviewed.
https://www.nature.com/articles/s41746-026-02492-x

8. U.S. Food and Drug Administration. Methods and Tools for Effective Postmarket Monitoring
of Artificial Intelligence (AI)-Enabled Medical Devices. Research program overview.
https://www.fda.gov/medical-devices/medical-device-regulatory-science-research-programs-conductedosel/methods-and-tools-effective-postmarket-monitoring-artificial-intelligence-ai-enabled-medical-devices

9. U.S. Food and Drug Administration. Marketing Submission Recommendations for a
Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software
Functions. Final guidance. August 2025. Official overview and full guidance sections III, IV.B–
C, V.A and VIII reviewed; printed pp. 5–9, 11 and 30.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submissionrecommendations-predetermined-change-control-plan-artificial-intelligence
https://www.fda.gov/media/166704/download

10. U.S. Food and Drug Administration. Digital Health and Artificial Intelligence Glossary —
Educational Resource. Synthetic Data entry. Educational definitions, not agency guidance.
https://www.fda.gov/science-research/artificial-intelligence-and-medical-products/fda-digital-health-and-artificialintelligence-glossary-educational-resource

Page 18 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

11. NIST/SEMATECH. e-Handbook of Statistical Methods. Section 1.3.5.2, Confidence Limits
for the Mean.
https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm

12. Sharma A, Miner AS, Atkins DC, Althoff T. A Computational Approach to Understanding
Empathy Expressed in Text-Based Mental Health Support. Proceedings of EMNLP. 2020:5263–
5276. doi:10.18653/v1/2020.emnlp-main.425. Abstract reviewed.
https://aclanthology.org/2020.emnlp-main.425/

13. Yang E, Ko S, Woo H. The Reliability of Human Evaluation of Large Language Models in
Health Care Settings: Scoping Review. J Med Internet Res. 2026;28:e98184. doi:10.2196/98184.
Publisher-indexed excerpts and citation record checked; full text was inaccessible.
https://www.jmir.org/2026/1/e98184/
https://pubmed.ncbi.nlm.nih.gov/42618044/

14. Williams G, Rutunda S, Nzabakira F, Mateen BA. Human evaluators vs. LLM-as-a-Judge:
toward scalable evaluation of GenAI in global health. npj Digital Medicine. Published July 20,
2026. doi:10.1038/s41746-026-02992-w. Peer-reviewed accepted article, early-access version.
Publisher abstract and publication notice reviewed.
https://www.nature.com/articles/s41746-026-02992-w

15. U.S. Food and Drug Administration, Health Canada, Medicines and Healthcare products
Regulatory Agency. Good Machine Learning Practice for Medical Device Development:
Guiding Principles. October 2021. Ten principle statements reviewed; principles 6 and 8 cited.
https://www.fda.gov/media/153486/download

16. Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies
using large language models. Nat Med. 2025;31(1):60–69. doi:10.1038/s41591-024-03425-5.
Published checklist reviewed; items 6c and 7a cited. Explanatory material not appraised.
https://www.nature.com/articles/s41591-024-03425-5

17. Park SH, Suh CH, Lee JH, Kahn CE, Moy L. Minimum Reporting Items for Clear Evaluation
of Accuracy Reports of Large Language Models in Healthcare (MI-CLEAR-LLM). Korean J
Radiol. 2024;25(10):865–868. doi:10.3348/kjr.2024.0843. Item 2 reviewed. The 2025 update
(Park SH, et al. Korean J Radiol. 2025;26(12):1123–1132. doi:10.3348/kjr.2025.1522) retains
stochasticity management as a reporting category; its text on that item was reviewed and is
consistent with the statement here.
https://kjronline.org/DOIx.php?id=10.3348%2Fkjr.2024.0843

18. Ko JS, Heo H, Suh CH, Yi J, Shim WH. Adherence of Studies on Large Language Models
for Medical Applications Published in Leading Medical Journals According to the MI-CLEARLLM Checklist. Korean J Radiol. 2025;26(4):304–312. doi:10.3348/kjr.2024.1161. Abstract
reviewed; 24 of 159 studies documented stochasticity handling.

Page 19 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

https://kjronline.org/DOIx.php?id=10.3348%2Fkjr.2024.1161

19. U.S. Food and Drug Administration. Artificial Intelligence-Enabled Device Software
Functions: Lifecycle Management and Marketing Submission Recommendations. Draft
guidance, not for implementation. January 7, 2025. Sections VI.B, VIII, X.A and XI and
Appendix C reviewed; printed pp. 14, 19, 22–23, 28, 32–33 and 45–46.
https://www.fda.gov/media/184856/download

20. Bellibatlu RR. Public comment FDA-2026-N-7874-0095. Posted September 15, 2026.
Comment text and attachment reviewed; attachment pp. 1–3, sections "What I am
recommending" and Questions 10, 12, 13 and 19.
https://www.regulations.gov/comment/FDA-2026-N-7874-0095

21. VitaSignal. Public comment FDA-2026-N-7874-0048. Posted September 1, 2026. Comment
text reviewed; item 2.
https://www.regulations.gov/comment/FDA-2026-N-7874-0048

22. UltraAI. Public comment FDA-2026-N-7874-0126. Posted September 30, 2026. Attachment
partly reviewed; section II.B.
https://www.regulations.gov/comment/FDA-2026-N-7874-0126

23. Clearstep Inc. Public comment FDA-2026-N-7874-0100. Posted September 15, 2026.
Attachment reviewed; sections II.B (Question 9) and II.D (Questions 14 and 15).
https://www.regulations.gov/comment/FDA-2026-N-7874-0100

24. Eltayeb Y. Public comment FDA-2026-N-7874-0091. Posted September 15, 2026. Comment
text and attachment reviewed; attachment sections 3, 6 and 7. The commenter discloses a
commercial interest.
https://www.regulations.gov/comment/FDA-2026-N-7874-0091

25. NIST/SEMATECH. e-Handbook of Statistical Methods. Section 7.2.2.2, Sample sizes
required; section 2.4.4.1, Analysis of repeatability.
https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

26. U.S. Food and Drug Administration. General Principles of Software Validation; Final
Guidance for Industry and FDA Staff. January 11, 2002. Sections 4.7 (p. 12) and 5.2.5 (pp. 24–
25). Section 6 is superseded by the September 24, 2025 Computer Software Assurance guidance;
the cited sections remain current.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/general-principles-software-validation

12. Signatory and public contact
Stephen J. Watt
Founder and Principal
People’s Evidence Lab Inc.

Page 20 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Public contact: stevewatt13@peoplesevidencelab.com

Page 21 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Appendix A. Glossary and terminology justification
Scope: terms used in PEL’s recommendations. Version 0.3, October 15, 2026. Definitions are paraphrases unless
identified as contextual interpretations or scope notes. Source authority and limits are shown below. FDA
discussion-paper usage is not an adopted standard. Appendix B gives PEL’s proposed working meanings for three
descriptive expressions.

Term Definition used and source Authority and qualification
Generative artificial AI that generates content from learned patterns.
FDA explanation; paraphrase.
intelligence Includes text and other media. [1, §III]
GenAI-enabled Product containing a medical-device software FDA explanation; technology alone
device function enabled by generative AI. [1, §III] does not establish device status.
Model trained on extensive text to learn language FDA explanation; component model
Large language model
patterns and generate answers. [1, §III] differs from configured device.
Used in the context of the cited FDA source; PEL
does not propose a separate definition. The purpose
Source-context note, not a local or
Intended use of an assessment and a particular performance
legal definition.
claim are not substitutes for the device’s intended
use. [1, §§I–III]
Structured non-clinical testing of the deployed or
Device benchmarking representative device configuration using defined FDA consultation usage.
tests and datasets. [1, §V.B]
Whether relevant performance holds across
Generalizability expected changes in users, inputs and conditions. FDA consultation usage.
[1, Appendix A]
Whether a test measures its intended concept. [4, I- NIST explanation; does not establish
Construct validity
A] clinical benefit.
Further evidence about performance in actual or FDA proposed approach; not always
Clinical confirmation
clinically representative use. [1, §V.C] a prospective trial.
Contextual reading of FDA;
Explicit criteria used to judge and score outputs. [1,
Scoring rubric presence alone does not establish
§V.B.2]
validity.
Conditions set beforehand for deciding whether Contextual reading of FDA; no
Acceptance criteria
performance is adequate. [1, §V.B.2] universal cutoff.
Care, judgment, system or output against which Contextual reading of FDA; choice
Comparator
performance is assessed. [1, §V.D.1] requires justification.
Contextual interpretation of FDA
Output for which several answers may be
Open-ended output usage; acceptable alternatives still
acceptable. [1, §V.D.1]
require clinical criteria.
Artificially generated data intended to reproduce FDA educational definition; does
Synthetic data
structures or relationships in observed data. [10] not prove representativeness.
Range from a procedure with stated long-run Statistical usage; not a probability
Confidence interval
coverage under its assumptions. [11] assigned to the fixed true value.
Documentation specifying planned device FDA guidance usage; a proposed
Predetermined change
modifications and how they will be assessed. [9, plan is distinct from an authorized
control plan (PCCP)
§IV.C] PCCP.
FDA guidance usage; authorization
A PCCP reviewed and established through a device
Authorized PCCP and coverage of a particular change
marketing authorization. [9, §IV.C]
are separate questions.

Page 22 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Term Definition used and source Authority and qualification
A model for the number of events in a fixed number
Statistical usage; assumptions must
Binomial model of independent binary trials with common event
be justified.
probability. [6]
Reporting-guideline usage;
Randomness in generation that can produce distinguish generation randomness
Stochasticity different outputs when the same input is repeated. from changes in the model or input
[17] data. A fixed setting alone is not
evidence of identical outputs.
These terms identify distinct evaluation concepts. Appendix B gives PEL’s proposed working meanings for
expressions used to organize the recommendations; no new scientific construct or universal metric is proposed. Full
source details and visible URLs are in section 11; that reference list must accompany any separate copy of this page.

Appendix B PEL working definitions
The following descriptive expressions state how PEL uses these phrases in this response. They
do not introduce a new scientific construct, an FDA definition or a regulatory requirement.
Expression Working meaning and limits

The device as evaluated, including the model version, instructions, information sources,
Configured device tools, safeguards and generation settings in use. Specify the actual components relevant to
each test. This is PEL assessment shorthand, not an FDA definition.

A specific statement about what the configured device is expected to do, for whom and
Performance claim under which conditions, that the evaluation will assess. State the behavior, population and
setting. This is PEL planning usage, not a legal labeling determination.

Generating and assessing multiple outputs for the same defined case under recorded
Repeated-output
conditions to characterize variation relevant to the performance claim. Distinguish withintesting
case variation from variation between cases. No universal repeat count is implied.

These meanings clarify the existing recommendations in Sections 2–5. Appendix D defines care
categories only for its hypothetical example; those categories are not proposed as a general
clinical classification.

Appendix C. Supporting material

Bounded example from PEL’s exploratory empathy study
PEL’s exploratory study provides an example of repeated sampling. It was presented at a
September 2026 conference and is awaiting publication in the conference proceedings; it has not
been independently replicated and is not independent validation of these recommendations. The
full draft and underlying data were not reanalyzed for this response. [2]
The study used one paraphrased health-related support-seeking message from a weight-loss
context. It generated 50 answers for each of six model-and-setting conditions, yielding 300
answers. The models were ChatGPT-4o and Grok-4.1-Fast, accessed in December 2025 through
OpenRouter. Three generation settings were tested. These were programmatic calls, not the
consumer applications. The results concern those recorded conditions. [2, Methods]

Page 23 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Answers were scored for three forms of empathy expressed in text, using a published framework.
A second rater independently scored 72 answers, representing 24% of the corpus. The draft
reports imperfect agreement for two scored behaviors and complete agreement for the third. The
latter had no score variation, so its reported agreement statistic could not be calculated. Most
answers therefore retained only the primary rater’s labels. [2, Methods and Results; 12]
The draft reports that some scored behaviors were consistently present. Another was commonly
absent or varied across answers. Our inference is limited: repeated sampling can reveal patterns
that one answer cannot describe. These patterns do not establish whether additional questioning
was clinically needed or whether its absence harmed users. An empathy score is not a patient
safety outcome.
The study covered one scenario and two models. The highlighted descriptive measure was
selected after an initial analysis. Analyses were exploratory. Intervals were not adjusted for
assessing several results together. Agreement estimates were uncertain. No patient or
implementation outcomes were measured. The method is not a validated combined score. [2,
Discussion]
Source basis: the supplied proceedings draft’s Methods, Results, Discussion and declarations.
The study findings remain author-reported. A follow-on study (ERI Phase 2a) was prospectively
registered on October 1, 2026 (OSF Registries, https://doi.org/10.17605/OSF.IO/YRGMJ); it is
descriptive, has no results yet, and is not relied on here.

Appendix D. Worked example: repeated-output testing for an urgency-of-care claim
[This is a hypothetical illustration. The device, cases and all numbers are invented to show how
the recommendations in Sections 3 to 8 could fit together. They are not study results, proposed
thresholds or a recommended sample size.]
The device and its claim
A hypothetical configured device answers patients’ symptom questions and advises one of four
levels of care: emergency care now, same-day care, a routine appointment or self-care. The
sponsor claims that the device advises emergency care for presentations that a clinical reference
panel classifies as emergencies. Differently worded answers are acceptable if they advise the
same level of care. An answer that advises a lower level of care for an emergency case is the
prespecified unacceptable error. (Sections 3 and 4) For this example, emergency care now means
immediate emergency assessment; same-day care means assessment that day without the
emergency-now instruction; a routine appointment means non-urgent planned assessment; selfcare means home management without a scheduled clinical assessment. These are examplespecific categories, not validated triage rules.
Test design
The sponsor tests the device as deployed and records the model version, instructions, information
sources, safeguards, generation settings and test dates. The test set contains 200 clinically

Page 24 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

reviewed cases; the reference panel classifies 40 of them as emergencies. Each case is submitted
20 times under the same conditions, giving 4,000 outputs. A real sponsor would justify both
numbers using the covered presentations, clinically important error frequency and required
precision. The counts here are illustrative, not an adequate-design claim. Repeats add
information about each case; they do not add new patients. (Sections 3 and 5)
For the illustration, qualified clinicians initially label cases independently, using a prespecified
clinical standard and blinded to device outputs. The reference panel is structurally independent of
the sponsor and third-party model developer. Record qualifications, relationships and conflicts.
Resolve disagreements through documented adjudication, using an additional qualified reviewer
where needed; retain initial labels, reasons and unresolved alternatives. The invented counts
assume the 200 cases have resolved reference categories. In a real test, unresolved cases remain
visible and require a prespecified analysis; consensus is not proof of correctness.
Scoring
Each output is classified as one of the four levels of care, or recorded as a refusal, an incomplete
answer, a technical failure or an unclassifiable response. Record scheduled attempts, completed
outputs and evaluable outputs separately. Prespecify how each unsuccessful category enters the
analysis; do not silently exclude it. In this illustration, an emergency-case attempt without clear
emergency-care advice counts as a claim failure. If an automated classifier does the scoring,
validate it on held-out clinician-scored outputs, report consequential disagreements and
uncertainty, and retain scorer provenance. (Section 6)
The passing rule is separate from the error definition. Before testing, specify whether each claim
is judged by the output-error frequency, the proportion of cases with any error in a fixed number
of repeats, or both. For each chosen measure, justify the clinical limit and how its uncertainty
will enter the decision—for example, requiring a specified upper confidence limit to remain
below a justified maximum error rate. Define any serious-error rule and the treatment of
unsuccessful attempts. No numerical limit is selected here, so these invented results do not
establish a pass.
What the sponsor reports
The hypothetical results below show why the average alone is not enough. All 4,000 scheduled
attempts produce classifiable outputs; refusals, incomplete answers, technical failures and
unclassifiable responses each number zero. Overall, 3,880 of 4,000 outputs (97%) match the
reference level of care. Of the 800 outputs for emergency cases, 9 (1.125%) advise a lower level
of care and 791 match the emergency reference. The 3,200 outputs for other cases include 3,089
matches and 111 other care-level mismatches. Thus the 120 mismatches comprise 9 emergency
under-escalations and 111 other mismatches; an overall average must not hide the first group.
One invented chest-pain case describes new persistent chest discomfort with breathlessness. Its
assumed clinical reference is emergency care now. Across 20 repetitions, 16 outputs advise
emergency care now and four advise self-care. Two other emergency cases contribute three and
two lower-level answers, respectively; their remaining 17 and 18 outputs advise emergency care.
These three cases account for all nine under-escalations. The chest-pain switch illustrates a

Page 25 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

change in advised action, not harmless rewording; the vignette and labels are not published
findings or a diagnostic rule.
Counted by case, 3 of 40 emergency cases (7.5%) received at least one lower-level answer in 20
repeats. Under an illustrative binomial model for independently sampled emergency cases with a
common probability of this event, the exact two-sided 95% confidence interval is about 1.6% to
20.4%. The event is at least one error in a fixed 20-repeat test, not an error in a single patient
encounter. Report the repeat count with the case-level estimate; changing it changes the event
being measured. The interval assumes correct classification and does not establish
representativeness. A real sponsor must justify the sampling model and uncertainty or use an
appropriate alternative analysis. Report relevant patient-group results separately. (Sections 3 and
5)
After a modification
The sponsor later replaces the underlying model with a new version, or changes the device’s
information sources. It reruns the same 200 cases, 20 times each, with unchanged scoring and
documented conditions, comparing both versions where available. It compares lower-level
answers for emergency cases, by case and by output, against clinically justified limits set before
results are reviewed. It adds cases for plausibly affected presentations and failure modes, and
broadens testing when the impact is uncertain. Finding no statistically significant difference
would not, by itself, show equivalent performance. The example does not determine whether the
change is covered by an authorized PCCP. (Section 8)
Monitoring in use
In use, the sponsor reruns the retained cases on a schedule set by the consequences of a missed
emergency. It also reviews a documented sample of real conversations, including any reported
cases in which emergency care was not advised. A single serious error of this kind may require
immediate investigation, without waiting for a statistical threshold. (Section 7) The clinical
safety lead assesses the signal, the quality lead coordinates investigation and revalidation, and
the accountable product or medical decision maker decides whether to restrict or suspend the
affected function and what evidence is needed before resuming it. Continue this monitoring after
a model update; successful repeat testing alone does not establish safe use.
What this example does not show
The example does not set a number of cases or repeats, an acceptance limit or a monitoring
interval. It does not show that the advised level of care predicts patient outcomes. Its numbers are
not drawn from any device or study. The reference labels are assumptions of the illustration, and
no passing decision can be made without a clinically justified rule and adequate evidence.

Page 26 of 27
People’s Evidence Lab Inc. | Docket FDA-2026-N-7874

Appendix E Illustrative sponsor reporting form
PEL proposal for one claim and one configured device version. Not an FDA-required form.
Complete each field or explain why it does not apply; link supporting protocols and results.
Hypothetical Appendix D counts illustrate reporting, not a pass.
Claim and clinical context: [behavior and error definition; intended patients, users, institutions and
settings; clinical consequences; applicable FDA elements]
Device and test identity: [model/version; instructions; information sources; safeguards; generation
settings; test dates; baseline/update identifier]
Cases and repetitions: [sampling frame; distinct cases by source/group; repeats per case; sampling
justification; synthetic and real data separately]
Attempt accounting: [scheduled attempts; completed outputs; evaluable outputs; refusals, incomplete
answers, technical failures and unclassifiable responses; exclusions and reasons]
Reference and scoring: [clinical standard; panel qualifications, relationships and blinding; independent
initial labels; adjudication and unresolved cases; scorer version and validation on outputs withheld from
scorer development and tuning]
Results and uncertainty: [category/error counts with denominators by output and case; repeats defining
the case event; subgroup results; interval method, confidence level and dependence assumptions]
Passing rule and decision: [prespecified measure, clinically justified limit and uncertainty rule; seriouserror and unsuccessful-attempt rules; observed result, unmet criteria and decision rationale]
Change and monitoring: [comparable before/after tests; affected cases; retained and current-use sampling;
signal/action criteria; decision owners; restriction, revalidation, suspension and resumption evidence]
Evidence and accountability: [protocol/result locations; unresolved limitations; preparer, clinical
reviewer, statistical reviewer and accountable decision maker; dates and approvals]
Report what was observed separately from what is assumed. A complete form does not establish
adequate sample size, clinical validity, regulatory eligibility or device safety.

Page 27 of 27