ESSKA AI & Innovation Committee (Felix C. Oettl)

The ESSKA AI & Innovation Committee supports FDA’s competency-based approach to generative AI-enabled devices and asks for one addition: define the human standard that devices are measured against, and let clinical specialties supply it.

FDA’s process provisions are sound: prespecified acceptance criteria, independent adjudicators, and reassessment after defined events. They operate on a comparator, a guideline or a panel of clinicians, whose definition and stability the paper leaves open. Three points show why this matters.

The comparator is not one object. For total knee arthroplasty, agreement between the AAOS Appropriate Use Criteria and arthroplasty surgeons was close to zero (weighted kappa 0.06), and agreement across 25 osteoarthritis guidelines ranged from minus 0.14 to 0.91.
The comparator moves. After endoscopists adopted AI-assisted colonoscopy, their adenoma detection without it fell from 28.4% to 22.4%, so a comparator measured in 2026 may misstate the standard in 2030.
The framework examines knowledge and reasoning but has no counterpart to supervised practice, which is where surgical competence is built.

We therefore ask FDA to require four comparator disclosures, none needing new regulatory machinery: the panel’s own agreement, the comparator’s date and provenance, comparator re-measurement as a postmarket trigger, and comparison granularity fixed in advance. We also ask FDA to recognize staged, task-by-task release as the step after shadow deployment, to link postmarket monitoring to registry outcomes, and to scope agentic devices by workflow, not by product.

The attached statement answers all 26 questions.

Response to the FDA Discussion Paper "Considerations for the
Regulation of Generative AI-Enabled Medical Devices"
Submitted by: ESSKA AI & Innovation Committee (European Society of Sports Traumatology, Knee
Surgery and Arthroscopy)
Docket: FDA-2026-N-7874
Comment period closes: 19 October 2026

Summary of the Committee's position
We welcome the discussion paper. Its process provisions are concrete and sound: test methods and
acceptance criteria fixed before testing, adjudicators independent of the sponsor, and reassessment
after both a defined cadence and defined events. Our concern is with what those provisions operate on.
The paper builds its evaluation on a human standard, a guideline or a panel of clinicians. It does not
say how that standard is defined, how stable it is, or who supplies it. Our answers to the 26 questions
return to five points.
1. The competency analogy covers the examination but not the apprenticeship.
Benchmarking and clinical confirmation correspond to licensing. Supervised practice with
graduated independence has no counterpart in the framework, and in surgery that is where
most competence is built. Staged, task-by-task release can make it concrete (Question 7).
2. The human standard is neither single nor constant. Guideline and panel comparators can
disagree almost completely, and the standard shifts as clinicians work with the devices being
tested.
3. Four comparator requirements need no new regulatory machinery. Sponsors should report
the comparator panel's own agreement and document its provenance. They should fix its
granularity in advance and treat its re-measurement as a postmarket trigger.
4. Outcome-linked monitoring is the missing mode. Registries already exist in orthopaedics.
The proposed postmarket methods measure output fidelity, not whether patients did better.
5. The scoping unit is the product, and it should be the workflow. Function-level scoping
cannot see orchestrated, agentic or intraoperative use.
Questions are answered in the paper's own numbering.

Section IV: Assessment of risk
Question 1. Does the two-axis framework capture the dimensions most relevant to risk?
It is a sound starting point, and we would add three dimensions.
Reversibility and time pressure belong on the consequences axis. An error that cannot be undone, in a
setting where it cannot be caught, is a different consequence from one that is corrected at the next visit.
Clinicians exposed to incorrect suggestions under time pressure have reversed correct initial diagnoses
(Heudel et al. 2026), and the conditions of that evidence are those of the operating room.

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 1
Effective reviewability should qualify the "HCP-supervised" category on the activity axis. Supervised
and reviewable are not the same condition. In the operating room the clinician's hands are occupied,
time is short and the action does not reverse, so nominal supervision may provide little review.
Comparator stability is a dimension of evidence rather than of risk, but it belongs in the framework. How
well qualified clinicians agree on the task determines how much evidence a performance claim requires
(see Questions 8 and 14).

Question 2. How should the spectrum from "non-directive" to "action-directing" be
accounted for?
The Committee agrees that directiveness should be judged by substance and context, not by wording.
The characteristics we would treat as risk modifiers are:
• whether the output offers a single action or a set of options;
• the specificity of the action (a dose, a date, a procedure);
• personalization to the user's circumstances;
• whether the basis for the output is shown and traceable to sources;
• whether the user can act on it without a further human step.
We suggest FDA publish a worked ladder of directiveness levels, in the manner of its lisinopril example,
for several specialties. Manufacturers should then classify a function by its behaviour across tested
conversational trajectories and not by its labelling. A "talk to your doctor" statement should not lower the
assigned level of an output that otherwise directs an action.

Question 3. When does delivering clinical information to patients present different or higher
risk?
The risk is higher where the patient cannot evaluate the output and acts on it in a way that delays
needed care or leads to inappropriate self-treatment. It is not uniformly higher. Postoperative questionanswering is an example where benefit is plausible: locally deployed language models showed
promising performance on postoperative orthopaedic patient questions, although that work was an in
silico analysis with only moderate agreement between blinded reviewers (Lanter et al. 2026; Pujol et al.
2026).
We therefore advise against treating patient-facing status as a blanket uprating. Patient-facing functions
should move up the consequences axis only where an error leads to delayed or inappropriate care.
Safeguards that matter include enforced scope limits, escalation behaviour tested under minimization
by the user, readability tested across health literacy levels, and source traceability. These correspond to
elements S.1, S.2 and E.4 of Appendix A.

Question 4. Should the generalist/specialist distinction inform the assessment of risk?
Yes, as a matter of intended use. A function that extends specialist knowledge to a generalist is
valuable, and its risk depends on whether safe use requires specialist contextualization. The intended
user should be part of the intended use statement. Benchmark adjudicators should be matched to the
intended user and to the level of specialty at which the task is normally performed. Useful safeguards
are explicit uncertainty communication, referral triggers and traceability to primary sources (elements
S.3 and E.1).

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 2
Question 5. How should risk be assessed for multi-turn devices that migrate from nondirective to action-directing?
We propose a worst-credible-state rule: the risk tier of a conversational device is set by the most
directive state it reaches across realistic trajectories, not by its starting state. Evaluation would use
simulated multi-turn encounters, including adversarial and minimizing user personas, and would report
what proportion of trajectories reach an action-directing state and how often it should have deferred
instead.
Intended use for such a device is best characterized by its boundary behaviour and by technically
enforced scope limits, not by a description of the typical exchange. Failure modes in sequential systems
are properties of the sequence and not of any single turn (Oettl et al. 2025).

Question 6. How should under-escalation and over-escalation be weighed?
They should be reported separately and not combined into one score. We suggest two figures:
sensitivity of escalation for true emergencies, and the rate of unwarranted escalation. Each needs a
pre-specified threshold, and the trade-off needs a written rationale for the intended clinical context. In
orthopaedics, postoperative wound, thromboembolic and infection red flags are examples of contexts
where under-escalation should dominate the threshold. Because the trade-off is a value judgement that
varies by context, it should be disclosed and not left implicit, as with the comparator choice discussed
under Question 14. Postmarket utilization data can then show whether the chosen balance holds in
practice.

Section V: A competency-based approach to premarket evaluation
Question 7. Is benchmarking followed by clinical confirmation a useful and appropriate
framework?
It is useful and incomplete. The framework describes accurately how medicine examines and licenses
clinicians. The discussion paper itself names the other half of clinician evaluation: supervised practice in
real clinical environments with progressively greater independence. Benchmarking and clinical
confirmation together correspond to an examination and a licensing decision, and the supervised years
have no counterpart.
In surgery this asymmetry matters because no written examination has produced a surgeon.
Competence is assembled through graduated operative responsibility under a named consultant who
remains answerable for the outcome. That arrangement corrects errors inside the loop while the case is
still open, and it places the consequence on a person. The latter is the accountability gap Freyer et al.
(2025) identify. Licensure is also an imperfect model: it does not stop poor clinicians from practising,
and a device deployed at scale repeats the same error across thousands of patients. The bar for a
device should therefore be higher, not lower, and who answers for a wrong recommendation has to be
defined before deployment, not after the first harm.
For a generative device, supervised practice can be made concrete as staged release. Take a system
that answers patients' questions after anterior cruciate ligament reconstruction. First, it works only on
simulated encounters, including rare complications and patients who play down their symptoms.
Second, it runs in shadow mode in a real clinic: it drafts an answer to every incoming question, the draft

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 3
is not sent, and the surgeon's own reply serves as the comparison. Third, its answers reach patients
only after a named surgeon has approved them, and every correction is logged. Only then is review
relaxed, task by task. The system might be released to answer questions about wound care and
physiotherapy, while every message mentioning calf pain, fever or wound discharge still goes to the
clinical team. Each release can be withdrawn, and outcomes are tracked through the clinic's registry.
The discussion paper lists shadow deployment as one way to confirm performance before
authorization. It does not describe the step after it, graduated task-by-task release as evidence
accumulates, and in surgical training that is the step where competence is built. We therefore support
the framework as a necessary component and recommend that FDA add a staged-independence
element, discussed under Questions 11 and 17.

Question 8. How should the two-axis framework be used within the competency-based
approach?
The tier should determine which benchmark elements apply, how difficult the test conditions are, and
which clinical confirmation method is required. We suggest a further factor: the stability of the
comparator. Where qualified clinicians agree poorly on the task, a device cannot be shown to equal or
exceed them with the same evidence that suffices where they agree well. Reversibility and time
pressure (Question 1) should raise the tier.

Question 9. Would the benchmark structure provide adequate evidence? What is missing,
redundant or externally available?
The structure is a reasonable skeleton for knowledge, safety behaviour and communication. We see
four gaps.
• Procedural and intraoperative competence. Nothing in the current set of instruments
addresses procedural or intraoperative use, and the discussion paper's worked examples come
from clinic, triage and self-care.
• Comparator reliability. The benchmark should include measurement of human agreement on
the same items, so that a score can be interpreted.
• Effect on the user. Performance of the clinician with the device, and after it is withdrawn, is not
an element. Endoscopists who had worked with AI-assisted colonoscopy detected fewer
adenomas without it: the detection rate fell from 28.4% to 22.4% across 1,443 procedures, a
reduction of 6.0 percentage points (95% CI 1.6 to 10.5; p=0.0089) (Budzyń et al. 2025).
• Patient outcomes. None of the elements establishes whether patients did better.
We also see partial overlap between element E.1 (acknowledging knowledge limits) and element S.3
(calibration and deferral).
On external standards, reporting standards exist but evaluation standards for generative devices do
not. What exists is mostly a reporting layer: TRIPOD, CONSORT-AI and STARD-AI, and the
generative-specific TRIPOD-LLM (Gallifant et al. 2025) and CHART (Collaborative 2025). A reporting
guideline specifies how a completed study is described. It does not set a threshold, a comparator or an
evidentiary burden. The closest instrument to an evaluation standard is the CORE-MD clinical risk
score, which titrates evidentiary burden across valid clinical association, technical performance and
clinical performance (Rademakers et al. 2025). It presumes a target the device detects or predicts, and
extending it to systems that generate is a defined task with an identifiable owner.

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 4
Question 10. How should a sponsor establish that benchmark performance predicts realworld behaviour, and what role should sponsor-developed benchmarks play?
Public benchmarks suffer from contamination, saturation and limited representativeness, as the
discussion paper recognizes (Garcia et al. 2026). Construct validity should be demonstrated
empirically: the sponsor pre-specifies the relation between benchmark score and performance on
independently adjudicated real cases in the intended setting, and reports it. Sponsor-developed
benchmarks are acceptable if they are pre-specified, scored by adjudicators independent of the
sponsor, and accompanied by a sequestered subset held by a third party and rotated over time.
Benchmark scores should be reported alongside the agreement of the human comparators on the
same items.

Question 11. How should a sponsor select and justify a clinical confirmation approach?
Which device types need more?
The Committee agrees that a prospective study is not needed in every case. We propose that the
method follow the tier:
• Retrospective evaluation on real inputs and shadow deployment suit informational, reversible,
lower-consequence functions.
• Prospective evaluation is expected for action-taking functions, for irreversible actions, and for
intraoperative use.
• Standardized patient interactions are insufficient as the sole confirmation for patient-facing
escalation, because they cannot reproduce the real distribution of presentations.
• Release should then widen task by task. Shadow deployment is the paper's last stage before
authorization; we ask FDA to recognize the stage after it, in which review is relaxed for one task
at a time while every release remains withdrawable (Question 7).
The sponsor should characterize the anticipated real-world input distribution and show how the
confirmation sample covers it. One further method deserves a place in the list: registry-embedded
evaluation. The CORE-MD consortium recommended that large trials be managed through a registry
and that inclusion target more than 95% of eligible patients. It also recommended mandatory enrolment
where a device is approved on limited evidence as a breakthrough technology (Fraser et al. 2025). That
is a model for graduated authorization conditional on confirmatory data.

Question 12. How should statistically meaningful performance measurement be achieved,
and when may evidence be combined?
Sample size should be set with the noise of the comparator in view: a comparator with low inter-rater
agreement widens the sample needed to support any superiority or non-inferiority claim, and a noninferiority margin should be tied to that variability. Estimates should be paired against the comparator
on the same cases and reported with confidence intervals.
Benchmarking and clinical confirmation evidence should be reported separately by default. They may
be combined into one estimate only through a pre-specified model, with real-patient data governing the
estimate and with the synthetic-to-real distribution difference characterized and propagated into the
uncertainty.

Question 13. Where is synthetic data well suited or inadequate? What safeguards apply?
Synthetic inputs are well suited to stress-testing robustness: paraphrase variation, input ordering,
missing or contradictory information, and adversarial prompting (elements R.1 and S.2). They are

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 5
poorly suited to establishing performance for underrepresented subgroups, because a generator trained
on the same distribution inherits the same gaps. They are also poorly suited to rare presentations. A
simulator that leaves out rare presentations will certify a system that has never met them, so a sponsor
relying on simulated experience should state what the simulation contains and what it omits.
Safeguards we propose are: the generator should come from a different model class or developer than
the device; its output should be validated against real data before use; the sponsor should characterize
what the synthetic set contains and omits; and synthetic evidence should inform test design but should
not by itself support a subgroup claim.

Question 14. How should comparators and acceptance criteria be selected for open-ended
outputs?
This is the question to which we have the most to contribute. The discussion paper offers a guidelinegrounded rubric or a panel of clinicians as expressions of one standard. They are two different objects.
• Agreement between the AAOS Appropriate Use Criteria and arthroplasty surgeons'
recommendations for total knee arthroplasty was close to zero (weighted kappa 0.06; 95% CI
0.04 to 0.09). The criteria rated 309 of 397 patients (77.8%) as appropriate for surgery (Selph et
al. 2024). A sponsor choosing between those comparators is choosing between two thresholds.
• Across 25 osteoarthritis guidelines from eight countries and four international organizations,
weighted kappa ranged from minus 0.14 to 0.91 (Gray et al. 2024).
• Agreement is heterogeneous within a single joint. For cartilage compositional MRI in knee
osteoarthritis, reliability was mostly excellent, with most intraclass correlation coefficients above
0.8 across 58 studies and 2,007 knees (MacKay et al. 2018). For MRI identification of the
anterolateral ligament, interobserver agreement across 24 studies and 2,427 knees ranged from
0.143 to 1.0 (Andrade et al. 2019).
We therefore propose four requirements, none of which needs new regulatory machinery:
1. The comparator panel's own agreement should be reported alongside the device performance it
anchors. The statistic should suit the output (Fleiss' or weighted kappa for categorical
judgements, the intraclass correlation coefficient for continuous scores), with 95% confidence
intervals and on the same scale as the agreement between device and panel.
2. The comparator should be dated and given a provenance: who the clinicians were, when they
were assessed, where they trained, and on what case mix. Like a clinical practice guideline, it
should carry a date by which it must be reviewed. Guidelines in the German AWMF register, for
example, expire after at most five years.
3. Re-measurement of the comparator should be a postmarket triggering event (Question 19).
4. The granularity of the comparator should be fixed in advance and match the granularity of the
clinical decision. Agreement rises when a multi-category judgement is collapsed to a binary.
Surgeons may disagree on whether a meniscal tear is horizontal, radial or complex, yet agree
closely on whether it can be repaired. A device scored on the full classification and one scored
on the repair decision are measured against different standards.
For some treatments, such as robotic-assisted knee arthroplasty, there is not yet an agreed standard to
compare against at all.
On the specific sub-questions: the sponsor should state whether the standard of care or the median
clinician in practice is the reference, and justify the choice, because the two are different thresholds.
The comparator clinicians should match the intended user of the device. Human-AI team performance
should be the basis of evaluation whenever the intended use is supervised, and device-alone

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 6
performance should also be reported, since the evidence on these differ. In one randomized trial
physicians using a language model barely improved, while the model alone scored 16 percentage
points higher than clinicians using conventional resources (95% CI 2 to 30; p=0.03) (Goh et al. 2024). A
UK replication found assisted clinicians 21.3 percentage points below the model alone (Healy et al.
2026). Pooled human-AI collectives, combining 40,762 physician differential diagnoses with five models
across 2,133 vignettes, outperformed both constituents (Zöller et al. 2025). How the two are combined
may matter more than which is placed above the other.
The standard is also not constant. If clinicians working with these systems improve, a comparator
measured in 2026 will understate the standard by 2030. If they deteriorate, it will overstate it (Budzyń et
al. 2025; Oettl et al. 2026a).

Question 15. Can performance be assessed against the care likely to occur without the
device?
Yes, and for devices whose value is access, this is often the more relevant comparison. The
counterfactual might be unaided generalist judgement, delayed specialist review, or no intervention.
Shadow deployment is a natural way to observe it, because the decision actually made is recorded.
Registry and routine-care data can supply the counterfactual for outcomes. One caution: a
counterfactual comparator can make a weak device look acceptable where existing care is poor, so we
recommend that a minimum performance floor on the safety elements (S.1 to S.3) apply regardless of
the comparator.

Question 16. Should independent third parties be involved?
Yes, selectively. The most valuable roles are sequestered evaluation datasets maintained
independently of the manufacturer, and independent expert adjudication. Specialty societies are well
placed to supply the rubrics, the pool of qualified adjudicators and, through registries, the outcome data.
Qualification and independence criteria should include specialty training matched to the task, no
financial ties to the sponsor or the foundation model developer, and declared conflicts of interest. This
applies to scientific societies as much as to commercial bodies, and we note that a society offering
these services must itself meet the criteria. Program design should include several recognized bodies
and not a single gatekeeper, published methods, fee transparency, and rotation of sequestered sets, so
that the program does not limit competition or favour incumbents. A third-party determination should be
an input to FDA review and not a substitute for it. We see little value in third-party participation for tasks
that cannot be defined before testing.
Who should reconcile differences between comparators? Not the sponsor, and not the regulator alone.
Formal consensus between clinicians is an established method that specialty societies already run, and
both regulators have a route to adopt the result: FDA recognizes voluntary consensus standards, and
the Medical Device Regulation allows common specifications where harmonized standards are lacking.
We ask FDA to state that clinical comparators for generative devices belong to that machinery, and to
invite societies to supply them. Orthopaedics can start from existing assets: national arthroplasty and
ligament registries, decades of measured inter-observer agreement, the STORM-AI reporting guideline
(Oettl et al. 2026b), and a role in European methodology work through EFORT.

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 7
Question 17. Does the approach apply to multimodal, generative or predictive world-model
devices?
The logic applies; the current benchmark elements do not transfer unchanged. Multimodal systems
need sequestered linked test sets, and the variability of human interpretation of the underlying modality
(Question 14) must be measured. Event-based foundation models predict patient trajectories and not
single outputs, so the relevant error is a diverging timeline rather than a wrong answer (Bouterse et al.
2026). They require trajectory-level calibration and prospective validation against observed outcomes.
Simulated evidence exposes a regulatory gap. FDA's guidance on credibility of computational modelling
states that it does not apply to standalone data-driven models and, for hybrids, applies to the firstprinciples aspects only (FDA 2023). CORE-MD states that in-silico trials cannot substitute for clinical
investigation and treats them as out of scope (Fraser et al. 2025). A generative device validated on
simulated patients is therefore excluded twice: as data-driven by the FDA guidance, and as in-silico
evidence by CORE-MD. The machinery for deciding when a simulation may stand in for an experiment
exists, and it stops at precisely the class of model that most needs it.
We recommend a data-driven annex to the existing credibility framework that asks the question the
guidance already asks: what is the context of use, and what evidence supports the model within it.
Coverage of the simulated distribution would be treated as an evidentiary matter. Simulation is the first
stage of the staged release described under Question 7, and it exposes no patient to early errors.

Section VI: Postmarket monitoring
Question 18. Under what conditions is greater reliance on postmarket monitoring
appropriate?
We see three necessary conditions. The device's function sits at limited or moderate consequence and
the harm it can cause is reversible. The monitoring program measures patient outcomes through linked
data and not only output fidelity. And the plan specifies thresholds and triggers in advance, with a
defined route to restriction or withdrawal. It would not be appropriate for action-taking functions, for
irreversible or intraoperative use, or where the time to detect harm exceeds the time over which harm
accrues. Approval on reduced evidence should carry mandatory enrolment in confirmatory follow-up, as
CORE-MD recommended for breakthrough devices (Fraser et al. 2025).

Question 19. What should postmarket performance evaluation include, and how should
cadence and triggers be set?
The three proposed approaches (periodic re-benchmarking, sample-based clinician review and
degradation monitoring) are reasonable. All three measure output fidelity, and none establishes
whether patients did better. Registries do, and the infrastructure in orthopaedics is already running.
Outcome linkage is the monitoring mode the framework lacks.
We would add three elements:
• Comparator re-measurement as a trigger. This modestly extends the event triggers the paper
already proposes, such as changes to the underlying model. We propose three further triggers:
the comparator's review date, publication of a revised guideline in the device's domain, and
evidence that clinicians using the device have changed their own performance. If the device no
longer meets its acceptance criterion against the renewed comparator, its performance claim

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 8
 and labelling should be revised. The corrective pathway for any postmarket performance
shortfall applies, not automatic withdrawal. It also fits the drift-threshold approach to mandatory
human review described in the orthopaedic AI implementation literature (Longo et al. 2026).
• Monitoring of the user. Deskilling is a measurable postmarket effect (Budzyń et al. 2025), and
the apprenticeship model depends on a human safety net that routine use may erode. Deskilling
and never-skilling, the failure of trainees to acquire proficiency through premature reliance, are
the failure modes to design against (Oettl et al. 2026a).
• Risk-tiered cadence with event triggers: a model version change, a shift in the input
distribution, a cluster of adverse events, or a comparator re-measurement.

Question 20. Can monitoring be facilitated by machine-based supervisory agents?
Partly. A supervisory agent is useful for screening volume and flagging cases for human review. It
should not be the sole monitor. We would ask FDA to consider four points. The supervisor should be
independent of the monitored model in architecture and developer, because correlated errors between
models of the same class defeat the purpose (see Question 13). It should be evaluated with the same
discipline as any device, including seeded known failures. Its output should be a flag and not a
decision. And a sampled human audit of the supervisor should continue.

Question 21. What roles should other stakeholders play without diffusing manufacturer
accountability?
The manufacturer should own the monitoring plan, the thresholds and the legal responsibility. Other
parties supply inputs under defined terms:
• Clinicians and institutions: incident reporting, participation in sampled adjudication, and
contribution of registry data.
• Professional societies: maintenance of rubrics and adjudicator pools, operation or coordination
of registries, and reporting standards.
• Standards bodies: recognized evaluation and reporting standards.
Accountability stays with the manufacturer if each role is written into the monitoring plan and the
labelling. A stakeholder's input then becomes an obligation on the sponsor to collect and act on, not a
shared responsibility. At the point of care, a named accountable clinician should be identified for each
deployment, which addresses the consequence gap noted under Question 7. In staged release, the
approving surgeon answers for the output during the supervised stages, as a consultant answers for a
resident. Once a task is released, responsibility for it shifts to the manufacturer. In Europe, the revised
Product Liability Directive (Directive (EU) 2024/2853) explicitly covers software, including AI, placed on
the market from December 2026.

Question 22. How should re-benchmarking be scaled to the nature of a modification?
We suggest three bands.
• Documentation within the quality management system: changes with no plausible effect on
behaviour, such as interface or latency changes.
• QMS-managed with mandatory re-benchmarking: prompt, retrieval or guardrail changes
within the validated scope. The safety elements (S.1 to S.3) and robustness (R.1) should be rerun each time, because variation in safety-critical behaviour can be seen as failure, and small
changes can move it.

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 9
 • FDA review before implementation: replacement of the underlying model, expansion of
intended use, addition of an action-taking capability, or a new patient population.
The comparator should be re-measured at defined intervals so that the baseline itself is not stale.
Candidates for a PCCP include model version updates within a qualified family and retrieval corpus
updates, each with pre-specified benchmark thresholds.

Question 23. How can PCCP concepts be adapted when future modifications cannot be fully
prespecified?
By prespecifying the test and not the change. A PCCP for a generative device can fix the benchmark,
the sequestered set, the comparator, the acceptance thresholds and the triggers. The content of the
modification stays open within a bounded envelope. The envelope fixes the intended use, the risk tier
and the autonomy level. Autonomy should be treated as a permission assigned per action and not as a
capability the system possesses, so any change that adds a permission falls outside the plan and
requires review.

Question 24. How should manufacturers detect and respond to changes initiated by thirdparty foundation model developers?
We would combine contractual, technical and regulatory mechanisms.
• Contractual: version pinning where available, minimum advance notice of changes and
deprecations, and access to change logs.
• Technical: automated regression benchmarking on every version change, shadow operation of
a new version before release, and drift monitoring in production.
• Regulatory: a PCCP that covers switching among pre-qualified models. Where the developer
will not commit to version pinning or notice, the sponsor should be assumed to carry higher
residual risk, and this should influence the tier.

Section VII: Other topics
Question 25. Would voluntary Foundation Model MAFs be practical, and what alternatives
exist?
They would be practical if scoped narrowly and kept current by design. Content should include the
items in footnote 24 and, in addition, the version history, deprecation schedule, refusal and content
policy changes, and healthcare-relevant safety evaluations. Each entry should be dated and versioned,
with an expiry, so that a stale file cannot support a current review.
Developers' limited incentive is the main weakness. FDA could create one by giving sponsors who
reference a current MAF a clearer, lighter review path for the foundation-model portion. A MAF should
be a reference input and not a substitute for device-level testing, consistent with the paper's statement
that the sponsor remains responsible for demonstrating safety and effectiveness. An alternative is a
contractual disclosure obligation on sponsors, so that the sponsor secures the necessary information
from the developer as a condition of supply. The European position is a useful contrast: the AI Act
treats general-purpose models as a distinct category, but those obligations fall on the model developer,
not on the device built on it. A MAF is one way to carry that information into device review.

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 10
Question 26. What additional considerations apply to agentic devices?
The unit of evaluation should be the workflow step and not the product. Failure modes of agentic
systems are properties of the sequence and not of any one step (Oettl et al. 2025), and the benchmarks
needed to assess them do not yet exist (Oettl et al. 2026c). We recommend that acceptance criteria for
agentic devices include:
• tested compliance with human-oversight checkpoints before irreversible or high-consequence
actions (element A.1);
• autonomy assigned per action and recorded in the labelling;
• complete action logs available for audit;
• evaluation of the whole trajectory, including recovery from tool failures and resistance to prompt
injection.
The current scoping approach cannot see these systems. One device was cleared in December 2025
as a Class II drug-dose calculator (510(k) K253281). It is marketed as the first software as a medical
device to use patient-facing large language models. Its clearance record does not mention language
models, generative methods or a conversational component. Both descriptions can be accurate,
because what was reviewed was the calculator. That is function-level scoping working as designed, and
it shows that such scoping cannot see an orchestrated workflow whose steps are separately
unremarkable but whose trajectory changes what a clinician does. The only bridge the discussion paper
names, an action sequence ending in control of another medical device, is the narrowest connector
available. We ask FDA to consider workflow-level scoping where the trajectory of an orchestrated
system changes clinician behaviour.

Closing
The ESSKA AI & Innovation Committee's central request is that the human standard be treated as an
object of evidence in its own right. The FDA's process provisions are sound, and the four comparator
requirements above would make them operable at little regulatory cost. They would also supply the
documentation that European human-oversight requirements under the Artificial Intelligence Act expect.
One set of artefacts could then serve both jurisdictions while neither has committed to a path.
Our answer to who defines the standard is that clinicians should, through their scientific societies,
working groups, guidelines and formal consensus projects. The Committee offers to contribute the three
inputs any standard requires: rubrics defining adequate clinical reasoning in orthopaedic and sportsmedicine domains, qualified and conflict-declared adjudicators, and access to registry-linked outcome
data. We would welcome the opportunity to discuss any of these answers with CDRH.
Felix C. Oettl, MD, Chair, on behalf of the ESSKA AI & Innovation Committee

References
Andrade R, Rebelo-Marques A, Bastos R, Zaffagnini S, Seil R, Ayeni OR, Espregueira-Mendes J (2019)
Identification of normal and injured anterolateral ligaments of the knee: a systematic review of magnetic
resonance imaging studies. Arthroscopy 35:1594-1613. doi:10.1016/j.arthro.2018.10.120
Bouterse AM, Pruneski JA, Oettl FC, Zsidai B, Longo UG, Tischer T, Hirschmann MT, Samuelsson K (2026) From
patient notes to prognostication: the revolutionary potential of event-based foundation models in orthopaedics.
Knee Surg Sports Traumatol Arthrosc. doi:10.1002/ksa.70428

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 11
Budzyń K, Romańczyk M, Kitala D, et al. (2025) Endoscopist deskilling risk after exposure to artificial intelligence
in colonoscopy: a multicentre, observational study. Lancet Gastroenterol Hepatol 10:896-903.
doi:10.1016/s2468-1253(25)00133-5
Chatbot Assessment Reporting Tool Collaborative (2025) Reporting guideline for chatbot health advice studies:
the CHART statement. BMJ Medicine 4:e001632. doi:10.1136/bmjmed-2025-001632
European Parliament and Council (2024) Directive (EU) 2024/2853 on liability for defective products.
Food and Drug Administration (2023) Assessing the credibility of computational modeling and simulation in
medical device submissions. Center for Devices and Radiological Health.
Fraser AG, Buccheri S, Byrne RA, et al. (2025) Recommended methodologies for clinical investigations of highrisk medical devices: conclusions from the European Union CORE-MD project. Lancet Reg Health Eur 58.
doi:10.1016/j.lanepe.2025.101460
Freyer O, Jayabalan S, Kather JN, Gilbert S (2025) Overcoming regulatory barriers to the implementation of AI
agents in healthcare. Nat Med 31:3239-3243. doi:10.1038/s41591-025-03841-1
Gallifant J, Afshar M, Ameen S, et al. (2025) The TRIPOD-LLM reporting guideline for studies using large
language models. Nat Med 31:60-69. doi:10.1038/s41591-024-03425-5
Garcia V, Sidulova M, Badano A (2026) Performance assessment strategies for language model applications in
healthcare. Artif Intell Life Sci 9:100162. doi:10.1016/j.ailsci.2026.100162
Goh E, Gallo R, Hom J, et al. (2024) Large language model influence on diagnostic reasoning: a randomized
clinical trial. JAMA Netw Open 7:e2440969. doi:10.1001/jamanetworkopen.2024.40969
Gray B, Gibbs A, Bowden JL, et al. (2024) Appraisal of quality and analysis of the similarities and differences
between osteoarthritis clinical practice guideline recommendations: a systematic review. Osteoarthritis
Cartilage 32:654-665. doi:10.1016/j.joca.2024.02.890
Healy J, Kossoff J, Lee M, Hasford C (2026) Human-AI collaboration in clinical reasoning: a UK replication and
interaction analysis. Diagnosis (Berl). doi:10.1515/dx-2025-0176
Heudel PE, Crochet H, Filori Q, Bachelot T, Blay JY (2026) Artificial intelligence in medicine: a scoping review of
the risk of deskilling and loss of expertise among physicians. ESMO Real World Data Digit Oncol 12:100693.
doi:10.1016/j.esmorw.2026.100693
Lanter L, Masel S, Hochreiter B, et al. (2026) Promising performance of locally deployed large language models
for postoperative orthopaedic patient questions: an in silico analysis. J Exp Orthop 13:e70813.
doi:10.1002/jeo2.70813
Longo UG, Merone M, Schena E, et al. (2026) A practical guide to the implementation of AI in orthopaedic
research, part 4. J Exp Orthop 13:e70863. doi:10.1002/jeo2.70863
MacKay JW, Low SBL, Smith TO, Toms AP, McCaskie AW, Gilbert FJ (2018) Systematic review and metaanalysis of the reliability and discriminative validity of cartilage compositional MRI in knee osteoarthritis.
Osteoarthritis Cartilage 26:1140-1152. doi:10.1016/j.joca.2017.11.018
Oettl FC, Pruneski J, Zsidai B, et al. (2026a) From de-skilling to up-skilling: how artificial intelligence will augment
the modern physician. J Exp Orthop 13:e70677. doi:10.1002/jeo2.70677
Oettl F, Zsidai B, Yu Y, et al. (2026b) Standardised Transparent Orthopaedic Reporting and Modelling for AI
(STORM-AI): guidelines for reporting artificial intelligence studies in orthopaedics from the ESSKA AI Working
Group. J Exp Orthop 13. doi:10.1002/jeo2.70702
Oettl FC, Pruneski J, Zsidai B, et al. (2025) Artificial intelligence agents in orthopaedics: concepts, capabilities and
the road ahead. Knee Surg Sports Traumatol Arthrosc 33:4475-4483. doi:10.1002/ksa.70109
Oettl FC, Pruneski JA, Zsidai B, Yu Y, Hirschmann MT, Samuelsson K (2026c) Small language models: the big
play for agentic artificial intelligence in orthopaedics. Knee Surg Sports Traumatol Arthrosc 34:398-401.
doi:10.1002/ksa.70126
Pujol O, Ferrer R, Coelho A, et al. (2026) Four general-purpose large language models show comparable
performance in specialised total knee arthroplasty clinical questions. Knee Surg Sports Traumatol Arthrosc.
doi:10.1002/ksa.70544

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 12
Rademakers FE, Biasin E, Bruining N, et al. (2025) CORE-MD clinical risk score for regulatory evaluation of
artificial intelligence-based medical device software. NPJ Digit Med 8:90. doi:10.1038/s41746-025-01459-8
Selph TJ Jr, Suleiman LI, Pagadala MS, Bergman R, Franklin PD, Edelstein AI (2024) Agreement of the American
Academy of Orthopaedic Surgeons Appropriate Use Criteria with treatment recommendations from
arthroplasty surgeons. Arthroplast Today 27:101386. doi:10.1016/j.artd.2024.101386
Zöller N, Berger J, Lin I, et al. (2025) Human-AI collectives most accurately diagnose clinical vignettes. Proc Natl
Acad Sci U S A 122:e2426153122. doi:10.1073/pnas.2426153122

ESSKA AI & Innovation Committee | Docket FDA-2026-N-7874 | Page 13