Palantir Technologies

See attached file(s)

Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993

Introduction
Palantir Technologies Inc. appreciates the opportunity to provide feedback on the FDA’s
discussion paper, Considerations for the Regulation of Generative AI-Enabled Medical
Devices. We support a risk-based, function-focused, and total-product-life-cycle
approach that preserves patient safety while enabling timely access to beneficial
technology. Our comments reflect our experience developing and deploying traditional
software systems that integrate Artificial Intelligence to support complex, human-led
workflows at hospitals across the country and globally; we have avoided addressing
issues outside our area of expertise.

Executive summary
We see GenAI’s value in healthcare arising from its focused application in real
operational contexts to support health outcomes, benefit patient journeys, and improve
conditions for frontline healthcare workers. Risks and benefits, therefore, cannot be
understood by treating GenAI as a chatbot operating in a vacuum, or by relying on
model guardrails; they instead depend on how an AI-enabled workflow is designed as
an end-to-end system, how it is integrated into complex heterogeneous technical
environments, and how it is utilized within a human-centered healthcare workflow.

This means that manufacturers and deployers of GenAI in healthcare must assess its
function within its intended deployment environment: what it does, who relies on it,
where it sits in the workflow, what actions may follow, and how errors and safeguards
may affect patients, care delivery, and worker wellbeing. Consequently, the risk
assessment and evaluation of GenAI systems should not be reduced to overly simplistic
dimensions. Instead, guidance should be explicit on the type of robust approach needed
to assess the nuanced avenues of risk from the broader workflow and health-system
context in which that activity occurs.

Additionally, any assessment must be of the fully integrated system, not only the model
or its outputs. This means devices and applications should be assessed across data
governance and source-data quality; provenance and transformations; system design
and orchestration; models and tools; interoperability and connections to records and
workflows; front-end interfaces, human review, monitoring, change management, and
more. Our experience is that risks often arise at the points of interaction across a
system’s components.

This document is approved for public dissemination.
These principles are intended to apply across healthcare applications: from lower-risk
GenAI devices that support appointment scheduling to higher risk systems involved in
clinical diagnostics. The FDA should tailor evidence requirements and safeguards to the
risk of the specific device’s function, intended use, and operational context, while
expanding its framework beyond device activity alone to address system-level and
lifecycle considerations, from source data through the user interface and postdeployment operation.

Risk assessment for GenAI-enabled medical devices
1 - Does the two-axis risk framework, organized around device activity and the
consequence of relying on an incorrect output, appropriately capture the dimensions
most relevant to the risk of a GenAI-enabled software function? If there are additional
dimensions—such as the reversibility of a resulting action, the availability of
downstream safeguards, the time pressure of the deployment setting, or the traceability
of the output (i.e., to primary source materials)—that should be represented in a risk
framework, please describe and provide examples of how they should be represented.

We believe that the two-axis risk framework provides only a starting point for the
assessment of risk for GenAI devices. In addition to the other dimensions mentioned in
the prompt for this question, which we believe are valid for a holistic risk assessment,
there are many other dimensions that could be appropriate, such as the fidelity and
quality of source data, the reliability of the AI writing back to systems of record, the
heterogeneity of the workflow across different sites, the result of the workflow on the
broader patient journey, and many more considerations. This aligns with our core belief
that AI, and technology as a whole, does not exist in vacuum, but rather is inextricably
tied to its contexts of application, its operational uses, and the full data operating
environment.

Our experience deploying GenAI across diverse contexts, including healthcare, has
taught us that it is difficult to create a single framework that captures the assessment of
all possible avenues of risks for a deployment a priori. Our feedback is that, to account
for the highly complex and contextually dependent nature of deploying GenAI in
healthcare, a principals-based assessment of individual deployment risks, including
evaluating severity and likelihood, is necessary to identify the risks and required
mitigations.
The framework for assessing risk then becomes one of orthopraxy, where developers
should have guidance on the methodology to conduct a nuanced assessment—
beginning with a system’s data inputs and best practices around sound data science
and governance, and running the full length of the application through to effectively
performing a thorough clinical risk assessment.

This process should begin by identifying the overall potential risk (e.g. worst-case
scenario) of the workflow in question, where the two-axis approach can be used, to
inform the required rigor for the mitigations at the next level of review. Once the overall
risk is understood, developers can perform something like a structured what-if technique
(SWIFT) assessment or other recognized approaches to identify specific risks or
hazards posed by the GenAI device and develop needed mitigations. These individual
hazards then could have their risk scores influenced by the degree of autonomy that is
involved in their specific processing, an estimate for the likelihood of occurrence, and
the severity of consequence should it manifest.

We firmly believe that the assessment of likelihood and severity, rather than just
autonomy and severity, creates a more proportionate assessment of the consequence
that a potential hazard ultimately presents for a device, as manual aspects of a workflow
could carry more risk than automated ones depending on wider circumstances. Not all
these levels of review will be mandatory for each application or device - they should be
intentionally flexible according to the full context of the workflow and its setting. As
noted above, for example, this framework should be different for a workflow that
leverages GenAI in entity resolution for patient scheduling, versus a workflow that
summarizes notes and provides clinical recommendations.

If a quantifiable risk score is needed by CDRH, then an assessment approach like this
could have the individual hazards scored based on their likelihood and potential severity
to create a risk score that could be tallied across the hazards of the device, thereby
incorporating the nuanced risks into a straightforward manner to create a cumulative
risk score that more accurately captures the multi-dimensional risk a device truly
presents.

5 - For multi-turn conversational GenAI-enabled devices that may migrate from
providing “non-directive” information to “action-directing” information over the course of
an exchange, how should risk be assessed across realistic conversational trajectories?
How could the intended use of such a device be characterized when its behavior is
emergent across a conversation?
There is no silver bullet for effective risk assessment of GenAI in healthcare devices or
workflows - this question emphasizes the necessity of assessing risk holistically at
workflow design, and across all components of the fully integrated system.

In the case of this example, this means fully interrogating the overall operational context
of the system and the role of AI within it. At this point, not only should risks be
assessed, but also controls and rails established to govern the exact degree of
autonomy, decision-making, and tools the AI is able to execute. These controls will only
be as effective as the underlying governance of the fully integrated system, such as
what data models are able to access, what tools they can leverage, what decisions they
can make, where and when human validation and approval will be required, the audit
logging and evaluation system around accuracy, and more. Importantly, many of these
technical controls root back to best practices in privacy and data governance and must
be thought of and designed at a foundational level of the technological system itself. For
example, model access controls to data will only be airtight if those same controls apply
to all data that enters a system, solved far upstream of the introduction of AI.

Risks of migration across a conversation trajectory should be viewed through this lens -
they are addressed upstream at the point of workflow design and operationalization.

Palantir has the strong belief that chat interfaces for users to interact with GenAI are
only a limited component of a wider device. Especially in the context of healthcare,
where the AI will need to interact with diverse data sources and produce specific
evaluations, we must start with the overall system function and granular assessment of
the degree of human intervention needed across stages of the operational lifecycle.

A Competency-Based Approach for Premarket Evaluation of
GenAI-Enabled Devices
7 - Is the competency-based approach described above, i.e., device benchmarking
followed by clinical confirmation, a useful and appropriate framework for evaluating
GenAI-enabled devices?

We believe that a competency-based approach is a useful tactic to perform the
evaluation of the medical performance of GenAI devices. A couple important aspects
we want to emphasize to ensure competency assessments are reliable are: 1) it is done
as much as possible using real data and 2) that at least some of the assessments
cannot be treated as a point in time exercise and must be done when a device is
deployed in a new setting, like a new hospital.

This is because GenAI devices will be deployed in settings where local context can
have a significant influence on the performance and behavior of the system. Seemingly
small variations in the input data, user practices, and interoperability of other interacting
systems can have a significant impact on the performance of a GenAI device. To
account for this reality, we advocate that certain benchmarking and clinical confirmation
assessments be part of standard deployment steps at new sites. This could include
approaches like a streamlined retrospective assessment conducted on real patient data
or a mandatory period of shadow deployment at each new site. These would not have
to be as comprehensive as the assessments done in a pre-market context in the earliest
live deployments, but they should be proportionate to the severity of the risk of the
clinical tasks being supported by the GenAI system and provide assurance that it can
be appropriately configured to address local nuances.

8 - How might the two-axis risk framework described in Section IV be considered within
the competency-based approach described above to help determine the level of
evidence needed for a premarket submission?

Our feedback above to question one provides context on a suggested additive approach
to the two-axis framework. With that response in mind, we believe that the hazard
assessment approach can provide a clear pathway to create thresholds for the level of
evidence required for premarket submission. CDRH can provide clear guidance if the
likelihood and severity of a hazard produces a risk score above a certain level, which in
turn requires more robust evidence gathering to support safety assurance of the
system’s mitigations and performance of the tasks related to that hazard. The benefit of
this approach is that the burden of evidence is proportionally assigned to the critical
components of a system dealing with the riskiest aspects of the device, rather than
classifying an entire device as high/low risk which leads to less granular assurance
burdens.

9 - Would a benchmarking structure such as the one described above be likely to
provide adequate evidence of clinical knowledge, analytic capabilities, safety behavior,
communication, and generalizability to support a reasonable assurance of safety and
effectiveness? Are there elements that are missing, redundant, or inappropriately
categorized? Are there externally developed standards that could be leveraged?

The benchmarking structure described in the discussion paper would provide an
important baseline for evaluating a GenAI device while it is being developed and before
deployment, but would not, by itself, provide sufficient evidence for assuring safety and
effectiveness following actual deployment. Benchmarking can establish whether the
final user-facing device demonstrates defined competencies under controlled and
repeatable conditions, and it can support comparison across versions of the application
to assess drifts in behavior or performance. However, as we have already mentioned,
GenAI devices may behave differently when exposed to the variation, incompleteness,
workflow constraints, user behaviors, and population characteristics present in actual,
real-world healthcare settings. Therefore, benchmarking should be considered a
necessary component of a broader evaluation and evidence gathering approach, but
not as a substitute for real-world evaluation or clinical confirmation.

The proposed benchmarking elements are generally appropriate, but we propose that
they are better understood as cross-cutting characteristics to be evaluated and not a
single competency for the device as a whole. For example, robustness, subgroup
performance, uncertainty of communication, and user comprehension should be
assessed across each relevant clinical or generally consequential task that comprises
the workflow of the end-to-end system.

We also have an informed opinion on the actual testing and evaluation approaches to
handle non-determinism and reliably assess robustness. A GenAI process can output
variable responses even with the same input, and irregularities in input, like those in the
real world, can lead to difficulties in predicting behaviors.

With this in mind, we advocate that the assessment of any GenAI function should
undergo perturbation testing. Perturbation testing can be broadly utilized to assess the
impact of typical data quality issues in health data like typos, synonyms, or inconsistent
clinical coding as well as more consequential assessments on the impact different
demographic inputs like gender, ethnicity, or disability have on performance. While the
specific measure to benchmark against should be specific to a device’s intended use,
the approach of using perturbation testing as a standard approach to handle the nondeterminism of GenAI systems should be encouraged.

11 - CDRH is considering that clinical confirmation for a GenAI-enabled device might
not require a prospective clinical study in every case, and has described above a range
of approaches of increasing rigor and patient exposure. How might a sponsor select and
justify a confirmation approach tailored to a device’s intended use and proportionate to
the device’s risk profile? Are there device types or risk profiles for which one or more of
these approaches would be insufficient or inappropriate? How might the anticipated
distribution of real-world inputs be taken into consideration? Are there other methods of
clinical confirmation that might help inform the evaluation of GenAI-enabled devices?

In the case of certain devices or uses that are relied upon for consequential, difficult-toreverse decisions, it would likely not be appropriate to avoid a traditional prospective
evaluation. That said, Palantir is generally supportive of such clinical confirmation routes
for devices that produce easily reviewable outcomes, with reliable routes for human
intervention.
To repeat a point made previously, the specific approach for clinical confirmation should
be tailored to the circumstances of the deploying institution, not solely to an abstract
device risk classification. For example, retrospective analysis may be informative where
an institution has sufficiently representative, high-quality historical data, but may be
misleading where data is incomplete, biased, or unrepresentative of future use.
Conversely, shadow deployment may better capture workflow and population effects but
may be impractical for an understaffed institution that cannot provide the resources
needed for monitoring and adjudication. Lack of institutional capacity should not
automatically justify a less rigorous evaluation; it should prompt consideration of
alternative designs, additional sponsor-provided support, a narrower intended use,
staged deployment, or selection of a better-resourced validation setting. FDA should
permit sponsors and institutions to justify a tailored confirmation plan that explains the
evidentiary strengths and limitations of the available approach, how gaps will be
mitigated, and what safeguards will apply during initial use.

15 - Are there ways in which performance might be assessed relative to the care,
technology, or course of action likely to occur in the absence of the device, rather than
to the comparators described in this section? What approaches might be used to
identify and justify a comparator such as unaided clinical judgment, delayed specialist
review, or no intervention?

Palantir believes in a “field to learn” approach to developing and testing new
technologies on consequential problems such as healthcare. New systems should be
taken out of a vacuum and evaluated in the real operational contexts in which they are
intended to operate, by the users who will ultimately rely on them and using
representative real-world data. This approach helps identify risks that may emerge only
in scaled or production use, develop appropriate mitigation measures, and establish the
actual performance of the existing process, which is often a baseline that is not fully
understood until an alternative is proposed. It therefore supports assessing performance
against the care process, technology, or course of action most likely to occur in the
absence of the device, rather than treating the status quo as a risk-free comparator.

For GenAI-enabled devices used primarily for administrative functions with limited
clinical decision support, established quality-improvement methods may provide an
appropriate way to compare the status quo with the process incorporating the device.
For example, in evaluating a GenAI-assisted workflow that converts clinical notes into
discharge summaries, an institution could assess error rates, accuracy, completeness,
turnaround time, delays, rework, escalation rates, staff workload, and downstream
clinical or operational outcomes both before and after deployment. The study should
define relevant failure modes, unintended effects, escalation pathways, and
downstream consequences, and should examine the existing human-driven process as
carefully as the new workflow. Where feasible, a concurrent control group, phased or
stepped-wedge rollout, matched cases, or an interrupted time-series design could help
distinguish device effects from other changes.

The acceptable error rate for the new system will depend on the context, the type of
decision, the consequences of failure, and the role of the device in the workflow. Those
thresholds should be established by the relevant oversight authorities and domain
owners, who must also retain responsibility for determining whether the system is
appropriate for use and for responding if it fails. It is our belief that the tolerance for error
rates should be much lower for any system that meaningfully replaces human oversight
with a form of AI-assisted decision making, even if the existing human-driven process is
more error prone than the new solution.
The assessment of new technology should intentionally be holistic to include the current
operational context of the system in place, including its challenges and gaps, as well as
the risk of continuing without the new system. In this case, a Total Economic Value
(TEV) framework can serve as a helpful framework and heuristic for this assessment,
recognizing that there should be significant flexibility to accommodate the specific
operational setting, the device, its intended purpose, the existing system, the users, the
role of autonomy in decision making, and more.
Postmarket monitoring for GenAI-enabled devices
18 - CDRH is considering whether it may be appropriate to accept greater premarket
uncertainty regarding a GenAI-enabled device’s benefit-risk profile through greater
reliance on postmarket monitoring. Under what conditions might such an approach be
appropriate, and what characteristics of a monitoring program would need to be in place
to justify reduced premarket evidence? Are there device types or risk profiles for which
this approach would not be appropriate?

Palantir supports the approach to shifting the burden of collecting evidence from
premarket evaluation to postmarket monitoring for GenAI devices, with some important
considerations. A program that enables this approach should have an initial limited
incubation or piloting period for new devices being assessed through enhanced
postmarket surveillance. This phased approach would enable the device to be deployed
in a real-world setting where this complex technology can be more accurately assessed
while limiting the potential risk surface. The aim of the early deployment period should
be to identify factors such as performance bounds and the efficacy of risk controls. After
a pre-determined period, there should be an additional regulatory check to assess the
efficacy and safety of the device before it can achieve full regulated status and available
to the broader market.

The most important commitments to ensure monitoring programs are effective for
devices released in an early-access state like this are that the manufacturer commits to
having engineers directly embedded with the adopting healthcare institution and that the
institution itself has committed to providing clinical and technical resources to establish
the most appropriate evaluation system for their circumstances. We firmly believe this
type of forward deployed engineering model that closes the observation and iteration
cycle between manufacturer and customer is the gold standard to responsibly and
safely develop and deploy GenAI to live complex environments like healthcare.

20 - Please comment on whether the proposed approaches to postmarket monitoring
can be facilitated by machine-based supervisory agents. What considerations, including
the evaluation and reliability of the supervisory agent itself, should CDRH take into
account for such an approach?

Palantir supports the use of machine-based supervisory agents, sometimes known as
LLM-as-a-judge workflows, to facilitate postmarket monitoring particularly for scalable
and repeatable checks. For example, an evaluator model may assess whether an
output satisfies clearly defined criteria, such as factual support, completeness,
adherence to a specified format, or appropriate escalation behavior. However, these
types of agentic evaluators need to be carefully designed with an appreciation of how
they behave. These evaluators should be designed with task-specific prompts,
representative multi-shot examples, and preferably binary pass/fail criteria that produce
actionable results rather than ambiguous numeric scores, like rating on a scale of 1-5,
which can lead to assessments that are difficult to action or explain. Before production
use, the evaluator should be validated against a domain-expert curated “golden” dataset
with relevant ground-truth or reference data, deterministic tests where appropriate,
testing for false positives, false negatives, subgroup variation, and sensitivity to prompt
or model changes. The evaluator’s version, criteria, thresholds, inputs, outputs, and
escalation decisions should be auditable and monitored over time.

Other practical constraints should be assessed when deploying agentic evaluators in
live workflows. These could include cost, which could be very high if an evaluation is set
to happen for all outputs of a device in production, latency, if the time to evaluate the
output live could itself increase the risk of the clinical workflow, as well as the agentic
evaluator potentially leading to users over-relying on its assessment and diminishing the
quality of end user reviews. Agentic evaluators should be treated as one facet of a
broader risk-based monitoring approach to generate ongoing performance measures
against well-defined criteria.

21 - What roles might clinicians, healthcare institutions, professional societies,
standards-setting bodies, and other stakeholders appropriately play in postmarket
monitoring, and how can these roles be structured without diffusing manufacturer
accountability?

We believe that a particularly important role at healthcare institutions for the effective
postmarket monitoring of GenAI devices will be the institution’s IT staff. Firstly, GenAI
devices, and especially more advanced autonomous agents, will rely significantly on an
institution’s local technical infrastructure to interact with resources to orchestrate the
delivery of care or other appropriate functions. Secondly, due to its non-deterministic
nature, the output of GenAI is likely to display variability across different deployment
sites due to varied input data and operational practices, which will likely require some
bespoke monitoring to be established at individual sites to account for local nuances. As
GenAI devices also become more pervasive, there will eventually come a time when
two or more GenAI systems from different manufacturers at an institution interact with
each other to perform complex tasks. Rather than diffusing manufacturer responsibility,
if healthcare institutions invest in both IT infrastructure and IT staff training to support
robust monitoring, they will have the tools to hold manufacturers accountable for how
their devices behave at their institution. The FDA and health institutions should require
GenAI device manufacturers to provide standardized telemetry data that can be
integrated with existing IT monitoring systems to facilitate this empowerment.
24 - For devices built on third-party foundation models, changes to the underlying model
may be initiated by the third-party model developer rather than the device manufacturer.
How can a manufacturer detect, evaluate, and respond to such changes in a timely
manner? What mechanisms—for example, contractual, technical, or through a PCCP—
could provide reasonable assurance that third-party developer-initiated changes do not
compromise the safety or effectiveness of the device?

We believe that it is essential for any organization to have the maximum insight possible
into the models it employs and further believe that it must retain the authority to govern
how those models are used, the data to which they are exposed, and - crucially - full
ownership over what is created in the employment of those models.

Reasonable assurance begins with timely, actionable change information. Contracts
with third-party model developers should require advance notice of material model, API,
safety-policy, or service changes; version-specific documentation and change logs;
notice of deprecation or service discontinuation; and cooperation with the
manufacturer’s validation and incident investigations. Where feasible, agreements
should also support version pinning, a transition period, the ability to defer or reject an
update, and continued access to a previously validated version for rollback. These
provisions should make clear that a provider’s general-purpose release process does
not, by itself, authorize a change in the device’s production environment.

Open-weight models may offer additional control, but not an automatic assurance of
safety or effectiveness. Where the manufacturer has the rights and technical capability
to deploy a model in an environment it controls, it can preserve the evaluated model
artifact, verify its provenance and integrity, pin versions, and decide when—if ever—to
adopt a new release or fine-tune. This can reduce exposure to unannounced changes
by a hosted provider. The manufacturer should still assess changes to model weights,
fine-tuning data, inference software, dependencies, safety layers, and deployment
infrastructure, since any of these may affect device behavior. An open-weight model
accessed through a third-party managed service may remain subject to providerinitiated changes and should be governed accordingly.