Solan Health, Inc. (Kaushal Solanki)

Solan Health, Inc. submits the attached comments on FDA’s discussion paper, Considerations for the Regulation of Generative AI-Enabled Medical Devices, Docket FDA-2026-N-7874. Please see the attached PDF for our full responses to the discussion questions.

Kaushal Solanki, PhD
CEO and Principal Scientist
Solan Health, Inc.

Comments of Solan Health, Inc. on Considerations for the Regulation
of Generative AI-Enabled Medical Devices
Docket: FDA-2026-N-7874
Submitted by: Solan Health, Inc.
Signatory: Kaushal Solanki, PhD, CEO and Principal Scientist
Dear FDA Digital Health Center of Excellence,
Solan Health appreciates the opportunity to comment on the discussion paper [1]. We are
developing AI tools for longitudinal care of cardiovascular, kidney and metabolic disease.
Our founder previously founded Eyenuk, Inc. and led development of the EyeArt
autonomous diabetic retinopathy screening system (510(k) K200667), including its
prospective multicenter pivotal evaluation [2]. EyeArt is not a generative AI device. We
refer to it because several questions the paper treats as new have already been worked
through for an autonomous AI device, and that experience shapes the comments below.
We support the paper’s direction. A risk-based, total product life cycle approach with
benchmarking and clinical confirmation is a sound basis. Our main recommendations are:
1. Evaluate GenAI devices by the clinical actions their outputs produce, not by the
wording of the outputs. Grouping open-ended outputs by the action they lead to makes
them countable, and established statistical methods then apply.
2. Add the mode of human-AI interaction, and the user’s ability to detect an error, to the
risk framework. “Human in the loop” describes several different designs, not one point
on a continuum.
3. Retain statistical rigor within competency-based evaluation. Use clinically justified
endpoints, sample sizes based on power or precision, and independent reference
methods appropriate to the claim.
4. Use an expert panel or clinical reference standard as the primary comparator. Measure
the median clinician against the same reference, rather than treating the median
clinician as the standard.
5. Support a staged path toward greater autonomy, with prospective evidence
addressing the responsibility transferred to the device. Each stage could generate
prospective evidence for the next, under a predetermined change control plan.

1. Evaluate outputs by the action they produce (Questions 2, 7, 12, 14)
Open-ended GenAI outputs are difficult to evaluate exhaustively. We suggest organizing
evaluation around the clinical decisions and actions the outputs are intended to support,
while also assessing how wording affects understanding and behavior.
For any informational or action-directing function, the sponsor should state the action
pathway the output is intended to support. Many differently worded outputs then fall into a
small set of action categories, such as refer or do not refer, increase, hold or decrease a
dose, escalate or reassure, and order or do not order a test. Within each category, minor
wording differences matter less. Between categories, differences are clinically meaningful.
This makes many outputs amenable to established statistical methods. Sensitivity and
specificity are appropriate for some decisions; other tasks may require measures of
quantitative error, appropriateness, timeliness or successful completion. Endpoints,
acceptance criteria and sample sizes should follow the intended claim and clinical
consequence.
This also answers Question 2. Directiveness is best judged by what the user is likely to do
next, not by whether the output uses words such as “consider” or “recommend.” We agree
with the paper that a disclaimer does not make an output less directive. A sponsor should
be able to show, through human factors testing with intended users, which action an output
actually leads to. If an informational output is not intended to support any action, the
sponsor should explain how it is intended to help.
Unsafe or conflicting actions should be counted as failures. Outputs outside the
prespecified categories should undergo independent adjudication under predefined rules,
with their handling reported transparently. A valid response omitted from the categories
may reveal a limitation of the evaluation scheme. Such outputs should remain accounted
for in the analysis.

2. Risk assessment (Questions 1 to 6)
Question 1. The two-axis framework is a useful heuristic, and the consequences axis is well
defined. Our view is that the activity axis needs more work. It treats the degree of device
independence as the main variable. Independence is one factor, but the way the device is
built into the human’s decision matters as much. Designs that look similar on the activity
axis include the following.
A device can present an output that a clinician reviews before acting, which is the case the
paper mainly discusses. It can work in the background and speak up only when it detects a
likely human error, as a spell checker does. It can intervene only when a problem is
perceived, as lane-keeping assistance does in a car. It can act as an independent second
reader, with a third party arbitrating when the human and the device disagree. It can also
review completed work for quality assurance, such as checking registry entries or closed
charts after the fact.
These designs differ in exposure to automation bias, in what happens when the device fails
silently, and in who is positioned to catch an error. A background checker can still change
human performance through reliance, interruptions or false alerts, so each interaction
mode should be evaluated against the same workflow without the device. For independentreader designs, cases where the human and the device agree should also be checked
against an independent reference, since both may agree on a wrong answer. We
recommend representing interaction mode as its own dimension, or as a structured
modifier of the activity axis. We also support the additional dimensions the paper lists
(reversibility, downstream safeguards, time pressure and traceability). We would add the
intended user’s realistic ability to detect an incorrect output under representative
workload. Performance of the human and device together under real-world workload is
hard to predict from bench testing, which is one reason clinical confirmation matters.
Question 2. See Section 1. Automation bias is a risk at every level of directiveness,
especially under real-world workload, and it varies across users. Human factors testing
should therefore include the range of intended users, not only the most experienced.
Question 3. Patient-facing functions can create harm in both directions: false reassurance
that delays care, and undue anxiety that drives unnecessary care. Adapting language to a
patient’s health literacy can help. However, any adaptation should change how information
is expressed, not the safety-critical content or the action it leads to. The sponsor should
show that adaptations preserve safety-critical information and support appropriate
understanding and action across intended user groups. Inferring a patient’s sophistication
is itself a possible source of subgroup bias and should be tested as such.
Question 4. When the intended user lacks the expertise to evaluate an output
independently, that user cannot act as a safeguard against an incorrect output. In that case
the function is, in practice, autonomous with respect to the specialist judgment it replaces,
and it should be evaluated as an autonomous function. Evaluation should not assume that
such users will detect errors in that judgment. Other safeguards available to them should
receive credit only for what they demonstrably catch in the intended workflow. This is
consistent with the paper’s footnote 13 linking this question to the independent-review
criterion for clinical decision support [4]. There is a precedent: Autonomous diabetic
retinopathy screening was designed for use in primary care, and it was evaluated on the
assumption that the operator would not apply ophthalmic judgment to the result [2].
Extending specialist knowledge to generalist settings is a real benefit, and this approach
allows it with evidence matched to the actual risk.
Question 5. The intended use of a conversational device should be defined by the set of
actions it is permitted to direct or take across a whole encounter. Evaluation should use
complete simulated conversations as the unit of analysis, including conversations that start
within scope and drift toward an excluded action. A safe first answer should not hide an
unsafe later one. We support independent adversarial testing (red-teaming) of multi-turn
behavior, as Appendix A elements S.2 and R.1 anticipate.
Question 6. Under-escalation and over-escalation can be measured with existing tools:
sensitivity and specificity for the escalation decision against an expert panel or other
reference standard, plus time to escalation. Because the two errors are not commensurable,
we recommend against combining them into one score. The sponsor should prespecify a
minimum sensitivity for conditions that need urgent care, justified clinically for the
intended setting. Specificity should then be required to meet a separately justified limit that
reflects the burden on patients and the health system. Diabetic retinopathy screening offers
a worked example: referral sensitivity and specificity were prespecified as separate
endpoints against an independent reading-center standard [2].
3. A competency-based approach should keep statistical rigor (Questions 7 to
10, 12, 13, 17)
Question 7. Benchmarking followed by clinical confirmation is a useful framework, and we
support evaluating the final device as configured for deployment. Our main concern is that
the licensure analogy could be read as permission to accept exam-sized evidence. The
analogy cuts the other way. A human examinee can answer only a limited number of
questions. A device can be tested on many thousands of cases, and on repeated runs of each
case. What limits device testing is the human effort needed to create realistic cases and
reliable reference standards. That effort should be matched to risk, not capped by analogy
to human examinations.
Calling an approach competency-based does not change the need for statistical analysis.
Primary endpoints should be tied to the intended use and indications for use, and sample
sizes should be justified by the study objective, using power, confidence-interval precision
or bounds on clinically important failure rates as appropriate. The action mapping in
Section 1 makes this practical. Synthetic data can supplement testing if it is generated
under prespecified procedures and its realism is evaluated.
Question 8. The two-axis framework, with the interaction-mode dimension above, could set
the evidence tier. Lower-risk functions could rely mainly on benchmarking. Higher-risk
functions should require powered benchmarking of safety-critical actions plus clinical
confirmation. Autonomous action-taking functions of high consequence should require
prospective confirmation in the intended setting.
Question 9. Benchmarking can be done in layers. General capabilities that support the
device, such as clinical knowledge and general analytic skill, can be shown with simpler
benchmarks, with testing depth reflecting their contribution to clinical risk. Behaviors that
directly affect safety and effectiveness, such as escalation, refusal, dosing and diagnostic
conclusions, need more extensive benchmarking with statistically derived sample sizes.
Well-designed benchmarking of this second kind can be combined with clinical
confirmation. Existing FDA guidance on diagnostic accuracy studies [7] and on human
factors engineering [5] already provides much of the methodology.
Question 10. Public benchmarks are a reasonable starting point and support comparisons
across products. They cannot by themselves support conclusions about safety and
effectiveness, because of risk of contamination and saturation. Those conclusions should
rest on sequestered benchmarks held by an independent party, or on data the sponsor can
show was not available during development, for example data created after the model’s
training and development cutoff. Sponsor-developed benchmarks should be acceptable
when they are built under prespecified protocols, with documented case selection and
reference-standard procedures, and kept separate from the development team.
Question 12. Statistically meaningful measurement starts with prespecifying the unit of
analysis (patient, episode or decision) and the primary action-level endpoints. Repeated
generations from the same input, and multiple decisions for the same patient, are
dependent observations. Confidence intervals should account for this. Synthetic data can
establish performance on specific technical tests and supplement clinical evidence,
particularly for rare or adversarial scenarios. Clinical performance claims should be
supported by real-patient evidence. Benchmark and clinical results can contribute to the
same submission without being pooled into a single estimate.
Question 13. Synthetic data is well suited to rare and safety-critical edge cases, boundary
and adversarial testing, and perturbations such as missing units or contradictory inputs.
Safeguards include clinician review of realism, generating cases with a model of a different
class from the device, and checking that failures seen in real patients are reproduced in the
synthetic set. Using a different model to generate cases does not by itself establish
independence from shared data or shared clinical blind spots.
Question 17. Measurement, imaging and signal-processing functions already have
established validation methods, such as agreement with a reference measurement, bias and
precision. These methods should apply whether or not the function uses transformers or
other generative architectures, consistent with the paper’s statement that FDA regulates
devices, not the technology inside them.

4. Comparators, clinical confirmation and third parties (Questions 14 to 16)
Question 14. For open-ended outputs, we recommend comparing what happens next, that
is, the action the output leads to, against a reference. Where a clinical reference standard
exists, it should be used. Where it does not, a panel of qualified, independent experts should
set the reference, with the rigor of the panel matched to the risk of the device.
Median clinician performance should not, by itself, define the acceptable clinical standard.
Where feasible, both the device and a representative sample of practicing clinicians should
be measured against the same clinical reference. The care comparator should reflect the
intended setting, and any non-inferiority margin should be prespecified and clinically
justified. Whether generalists or specialists form the comparison group should follow the
intended user and the specialist judgment the device replaces (see Question 4).
The device should be evaluated in the role it is intended to play. If it is intended to work
autonomously, it should be tested autonomously. If it is intended to work with a clinician,
the human-AI team should be tested in a scenario close to intended use, including realistic
workload, and compared against the clinician working without the device.
Question 15. Comparing against the care likely to occur without the device is appropriate
where the alternative is in fact no care or delayed care, for example screening in
populations with low screening rates. In that setting, benefit should be assessed against the
actual alternative, while still showing an acceptable benefit-risk profile and justified limits
on serious errors. The sponsor should show which care the device adds to and which it
replaces.
Question 16. We welcome a role for qualified independent third parties, especially for
higher-risk devices. Benefits include greater trust in validation data and reuse of the same
sequestered data by multiple manufacturers. Third-party testing should not be mandatory.
Equivalent sponsor-run approaches under prespecified protocols should remain available.
Accredited testing bodies should be accessible to all manufacturers, large and small, on
transparent terms.

5. Graduated autonomy, postmarket monitoring and change control (Questions
18 to 24)
Question 18. Accepting greater premarket uncertainty can be appropriate if it is justified
and bounded. Accepting greater unmitigated risk is not. We welcome greater use of
postmarket evidence where the remaining uncertainty is defined, outcomes can be
observed in time to act, exposure can be limited, and a corrective action is ready. It is not
appropriate where an error can cause irreversible harm before monitoring could detect it.
A practical form of this is graduated autonomy, which mirrors the progressive
independence of clinical training that the paper describes. A device could first be deployed
with clinician confirmation of each action, generating prospective evidence for a later stage.
That evidence should include independent assessment, relevant clinical outcomes, and the
frequency and importance of clinician corrections and rescue actions. Agreement with the
confirming clinician alone is insufficient, since it can reflect anchoring or shared error.
Evaluation should address the configuration proposed for greater autonomy, including the
consequences of removing supervision. FDA should clarify which transitions could be
included in an authorized PCCP and which would require a new marketing submission.
Question 19. We support re-testing on retained benchmarks to detect regression and on
protected new cases to assess emerging failure modes. Monitoring should combine
operational signals, which may be continuous, with periodic independent review of
sampled cases and clinical outcomes, which are often delayed. Frequency should reflect
exposure, time to harm and the availability of reliable outcome data. Material changes to
the model, prompts, retrieval sources, tools or intended population should trigger
proportionate reassessment. Higher-risk devices need earlier warning and correctiveaction thresholds that can be acted on before avoidable harm occurs.
Question 20. Machine-based supervisory agents, including AI agents, can help scale
monitoring by flagging suspicious outputs for review. Monitoring should not rely on them
alone. A supervisory agent should be evaluated like any other test method, with its own
missed-error and false-alert rates measured against an appropriate independent clinical
reference, using qualified adjudicators where needed. Where the supervisor and the device
share an underlying model, their errors may be correlated, and this should be documented
and addressed.
Question 21. The manufacturer remains accountable for device performance and corrective
action. Clear role definitions will help every stakeholder. We recommend that the
responsibilities of manufacturers, deploying institutions and clinicians be defined in
labeling and deployment agreements, informed by the device’s benefit-risk profile, the
known error rates of the standard of care it supports or replaces, and the performance
accepted at authorization. Questions of legal liability are largely outside FDA’s remit, but
clear allocation of operational responsibilities would reduce confusion.
Question 22. The PCCP is a strong mechanism for GenAI devices and should be used to
allow streamlined improvement [6]. The premarket benchmark is a natural baseline. The
extent of re-testing should follow the expected impact of a change and the benefit-risk of
the device. Changes demonstrated not to affect clinical behavior or safety may warrant
limited verification and quality-system documentation. Changes to presentation should not
be presumed clinically neutral. Changes to the model version, prompts, retrieval corpus or
tool permissions should require re-benchmarking of the affected actions before
deployment plus a retained core safety suite, and clinical confirmation where agreed in
advance. A new population, new clinical task or higher level of autonomy changes the
evidence question itself and should not be handled as a routine modification.
Question 23. Where future changes cannot be fully specified, a PCCP can still describe a
bounded class of permitted change, the clinical actions that must not change, the validation
and acceptance criteria every change must pass, and a fallback. It should not provide openended permission for unspecified changes. We encourage FDA to clarify whether particular
staged-autonomy transitions could qualify, consistent with intended-use limits, and what
evidence and review would be needed otherwise [6].
Question 24. The manufacturer is responsible for detecting, evaluating and mitigating
changes to third-party foundation models, and for reporting them where required.
Practical controls include pinning model versions, contractual advance notice of changes,
access to a candidate version for re-benchmarking before deployment, automated checks
that the model version in use matches the validated one, and a documented fallback if a
version is retired.

6. Foundation Model Master Files and agentic systems (Questions 25 and 26)
Question 25. Voluntary Foundation Model MAFs are a good idea. They would reduce
repeated requests to model developers and give review teams consistent information.
Model developers also have an incentive to take part, since a well-documented model is
easier for device manufacturers to adopt. Files should be version-specific and kept current.
Using a MAF should not imply that the foundation model is authorized for any clinical use,
and it should not reduce the device manufacturer’s responsibility to validate the device.
Question 26. For agentic systems that decide next steps or take actions, the competencybased approach in the paper and FDA’s existing framework for autonomous AI devices both
apply. The action-based evaluation in Section 1 is especially important here. Every action
an agent can take should be listed and characterized, and each one should be validated
systematically, with sample sizes based on power calculations and matched to the
consequence of that action. Irreversible actions warrant the strictest acceptance criteria.
Authorization should apply to the specific patient, action and parameters being executed,
and should not carry over from earlier in a session. Testing should cover complete action
sequences, including tool failures and recovery, so that responsibility for unfinished care is
never left unassigned.
Closing
Many of the questions the paper raises can be answered by applying established evaluation
methods to the actions GenAI devices produce. We would welcome the opportunity to
contribute to worked examples and submission templates, and we thank CDRH for its open
approach to this topic.
Sincerely,
Kaushal Solanki, PhD
CEO and Principal Scientist
Solan Health, Inc.

References
[1] U.S. Food and Drug Administration. Considerations for the Regulation of Generative AIEnabled Medical Devices: Discussion Paper and Request for Feedback. August 2026.
https://www.fda.gov/media/194242/download
[2] Ipp E, et al. Pivotal Evaluation of an Artificial Intelligence System for Autonomous
Detection of Referrable and Vision-Threatening Diabetic Retinopathy. JAMA Network Open.
2021;4(11):e2134254. doi:10.1001/jamanetworkopen.2021.34254
[3] U.S. Food and Drug Administration. Clinical Decision Support Software: Guidance for
Industry and Food and Drug Administration Staff. January 2026.
https://www.fda.gov/media/109618/download
[4] U.S. Food and Drug Administration. Applying Human Factors and Usability Engineering
to Medical Devices: Guidance for Industry and Food and Drug Administration Staff. August
2026. https://www.fda.gov/media/80481/download
[5] U.S. Food and Drug Administration. Marketing Submission Recommendations for a
Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software
Functions. August 2025. https://www.fda.gov/media/166704/download