March Global Inc DBA Basion

March Global Inc., doing business as Basion, builds a patient-facing AI assistant for people and families living with rare and complex conditions, together with the evaluation infrastructure we use to test it. Our full comments are attached. In summary:
Evaluate the deployed device (Q7). We support the competency-based approach and its principle that the final user-facing device, as configured for deployment, is what gets evaluated. Published studies show that results for a base model or a different access mode do not carry over to deployed products.
Measure safety-critical behavior directly (Q9, Q10). Pair outcome benchmarks with prespecified process measures taken from execution logs. In our testing, one configuration change left a public benchmark score unchanged while the rate at which our assistant consulted the patient’s record fell from 62% to 46%.
Add record faithfulness (Q1). For devices that act on a patient’s own data, assess whether each patient-specific statement matches the record as it stood at that moment, alongside traceability to primary sources.
Do not average away rare presentations (Q6, Q10). Report results stratified by condition prevalence, and score escalation against the patient-specific plan where the record contains one, reporting under-escalation and over-escalation separately.
Safeguard synthetic data (Q13). Build synthetic cases from curated, citable knowledge sources, render them in standard record formats, and use different model families to generate, simulate and judge than the one inside the device.
Use two comparators for rare conditions (Q14, Q15). Report performance against a specialist panel, which sets the knowledge standard, and against the realistic alternative, which is often a general-purpose chatbot or a long wait for a specialist. Evaluate patient-facing devices as used by patients and caregivers, not only alone.
Constrain supervisory agents (Q20). Allow them in postmarket monitoring with read-only access, evaluation against the record as it stood, device logs treated as claims to verify, evidence before verdicts, human decisions on fixes, and independent validation of the supervisory agent itself.
Re-measure after any change (Q22, Q24). Treat configuration and cost changes, not only model and prompt changes, as triggers for re-measuring safety-critical behaviors against the premarket baseline.
Test agentic devices end to end (Q26). Benchmark from raw source documents through the full pipeline, since extraction errors are invisible when an evaluation supplies clean structured inputs.
Involve patients and independent parties (Q11, Q16). Include patients and caregivers as adjudicators of communication quality and as standardized patients, and accredit multiple independent third-party providers, including condition-specific ones.
We would welcome the opportunity to share our methods and to take part in pilots of competency-based benchmarking for patient-facing devices.

Comments on Docket No.
FDA-2026-N-7874
Submitted electronically via Regulations.gov
Dockets Management Staff (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Rm. 1061
Rockville, MD 20852
October 19, 2026
Re: Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion
Paper and Request for Feedback (Docket No. FDA-2026-N-7874)
Dear Members of the Digital Health Center of Excellence:
March Global Inc., doing business as Basion, appreciates the opportunity to comment on the
discussion paper. Basion builds a patient-facing AI assistant for people and families living with
rare and complex conditions. It organizes a patient's medical records into a structured profile,
grounds its answers in that record and in curated biomedical sources, cites the source of each
claim, and is designed to work alongside a patient's clinicians, not to replace them. To test it, we
built evaluation infrastructure that runs the deployed assistant through simulated, multi-turn
conversations with synthetic patients who have complete medical records, and an automated
reviewer that checks the assistant's statements against the patient's record. Our comments
draw on that work.
We focus on patients with rare diseases, complex multisystem conditions, and conditions for
which standard treatments have been exhausted. According to the National Center for
Advancing Translational Sciences, about 30 million people in the United States have one of
roughly 10,000 known rare diseases; half of those diagnosed are children, diagnosis commonly
takes 3 to 15 years, and 95% of rare diseases lack an FDA-approved treatment. These patients
are underrepresented in model training data and in public benchmarks. Evaluation methods
tuned to common presentations can miss the failures that matter most for them, and our
comments are aimed at closing that gap.

Summary of recommendations
1.​ Support the competency-based approach and its principle that the final user-facing
device, as configured for deployment, is what gets evaluated (Question 7).
2.​ Pair outcome benchmarks with prespecified process measures of safety-critical behavior,
taken from the device's execution logs (Questions 9 and 10).
3.​ For devices that act on a patient's own data, add record faithfulness to the risk framework
and the benchmarking elements, alongside traceability to primary sources (Question 1).
4.​ Require results stratified by condition prevalence, and score escalation against
patient-specific plans where the record contains one (Questions 6 and 10).
5.​ Build synthetic test data from curated, citable knowledge sources, and separate the
models that generate, simulate and judge from the model under test (Question 13).
6.​ For rare conditions, compare performance with both a specialist panel and the realistic
alternative available to the patient, and evaluate patient-facing devices as used by
patients (Questions 14 and 15).
 7.​ Allow machine-based supervisory agents in postmarket monitoring under specific design
constraints, which we describe from operating one (Question 20).
8.​ Treat configuration and cost changes, not only model and prompt changes, as triggers for
re-measuring safety-critical behaviors (Questions 22 and 24).
9.​ Benchmark agentic devices from raw source documents through the full pipeline
(Question 26).
10.​Include patients and caregivers as adjudicators and standardized patients, and accredit
multiple independent third-party providers (Questions 11 and 16).

Risk assessment (Questions 1, 3, 5 and 6)
Record faithfulness as a risk dimension (Question 1). The paper names traceability of an
output to primary source materials as a candidate risk dimension. We support it, and we suggest
a related dimension for devices that act on patient-specific data: record faithfulness, meaning
whether each patient-specific statement matches the patient's record as it stood when the
statement was made. A device can cite a sound guideline and still harm a patient by misstating
that patient's allergy, dose or diagnosis. Traceability tells a user where a claim came from;
record faithfulness tells the user whether the claim is true of them. Both reduce the chance that
a user relies on an incorrect output, so a device that shows the source of each patient-specific
statement could be placed lower on the consequences axis, provided benchmarking shows that
its attributions are accurate.
Patient-facing functions (Question 3). We agree with CDRH that adding a disclaimer does not
make an action-directing output less directive. We would not presume that patient-facing
functions carry higher risk as a class. In rare disease, the patient or parent is often the person in
the room who knows the most about the condition, and the realistic alternative to a well-built tool
is often a general-purpose chatbot or no help at all. We suggest that risk assessment credit
specific mitigating features, and that benchmarking test whether each feature works, not only
whether it is present:
●​ citations for each claim, to the patient's record or to primary literature, that a patient can
show a clinician;
●​ explicit statements of uncertainty when evidence is limited or contested (element S.3);
●​ handoff features that prepare the patient to raise a question with a clinician, such as a
summary of the relevant record entries;
●​ adherence to patient-specific care plans already in the record, such as an emergency
protocol.
Conversational trajectories (Question 5). We support assessing risk across realistic
conversational trajectories. In practice this requires a simulated patient with a persistent,
complete record; conversations that run many turns and, for some devices, several sessions;
and scoring at each turn, not only at the end, so that the turn where a device moves from
information to direction can be identified. The intended use of such a device could then be
described as a defined set of trajectory classes, each with its own acceptance criteria.
Escalation that depends on the record (Question 6). For many rare conditions, the correct
escalation depends on the individual record, not on population triage rules. A fever in a child
with adrenal insufficiency or a urea cycle disorder can call for an emergency plan that would be
unnecessary for most children with a fever, and families of children with such conditions often
carry a written emergency protocol. A benchmark that gives the device no patient record cannot
test this behavior. We recommend that escalation benchmarks include record-dependent
scenarios, that under-escalation and over-escalation be reported separately, and that the
scoring standard be the patient-specific plan wherever the record contains one.

Premarket evaluation (Questions 7, 9, 10, 11, 13, 14, 15
and 16)
Evaluate the deployed device (Question 7). We support the competency-based approach,
and in particular the principle that the final user-facing device, as configured for deployment, is
what gets evaluated. Two recent studies show why this matters. In a randomized study with
1,298 participants, LLMs tested alone identified the relevant conditions in 94.9% of scenarios,
while participants using the same LLMs identified them in fewer than 34.5%, no better than a
control group; the authors report that standard medical knowledge benchmarks and simulated
patient interactions did not predict these failures (Bean et al., 2026). That finding applies to
simulation of the kind we use, and it is why we recommend below that simulation be paired with
testing by representative users. In a study of 30,000 responses to suicide-related prompts,
very-high-risk prompts received a direct answer in 24.8% of responses through model APIs and
in 4.6% through consumer apps, and the authors conclude that safety observed in a consumer
app should not be assumed to hold for apps built on the API (Cantor et al., 2026). Many
GenAI-enabled devices will be built on foundation model APIs.
Process measures alongside outcome scores (Questions 9 and 10). An outcome score can
stay flat while a safety-critical behavior gets worse, if the benchmark does not exercise that
behavior. In our own testing, lowering one configuration setting (the reasoning-effort level of the
underlying model) cut the cost per conversation by about 30% and produced no measurable
change in our score on a widely used public benchmark, HealthBench. Over the same runs, the
rate at which our assistant checked the patient's record before answering fell from 62% to 46%
(246 conversations per arm, p = 0.0003), and it rose to 76% at the highest setting. Because the
benchmark conversations come with no patient record, skipping that check cost nothing on the
score. In deployment, answering a medication question without checking the record is how a
documented allergy or drug interaction is missed. We recommend that benchmarking pair
outcome scores with prespecified process measures for safety-critical behaviors, measured
from the device's execution logs, each with its own acceptance criterion. Examples include
whether the device consulted the patient's record before giving medication-related information,
whether it escalated on defined red flags, and whether it retrieved a source before making a
factual claim.
Benchmark validity (Question 10). We recommend:
●​ a sequestered holdout set maintained by a party independent of the sponsor and
refreshed on a set schedule, with methods and rubrics published and test items kept
private;
●​ acceptance of sponsor-developed benchmarks when they are prespecified and reported
alongside the independent holdout, with the gap between the two reported;
●​ results stratified by condition prevalence, so that strong performance on common
presentations cannot hide failures on rare ones;
●​ evidence of construct validity in the form of agreement between benchmark results and
clinician adjudication of real or standardized-patient interactions with the same device.
Synthetic data (Question 13). Synthetic data is well suited to rare conditions, and often the
only practical option, because each condition has few real cases and subgroups cannot be
sampled. It is inadequate when a model of the same class as the device writes cases from its
own knowledge, because the cases then share that model's gaps. We recommend these
safeguards:
●​ build cases from curated, citable sources, such as Human Phenotype Ontology
annotations, OMIM and Orphanet disease definitions, ClinVar variant records and
published case reports, and record the source of every clinical fact in each case;
●​ render cases in standard record formats (we use FHIR) together with realistic source
documents, so that the device takes in a synthetic patient the same way it takes in a real
one;
●​ use model families for case generation, patient simulation and judging that differ from the
one inside the device;
●​ have qualified clinicians review a random sample of cases against prespecified criteria,
and report the rejection rate;
●​ estimate the gap between synthetic and real performance on a calibration set of real or
published cases.
Comparators (Questions 14 and 15). For rare conditions, two comparators answer different
questions, and we recommend that sponsors report both. A panel of specialists in the condition
sets the knowledge standard, since a generalist panel will often not know the correct answer for
an ultra-rare disease. The realistic alternative sets the benefit standard: for many of these
patients, the alternative to the device is a general-purpose chatbot, an internet search or a long
wait for a specialist, not a specialist consultation. Patient-facing devices should also be
evaluated as a human-AI team, meaning the patient or caregiver using the device, as well as
alone, since Bean et al. show that the two can differ widely.
Independent third parties and patient involvement (Questions 11 and 16). We support a
role for qualified independent third parties, especially in maintaining sequestered datasets and
in adjudication. Suggested independence criteria are no financial interest in the device under
test or in its foundation model developer, disclosure of any competing products, and published
methods and rubrics with test items kept sequestered and refreshed. To avoid a bottleneck that
limits competition, the program could accredit multiple providers, as the ASCA program does,
and allow condition-specific providers, such as those working with patient foundations and
specialist networks, to qualify for narrow clinical domains. We also recommend that patients and
caregivers serve as adjudicators of communication quality (element E.4) and as standardized
patients in clinical confirmation. They are the intended users, and they judge whether an output
is understandable differently from clinicians. [Optional: Basion develops evaluation tools of this
kind and may seek to provide them for devices in which it has no interest.]

Postmarket monitoring (Questions 20, 22 and 24)
Supervisory agents (Question 20). We operate an automated reviewer that checks a sample
of our assistant's conversations against the patient's record, and we support allowing this
approach under design constraints. These are the constraints we found necessary:
●​ Read-only by construction. The reviewer has no tool that can write to the record or
change the device, so it cannot alter what it is checking.
●​ Evaluation against the record as it stood. The reviewer checks each statement against the
record as of the moment the statement was made. In an earlier version, where this was
optional, the same conversation received scores of 5, 5 and 4 out of 5 on one run and 2, 2
and 1 on another, because the reviewer sometimes judged earlier statements against
later data.
 ●​ Device logs treated as claims to verify. A device's own execution trace can contain wrong
or fabricated tool results, so the reviewer does not treat it as ground truth.
●​ Evidence before verdict. A low safety score requires a filed issue that quotes the
statement and the conflicting record entry.
●​ Defects classified by origin (instructions, tools, data ingestion or infrastructure) and
grouped by type, so that a pattern across many conversations is handled as one problem.
●​ Proposals, not changes. The reviewer can propose a fix, and a person decides. A
deterministic rule set assigns each proposed change a risk tier, and the model can raise
that tier but never lower it.
We also recommend that a supervisory agent itself be validated against independent clinician
adjudication on a sample of cases, that it be re-validated when its own underlying model
changes, and that, where possible, it run on a different model family from the device it
supervises.
Changes that trigger re-measurement (Questions 22 and 24). The configuration example
above is also a change-control example: a cost setting that a sponsor might classify as minor
changed a safety-critical behavior without changing the headline score. We recommend that any
change, including changes to configuration settings, retrieval parameters, prompts, tool
definitions or the underlying model version, trigger re-measurement of a prespecified set of
safety-critical process measures against the premarket baseline, with full re-benchmarking
reserved for changes that move those measures outside prespecified bounds. For third-party
foundation models, manufacturers can pin model versions, require advance notice of model
changes and deprecations in their contracts, and complete the re-measurement before
switching versions. A device designed to work with more than one foundation model can also
compare candidate models on the same benchmark before a switch.

Agentic devices (Question 26)
One failure class deserves explicit attention. Agentic devices often build their own structured
view of a patient by extracting data from documents. When extraction goes wrong, for example
when a diagnosis mentioned only as a possibility is recorded as an active condition, the device
can then reason faithfully from incorrect data, and an evaluation that supplies clean structured
inputs will never see the error. We recommend that benchmarking of agentic devices start from
raw source documents and exercise the full pipeline, and that the evaluation record which tools
the device called and what they returned, so that a correct final answer reached through an
incorrect path is still detected.

Closing
We thank CDRH for opening this discussion early. We would welcome the opportunity to share
our methods in more detail and to take part in any pilot of competency-based benchmarking for
patient-facing devices, particularly for rare and complex conditions.
Respectfully submitted,
March Global Inc., d/b/a Basion
2261 Market Street STE 22009, San Francisco, CA 94114
support@basion.ai
References
●​ Bean AM, Payne RE, Parsons G, et al. Reliability of LLMs as medical assistants for the
general public: a randomized preregistered study. Nature Medicine. 2026;32(2):609-615.
doi:10.1038/s41591-025-04074-y
●​ Cantor J, Huang E, Zhang LA, et al. Suicide-related responses from AI chatbots through
application programming interfaces and consumer-facing apps: observational
cross-sectional comparative study. Journal of Medical Internet Research.
2026;28:e97968. doi:10.2196/97968
●​ National Center for Advancing Translational Sciences. Delivering Hope for Rare Diseases
(fact sheet). January 2023.
https://ncats.nih.gov/sites/default/files/NCATS_RareDiseasesFactSheet.pdf
●​ OpenAI. HealthBench. May 2025. https://openai.com/index/healthbench