Froda AI (Shahriar Hossain)

On behalf of Froda AI, please find attached our comment on Docket FDA-2026-N-7874. It addresses Questions 5, 19, 20, 21 and 24, focusing on runtime controls for generative and agentic AI-enabled devices. It covers defining intended use through clear boundaries, enforcing safety-critical responses through deterministic rules outside the model, ensuring reliable supervision, maintaining clear accountability across stakeholders, and detecting changes to third-party foundation models.

Public Comment: Docket FDA-2026-N-7874
Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and
Request for Feedback

Submitted by: Shahriar Hossain, Founder, Froda AI (froda.ai)

Date: October 19, 2026

Questions addressed: 5, 19, 20, 21, 24

About this comment
To the Center for Devices and Radiological Health:

I am the founder of Froda AI, a startup that builds runtime governance software for AI agents. Our
software sits between an agent and the tools it uses, and decides, before each action runs, whether to
allow it, block it, or hold it for a human. We have also open-sourced gitfence, a set of deterministic
guardrails for AI coding agents.

I am not a clinician, and Froda does not build medical devices. I have limited this comment to the
questions where our everyday engineering work gives me something specific to offer: what happens at
runtime, between a generative model and the outputs and actions it produces.

Disclosure. Froda sells software in this category, so I have a commercial interest in how FDA treats
runtime controls. I propose no role for Froda in any FDA program, and I note the interest so the Center
can weigh these comments accordingly.

Summary of recommendations
Question Issue Recommendation

5 Conversations can drift across Define intended use by boundaries; let models
boundaries that no single turn detect a boundary but enforce the response with
crosses deterministic rules; replay recorded conversations
as test cases

19 Problems may surface only at the Track how often rules block, escalate or are
next scheduled review overridden, and treat sudden changes as a trigger
for further testing

20 A supervisory model may share Use models only to flag risks, and let fixed rules
the blind spots of the model it decide the response. Keep those rules separate
watches from the models, stop safely when supervision fails,
and test the rules for gaps and conflicts before
release.

Public Comment, Docket FDA-2026-N-7874 | Shahriar Hossain, Froda AI Page 1 of 5
 Question Issue Recommendation

21 Involving many stakeholders can Keep the manufacturer responsible; let institutions
blur accountability tighten rules but not weaken them; record who
made each change

24 Third-party models can change Pin versions; declare the model in a bill of materials;
without the manufacturer's verify it at runtime; run quick regular checks;
knowledge require advance notice of changes

Response to Question 5: Conversational trajectories
The problem

Medical device frameworks usually tie intended use to fixed software requirements that can be tested
before release. A conversational device does not work that way, because the user shapes the
conversation as it goes. As Section IV notes, a device can start out informational and become actiondirecting over an exchange, even when no single turn looks out of scope. Testing every possible path is
not feasible, and testing turns one at a time misses exactly this kind of gradual drift.

Recommendations

1. Define intended use by setting clear boundaries, not by predicting every conversation. A
conversational device’s intended use can be defined by what it is allowed to do and where its
limits lie. For example, it might provide general wellness information but not interpret a
patient’s own lab results. The manufacturer should define and validate in advance how the
device must respond when a conversation crosses each boundary, such as referring the user to a
clinician, displaying a predefined safety message, or refusing the request and directing the user
elsewhere. This makes the device’s intended use easier to test, even when conversations take
unexpected turns.
2. Separate detecting a boundary from deciding what happens next. Detecting when a
conversation is approaching a boundary is a judgment call made by the model or a separate
classifier. How reliably it detects these situations can be tested under elements S.2 and R.1. The
response, once a boundary is detected, should be enforced by a deterministic layer outside the
generative model. That way the response cannot be argued away by a user, worn down over a
long conversation, or changed by a foundation model update. Premarket testing then checks a
small set of rules, such as "a conversation that reaches domain X always triggers protocol Y,"
instead of an unbounded set of paths.
3. Record conversation trajectories and reuse them for testing. Each time a conversation
approaches a boundary, gets escalated or is overridden by a human, the system should record
what happened, along with the relevant session context and the policy and model versions in
use. Over time, these records, de-identified where appropriate, build a collection of real-world
conversations, including cases where things started to go off track. Testing new rules or model
versions against these conversations helps catch problems that might otherwise be missed
because they were not anticipated in the original test scenarios.

Public Comment, Docket FDA-2026-N-7874 | Shahriar Hossain, Froda AI Page 2 of 5
Response to Questions 19 and 20: Supervisory agents and postmarket monitoring
The problem

Question 20 asks whether machine-based supervisory agents can support postmarket monitoring. They
can, but a supervisor built from a generative model of the same class as the device may share the
device's blind spots. If the same kind of model both acts and judges, one prompt or one model update
can defeat both at once. In addition, a supervisor that only watches and reports will usually identify a
problem after a patient has already received the output.

Recommendations

1. Use models to spot problems, but let clear rules determine what happens next(Question 20).
A supervisory model can help identify potential emergencies, out-of-scope requests or
conversations that are starting to drift. But whether to allow, block, pause or escalate should be
determined by explicit rules that the manufacturer sets, reviews and versions. A runtime
governance layer can enforce these rules between the device’s models and the actions they
trigger, whether that layer is built in-house or provided by a third party.
2. Keep enforcement separate from the models (Question 20). The rules that control what the
device can do should sit outside both the device’s model and the supervisory model. This way,
an update to either model cannot silently change the rules. Neither model should be allowed to
modify its own rules.
3. Fail safely when supervision is unavailable (Question 20). The supervisory model may be
unavailable, time out or fail to reach a reliable decision. The runtime layer itself may also fail to
process an action. In any of these cases, the action should be stopped and the system should
switch to a validated safe state, such as handing control over to a clinician or displaying a
predefined safety message. Missing supervision should never be treated as permission to
proceed.
4. Test rules before putting them into use (Question 20). Explicit rules are easier to verify than a
generative model, but they still need to be tested against realistic situations. Before new rules
are deployed or existing ones are changed, the enforcement layer should let teams test them
against a broad set of scenarios, including recorded real-world interactions. This helps catch
cases where the rules fail to respond as expected before patients are affected. Every rule change
should be tested before release, just like any other device update.
5. Detect conflicts between rules (Question 20). As more rules are added, they may overlap or call
for different outcomes in the same situation. This can lead to inconsistent results depending on
which rule takes priority. The enforcement layer should identify these conflicts before
deployment and flag them for review. If a conflict still occurs at runtime, a defined order of
precedence should apply, such as the most protective outcome taking priority.
6. Use enforcement activity to spot changes in real-world behavior (Question 19). Tracking how
often the rules block an action, escalate a case or are overridden by a clinician can help show
how the device is performing in practice. A sudden change in these rates, especially after a
model update, could signal a problem and prompt further testing or clinician review before the
next scheduled check.

Public Comment, Docket FDA-2026-N-7874 | Shahriar Hossain, Froda AI Page 3 of 5
Response to Question 21: Accountability across stakeholders
The problem

The same issue affects postmarket monitoring. When something goes wrong in a conversation, the
manufacturer, the deploying institution and the clinician each hold part of the picture. Without a shared
record that ties each decision to the party responsible for it, involving more stakeholders makes
accountability less clear, not more.

Recommendation

1. Let responsibility follow policy decisions. The manufacturer remains responsible for defining
the rules and deciding when a case must be escalated. Hospitals and other institutions can make
these rules stricter to fit their needs, but should not be able to weaken them. Every change
should be recorded along with the name of the person or organization that made it. Clinicians
and professional societies can help define the domain categories and the criteria for escalation.
Keeping a record of each party’s contributions makes it clear who is responsible for what.

Response to Question 24: Changes to third-party foundation models
The problem

A device can start behaving differently without its manufacturer changing anything. A hosted model may
be updated behind an unchanged name, retired, or given new safety filters. Separately, a device
delivered as a sealed system gives the deploying institution no way to confirm which model is actually
running. In both cases, the documented configuration and the real behavior can drift apart without
anyone noticing.

Recommendations

1. Pin model versions. Where a provider offers fixed snapshots, the device should call a specific
version rather than a name that points to whatever is current.
2. Declare the model in a bill of materials. Building on FDA’s software bill of materials (SBOM)
requirements for cyber devices, manufacturers could also document each AI model, its version,
and a way to verify its identity: a weights hash for self-hosted models or an API snapshot
identifier for hosted ones.

Component Example entry

Foundation model (self-hosted) Llama 3.2 11B Vision, weights hash sha256:4f2a…

Foundation model (hosted) Provider model, snapshot ID 2026-05-14

Fine-tuning Sponsor clinical dataset, version 4

A hospital receiving a sealed agentic system usually cannot inspect its internal components. A declared
bill of materials gives the hospital and FDA a reference to check the running system against, rather than
relying on a model name alone.

Public Comment, Docket FDA-2026-N-7874 | Shahriar Hossain, Froda AI Page 4 of 5
 3. Verify at runtime. Where model calls pass through a governance layer, the model identifier on
each call can be checked against the declared bill of materials, and any mismatch flagged
automatically.
4. Use quick, regular checks to know when full testing is needed. A hosted model can change its
behavior without changing its version name, so version identifiers alone are not enough.
Running a full set of tests too often can be expensive, but a small set of safety-critical prompts
from the original premarket tests can be run daily at low cost. If the results change, that is a
signal to run the full set of tests described in Section VI, rather than waiting for the next
scheduled check.
5. Require advance notice of model changes. The device manufacturer is responsible for any
model change, regardless of who makes it. Agreements with model providers should require
advance notice of updates and of when a model will be discontinued. Manufacturers should in
turn notify hospitals and other institutions using the device of any model change before it takes
effect. Contracts help manage these changes, but they do not replace the need for technical
checks.

Closing
The discussion paper is right that GenAI-enabled devices cannot be evaluated by testing every input. My
suggestion is that some of the hardest guarantees, such as escalation, scope limits and stopping before
an irreversible action, do not have to depend on the model at all. Placing them in a deterministic layer
outside the model makes them testable before market and observable after it. In our experience
governing AI agents in enterprise settings, this separation is practical to build today, and I expect it to be
at least as useful in clinical settings.

I would be glad to provide more detail on any of these points.

Respectfully submitted,

Shahriar Hossain

Founder, Froda AI

froda.ai

Public Comment, Docket FDA-2026-N-7874 | Shahriar Hossain, Froda AI Page 5 of 5