Does the approach still work for devices built on other model architectures, such as multimodal vision-language models and world models?

Question 17: Other model types The three-layer approach can apply to different model types. A multimodal device needs tests for: Incorrect links between text, images, and other data. Missing or conflicting inputs. Incorrect patient or time-point matching. Unsupported statements about an image or signal. Changes in imaging equipment or acquisition method. A predictive world model needs tests for: Error across multiple predicted steps. Incorrect counterfactual results. Uncertainty in future states. Failure when real events differ from predicted events. The final device remains the main unit of evaluation.

Question 17 - Applicability to multimodal, vision-language, and world-model architectures The competency-based framework should remain architecture-neutral. The evaluation target should be the clinically consequential behavior of the final configured device, not the internal model class. Multimodal and world-model systems may require additional attention to cross- modal consistency, patient/encounter association, provenance of each modality, temporal alignment, error propagation between modalities, and the possibility that one modality silently dominates another. As architectures evolve, the assurance framework should therefore focus on intended function, input/output behavior, configuration state, failure modes, and downstream authority rather than tying regulatory expectations to a particular model architecture. 9 SECTION VI - POSTMARKET MONITORING

Question 17 — Can competency-based evaluation apply across model architectures? Regulate the clinical capability. Let the architecture change underneath it. A foundation model, a multimodal model, and whatever comes after it should all face the same five questions: What clinical function does it perform? What information does it use? What authority does it have? What happens when it's wrong? How does it behave outside its scope? Architecture-specific rules will be obsolete before they're finalized; competency-based rules will not. Part III — Postmarket Monitoring

Question 17 — Other model architectures I would encourage caution in extending the framework by analogy. The competency-based approach is workable for language-centric devices in part because a meaningful, if flawed, ecosystem of language benchmarks exists. For multimodal vision-language models and generative or predictive world models, the benchmarking assets are substantially thinner, and I do not have verified information on what is currently available for clinical use cases in these modalities. The structural implication is that where benchmarking assets are immature, clinical confirmation must carry proportionally more weight. I recommend CDRH state this scaling principle explicitly rather than allowing a uniform framework to imply that a thin benchmarking record is as informative as a rich one. Otherwise the framework’s evidentiary flexibility will be claimed most aggressively exactly where the underlying evidence base is weakest. III. Postmarket monitoring (Section VI; Questions 18–24)

FDA Question 17 - Other model architectures Trace ID. TR-Q17 | FDA Q17; Sec. V.E; App. B; pp. 18-19 / 28-29 BCR response. Keep the closure architecture technology-neutral but add architecture-specific failure branches: multimodal consistency, world-model state/long-horizon error, clinically relevant morphology preservation, and action-state verification for autonomous controllers. BCR rule basis. BCR-R01,R06,R10,R17 Solution-stack link. S1,S2,S3,S6,S9 Closure evidence. Architecture-specific tests: multimodal consistency, world-model state error, action-state verification, etc. Pass / re-open. Architecture-specific failure modes are mapped to evidence Re-open when: Underlying architecture/modality/tooling change.

Response to Question 17: Multimodal Architectures Multimodal devices — those integrating imaging AI with clinical language model components, for example — require explicit framework adaptation. A device that analyzes echocardiographic imaging while simultaneously processing the patient's clinical history through a language model presents a combined risk profile that cannot be adequately assessed by treating each modality in isolation. Each modality must satisfy its domain-specific benchmarking requirements, and the integrated device must demonstrate that combined performance is not materially degraded relative to each component's standalone performance. FDA should issue a specific companion guidance document for multimodal architectures, developed in consultation with imaging, clinical AI, and specialty professional society stakeholders. V. SECTION VI: POSTMARKET MONITORING — RESPONSES TO DISCUSSION