Donald Casey

My comments are contained in the attached documents and relate to the potential legal liability risks to healthcare delivery organizations associated with adverse events that have a potential causal link from the use of artificial intelligence methods for clinical care of patients. For the first document, the questions highlighted in red and yellow are mine and the outputs were generated by Microsoft Co-Pilot. I often serve as a corporate negligence expert witness in medical malpractice cases and have shared this with malpractice defense attorneys who have confirmed a reasonable validity of these concerns.
The second document was recently published in the Pennsylvania Gazette, the alumni magazine for the University of Pennsylvania, which I believe reinforces these future and largely unaddressed concerns.
Sincerely,
Donald Casey MD MPH MBA MACP FAHA, Adjunct Professor of Healthcare Quality and Safety and Population Health at the Thomas Jefferson University College of Population Health and Affiliate Faculty, University of Minnesota Institute for Healthcare Informatics.

Reflections on Lab 2.0 2026
1 1. What sort of Enterprise and Corporate Negligence Risks might arise with the deployment
2 of patient-facing AI generated and/or developed APIs and CDS?
4 From Co-Pilot:
5 Below is a clear, structured synthesis of enterprise-level and corporate-negligence risks that
6 arise when health systems deploy patient-facing AI, including AI-generated APIs, LLM-driven
7 chat interfaces, and clinical decision support (CDS) tools. I’m framing this the way a hospital
8 attorney, a quality leader, and a digital-health safety engineer would jointly analyze it — which fits
9 your style of benchmarking legal doctrine against modern AI governance.
11 🏥 Enterprise & Corporate Negligence Risks for Patient-Facing AI and CDS
12 Hospitals have non-delegable duties under corporate negligence doctrine (e.g., Thompson v.
13 Nason in Pennsylvania), and those duties now extend to AI-enabled care delivery. When a
14 health system deploys patient-facing AI, it assumes responsibility for:
15 • The safety of the technology
16 • The competence of those who use it
17 • The policies and oversight governing it
18 • The foreseeability of harm arising from its use
19 AI changes the risk surface dramatically.
20 Below is a structured map of the major risk domains.
22 ⚠️ 1. Clinical Safety & Misdiagnosis Risk
23 AI that interacts directly with patients can create direct clinical harm if:
24 • It provides incorrect triage advice (e.g., telling a patient chest pain is “likely anxiety”).
25 • It generates hallucinated medical guidance that patients treat as authoritative.
26 • It gives incomplete or outdated recommendations due to model drift.
27 • It fails to escalate urgent symptoms because of poor prompt design or inadequate guardrails.
28 Corporate negligence exposure:
29 Failure to maintain safe systems, failure to oversee clinical practice, and failure to enforce
30 adequate policies.
32 🧩 2. Duty to Vet, Validate, and Monitor AI (VVUQ Failure)
33 Hospitals must ensure that AI tools are:
34 • Validated for intended use
35 • Tested for bias, drift, and reliability
36 • Monitored post-deployment
37 • Updated when evidence or guidelines change
38 If the organization deploys an AI tool without adequate verification, validation, uncertainty
39 quantification (VVUQ) or post-market surveillance, it risks:
40 • Negligent selection of technology
41 • Negligent retention of unsafe systems

Page 1 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
42 • Failure to maintain safe equipment (AI is now “equipment” under corporate negligence)
44 🔐 3. Data Governance, Privacy, and Security Failures
45 Patient-facing AI often requires:
46 • API integrations
47 • Real-time EHR data
48 • Identity verification
49 • Secure messaging
50 Risks include:
51 • PHI leakage through prompts or model outputs
52 • Insecure APIs exposing patient data
53 • Model inversion or membership inference attacks
54 • Unauthorized secondary use of patient inputs
55 Corporate negligence exposure:
56 Failure to maintain safe facilities and equipment; failure to enforce adequate rules and policies.
58 🧠 4. Automation Bias & Human-Factors Failures
59 Even patient-facing AI influences clinician behavior indirectly.
60 Risks include:
61 • Patients presenting AI-generated summaries that clinicians over-trust
62 • Clinicians relying on AI-generated triage notes without verification
63 • Alert fatigue from AI-driven messaging systems
64 • Misinterpretation of probabilistic outputs by patients or staff
65 Corporate negligence exposure:
66 Failure to oversee medical practitioners and ensure safe workflows.
68 ⚖️ 5. Scope-of-Practice & Unauthorized Medical Advice
69 If patient-facing AI:
70 • Gives diagnostic or treatment advice
71 • Suggests medication changes
72 • Interprets labs or imaging
73 • Provides risk scores without context
74 …it may cross into unlicensed medical practice.
75 Hospitals are liable if they:
76 • Allow AI to function as a de facto clinician
77 • Fail to clearly disclose limitations
78 • Do not implement guardrails preventing clinical advice
80 🧭 6. Inadequate Transparency, Disclosures, and Consent
81 Patients must understand:
82 • That they are interacting with AI

Page 2 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
83 • What the AI can and cannot do
84 • How their data is used
85 • That outputs may be incomplete or probabilistic
86 Failure to provide adequate disclosure can create:
87 • Informed-consent failures
88 • Misrepresentation
89 • Reliance damages if patients act on AI advice
91 🏗️ 7. Vendor Management & Third-Party Liability Gaps
92 Hospitals remain liable even if the AI is:
93 • Built by a vendor
94 • Hosted externally
95 • Integrated via API
96 • “Black-box” or proprietary
97 Risks include:
98 • Inadequate vendor vetting
99 • Poorly negotiated indemnification
100 • Lack of transparency into training data
101 • Failure to monitor vendor updates or model changes
102 Corporate negligence exposure:
103 Negligent selection and retention of technology partners.
105 📉 8. Equity, Bias, and Disparate Impact
106 Patient-facing AI can worsen disparities if:
107 • It performs worse for certain demographic groups
108 • It uses biased training data
109 • It misinterprets language, dialect, or cultural context
110 • It provides inequitable triage recommendations
111 Hospitals face:
112 • Civil rights exposure (e.g., Title VI)
113 • Reputational harm
114 • Regulatory scrutiny (OCR, FTC, state AGs)
116 🧩 9. Operational & Workflow Breakdown
117 AI can create systemic risks:
118 • Over-triaging or under-triaging patients
119 • Flooding clinicians with AI-generated messages
120 • Creating bottlenecks in call centers or urgent care
121 • Misrouting patients to inappropriate care settings
122 These failures fall under the hospital’s duty to maintain safe and effective operations.
Page 3 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
124 🧱 10. Governance Failures
125 Perhaps the largest category.
126 Risks include:
127 • No AI oversight committee
128 • No model inventory
129 • No risk-stratification framework
130 • No safety case documentation
131 • No incident reporting pathway for AI failures
132 • No monitoring of real-world performance
133 This is classic corporate negligence: failure to enforce adequate rules and policies.
135 🧭 Want to go deeper?
136 If you want, I can map these risks to:
137 • A hospital-level AI risk matrix
138 • SaMD vs. non-SaMD regulatory implications
139 • A corporate negligence checklist for AI governance
140 • A model safety case template tailored to patient-facing AI
141 • How NCQA, CMS, and HTA bodies are beginning to evaluate AI-enabled care
142 Just tell me the direction you want to take this.
144 2. How will Informed Consent be structured, delivered and documented when AI generated
145 and/or developed patient facing APIs and CDS apps are used in daily clinical practice?
147
149 From Co-Pilot:
151 Below is a structured, forward-looking model of how informed consent will need to be
152 designed, delivered, and documented when AI-generated or AI-developed patient-facing
153 APIs and CDS apps become routine in clinical practice. This is one of the most unsettled areas
154 in digital-health governance, and health systems are already being pushed toward a new consent
155 architecture that blends clinical ethics, SaMD-style transparency, and algorithmic accountability.
156 I’ll give you a modular, evidence-aligned framework that a hospital could actually operationalize.
158 🧭 1. The Core Shift: From “Procedure Consent” to “AI System Consent”
159 Traditional informed consent focuses on clinical interventions.
160 AI introduces a new object of consent: the digital system itself.
161 Patients must understand:
162 • They are interacting with an AI system
163 • What the AI is designed to do
164 • What it cannot do

Page 4 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
165 • How it uses their data
166 • How its outputs may influence care
167 • What risks and uncertainties exist
168 This is closer to SaMD-style labeling than classic clinical consent.
170 🧱 2. The Three-Layer Consent Architecture
171 A practical model emerging across health systems, regulators, and ethics bodies is a three-layer
172 consent structure:
173 Layer 1 — System-Level Disclosure (Global Consent)
174 Delivered once, usually at onboarding to a portal or app.
175 Includes:
176 • The fact that AI tools are used in the system
177 • The types of AI (triage bots, symptom checkers, CDS, summarizers)
178 • The general risks (errors, limitations, probabilistic outputs)
179 • Data use and privacy practices
180 • The role of clinicians as final decision-makers
181 • How to opt out of AI-mediated interactions
182 This is analogous to consenting to the use of a patient portal or remote monitoring program.
184 Layer 2 — Context-Specific Micro-Consent (Just-in-Time)
185 Triggered at the moment an AI tool is used for a specific purpose.
186 Examples:
187 • “This triage tool uses AI to assess your symptoms. It may be wrong. A clinician will review if
188 needed.”
189 • “This medication-advice feature is AI-generated and not a substitute for professional medical
190 judgment.”
191 • “This risk score is produced by an algorithm trained on population-level data and may not
192 reflect your individual circumstances.”
193 Micro-consent is short, contextual, and embedded in the workflow.
195 Layer 3 — High-Risk AI Consent (Enhanced Consent)
196 Required when AI meaningfully influences diagnosis, treatment, or triage.
197 Examples:
198 • AI-supported cancer risk prediction
199 • AI-driven imaging interpretation
200 • AI-based treatment recommendations
201 • AI-mediated triage for chest pain, stroke, sepsis, etc.
202 Enhanced consent includes:
203 • Explanation of the model’s purpose
204 • Known limitations and uncertainties
205 • Alternatives (e.g., clinician-only evaluation)

Page 5 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
206 • How the clinician will use or override the AI
207 • Risks of false positives/negatives
208 • Patient’s right to decline AI involvement
209 This mirrors high-risk SaMD labeling and shared decision-making.
211 🧩 3. How Consent Will Be Delivered
212 Expect a hybrid of modalities:
213 A. In-App or Portal-Based Consent
214 • Click-through disclosures
215 • Layered explanations with expandable sections
216 • Short videos or interactive modules
217 • Micro-consent pop-ups for high-risk tasks
218 B. Clinician-Mediated Consent
219 For high-risk AI, clinicians will:
220 • Explain the AI’s role
221 • Clarify uncertainties
222 • Document the discussion
223 • Provide alternatives
224 C. Automated Logging
225 AI systems will automatically record:
226 • When disclosures were shown
227 • What version of the model was active
228 • Whether the patient acknowledged the notice
229 • Whether the patient opted out
230 This becomes part of the audit trail.
232 🧾 4. How Consent Will Be Documented
233 Documentation will need to be machine-readable, time-stamped, and version-controlled.
234 Expect:
235 A. EHR Integration
236 • Consent events stored as structured data
237 • Links to the specific AI model version used
238 • Clinician notes for enhanced consent encounters
239 B. API-Level Logging
240 For patient-facing APIs:
241 • Every AI interaction is logged
242 • Consent acknowledgments tied to session IDs
243 • Model versioning captured for legal defensibility
244 C. CDS Hooks / FHIR-Based Documentation
245 FHIR resources (e.g., Consent, Provenance, AuditEvent) will be used to:
246 • Record consent

Page 6 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
247 • Track model provenance
248 • Capture the AI’s influence on clinical decisions
249 D. Enterprise-Level Consent Registry
250 Large systems will maintain a centralized consent ledger that tracks:
251 • Global AI consent
252 • Tool-specific opt-outs
253 • High-risk AI consent events
254 • Revocation history
255 This becomes essential for corporate negligence defense.
257 ⚖️ 5. Legal and Ethical Requirements Driving This Structure
258 Several forces are converging:
259 • Corporate negligence doctrine (duty to maintain safe systems)
260 • FTC expectations for transparency in automated decision-making
261 • OCR expectations for algorithmic fairness and data use
262 • EU AI Act (even if not binding, it sets global norms)
263 • SaMD labeling principles from FDA/IMDRF
264 • NCQA and CMS movement toward algorithmic accountability
265 • Case law emerging around AI-mediated harm
266 The direction is clear:
267 AI use must be disclosed, understandable, and documented in a defensible way.
269 🧭 6. What This Means for Daily Clinical Practice
270 Expect the following to become routine:
271 • Patients see a banner: “This system uses AI.”
272 • Symptom checkers and triage bots show micro-consent notices.
273 • High-risk AI triggers a clinician-mediated consent step.
274 • EHRs automatically log AI involvement and consent.
275 • Patients can opt out of certain AI tools.
276 • Hospitals maintain a model inventory tied to consent events.
277 This is the future of AI-aware clinical governance.
279 If you want, I can take this further
280 I can build:
281 • A model informed-consent template for patient-facing AI
282 • A risk-tiered consent matrix (low, moderate, high-risk AI)
283 • A policy blueprint for a hospital AI consent program
284 • A documentation workflow for EHR and API integration
285 Just tell me which direction you want to explore.
286 Below is a structured, high-resolution answer tailored to your style of benchmarking clinical
287 rigor, legal doctrine, and computational credibility. You’re essentially asking:

Page 7 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
289 3. Which sort of iterative evaluation frameworks for validity, reliability and usability of patient
290 facing APIs and CDS apps that are used in daily clinical practice via methods such as
291 Verification, Validation and Uncertainty Quantification (VVUQ)?
293 From Co-Pilot
294 What iterative evaluation frameworks can ensure validity, reliability, and usability of
295 patient-facing APIs and CDS apps—especially when using VVUQ-style methods?
296 This is exactly where health-care AI is heading: continuous, lifecycle-based evaluation rather
297 than one-time validation.
298 I’ll give you a modular, clinically grounded framework that a health system could actually
299 operationalize.
301 🌐 1. The Core Principle: Continuous, Lifecycle-Based Evaluation
302 Patient-facing AI and CDS apps cannot rely on “validate once, deploy forever.”
303 Instead, they require iterative, cyclic evaluation across the entire model lifecycle:
304 • Pre-deployment verification
305 • Clinical validation in controlled settings
306 • Real-world performance monitoring
307 • Post-market surveillance
308 • Model drift detection
309 • Re-verification after updates
310 This is the VVUQ mindset applied to clinical software.
312 🧩 2. A Practical Iterative Evaluation Framework (VVUQ-Aligned)
313 Here is a structured framework that integrates Verification, Validation, and Uncertainty
314 Quantification with usability, safety, and clinical governance.
316 A. Verification (Is the system built correctly?)
317 Focus: correctness of implementation, data pipelines, and computational logic.
318 Methods:
319 • Unit testing of API endpoints and CDS logic
320 • Static and dynamic code analysis
321 • Model reproducibility checks
322 • Data lineage and provenance verification
323 • Boundary-condition testing
324 • Adversarial input testing (e.g., malformed symptoms, ambiguous phrasing)
325 • Version-controlled model cards documenting changes
326 Iterative cycle:
327 • Re-verification after every model update
328 • Automated regression testing for every API release

Page 8 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
329 • Continuous integration pipelines with safety gates
331 B. Validation (Does the system represent clinical reality?)
332 Focus: clinical accuracy, generalizability, and real-world fidelity.
333 Methods:
334 • Retrospective validation using curated datasets
335 • Prospective clinical validation in controlled pilots
336 • Silent-mode evaluation (AI runs in background without influencing care)
337 • External validation across diverse populations
338 • Bias and fairness audits
339 • Calibration testing (e.g., Brier score, reliability diagrams)
340 • Human-AI comparison studies
341 Iterative cycle:
342 • Re-validation triggered by:
343 o Model updates
344 o Shifts in clinical guidelines
345 o Changes in population characteristics
346 o New evidence of bias or drift
348 C. Uncertainty Quantification (How confident is the system?)
349 Focus: quantifying and communicating uncertainty to clinicians and patients.
350 Methods:
351 • Probabilistic outputs with confidence intervals
352 • Conformal prediction for calibrated uncertainty
353 • Ensemble modeling to estimate variance
354 • Out-of-distribution detection
355 • Uncertainty-aware triage thresholds
356 • Flagging low-confidence outputs for human review
357 Iterative cycle:
358 • Continuous monitoring of uncertainty distributions
359 • Alerts when uncertainty increases (possible drift)
360 • Updating thresholds based on real-world performance
362 🧪 3. Usability & Human-Factors Evaluation (Critical for Patient-Facing Tools)
363 AI that interacts directly with patients must undergo iterative usability testing:
364 Methods:
365 • Cognitive walkthroughs
366 • Think-aloud protocols
367 • Task-completion studies
368 • A/B testing of interface changes
369 • Accessibility audits

Page 9 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
370 • Health literacy testing
371 • Cross-cultural and language-variant testing
372 Iterative cycle:
373 • Re-testing after UI changes
374 • Monitoring patient-reported usability issues
375 • Continuous improvement based on real-world feedback
377 📊 4. Real-World Performance Monitoring (Post-Deployment VVUQ)
378 This is where most health systems are weakest today.
379 Methods:
380 • Automated drift detection (data, concept, and performance drift)
381 • Real-time error logging
382 • Safety signal detection (e.g., unexpected triage patterns)
383 • Outcome tracking (e.g., ED visits after AI triage)
384 • Equity monitoring across demographic groups
385 • Human override analysis (when clinicians disagree with AI)
386 Iterative cycle:
387 • Monthly or quarterly performance reviews
388 • Triggered re-validation when drift is detected
389 • Continuous updates to model cards and safety cases
391 🧱 5. Safety Case Frameworks (The Emerging Standard)
392 Borrowed from aviation and medical device engineering.
393 A safety case is a structured argument with evidence that the system is safe for its intended use.
394 Components:
395 • Hazard analysis
396 • Risk controls
397 • Verification evidence
398 • Validation evidence
399 • Uncertainty quantification
400 • Monitoring plan
401 • Incident response plan
402 Iterative cycle:
403 • Safety case updated with every model change
404 • Reviewed by an AI governance committee
405 This is becoming the backbone of enterprise AI governance.
407 🧭 6. Regulatory-Aligned Frameworks That Support Iterative Evaluation
408 These frameworks are not identical, but they converge on the same lifecycle-based approach:

Page 10 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
Relevance to Iterative Components
Framework
Patient-Facing AI
FDA Total Product Continuous monitoring,
SaMD-like CDS post-market updates
Lifecycle (TPLC)
Risk-tiering, clinical Ongoing performance evaluation
IMDRF SaMD Framework
evaluation
Monitoring, documentation, human
NIST AI RMF Enterprise AI governance factors

ISO/IEC 62304 Software lifecycle Verification & validation cycles

ISO/IEC 14971 Risk management Hazard analysis, mitigation, review

NIST SP 1270 Bias & fairness Continuous bias monitoring

Logging, transparency,
EU AI Act High-risk AI post-market surveillance

409 These frameworks are increasingly being blended into hospital AI governance programs.
411 🧩 7. A Unified Iterative Evaluation Model (What Hospitals Will Actually Use)
412 Here is a practical, integrated model:
413 1. Pre-Deployment
414 • Verification suite
415 • Clinical validation
416 • Usability testing
417 • Safety case creation
418 • Bias audit
419 2. Deployment
420 • Controlled rollout
421 • Silent-mode evaluation
422 • Human-in-the-loop oversight
423 3. Post-Deployment
424 • Drift detection
425 • Uncertainty monitoring
426 • Real-world outcome tracking
427 • Incident reporting
428 • Continuous re-verification
429 4. Update Cycle
430 • Triggered by:
431 o Model updates

Page 11 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
432 o New evidence
433 o Drift
434 o Safety signals
435 • Re-validation
436 • Safety case update
437 • Governance review
438 This is the VVUQ lifecycle applied to clinical AI.
440 If you want, I can take this further
441 I can build:
442 • A hospital-ready VVUQ evaluation checklist
443 • A risk-tiered evaluation matrix for patient-facing AI
444 • A model safety case template
445 • A governance workflow for continuous AI monitoring
446 • A comparison of VVUQ vs. traditional clinical validation
447 Just tell me which direction you want to explore next.
448 4. How, by whom and by what criteria will the selection and evaluation of “The Human in the
449 Loop” overseeing the development and implementation of AI/ML be implemented,
450 measured, and held accountable?
452 From Co-Pilot
Key Primary Accountability
Dimension Core criteria signals
decisions actors
What the
AI Written role
human in the
governance Override authority, descriptions,
loop (HITL) is
Role definition committee, scope of review, policies, RACI
actually
clinical escalation rules maps
responsible
leadership
for
Who is Clinical Credentialing
Service line
allowed to competence, AI records, training
chiefs,
Selection serve as literacy, completion,
credentialing,
HITL for a conflict-of-interest, privileging
HR
given AI/CDS workload capacity
How HITL is CDS logs,
Clear touchpoints,
embedded MLOps, turnaround
UI design,
Implementation into informatics, times, override
explainability,
workflows quality/safety rates
time-to-review
and systems

Page 12 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
Key Primary Accountability
Dimension Core criteria signals
decisions actors

How HITL Safety outcomes, Dashboards,
Quality, risk safety case
performance appropriateness of
Measurement management, updates,
and impact overrides, equity,
data science incident reviews
are evaluated usability
How
Adherence to Peer review,
individuals Medical staff
policy, remediation,
and the office,
Accountability documentation governance
organization compliance,
quality, response reports
are held board
to signals
responsible
454 1. How the human-in-the-loop role is structured
455 a. Multi-layered oversight, not a single hero
456 Modern governance is moving from vague “human oversight” to structured, multi-layered HITL
457 embedded across the AI lifecycle—design, deployment, monitoring, and decommissioning.
458 Preprints
459 Typical layers:
460 • Design-time HITL:
461 Who: clinical domain experts, informaticists, data scientists.
462 What: define indications, exclusions, guardrails, and acceptable failure modes; review training
463 data and labeling.
464 • Run-time HITL (frontline clinicians or triage staff):
465 What: review AI outputs, override when inappropriate, escalate ambiguous or high-risk cases,
466 document rationale.
467 • Post-deployment HITL:
468 Who: AI safety/quality teams, governance boards.
469 What: monitor drift, incident patterns, equity impacts, and feed back into model updates and
470 policy.
471 b. Explicit authority and obligations
472 Policies must specify that HITL actors:
473 • Have clear override authority and are never bound to follow AI recommendations. Preprints
474 • Must escalate when uncertainty is high or cases are out-of-distribution. European/American
475 Journals - EA Journals
476 • Are responsible for documenting key decisions when AI materially influences care.
478 2. By whom the HITL is selected
479 Selection should look more like credentialing and privileging than informal “super-user”
480 designation.

Page 13 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
481 a. Core selection criteria
482 • Clinical competence: board certification or equivalent for the domain where AI is applied
483 (e.g., cardiology, ED triage).
484 • AI/CDS literacy: demonstrated understanding of model scope, limitations, uncertainty, and
485 common failure modes. European/American Journals - EA Journals
486 • Workflow proximity: actually works in the setting where AI is used (e.g., ED nurses for triage
487 bots).
488 • Cognitive capacity & workload: protected time or realistic workload so oversight is not
489 purely nominal.
490 • Ethical and regulatory awareness: familiarity with documentation, bias, and escalation
491 obligations.
492 b. Who makes the selection
493 • Service line leadership (e.g., chief of medicine, ED director) nominates HITL roles.
494 • Medical staff/credentialing bodies formalize privileges for “AI/CDS oversight” as a specific
495 competency.
496 • AI governance committee confirms alignment with system-wide policies and risk tier of the
497 tool. Preprints
499 3. By what criteria HITL implementation is evaluated
500 Think of this as performance metrics for human–AI collaboration, not just for the model.
501 a. Safety and quality metrics
502 • Adverse event rates where AI was involved vs. not involved.
503 • Near-miss and override analysis:
504 o Frequency of overrides
505 o Appropriateness of overrides (e.g., were they associated with preventing harm?)
506 European/American Journals - EA Journals
507 • Escalation performance: time from AI output to human review in high-risk scenarios.
508 b. Appropriateness and calibration
509 • Concordance with expert panels on sampled cases.
510 • Use of uncertainty: whether HITL actors appropriately treat low-confidence outputs as
511 advisory rather than definitive. European/American Journals - EA Journals
512 c. Equity and bias
513 • Differences in overrides, escalations, and outcomes across demographic groups; whether
514 HITL behavior mitigates or amplifies algorithmic bias. Preprints
515 d. Usability and cognitive load
516 • Time burden per AI-review event.
517 • Self-reported trust, clarity, and cognitive strain in interacting with the AI. Technology Rivers
519 4. How HITL accountability is implemented and measured
520 a. Embedding into MLOps and audit trails
521 Every AI-influenced decision should leave a traceable record:

Page 14 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
522 • Which model and version produced the output.
523 • Which human reviewed it (identity/role).
524 • Whether they accepted, modified, or rejected it.
525 • Any documented rationale for overrides in high-risk cases. European/American Journals - EA
526 Journals
527 These logs feed:
528 • Safety cases and regulatory documentation.
529 • Peer review and quality-improvement cycles. Preprints
530 b. Governance and peer review
531 • HITL performance is periodically reviewed by quality and medical staff committees, similar
532 to review of procedural complications.
533 • Recurrent patterns (e.g., systematically ignoring uncertainty flags, failing to escalate) trigger:
534 o Targeted retraining
535 o Temporary restriction of HITL privileges
536 o Policy or UI redesign if the problem is systemic rather than individual.
537 c. Organizational accountability
538 The institution remains responsible for:
539 • Defining the HITL model (where humans must be in the loop and with what authority).
540 Technology Rivers
541 • Resourcing it (staffing, training, time).
542 • Monitoring it (dashboards, incident learning systems). Preprints
543 This ties directly into corporate negligence: failure to select, train, and support adequate human
544 oversight becomes an enterprise-level breach, not just an individual error.
546 5. A concise, operational blueprint
547 If you wanted to make this real in a health system, the steps would be:
548 1. Define HITL roles per AI tool and risk tier (triage, imaging, documentation, etc.).
549 2. Create selection and privileging criteria (clinical + AI literacy + workload).
550 3. Mandate training and competency assessment before granting HITL privileges.
551 4. Instrument the system so every AI-human interaction is logged and analyzable.
552 5. Build dashboards for overrides, safety events, equity, and workload.
553 6. Tie HITL performance into existing peer-review and quality-improvement structures.
554 7. Update the model safety case and governance reports with HITL metrics on a fixed
555 cadence.
556 If you’d like, we can next design a risk-tiered HITL matrix (e.g., what “human in the loop” means
557 for low-, medium-, and high-risk AI in a hospital).
559 5. How should a “top-down” versus “bottom up” implementation and evaluation strategies
560 be designed to assure adequate and continous input and feedback from key end-user
561 clinical and patient stakeholders who are the recipients of care that is supported by
562 patient facing APIs and CDS apps that are used in daily clinical practice?

Page 15 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
564 From Co-Pilot
Top-down strategy Bottom-up strategy How they
Aspect should connect
(enterprise-driven) (end-user–driven)
Co-design
Board, C-suite, system AI Clinicians, patients, charters and
Primary driver:
governance frontline staff priorities

Fit to workflow, trust, Jointly define
Alignment, safety, “success” and
Main purpose: usability, real-world
standardization, risk control guardrails
value

Policies, standards, model Pilots, PDSA cycles, Shared metrics
Key and feedback
inventory, risk tiers, user councils,
mechanisms: channels
evaluation frameworks qualitative feedback
Upward: lived
Downward: requirements, experience, incident Closed-loop
Feedback flow: learning cycles
constraints, templates reports,
workarounds
Professional duty, Integrated
Corporate negligence, dashboards and
Accountability: patient advocacy,
regulatory, reputational safety cases
local quality
566 1. Design principles: you need both directions by design, not by accident
567 Top-down alone gives you governance but not adoption.
568 Bottom-up alone gives you enthusiasm but not safety or consistency.
569 A robust strategy for patient-facing APIs and CDS should:
570 • Start with enterprise guardrails (risk tiers, safety requirements, evaluation expectations).
571 Frontiers
572 • Embed structured, continuous stakeholder engagement (clinicians + patients) across the
573 lifecycle, not just at launch. Frontiers
574 • Make feedback a formal input into model updates, de-implementation decisions, and
575 governance reports.
576 Think of it as a bi-directional control system, not a hierarchy.
578 2. Top-down implementation and evaluation: what it should do
579 A. Set the frame
580 • Define risk tiers for patient-facing tools (e.g., education vs triage vs treatment-shaping CDS).

Page 16 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
581 • Mandate minimum evaluation standards (verification, clinical validation, usability, equity
582 checks, post-deployment monitoring). Frontiers
583 • Create an AI implementation process framework (e.g., adapted from QIF / TPLC / NIST AI
584 RMF) that every project must follow. Frontiers
585 B. Institutionalize stakeholder input
586 Top-down design should require bottom-up input:
587 • Charter clinician and patient advisory councils for AI, with defined roles in:
588 o Use-case selection
589 o Requirements definition
590 o Go/no-go decisions
591 o De-implementation recommendations
592 • Bake stakeholder checkpoints into the process:
593 o Pre-design: problem scoping with frontline and patients
594 o Pre-go-live: usability and workflow sign-off
595 o Post-go-live: review of real-world metrics and qualitative feedback
596 C. Enterprise-level evaluation
597 • Standard outcome set for all patient-facing AI:
598 o Safety (adverse events, near misses)
599 o Effectiveness (e.g., triage accuracy, adherence, readmissions) Monterail
600 o Equity (performance across groups)
601 o Experience (patient-reported and clinician-reported)
602 • Central dashboards that show these metrics by tool, service line, and population.
603 • Formal review cadence (e.g., quarterly AI safety and value review) with authority to pause or
604 retire tools.
606 3. Bottom-up implementation and evaluation: how to structure it
607 A. Clinician-driven bottom-up
608 • Local AI champions / super-users in each unit or service line who:
609 o Co-design workflows and prompts
610 o Run small PDSA cycles (Plan–Do–Study–Act) on real cases
611 o Collect “pain points” and workarounds systematically
612 • Frontline incident and near-miss reporting explicitly tagged as “AI-involved” to feed
613 governance. Stanford Medicine
614 • Regular debriefs (e.g., monthly) where clinicians review:
615 o Cases where AI helped
616 o Cases where AI misled
617 o Situations where they ignored or over-trusted AI
618 B. Patient-driven bottom-up
619 • Patient panels / community advisory boards specifically for digital and AI tools.
620 • In-app feedback channels:
621 o “Was this helpful?”

Page 17 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
622 o “Did this advice match what your clinician said?”
623 o Free-text comments routed to a review queue.
624 • Targeted qualitative work (interviews, focus groups) with populations most affected or at risk
625 of harm or exclusion. Frontiers
626 C. Local evaluation
627 • Units track their own micro-metrics (e.g., message volume, time saved, confusion
628 generated, extra visits) and feed them upward. Monterail
630 4. Designing the feedback loops: how top-down and bottom-up talk to each other
631 You want explicit, engineered loops, not ad-hoc emails.
632 Loop 1: Design & selection
633 • Top-down: defines strategic priorities and risk thresholds.
634 • Bottom-up: proposes use cases and flags real pain points.
635 • Mechanism: joint intake process where proposals must show both enterprise value and
636 frontline/patient sponsorship.
637 Loop 2: Implementation
638 • Top-down: provides templates (risk assessment, consent language, evaluation plan).
639 • Bottom-up: adapts workflows, tests usability, and signs off on “fit for use.”
640 • Mechanism: go-live checklist that requires documented clinician and patient review.
641 Loop 3: Post-deployment evaluation
642 • Top-down: monitors system-level metrics and safety signals.
643 • Bottom-up: supplies case narratives, workarounds, and perceived harms/benefits.
644 • Mechanism:
645 o Shared dashboards accessible to units
646 o Regular joint review meetings
647 o Clear triggers for: model retraining, UI changes, or de-implementation
648 Loop 4: Governance and learning
649 • Top-down: updates policies, frameworks, and model inventory based on what’s learned.
650 • Bottom-up: validates whether changes actually improve daily practice.
651 • Mechanism: versioned “AI playbook” that evolves with each cycle, co-authored by
652 governance and frontline reps.
654 5. Concrete design moves a health system could make
655 • Create an AI Implementation Council with equal representation from:
656 o Executives, legal, compliance
657 o Clinical leaders and frontline staff
658 o Patient/community representatives
659 • Adopt a standard AI implementation process (e.g., QIF-adapted) that requires stakeholder
660 engagement at each phase. Frontiers
661 • Instrument every patient-facing API/CDS with:
662 o Usage analytics

Page 18 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026
 Reflections on Lab 2.0 2026
663 o Outcome linkage (where feasible)
664 o In-app patient feedback
665 o Clinician override and incident logging
666 • Run structured learning cycles (e.g., every 3–6 months) where:
667 o Data + stories are reviewed together
668 o Decisions are made to scale, modify, or retire tools
669 o Lessons are codified into the next generation of projects
670 If you’d like, we can next build a visual “bi-directional governance map” or a checklist that
671 you could drop straight into a course module or policy draft.

Page 19 Created by Don Casey MD, MPH, MBA, MACP, FAHA 10/18/2026