Debaraj Shome Purkayastha

Commenter: Dr. Debaraj Shome Purkayastha, MS (General Surgery), surgical oncology resident at a tertiary cancer centre in Assam, India, serving a multilingual, resource-limited population. I comment as an individual; these views are not my institution’s.

Disclosures: (1) I am a freelance clinical evaluator for an India-based data-services vendor, assessing AI-generated clinical documentation. I receive no payment from any device manufacturer or foundation-model developer. (2) I am the developer of Aakhyan (https://www.aakhyan.health), a vernacular discharge-communication system for oncology patients, owned by my company SciScribe Solutions OPC (India). It is not marketed in the US and is not the subject of any FDA submission. I cite it below only to illustrate a design pattern.

I support the competency-based framework and recommend three changes.

1. VALIDATE LLM ADJUDICATORS BY ERROR TYPE (Q10, Q16, Q20)

The paper states that adjudicator independence and qualification apply when the adjudicator is an LLM. But an LLM judge can meet both and still be blind to a whole error class. Across eight LLM-judge designs on 500 AI clinical-note pairs, detection of added or altered content was 0.79-0.94, but detection of omissions was 0.50-0.63, near chance. Prompt changes and voting did not fix this; restructuring the task (list the facts the source establishes, then check each) did (Fox et al., arXiv:2608.31016). Omission is a primary harm mode for any function that summarises or rewrites clinical information, and an aggregate agreement score can hide it.

Recommendations:
(a) Validation of an LLM adjudicator against the human subsample should report sensitivity and specificity per error class (at minimum fabrication, omission, and misattribution or negation), with the subsample enriched for each class.
(b) Acceptance criteria should be set per error class, not as one agreement figure.
(c) The same should apply to machine-based supervisory agents in postmarket monitoring (Q20). Omissions rarely produce complaints, so a monitor blind to them would report a degrading device as stable.

2. NAME LANGUAGE EXPLICITLY IN SUBGROUP PERFORMANCE (Q9, Q13)

Element R.2 covers dialects, accents, colloquialisms and literacy, but not language itself, code-mixed input (routine in the US and India alike), or machine-translated output. Performance in low-resource languages can be far below English: fine-tuned speech recognition for Assamese still had a 43% word error rate (Das et al., arXiv:2607.17164). A device tested only in English could pass R.2 as written.

Recommendations:
(a) Name the language of interaction, input and output, as a subgroup dimension, including code-mixed input and translated output.
(b) Report performance separately for each language the labelling supports, and require explicit, safe behaviour for unsupported languages.
(c) Where a device translates, evaluate safety elements (S.1, E.4) in the target language with fluent clinicians, not by back-translation alone.
(d) On synthetic data (Q13): test sets made by translating English cases miss natural code-mixing and carry translation artefacts; include natively collected material for each supported language.

3. PATIENT-FACING OUTPUT DERIVED FROM CLINICAL DOCUMENTATION (Q3, Q5)

The paper notes that documentation and patient-outreach functions may fall outside device oversight. A common pipeline crosses that line unnoticed: an AI-generated note or discharge summary is simplified, often translated, and delivered as instructions the patient acts on. That final output is action-directing information given directly to a patient; a dropped or mistranslated red-flag symptom can cause severe harm. The paper’s principle of evaluating "the final user-facing device, as configured" should apply explicitly to such chains.

One mitigating pattern separates extraction from composition: the model only extracts content into structured fields, a clinician verifies them before anything is sent, and patient-facing text is rendered from fixed clinician-approved templates, so the model never writes free text to the patient. Aakhyan uses this pattern, with clinician review of every extraction and escalation to a consultant where needed (Shome Purkayastha, medRxiv, doi:10.64898/2026.04.15.26350965). It has no outcome data yet.

Recommendations:
(a) Clarify that risk for transformed patient-facing output is assessed on the final output, even when the upstream function is not a device.
(b) Under Q3, recognise constrained generation with clinician sign-off as a mitigation, while still benchmarking the extraction step, including for omissions.
(c) Under Q5, note that patient follow-up questions about discharge instructions readily become action-directing, and assess such conversations on that trajectory.