At a recent healthcare conference, I put six AI-specific threats on a slide and asked the room two questions. Do you recognize any of these? Heads nodded. Don’t tell me which ones, I said. Is there anything here you don’t recognize? Heads nodded again.
The second round of nods marks where the security program ends. Health systems recognize the threats their programs were built to stop: ransomware, phishing, insider misuse. The threats no one recognized came in with AI, and a review that stops at the login screen will not find them.
Healthcare reported the most expensive data breaches of any industry, according to the World Economic Forum. Shadow AI, meaning staff using AI tools nobody approved, is already widespread: a Wolters Kluwer Health survey found that 40% of healthcare professionals report encountering unauthorized AI tools at work, and roughly 1 in 5 admit to using them. A routine case: a care coordinator pastes a patient’s discharge summary into a public chatbot to tidy the wording. Nothing in a login review touches that moment, because the coordinator didn’t sign in to anything the health system manages.
The login boundary
A health system runs as one connected network. Care teams, records, imaging, devices, and cloud infrastructure share identity and access rules, and few teams have fully mapped the trust boundaries between them. A login review draws one of those boundaries at identity. It asks who may sign in and what role they hold. I call that line it draws the “login boundary,” and for a conventional application it does most of the work.
AI adds a new kind of workload to the stack. A user’s identity starts a request. The request carries prompts and context into a retrieval layer that pulls from the EHR and other documents, and a model turns that material into an answer. An agent, tool, or API may then act on the answer inside a clinical or operational workflow, while monitoring and audit record each step. The same chain sits behind a patient chatbot, an inbox draft, ambient documentation, and a scheduling assistant. Whatever the user sees, the risk sits in the links behind it.
The middle links behave differently from traditional software. Ask a model the same question twice and the answer can change. A document in the retrieval layer can carry instructions the model follows. A vendor can ship a model update overnight, and the tool behaves differently by morning with no change request on your side. A retrieval layer that searches with broad service permissions can hand the model records the person asking was never cleared to see. Audit now has to capture prompts and outputs as well as logins.
Take a security reviewer signing off on a new summarization tool. The sign-off confirms single sign-on and role-based access, and every line of it is accurate. All of it sits inside the login boundary. A review of the other links would ask what retrieval pulls and under whose permissions, what the vendor logs and keeps, which actions the agent may take without a person approving them, and who learns when the model changes. The sign-off answers none of those questions. If the tool misbehaves later, the review file shows a signature on a system examined at one link.
The reviewer followed the process correctly. The process inherits an assumption from conventional software testing, that the same input produces the same output, and a review that tests once and signs off carries that assumption forward.
Trusted AI takes five disciplines working together: security, privacy, safety and quality, governance and accountability, and continuous assurance. Each can fail while the others hold. An ambient documentation tool can protect every field of PHI and still draft a summary that drops a medication change. Privacy passes that tool and safety fails it. Governance then asks who owns the error once a clinician catches it, and continuous assurance asks who will notice the next one. A login review would have passed the tool and answered neither question.
Six threats
The older threats have not gone anywhere. AI adds six more, and each one has a clinical version.
- Prompt Injection
A referral document arrives with hidden text in it, and the summarization assistant reads the text as an instruction. Security teams call this prompt injection, and it needs no one inside the building.
- Sensitive-Data Leakage
The same assistant sends its prompts to a vendor that logs them, so patient details now sit in a system the covered entity cannot inspect.
- Retrieval and Data Poisoning
Someone alters a care guideline in a repository the retrieval layer trusts, or the layer simply pulls a superseded version. Either way, the summary reaches a clinician’s queue looking routine and rests on the wrong guidance.
- Model and Supply Chain Risk
A vendor retires the model version your summarization tool was validated on, and the replacement formats medication lists differently. Nobody on your side filed a change request, because nobody on your side made a change. Connectors, plugins, and APIs carry the same exposure.
- Excessive Agent Authority
A scheduling assistant meant only to read the schedule cancels a post-operative follow-up, because nobody limited what the agent could do.
- Model Drift
Summarization accuracy can decline as documentation patterns shift. No outage announces it, and the first sign can be a clinician who stops trusting the summaries.
A threat model built for this chain starts at the retrieval layer, asks what it will hand the model, and then follows the output to whatever acts on it. A login review looks at identity, and these six threats work on what happens after it. The patient meets only the output: a care gap flagged or missed, an outreach message that reads right or reads wrong.
A reviewer can only review what reaches the review queue. A department that buys a tool on its own, or a clinician who starts using one, never files a request, and the tool never reaches a login review. Shadow AI accumulates in that gap.
Part 2 of this blog series follows the care-management nurse who approves an AI-drafted message before it goes out to a patient, working inside a workflow that assumes someone upstream already protected her. It introduces risk tiering as the fix, lays out the first 90 days, and starts with an inventory. Can you name every AI system that touches PHI in your environment today, including the ones a department bought on its own?
