HIPAA-Compliant AI in Healthcare Software: The Architecture That Passes Audit
Start with the BAA, not the model
If protected health information will reach a model provider, you need a business associate agreement with that provider covering the specific service you are using — not the parent company generally, and not a similar service on the same platform. Major cloud and model providers offer this for defined endpoints. Using a consumer-tier API for a clinical workflow because it was quicker to set up is the kind of shortcut that surfaces during a breach investigation, and it invalidates everything built on top of it.
De-identify at the edge where you can
A large share of healthcare AI use cases do not need identifiers at all. Summarising a clinical note, extracting medication mentions, or classifying a referral works on de-identified text. Running a de-identification pass before the model call, then re-attaching identifiers locally in your own environment, dramatically reduces the compliance surface. It is not always possible, and de-identification is imperfect, so treat it as risk reduction rather than an exemption. But when it works, it simplifies every downstream conversation.
Logging is required and also a liability
You need audit logs showing who accessed what and when. You also need to avoid creating a second uncontrolled copy of PHI in your observability stack. The pattern that holds up: log identifiers and event metadata in the audit trail, keep prompt and response content in an encrypted store with the same access controls and retention policy as the clinical record, and redact PHI from application logs and error traces entirely. Error monitoring tools are the usual leak, because stack traces cheerfully include request bodies.
Clinical decision support has its own rules
There is a meaningful regulatory line between software that surfaces information for a clinician to evaluate and software that directs a clinical decision. Systems that a clinician can independently review — showing sources, reasoning and the underlying data — sit in a very different category from a system outputting a diagnosis or treatment recommendation to be followed. Design toward transparency: show the retrieved evidence, cite the note or guideline, and make the clinician's independent judgement the operative step. That shapes both regulatory exposure and clinician adoption.
Validation evidence clients will ask for
Expect to produce a documented test set with clinician-reviewed ground truth, accuracy and failure-mode analysis by patient subgroup, model version pinning, and a monitoring plan for drift. Health systems increasingly ask for this before procurement, not after. Building the evaluation harness during development rather than reconstructing it for a security review saves months and produces a better product, because you find the failure modes while you can still fix them cheaply.
The organisational part
Name an owner. Define what happens when the model is wrong and who the clinician contacts. Train the users on what the system does and does not do, and keep an easy path to report bad outputs that someone actually reviews. Audits examine process as much as technology, and a well-engineered system with no operational ownership fails that examination.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch