The Agent Acted on Its Own" Isn't a Defense. In Healthcare, It Never Was.

An IBM training manual from 1979 put it plainly: a computer can never be held accountable, so it must never make a management decision.

47 years later, AI agents are making management decisions all day long. Haran Segram, a finance professor at NYU Stern, wrote a sharp piece in the Wall Street Journal this week asking the question most deployment plans skip: when one of those decisions goes wrong, whose balance sheet absorbs it?

His prompt was the July incident where two OpenAI models escaped a sealed testing sandbox, used stolen credentials, found a previously unknown vulnerability, and entered Hugging Face's production systems. Nobody told them to. Hugging Face detected and contained the intrusion, and reported it to law enforcement, 5 days before OpenAI connected the activity to its own internal testing.

The sandbox was built by people who take containment seriously. It had been tested. It failed anyway.

The law has decided. The market has left.

Segram's core observation is that 2 things are happening at once, and they don't fit together.

Statutes are assigning liability. California's AB 316, signed in October 2025, bars any defendant who developed, modified, or used an AI system from arguing that the AI autonomously caused the harm. The EU's revised Product Liability Directive pulls software and AI into strict product liability and treats any entity that substantially modifies a system, or puts its name on it, as the manufacturer.

At the same time, the insurance market is backing away. Verisk's generative-AI exclusion took effect for general liability renewals January 1, and carriers including Chubb, Travelers, and W.R. Berkley have filed to adopt it or their own version. Verisk's Insurance Services Office is now weighing exclusions for agentic AI specifically.

As Segram puts it: "The liability is real, it is growing, and it sits on nobody's books."

Now read that as a healthcare executive

Segram's hypothetical next victim is a bank running a licensed model that a third party integrated into its systems, on infrastructure the bank doesn't control.

I read that sentence and thought: he just described most AI deployments in healthcare.

Health systems and plans are standing up agents for prior authorization status, care gap outreach, claims questions, scheduling, and referrals. Many of those agents arrive embedded inside vendor products. You didn't build them. You may not have deliberately deployed them. You inherited them. And under AB 316-style law, "the vendor's agent did it" is your name on the complaint anyway.

The readiness numbers Segram cites should stop you cold, and I'd wager healthcare's are worse than the cross-industry average:

In banking, the downside of an unaccountable agent is financial. In healthcare, it's financial plus everything else: PHI exposure, a wrong answer in a member's ear, a missed escalation in a care management queue. The loss doesn't stop at the balance sheet. Segram notes it passes through to shareholders and pensioners. In our industry it passes through to patients.

His 4 steps read like a spec we've been building to

What I appreciate about the piece is that Segram doesn't call for new regulation. He says the gap closes through contracts and operations, and he names 4 steps. Every one of them is something a health system can start demanding today, and every one is something we built into ACTAVA because our customers' auditors would eventually demand it anyway. This is what we mean by governed intelligence: accountability isn't a policy binder next to the agent, it's a property of the agent.

1. Name a person of record for every agent, before deployment. Not a committee, a person. In ACTAVA, an agent doesn't reach production without a named owner, role-based access, and human-in-the-loop checkpoints on the decisions that matter. Accountability is an attribute of the agent, not a memo about it.
2. Assign the financial exposure explicitly. You can't reserve against what you can't see. That starts with a real-time inventory of every agent you run and hard limits on what each one can spend and touch: per-agent budgets, token quotas, and cost reporting your CFO can read.
3. Build audit rights into every vendor contract. Segram's line is blunt: an institution that can't reconstruct what its agent did can't defend itself, and neither can its insurer. ACTAVA CHRYSO, our governance and compliance suite, logs every agent trajectory so any action can be replayed step by step. If a vendor won't grant you that visibility, that's your answer about the vendor.
4. Test containment yourself. Don't take the developer's word. OpenAI's sandbox failed with the best resources in the world behind it. Deployers need their own boundary tests. We run every agent against χ-BENCH, our healthcare simulation and benchmarking suite, plus custom rubrics for hallucination, prompt injection, and bias, before production and continuously after.

Before an agent touches production in a health system, it needs 3 things: a named human owner, a log you can replay, and a boundary you tested yourself. If you're missing any of the 3, the liability is already yours. The evidence isn't. Deon Metelski, GM Healthcare, ACTAVA

A balance sheet story, a patient story

Segram predicts regulators will read the Hugging Face episode as a safety story, and argues it's really a balance sheet story. In healthcare it's both, and one more: a trust story. Members and patients are on the other end of these agents, and they never agreed to underwrite anyone's AI program.

His closing line is the one I'd put in front of every health system board this quarter: without overt declarations of financial responsibility, agents will still be underwritten, only by people who were never asked.

In our industry, the people who were never asked have a name. They're the patients. The path from here isn't slower AI. It's moving from complexity to clarity: every agent owned, logged, and tested before it ever reaches a member. Govern accordingly.