What Is AgentOps? Continuous Evaluation for Healthcare AI Agents

You hired the agent. Now who reviews its performance? AgentOps is the operating discipline for running AI agents in production: tracing their work, evaluating live interactions at a risk-appropriate sampling rate, detecting regressions, enforcing policy, and measuring outcomes over time.

Most healthcare AI programs treat evaluation as a gate. The agent gets tested, the agent passes, the agent ships. Everyone moves on to the next use case.

That model works for software that behaves predictably under stable conditions. It does not work as well for an agent that reasons, calls tools, and may take a different trajectory on each run. An agent that scored 0.91 on task completion in March is not guaranteed to score 0.91 in September. The model may be unchanged while the data, tools, policies, and workflows around it have moved.

In this article, AgentOps means the operating discipline, not a particular tool or vendor. It borrows from DevOps and MLOps, then extends those practices for systems that make decisions and take actions. That continuous-lifecycle approach is consistent with the NIST AI Risk Management Framework, which calls for AI behavior to be monitored in production and risk management to continue throughout the system lifecycle.

Managing agents as a workforce

Here is a framing that lands with many operations leaders faster than an architecture diagram.

Every agent you deploy has something like a job description: a scope of authority, a set of tools it may use, and a performance standard. An organization would not hire 300 people and then decline to manage any of them. Yet many organizations adding agents have no equivalent operating function to govern the resulting estate.

AgentOps is that operating function. Continuous evaluation is its recurring performance review: applied to live traffic at a frequency proportionate to the workflow's risk, value, and cost.

The four moving parts

Strip AgentOps down and it comes to four jobs that must operate together throughout the agent lifecycle.

Miss any one of the four and the others lose much of their value. Monitoring without evaluation tells you that the agent is fast, not that it is right. Evaluation without governance produces a score nobody is accountable for.

How continuous evaluation works in ACTAVA

Continuous evaluation is not bolted onto one product. It runs through the ACTAVA suite and maps to the four pillars the platform was built around: Build, Test, Learn, Guide.

Thresholds are two different things

This distinction trips up many teams, so it is worth separating cleanly. In ACTAVA, evaluation thresholds and outcome targets answer different questions.

Control	Question it answers	Who owns it
Evaluation threshold Defined for each metric within a versioned collection	Did this evaluated run pass or fail? Is Faithfulness above the bar set for this agent?	Agent builders, clinical informatics, and the AI governance committee
Outcome target Baseline, target, and direction for each ROI metric	Is this agent making progress, moving backward, or in need of attention against the business case?	Finance, operations, and the executive sponsor

An agent can pass every behavioral evaluation and still miss its outcome target. It may be behaving as designed without delivering enough business value. Without both views on the same page, teams may not discover the gap until renewal.

Evaluation on production traffic, not just test cases

The word doing the work in continuous evaluation is continuous. Pre-launch benchmarking is table stakes. What matters is whether performance holds at run 40,000.

In ACTAVA, evaluation is wired into the agent runtime. After an eligible run completes, automatic evaluation uses the agent's configured collection and converts the recorded trajectory into a test case. Session, agent, version, query, response, and run context are carried through so the result remains attributable.

The evaluation layer then scores behavior against the configured metric thresholds. Separately, the outcome layer compares measured KPIs with their baselines and targets, surfacing progress and regression. Together, they answer two different questions: Is the agent behaving as approved? And is it producing the value expected?

Two design choices matter here.

• The same rubric works for single-turn and multi-turn evaluation. A metric written once can evaluate a one-shot answer or a 20-turn conversation, scoped to the final turn or the full conversation. Teams do not have to maintain two evaluation systems.
• Sampling is configurable per metric. Evaluate every eligible run where the stakes justify it or a defined fraction of traffic where they do not. Evaluation frequency should be proportionate to the workflow's risk, business value, and operating cost.

Three kinds of metric cover most of what a health system needs to check.

Metric type	How it judges	Good for
Rubric	LLM-as-judge against weighted criteria, at the metric or item level, including negative points for known failure patterns	Quality judgments that need reasoning: tone, completeness, and grounding in the retrieved record
Python	Custom code with access to conversation context	Deterministic checks: was the required field populated, was the correct tool called, and did the total reconcile?
Plan Compliance	LLM-as-judge against the system prompt or plan, returning a structured checklist of completed items	Multi-step workflows where finishing the job matters more than any single answer

For background agents with a defined plan, ACTAVA automatically adds Plan Compliance. Conversation agents receive Role Adherence instead. The default metric reflects the agent's operating mode rather than relying on someone to remember to add it.

Outcomes, not just behavior

Behavioral scores show how well the agent performed against a defined rubric. They do not tell you whether the work produced the intended business result.

That is why outcome capture runs alongside evaluation. Business outcomes can be captured automatically from agent runs and rolled up across windows from hourly to monthly, with each metric reporting how much of its value came from runtime measurement, emitted events, or manual entry.

Cost displacement is the clearest example. Rather than asserting a number in a slide, the calculation is explicit: events completed, multiplied by minutes saved per unit, converted to hours, multiplied by the loaded hourly rate, and then multiplied by an attribution coefficient that states how much of the saving is assigned to the agent.

What to ask your vendor, and yourself

If you are evaluating agentic AI now, these five questions help distinguish a compelling demonstration from a production operating model.

• What percentage of production runs get evaluated, and can you show me the metric definitions?
• Where is the threshold set for each metric, and who has authority to change it?
• How would I learn that this agent regressed, and how long would that take?
• Which named human reviews the exceptions, and what audit trail does that review leave?
• What business outcome is this agent credited with, and how much of that credit is attributed to the agent versus the surrounding process change?

Clear, evidence-backed answers to all five are a strong sign of production-grade AgentOps. Model benchmarks alone are not.

The part worth taking away

Agentic AI does not fail in healthcare only because of model quality. It also fails when organizations deploy systems that make decisions and then supervise them like static reports.

Continuous evaluation closes that gap. Score the work as it happens. Set thresholds with named owners. Watch for behavioral and outcome regressions. Tie performance to results you can defend. Keep a human accountable for the exceptions.

That is the discipline. The agents are already entering the building. The question is whether anyone is managing them.