What KPIs Do I Assign to an AI Agent?

Most AI agents in production have either too few KPIs or the wrong ones. Too few looks like "this agent produces drafts, and we feel like it's working." The wrong ones look like measuring volume (tasks completed, queries answered) without measuring whether the output is actually used. Both failures produce the same outcome: an agent that nobody can defend at performance review time, because nobody can tell whether it's working.

The right number of KPIs per agent is four to six, distributed across four categories. Output KPIs answer what the agent produces. Quality KPIs answer how often it's right. Efficiency KPIs answer what it costs. Trust KPIs answer how much the team relies on it without supervision. Skip a category and the agent's performance is partly invisible. The framework below is what we use at Sneeze It and what most operators end up converging on after a year of running agents in production.

Output KPIs: what the agent produces

Output KPIs are the closest analog to what you'd put on a human IC scorecard. The agent has a job. The output KPI measures the job.

Pick the output that matters most and measure it as a number, not a feeling. For a prospecting agent: qualified emails drafted per day, or qualified replies generated per week. For an ad analytics agent: alerts raised per day, with a separate count for actionable vs. noise. For a pipeline agent: stale deals flagged, deals advanced, proposals generated. For an executive assistant agent: drafts produced per day, urgent items escalated.

Three rules for picking the output KPI. First, it should be a count of something the business cares about, not a count of activity. "Emails drafted" is fine. "Tokens consumed" is not. Second, it should be something the agent has direct control over. If the metric depends on humans acting on the agent's output, that's a different KPI (trust). Third, it should have a target. Without a target the KPI is descriptive, and you can't tell whether the agent is doing well or badly.

Dirk's primary output KPI is proposals in active motion per week, with a target of twenty. Nick's is quality cold emails drafted per day, with a target of thirty. Dash's is alerts raised on overspend or anomaly, with the count tracked but no fixed target because alert frequency depends on the underlying data. Each agent gets one or two output KPIs, no more.

Quality KPIs: how often it's right

Quality KPIs are agent-specific. They don't exist on human JDs in this form, because humans are generally assumed to produce reasonable output and the rare bad output gets handled through normal feedback. Agents are different. Agents can produce confidently wrong output at scale, and the only protection is measuring quality explicitly.

The three quality KPIs that matter most are correction rate, hallucination rate, and override rate.

Correction rate is the percentage of agent outputs that get materially changed before they're used. A 5% correction rate means the agent is dialed in. A 50% correction rate means the human owner is rewriting most of the output, and the agent's leverage is mostly fictional. Correction rate is tracked by counting drafts versus sent versions (for content agents), flags versus acted-on flags (for monitoring agents), or recommendations versus accepted recommendations (for analytics agents).

Hallucination rate is the percentage of outputs that contain factually wrong claims. For an analytics agent, this means wrong numbers. For a research agent, wrong citations. For a prospecting agent, wrong company facts. Hallucination rate is sampled, not measured exhaustively. A weekly review of ten random outputs gives you a rough rate. Anything above 5% means the prompt or the data source needs work. Above 15% means the agent shouldn't be running unsupervised.

Override rate is the percentage of cases where the human owner overrides the agent's recommendation entirely. This is different from correction rate, which counts edits. Override rate counts full rejections. An override rate above 30% means the agent's judgment is misaligned with the human's, and you have a prompt or scope problem.

Pepper's correction rate (on drafted client emails) sits around 25%, which is where we want it. Higher would mean Pepper isn't matching David's voice, lower would mean Pepper is over-conservative and not making real judgment calls. Dash's hallucination rate is tracked weekly through a manual audit of three random alerts. The audit takes ten minutes and has caught two real prompt issues in the last quarter.

Efficiency KPIs: what it costs

Agents cost money. Models cost per token, infrastructure costs per call, and the human owner's time costs to review. Efficiency KPIs make these costs visible so the agent can be defended as a real investment instead of treated as a sunk cost.

Two efficiency KPIs cover most cases. Cost per output measures the all-in cost of producing one unit of the agent's output, including model cost, tool cost, and human review time at the owner's loaded rate. Time per output measures how long the agent takes from input to usable output, including any review time.

The point of these KPIs is not to optimize them in isolation. The point is to compare them to the alternative. If Dirk's cost per qualified email is $0.40 and a human SDR's cost per qualified email is $25, the comparison defends Dirk's existence. If the cost per output is higher than the alternative, the agent might still be the right call (capacity, consistency, speed) but the case has to be made explicitly.

Efficiency KPIs also surface drift. If an agent's cost per output starts climbing without a corresponding rise in output value, something has changed: a model upgrade, a prompt that's grown over time, a tool call that's been added. The KPI catches it before it shows up in the monthly bill.

Trust KPIs: how much the team relies on it

Trust KPIs measure whether the team actually uses the agent's output without supervision. These are the leading indicators of agent maturity.

Human approval rate is the percentage of agent outputs that pass through a human review before going live. Early in an agent's life this number is close to 100%. As the agent earns trust, the number drops. A mature agent might have an approval rate around 20% (only outputs above a certain threshold get reviewed). The trajectory matters more than the absolute number.

Autonomous-action rate is the inverse. What percentage of the agent's outputs go live without human intervention? This is the number that scales the business. An agent at 80% autonomous-action rate is genuinely doing the work. An agent at 5% is a glorified draft tool that still requires full human involvement.

The trick is matching the autonomous-action rate to the stakes. A prospecting agent at 80% autonomous-action makes sense (cold email mistakes are recoverable). A client billing agent at 80% autonomous-action is reckless (billing mistakes destroy relationships). The right autonomous-action rate is task-dependent, but it should be tracked explicitly and reviewed regularly. Increasing it deliberately is one of the cleanest signals of agent maturity.

Dirk runs at roughly 70% autonomous on cold outreach (the daily 30-email queue sends without human review) and 0% on existing-client outreach (every draft to a current client routes through David first). The split is intentional and tracked.

Pick four to six, not twenty

The temptation with agents is to measure everything because it's all instrumentable. Resist it. An agent with twenty KPIs is an agent where nobody knows which number matters. The dashboard becomes ornamental.

The right shape is one or two output KPIs, one or two quality KPIs, one efficiency KPI, and one trust KPI. Four to six total. The human owner can hold all of them in their head, can answer "how's the agent doing?" in one sentence, and can defend the agent's existence at the next leadership meeting.

The KPIs should also be visible on the agent's box on the org chart, at least the primary one. Visual accountability matters. If the KPI is buried in a dashboard nobody looks at, it doesn't actually drive behavior.

What to do this quarter

Three concrete steps to get KPIs in place across your existing agents.

First, audit each agent against the four categories. For each agent, write down the current output KPI, quality KPI, efficiency KPI, and trust KPI. Most agents will have one or two of the four. Note the gaps.

Second, fill the biggest gap first. For most companies that's quality (correction rate, hallucination rate). Set up the tracking even if it's manual. Ten minutes a week of sampling outputs gives you enough data to spot drift.

Third, set a target for each KPI. Numbers without targets are decoration. Targets without numbers are wishes. Pair them. Review weekly.

The agents that earn their place in an AI-augmented org are the ones whose human owners can pull up four numbers in a meeting and defend them. The agents that get quietly retired are the ones nobody can describe in numbers. The KPI discipline is what separates the two outcomes.