How Do I Track AI Agent Performance on the Org Chart?

Tracking AI agent performance on the org chart works the same way tracking human IC performance works, with three additions specific to agents. Run a weekly scorecard. Include the scorecard in your standing leadership meeting. Look at the trend over time, not just the current week. Then add: automate the KPI push so the data is fresh, track failure modes alongside the numbers, and review whether the agent's seat still makes sense once a quarter. That's the entire pattern. The work is in being consistent about it for long enough that the trend becomes visible.

Most companies running AI agents don't track performance like this. They check on agents when something breaks, look at a dashboard nobody trusts, or rely on the agent's human owner to "feel" whether things are working. All three approaches produce the same outcome: surprise. The agent works for nine months, then quietly degrades, then causes an incident that nobody saw coming because nobody was looking. The weekly scorecard pattern is what turns the surprise into trend data you can act on.

Weekly scorecard, same template as humans

The unit of agent performance tracking is the weekly scorecard. Same format as a human IC scorecard. Same week. Same review meeting. Same level of seriousness.

The scorecard has the agent's four to six KPIs across the top (output, quality, efficiency, trust, the same categories covered in the KPI post). Each row is a week. The numbers update on Sunday night or Monday morning so they're ready for the weekly leadership meeting. The human owner reviews the row before the meeting. Anomalies get noted in a comment column.

The scorecard template for an agent looks identical to a human's, with the column header set distinguishing agent-specific KPIs (override rate, hallucination rate) from output KPIs that could apply to either. Keep the template uniform. The reason is structural: the leadership team should be reading agent performance the same way they read human performance. Different templates produce different attention, and agents quietly slip below the threshold.

At Sneeze It, the agent scorecards live in a shared Google Sheet alongside the human IC scorecards. The view that the leadership team pulls up at the weekly meeting shows both, sorted by team, with agents indicated by an icon on the row. Nothing fancy. The visibility is the point.

Include agents in the leadership meeting

If the agent's performance isn't being reviewed in the same forum where human performance is reviewed, you don't actually have agent performance tracking. You have a side project that the human owner runs on their own time and that the rest of the leadership team never sees.

Pick the existing meeting. For most operating teams running EOS, this is the weekly L10. For Scaling Up shops, this is the weekly meeting on the OPSP cadence. For everyone else, this is whatever you call your standing leadership review. Slot the agent scorecards into the same place where human scorecards get reviewed. Same time allocation per agent (typically two to three minutes, longer if there's a red flag).

The conversation is the same conversation you'd have about a human IC. What's the trend on the KPI? What changed this week? What's the plan if the trend is bad? Who's doing the work to fix it? When is the next checkpoint? The human owner answers, the same way they would for a human report.

The two minutes per agent feels light. It is. The point is the trend over time, not the deep dive each week. The deep dive happens when the trend starts moving in the wrong direction, and the weekly cadence is what makes the trend visible early enough to act.

Trend, not snapshot

The single most important shift in tracking agent performance is moving from snapshot to trend. A snapshot ("Dirk produced 87 emails last week") tells you almost nothing. A trend ("Dirk has produced 85-92 emails for the last six weeks, with correction rate trending from 12% to 7%") tells you the agent is dialing in.

The way to make trend data exist is to store each week's scorecard row historically, not just the current week. The whole point of the scorecard is to be able to look at it over time. A four-week moving average reveals more than any single week's number ever will. A quarter of history reveals seasonal patterns, prompt change impacts, and slow degradations.

Trend tracking is also what catches the silent failures. An agent that drops from 92 emails to 86 in one week is a non-event. An agent that drops from 92 to 86 to 80 to 75 over four weeks is a real signal. The trend is the signal. The snapshot is noise.

A few teams build the trend visualization right into the scorecard. A small sparkline next to each KPI showing the last twelve weeks. It takes ten minutes to add and changes how the meeting feels. You stop reading numbers and start reading direction.

Track failure modes alongside the numbers

KPIs don't capture everything that matters about agent performance. The other half is failure modes: the specific ways this agent has gotten things wrong, the ways it might be drifting, the edge cases it handles poorly.

Maintain a running failure modes log per agent. Date, description of the failure, root cause, fix, status. The log lives alongside the agent's JD. New failure modes get added when they happen. Patched failure modes get marked as resolved. The log is reviewed monthly, not weekly (failure modes are more strategic than tactical).

The failure modes log catches the things the KPIs miss. An agent's KPIs can all be green while the agent is failing in a particular way that the KPIs don't measure. The log surfaces that. It also creates institutional memory: when a new team member takes over the agent's ownership, they read the failure modes log and inherit two years of hard-won knowledge instead of rediscovering it the painful way.

Automate the KPI push

The single biggest failure mode in agent performance tracking is the data going stale because someone has to manually update a spreadsheet every week. Within a month, the human owner forgets. Within three months, the scorecard is fiction. Within six months, the leadership meeting stops reviewing agents because the data is too old to trust.

The fix is automation. Each agent should push its KPI values to the central scorecard on a schedule. Not "we'll do it from time to time," on a schedule. Daily for the operational metrics, weekly for the rolled-up scorecard row.

At Sneeze It, this is handled by an agent named Tally whose entire job is reading KPI values from local sources and pushing them to OTP, the coordination layer that displays the org's scorecard. Tally runs four times a day on weekdays. The human owners don't have to remember anything. The numbers are fresh when the leadership team opens the meeting on Monday.

The KPI push pattern is the small piece of infrastructure that makes everything else work. Without it, the discipline of weekly review erodes within a quarter, and the whole tracking system collapses. With it, the discipline is sustainable indefinitely.

Implementing the push doesn't require a dedicated agent. A scheduled script that reads from each agent's data source (its logs, its database, its output history) and writes to a central scorecard works fine for small numbers of agents. The dedicated agent makes sense once you're tracking ten or more KPIs across multiple agents and the maintenance cost of individual scripts gets noticeable.

Quarterly: does the seat still make sense

Weekly is for KPI trend. Monthly is for failure modes. Quarterly is for the existential question: does this agent's seat still make sense at all?

Once a quarter, the human owner answers three questions for each agent they own. Is the agent producing value commensurate with its cost? Has the agent's scope expanded or contracted such that the KPIs no longer measure what matters? Is there a better-shaped agent that should replace this one?

These are not weekly questions. Asking them weekly produces churn. Asking them quarterly produces deliberate decisions. Agents that don't pass the quarterly check get scope-narrowed, KPI-updated, or retired. The retirement decision is the one most companies avoid making, and the quarterly cadence is what forces the conversation.

Jeff, an internal data scout agent at Sneeze It, was retired after a quarterly review made the case that the seat had been informally absorbed by three other agents. The retirement decision was uncomfortable. The discipline of the quarterly review was what made the decision land cleanly instead of dragging on.

What to do this quarter

Four steps to install the tracking discipline.

First, build the weekly scorecard template. One row per agent, columns for each KPI, a comment column, a sparkline column. Same template as your human IC scorecards. Keep it simple.

Second, slot the agent scorecard review into your existing weekly leadership meeting. Two minutes per agent. Same forum, same level of seriousness as human IC review.

Third, automate the KPI push from each agent's data source to the scorecard. Script, scheduled job, or dedicated agent. Whichever fits, but automate it. Manual data entry kills tracking within a quarter.

Fourth, run the first quarterly review at the next quarter boundary. Three questions per agent. Decisions documented. Retire any agent that doesn't pass.

Tracking agent performance is not glamorous and it is not optional. It is the work that turns a collection of clever AI experiments into an operating function the company can rely on. The discipline shows up in the scorecard. The scorecard shows up in the meeting. The meeting shows up in the decisions. That chain is what builds an AI-augmented org instead of an AI mess.