Beyond the Demo: How to Measure AI Agent Performance

Launching an AI agent is the easy part. Knowing whether it works, and how to improve it, is not.

There are more ways to change an agent than you have hours to test them in. You could start with a call instead of a text, ask the user for 4 documents at once or one at a time, have the agent reach out in the evening instead of the afternoon, change how the agent responds to the user’s questions, and the list goes on.

How do you know what lever to pull and if it was effective? Knowing whether this week’s version is better than last week’s is a challenge without a clear answer.

We wrote earlier about how we build agents credit unions can actually trust. This is the other half of that problem: how we tell whether they are working.

The two ways agent measurement goes wrong

The first is picking a metric that sounds right but that the agent does not control. Loan close rate is a number that executives care about. It is also a bad way to grade a document collection agent, because that agent has no say in underwriting, no say in pricing, and no say in whether the member signs. Grade on close rate and you will spend a quarter tuning agents against noise.

The second is measuring too many things. Once you have instrumented an agent, there are a multitude of metrics that can be measured: response latency, tool call counts, retry rates, transcript length, channel mix, time of day, you can measure it all. While a dashboard with 40 numbers on it feels rigorous, in practice it introduces more noise than signal.

We settled on three pillars to measure our agents: delivery, sentiment, and cost. Every agent we run is scored on all three.

Delivery: did the agent do the one thing it was sent to do?

Delivery is measured against the specific action the agent was deployed to influence, and nothing downstream of it. For a direct deposit switch agent example, that is the number of members who initiated a switch divided by the number of members it reached out to.

We are careful to choose an delivery metric that the agent can directly drive. If you cannot draw a straight line from something the agent did to the number moving, it is the wrong number.

This “direct” approach can be hard to sell to customers or executives. No one really cares about the percent of documents collected if it does not move the key business metrics, like loan funding rates or membership growth. We are not arguing that those key business metrics be ignored, we are only saying that they should be used in the correct context.

In the context of developing an agent to drive a certain user action, use the direct metrics. However, when you are evaluating the overall business impact of the agent, you should absolutely look at the indirect metrics as well.

Sentiment: how it felt to be on the receiving end

Sentiment is the pillar most teams skip, because it looks unquantifiable. Take a 1-10 scoring methodology. How do you distinguish between a 7 and an 8? If you ask 100 people who rated a conversation sentiment as a 7 “What would make this conversation an 8?”, you would get 100 different answers. 1-10 scoring and other numerical approaches look nice on the outside, but under the hood they are very subjective and inconsistent.

Sentiment is important to us. We don’t overlook it entirely just because it cant be precisely measured. Instead of a numerical scoring, we decided to take some inspiration from the Net Promoter Score (NPS). We like the NPS approach because it groups users into 3 broad buckets but still results in a measurable value. Here’s how we have adapted an NPS-like approach for our needs:

Every interaction is classified into one of three buckets: positive, neutral, or negative. The score is the % of positive minus the % of negative interactions. A model asked to sort a transcript into three well-defined buckets is correct far more often than one asked to choose from a ten point scale. In other words, it is much easier to choose between a “positive” and a “neutral” label than it is to decide between a 7 and an 8.

With our sentiment approach, you give up granularity you never really had for a more consistent and simpler classification.

Cost: what the outcome took to produce

Cost is the plainest of the three. Tokens, AI synthesis, telephony minutes, the compute behind every classification. Add it up per agent.

We measure cost on a per user basis, instead of per interaction. A member who converts on the second call and a member who converts on the tenth produce the same delivery score for the agent. However, one of them cost five times as much to get there.

This approach gives us insight as a finance metric (i.e. how much are agents costing us to run?) as well as a product metric (i.e. is the agent achieving its goal efficiently and with low user friction?)

Putting it into practice: a good sentiment score hiding a bad agent

The three pillars are in tension, and that is the point. Each one on its own can be satisfied by an agent doing something you would not want.

A change that lifts delivery while sentiment holds and cost per outcome drops is a real improvement. A change that lifts delivery while the other two degrade is a trade, and you should make it deliberately or not at all.

Here is what this looks like in practice. The scenario below is an illustrative example of how our measurement framework could be used.

Scenario

A document collection agent for loan applications is scoring well on sentiment and poorly on the other two. Members like talking to it. But not many of them are sending documents, and cost per completed file is high because a lot of calls are happening and not much is coming out of them.

Sentiment on its own would have told us this agent was working. The three pillars together told us something different: it was pleasant because it was avoiding the hard part of the conversation.

NOTE: For a more consistent visual, a higher value for cost in our charts correlates with a lower cost. This way, an increase is a positive move for all metrics.

Our First Iteration

The transcripts gave us a direction. The agent was too soft. It backed off at the first sign of friction and treated any hesitation as a signal to wrap up. A member says “I can’t find my document right now,“ and instead of offering to help them look, the agent starts its closing sequence: “No worries, I’ll call you back later.“

So we made some changes. Now, the agent always offers to help locate a document before it closes a call.

Initial Results and Second Iteration

Delivery moved up right away. Sentiment fell off a cliff. Cost rose.

The fix had created a loop. The agent would offer help finding the document, and if the member still could not find it, it would start the same help sequence over again instead of escalating to a human or pointing them to the member service line.

The second change was to add guardrails rather than a reversal. The agent was given up to three attempts at helping locate a document, with each attempt required to offer different guidance from the last. Then a handoff.

Final Results

Sentiment came back close to where it started, with a small give-back on delivery. Cost reduced as well, below the original value.

We would not be able to make targeted, informed iterations like these without an effective assessment framework. Our approach works because it looks at the whole picture while keeping the complexity low.

Why does this matter for Financial Institutions looking for an AI solution?

It’s easy to make a demo ready agent that covers a specific use case and handles pre-planned user questions/problems. But how do you know that it will work across many different user types, situations and use cases? How do you know that the vendor will make well-informed and effective decisions on how to improve the agent? Unless the agent has been tested and evaluated against well defined, holistic and straightforward framework, you don’t.

When evaluating an AI solution, be sure to ask how they measure their agents. What to look out for:

1. What metrics are not being tracked? If a vendor only quotes campaign success rates, are they ignoring user experience?

2. Are the metrics value-adds for your institution and can be directly influenced by the agent?

3. Are they taking a deliberate approach to what metrics are important or are they just measuring everything?

4. Does the vendor consider cost implications for their customers? An agent reaching out every 5 minutes may improve results, but it will cost you greatly (not to mention the user impact).

Measuring the effectiveness of an AI agent on the surface sounds like a secondary concern: the step that comes after you launch it. We argue that it should be a primary concern, because the metrics you measure by are the metrics you build by.

At Glide, we have a well-defined, measurable framework to ensure our AI agents perform across all dimensions. Does your AI vendor?