07212017

5 Key Metrics for Measuring Chatbot Success

Photo credit: Zapp2Photo/Shutterstock

You’ve built a chatbot and released it, but now what? A bot maker’s job is never done, and now that your bot has people talking, it’s time to assess how it can be improved.

You’ve built a chatbot and released it, so now what? A bot maker’s job is never done, and now that your bot has people talking, it’s time to assess how it can be improved. 

Poring over chatbot analytics might seem overwhelming, but it doesn’t have to be: At a glance, you can measure success for your chatbot by checking a handful of metrics. Let’s delve into the five metrics you’ll want to check first.

Engagement, retention and reach

These metrics are a great place to start, because they tell you at a glance how helpful or useful your bot is. Your reach tells you how many people have used it. This is important not just for measuring growth, but it’s also worth checking to understand the metrics we’ll explore below. If your reach is very small, for example, the sample size may not be big enough to draw conclusions from. 

Your engagement and retention rates, meanwhile, show how many people are actually using your chatbot. Your active users are those who read what the chatbot has to say, while your engaged users answer back. Together, fluctuations with these rates help you assess your chatbot’s shortcomings and measure its success.

When you’ve introduced a new feature or made changes to your bot, compare a before and after engagement rate to measure the success of the change. You should also monitor your retention rate over time; daily retention is most useful for media-based chatbots, while other services might benefit best by measuring seven- or 30-day retention.

Top users and top phrases

Who’s using your chatbot the most, and what are they saying? These questions are important to measuring chatbot success, because they help you identify what’s working and what isn’t. By understanding how your heaviest users are interacting with your bot, you can learn how to improve the overall experience for everyone else. This, in turn, can increase your chatbot engagement and retention rates.

Most users barely interact with a chatbot, and when they’ve barely interacted, they leave little data behind for determining what made them leave. Therefore, your power users’ activity is key to understanding the metrics of your chatbot.

Conversation steps

Every time a user says something and a chatbot responds, it counts the interaction as a step. The number of steps a user takes per conversation can help in measuring chatbot success when it comes to accomplishing a task. 

What this number means varies by bot. If your chatbot is designed to capture user attention and keep them engaged, your goal should be a high step count. Task-oriented utility bots should aim for a lower number of conversation steps, demonstrating they can help the user quickly. Basically, your step count helps you understand if your bot can maintain a customer’s attention and if it can quickly complete tasks. 

Click-through rate 

Like conversation steps, how click-through rates function as chatbot success metrics will depend on what you’re trying to do with the bot. Let’s say you’re a publisher who’s developed a chatbot to alert users of stories that interest them. If you’re sending users links to the content on your website or to the app, you’ll want your CTR to be high.

For other chatbots, a high CTR is a bad thing. This is the case for bots designed to provide a self-contained experience. For example, if many people using a customer support bot click through to find answers to their problems elsewhere, rather than have the bot solve the issue, it might mean the bot is doing a poor job.

Confusion triggers

One of the most obvious chatbot analytics to track, confusion triggers are moments where a chatbot totally fails to act on a user’s query. You’ve certainly experienced this moment before – the dreaded “Sorry, I don’t understand” message. 

This situation is incredibly frustrating to the user, because it requires them to try to rephrase their query until the bot understands. Sometimes it might prompt users to stop talking altogether. When assessing user churn and the actual content of conversations, look for confusion triggers to see how you can make improvements. 

It doesn’t stop there! This is just a sample of the chatbot analytics you’ll want to track. You can learn more about other metrics and how they can help your bot. 

What Next?

Recent Articles

Leave a Reply

You must be Logged in to post comment.