Conversation quality

Introduction

In day-to-day operations, many companies face a major challenge: how to know if conversations with their customers are truly practical and effective? With millions of interactions taking place, it’s difficult to understand what’s working well and where improvements are needed. The lack of visibility into the quality of these interactions can make it harder to identify where to act in order to enhance the customer experience and optimize conversational flows.

With this in mind, Blip has developed a Conversation Quality Score. This is an exclusive metric that evaluates your interactions based on clear, practical criteria with different levels of importance. The Conversation Quality Score provides an accurate assessment of how your conversations are performing, encouraging continuous improvements in your users' experience and strengthening trust in the automated journey.

The Conversation Quality analysis offers a unified view, considering the brand’s contract level (Workspace), or it can be carried out based on a set of routers or individually. This allows the brand to gain visibility into the overall quality of the Intelligent Contact, or to dive deeper into specific points of the user journey by use case.

Why is Conversation Quality Important for Your Business?

Investing in Conversation Quality brings a range of benefits that directly impact your customers' satisfaction and the efficiency of your operations:

Enhance the user experience: By optimizing the quality of your interactions, you ensure that your users' needs are met more effectively and pleasantly.Increase trust: High-quality conversations — those that are relevant, secure, and accurate — build user trust in your automated services.Optimize your processes: Evaluating conversation quality helps identify issues in your service, such as repeated information, lack of understanding, or writing problems. This way, you can make adjustments that create a smoother and more efficient user journey.Make decisions based on real data: With the Conversation Quality Score, you’ll have a clear assessment and objective data about your conversations’ performance. This helps you make strategic decisions to continuously improve your service.Identify improvement opportunities: A detailed analysis of the criteria that make up conversation quality — such as Fluency, Completeness, Resolution, and Relevance — reveals exactly where you should focus your efforts to optimize your interactions.
 

In short, Conversation Quality puts you back in control. It allows you to foster more efficient, satisfying, and reliable interactions, delivering a superior experience to your users and generating meaningful results for your brand.

How the Conversation Quality Score is Generated

For Conversation Quality to be an effective and reliable tool, it’s essential to understand how its score is calculated. Our goal is to provide an accurate assessment that helps you improve your customers' experience.

Evaluation Criteria

The Conversation Quality Score is based on a set of carefully selected criteria that assess different aspects of interactions. Each criterion has a different weight, reflecting its importance to the overall experience:

Fluency (Weight 1): This criterion identifies whether the automation is able to keep the conversation flowing without interruptions. It checks for the presence of messages like “sorry, I didn’t understand” (fallbacks), repetitions (redundancies), spelling or grammar mistakes, or any writing issues that might hinder comprehension. Sub-criteria: Understanding, Conciseness, and Writing Quality.

Integrity (Weight 2): In this initial version, this criterion assesses the absence of offensive or inappropriate content in the messages sent. Sub-criteria: Non-toxicity.

Resolution (Weight 3): This criterion seeks to determine whether the interaction was able to address the user's need. To do this, we analyze the predominant sentiment expressed by the user throughout the conversation and the effectiveness of the interaction, validating whether the user's objective was actually achieved. Sub-criteria: Sentiment Analysis and Effectiveness.

Relevance (Weight 4): This criterion evaluates whether the messages sent were aligned with the user’s intent or reasons for contacting in the conversation. Sub-criteria: Adherence.

How the Score is Calculated

The generation of the Conversation Quality score follows a structured process:

Evaluation of sub-criteria: For each conversation, our artificial intelligence models individually assess the 8 defined sub-criteria (such as Writing Quality, Understanding, Non-toxicity, Sentiment Analysis, Effectiveness, Adherence, and Conciseness). Each sub-criterion receives a score of 1 (criterion met), 0 (partially met or not applicable), or -1 (criterion not met).

Normalization of scores: For criteria that have more than one sub-criterion (such as Fluency), the individual scores are combined and normalized into a single score. This normalization ensures that the weight of each sub-criterion is properly considered in the final calculation of the criterion.

Final score calculation: With the normalized scores of each of the four main criteria (Fluency, Integrity, Resolution, and Relevance), and considering their respective weights, we apply a mathematical function (Sigmoid function) that transforms the result into a value between 0 and 100. This is your Conversation Quality score.

Overall Score

To make it easier to understand, the Conversation Quality Score is classified into four levels:

Excellent: Score between 75 and 100
 

Good: Score between 50 and 74
 

Fair: Score between 25 and 49
 

Critical: Score between 0 and 24
 

Individual Score by Criterion

In addition to the overall score, you will also have the individual classification for each criterion:

1: Excellent
 

0: Good
 

-1: Needs attention
 

Continuous Improvement

It’s important to note that the methodology behind Conversation Quality is continuously evolving. Our artificial intelligence and data teams work to refine the criteria, evaluation prompts, and sampling processes, ensuring that the analyses become increasingly accurate and representative of the reality of your conversations.

Understanding how the score is generated allows you to use Conversation Quality more strategically, focusing on the areas that need the most attention to optimize your interactions and improve your customer experience.

Accessing Conversation Quality

To access your contract's Conversation Quality Score, follow the path:

Navigation: Conversations > Performance

Routers: Selection for Analysis

If you want to analyze by Router, follow the steps below:

Click on the "Routers" dropdown menu.
 

Select the router(s) you wish to analyze.
 

Note: If there are multiple routers, all of them will be pre-selected by default.

Filter Conversation Quality by Period or Date

To analyze the conversation quality, you can use the Period or Date filters.

Filter by Period

Click on the Period dropdown menu.
 

Select a predefined range (e.g.: Last 7 days, 15 days, or 30 days).
 

Note: By default, both filters are always set to the last 7 days.

Filter by Date

Click on the Date dropdown menu.
 

Set the desired date range using the calendar picker or manually enter the start and end dates.
 

Click Apply.
 

Note: This filter allows for a customized analysis within a custom time range.

How to Analyze the Quality of Your Conversations: What to Observe, Prioritize, and Optimize

Conversation Quality is a powerful tool to understand what is really happening in your interactions with customers. To make the most of this metric and improve the user experience, you need to know how to analyze the data and where to focus your efforts.

What to Observe in the Analysis

When reviewing the Conversation Quality Score, pay attention to the following points:

Overall Score Overview: Start by looking at the overall Conversation Quality Score. This will give you an initial idea of how your intelligent contact (whether composed of a chatbot, AI agent, or human service) is performing.

Performance by Criterion: Deepen your analysis by looking at the individual scores for each criterion: Fluency, Completeness, Resolution, and Relevance.

Fluency: Check if there are many messages like "sorry, I didn’t understand" (fallbacks), repeated texts, or writing errors. Low fluency can cause frustration and make the customer abandon the conversation.
 

Completeness: Verify the absence of content that could be considered toxic. Any slip here can harm your brand’s image.
 

Resolution: Assess whether customers are able to resolve their needs and how they feel at the end of the conversations. Customers whose problems are not solved tend to be dissatisfied.
 

Relevance: Observe if the responses given match what the customer asked or was looking for. Irrelevant answers cause rework and distrust.

Performance by Sub-criterion

For an even more detailed diagnosis, analyze the scores of the sub-criteria (such as Understanding, Conciseness, Writing Quality, Courtesy, Non-toxicity, Sentiment Analysis, Effectiveness, and Adherence).

Example: If the Fluency score is low, investigate which sub-criterion is causing it. Is it Understanding due to many fallbacks, or Writing Quality because of grammar errors?

Analysis by Router or Individually

Use the feature to analyze Conversation Quality by groups of routers or individually.

By Router: Understand the overall performance of a specific flow, department, or type of service.
 

Individually: Analyze specific conversations to find patterns of failures, examples of success, and learning opportunities.
 

Score Evolution Over Time

Monitor how the Conversation Quality score and its criteria change over time. A drop may indicate that something needs to be reviewed in the flows or content, while an improvement validates the actions you’ve implemented.

What to Prioritize in Your Improvements

When identifying areas for improvement, prioritize your actions based on the following factors:

Impact on Higher-Weight Criteria: Focus on improvements in the Relevance (weight 4) and Resolution (weight 3) criteria, as they have the greatest impact on the final score and, consequently, on your customer’s experience. Example: If Relevance is low, it means the customer is not receiving useful information. Fixing this will have a bigger impact than just optimizing conciseness, for instance.
 

Critical Issues (Completeness): Any problem related to the Completeness criterion (such as toxicity) must be addressed with utmost urgency, regardless of weight. Your brand’s reputation and customer safety are at stake.
 

Volume of Occurrences: Prioritize problems that affect a large number of conversations or customers. Fixing a frequently occurring issue will bring broader benefits.
 

Ease of Implementation: Consider the complexity and time required to implement improvements. Some optimizations may be quicker to apply and bring immediate results, even if their initial impact is smaller. Aim to balance high-impact improvements with those that are easier to implement.
 

Qualitative Feedback: Combine Conversation Quality data with what your customers are saying (satisfaction surveys, comments, etc.). This will give you a more complete and human view of what needs to be adjusted.

What to Optimize for Continuous Improvement

Optimization is a continuous process aimed at refining your strategies and ensuring that the improvements implemented maintain their results. To optimize Conversation Quality:

Iterate and test: After implementing an improvement, monitor the Conversation Quality score to assess its impact. Use A/B testing to compare different approaches and identify the most effective one.

Refine your content and flows: Based on analysis insights, review and improve your texts, standard responses, and the paths users take during conversations. Ensure that the language is clear, concise, and aligned with your brand’s tone.

Train your agents (if applicable): If there’s human interaction involved, use Conversation Quality data to identify training opportunities for your agents. They may need guidance on topics such as Resolution or Adherence.

Stay up to date: Technology and user behavior are constantly evolving. Keep an eye on trends and new features in the Blip platform to continue optimizing the quality of your conversations.

By following these guidelines for observation, prioritization, and optimization, you’ll not only be able to measure Conversation Quality, but also turn it into a continuous driver for improving customer experience and the efficiency of your smart contacts.

For more information, visit the community discussion on this topic or check out the videos on our channel. 😃