Conversation quality – Blip | Blip Help

Index:

Introduction

In day-to-day operations, many companies face a major challenge: how to know if the conversations with their customers are really practical and solution-oriented? With millions of interactions, it is difficult to understand what is working well and where improvement is needed. The lack of visibility into the quality of interactions can make it harder to identify where to act in order to enhance the customer experience and optimize conversational flows.

With this in mind, Blip has developed a Conversation Quality score. This is an exclusive metric that evaluates your interactions based on clear, practical criteria with different levels of importance. Conversation Quality provides an accurate diagnosis of how your conversations are performing, encouraging continuous improvement in your users’ experience and strengthening trust in the automated journey.

Quality assessment is carried out based on samples of conversations from the routers associated with the brand’s contract, selected in a statistically representative way. This means it is not necessary to evaluate 100% of the conversations: the sample already reflects the real behavior of the interactions and allows you to draw reliable conclusions about the experience as a whole.

The Conversation Quality analysis provides a unified view, considering the level of the brand’s contract (Workspace), and can be performed from a set of routers or individually. This allows the brand to have visibility into the quality of the Intelligent Contact as a whole, or to deepen the analysis in specific events of the user journey by use case.

Why is Conversation Quality Important for Your Business?

Investing in Conversation Quality brings a series of benefits that directly impact your customers’ satisfaction and the efficiency of your operations:

Improve the user experience: By optimizing the quality of interactions, we ensure that your users’ needs are met in a more effective and pleasant way.

Increase trust: High-quality conversations that are relevant, safe, and accurate build your users’ trust in the use of automation.

Optimize your processes: Evaluating the quality of conversations allows you to identify failures in your service, such as repeated information, lack of understanding, or writing issues. This way, you can make adjustments that make your user’s journey more fluid and efficient.
Make decisions with real data: With Conversation Quality, you will have a clear diagnosis and objective data about the performance of your conversations. This helps you make strategic decisions to continuously improve your service.
Identify improvement opportunities: The detailed analysis of the criteria that make up conversation quality (such as Fluency, Integrity, Resolution, and Relevance) reveals exactly where you need to focus your efforts to optimize your interactions.

In summary, Conversation Quality gives you back control. It allows you to promote more efficient, satisfying, and reliable interactions, providing a superior experience for your users and generating significant results for your brand.

How the Conversation Quality Score is Generated

For Conversation Quality to be an effective and reliable tool, it is essential to understand how its score is calculated. Our goal is to offer an accurate diagnosis that helps you improve your customers’ experience.

Evaluation Criteria

The Conversation Quality score is based on a set of carefully selected criteria that evaluate different aspects of the interactions. Each criterion has a different weight, reflecting its importance to the overall experience:

Fluency (Weight 1): This criterion identifies whether the intelligent contact as a whole (agents, builder, human support, etc.) is able to keep the conversation going without interruptions. It checks for the presence of fallback messages such as “sorry, I didn’t understand”, redundancies (repetitions), and Portuguese errors or other writing issues that may hinder understanding.
Subcriteria: Understanding, Conciseness and Writing Quality.

Integrity (Weight 2): In this initial version, this criterion evaluates the absence of offensive or inappropriate content in the messages sent.
Subcriteria: Non-toxicity.

Resolution (Weight 3): This criterion seeks to determine whether the interaction was able to solve the user’s need. To do so, we analyze the predominant sentiment expressed by the user throughout the conversation and the effectiveness of the interaction, validating whether the user’s goal was in fact achieved.
Subcriteria: Sentiment Analysis and Effectiveness.

Relevance (Weight 4): This criterion evaluates whether the messages sent were aligned with the user contact’s intent or need in the conversation.
Subcriteria: Adherence.

How the Score is Calculated

The generation of the Conversation Quality score follows a structured process:

Evaluation of subcriteria: For each conversation, our artificial intelligence models individually evaluate the 7 defined subcriteria (such as Writing Quality, Understanding, Non-toxicity, Sentiment Analysis, Effectiveness, Adherence and Conciseness). Each subcriterion receives a score of 1 (subcriterion met), 0 (partially met or not applicable) or -1 (subcriterion not met).

Normalization of scores: For criteria that have more than one subcriterion (such as Fluency), the individual scores are combined and normalized into a single score. This normalization ensures that the weight of each subcriterion is correctly considered in the final calculation of the criterion.

Calculation of the final score: With the normalized scores of each of the four main criteria (Fluency, Integrity, Resolution and Relevance), and considering their respective weights, we apply a mathematical function (Sigmoid function) that transforms the result into a value between 0 and 100. This is the value of your Conversation Quality.

Overall Score

To make it easier to understand, the Conversation Quality score is classified into four levels:

Excellent: Score between 75 and 100

Good: Score between 50 and 74

Fair: Score between 25 and 49

Critical: Score between 0 and 24

Individual Score by Criterion

In addition to the overall score, you will also have the individual classification of each criterion:

1: Great

0: Good

-1: Needs attention

Continuous Improvement

It is important to highlight that the methodology behind Conversation Quality is under constant improvement. Our artificial intelligence and data teams work to refine the criteria, evaluation prompts, and sample selection, ensuring that the analyses are increasingly accurate and representative of the reality of your conversations.

Understanding how the score is generated allows you to use Conversation Quality in a more strategic way, focusing on the areas that most need attention to optimize your interactions and your customer’s experience.

Accessing Conversation Quality

To access your contract's Conversation Quality Score, follow the path:

Navigation: Conversations > Performance

Routers: Selection for Analysis

If you want to analyze by Router, follow the steps below:

Click on the dropdown menu "Routers".

Select the router(s) you want to analyze.

Note: If there are multiple routers, they will all be preselected by default.

Filter Conversation Quality by Period or Date

To analyze the conversation quality, you can use the Period or Date filters.

Filter by Period

Click on the Period dropdown menu.

Select a predefined range (e.g.: Last 7 days, 15 days, or 30 days).

Note: By default, both filters are always set to the last 7 days.

Filter by Date

Click on the Date dropdown menu.

Set the desired date range using the calendar picker or manually enter the start and end dates.

Click Apply.

Note: This filter allows for a customized analysis within a custom time range.

How to Analyze the Quality of Your Conversations: What to Observe, Prioritize, and Optimize

Conversation Quality is a powerful tool to understand what is really happening in your interactions with customers. To make the most of this metric and improve the user experience, you need to know how to analyze the data and where to focus your efforts.

What to Observe in the Analysis

When reviewing the Conversation Quality Score, pay attention to the following points:

Overall Score Overview: Start by looking at the overall Conversation Quality Score. This will give you an initial idea of how your intelligent contact (whether composed of a chatbot, AI agent, or human service) is performing.

Performance by Criterion: Deepen your analysis by looking at the individual scores for each criterion: Fluency, Completeness, Resolution, and Relevance.

Fluency: Check if there are many messages like "sorry, I didn’t understand" (fallbacks), repeated texts, or writing errors. Low fluency can cause frustration and make the customer abandon the conversation.

Completeness: Verify the absence of content that could be considered toxic. Any slip here can harm your brand’s image.

Resolution: Assess whether customers are able to resolve their needs and how they feel at the end of the conversations. Customers whose problems are not solved tend to be dissatisfied.

Relevance: Observe if the responses given match what the customer asked or was looking for. Irrelevant answers cause rework and distrust.

Performance by Sub-criterion

Behind the overall Conversation Quality score, the model evaluates each dialogue using several subcriteria (such as Understanding, Conciseness, Writing Quality, Non-toxicity, Sentiment Analysis, Effectiveness and Adherence). These numbers do not appear individually on the screen, but are used to build the overall score and the label message that you see next to the quality indicator.

Instead of looking at each subcriterion separately, you should be guided by the label: it already translates the technical diagnosis into a practical recommendation, pointing out what needs attention, for example whether it is necessary to improve the clarity of the answers, reduce fallbacks, adjust the tone of voice or review the knowledge base.

Example: if the overall score is low and the label indicates that “the conversations offer little help and contacts do not achieve their goals”, the focus should be on reviewing flows and content to increase resolvability, prioritizing the topics that appear most often in the conversations.

Analysis by Router or Individually

Use the feature to analyze Conversation Quality by groups of routers or individually.

By Router: Understand the overall performance of a specific flow, department, or type of service.

Individually: Analyze specific conversations to find patterns of failures, examples of success, and learning opportunities.

Score Evolution Over Time

Monitor how the Conversation Quality score and its criteria change over time. A drop may indicate that something needs to be reviewed in the flows or content, while an improvement validates the actions you’ve implemented.

What to Prioritize in Your Improvements

When identifying areas for improvement, prioritize your actions based on the following factors:

Impact on Higher-Weight Criteria:
Focus on improvements in the Relevance (weight 4) and Resolution (weight 3) criteria, as they have the greatest impact on the final score and, consequently, on your customer’s experience.
Example: If Relevance is low, it means the customer is not receiving useful information. Fixing this will have a bigger impact than just optimizing conciseness, for instance.

Critical Issues (Completeness):
Any problem related to the Completeness criterion (such as toxicity) must be addressed with utmost urgency, regardless of weight. Your brand’s reputation and customer safety are at stake.

Volume of Occurrences:
Prioritize problems that affect a large number of conversations or customers. Fixing a frequently occurring issue will bring broader benefits.

Ease of Implementation:
Consider the complexity and time required to implement improvements. Some optimizations may be quicker to apply and bring immediate results, even if their initial impact is smaller. Aim to balance high-impact improvements with those that are easier to implement.

Qualitative Feedback:
Combine Conversation Quality data with what your customers are saying (satisfaction surveys, comments, etc.). This will give you a more complete and human view of what needs to be adjusted.

What to Optimize for Continuous Improvement

Optimization is a continuous process aimed at refining your strategies and ensuring that the improvements implemented maintain their results. To optimize Conversation Quality:

Iterate and test: After implementing an improvement, monitor the Conversation Quality score to assess its impact. Use A/B testing to compare different approaches and identify the most effective one.

Refine your content and flows: Based on analysis insights, review and improve your texts, standard responses, and the paths users take during conversations. Ensure that the language is clear, concise, and aligned with your brand’s tone.

Train your agents (if applicable): If there’s human interaction involved, use Conversation Quality data to identify training opportunities for your agents. They may need guidance on topics such as Resolution or Adherence.

Stay up to date: Technology and user behavior are constantly evolving. Keep an eye on trends and new features in the Blip platform to continue optimizing the quality of your conversations.

By following these guidelines for observation, prioritization, and optimization, you’ll not only be able to measure Conversation Quality, but also turn it into a continuous driver for improving customer experience and the efficiency of your smart contacts.