Unit Tests – Blip | Blip Help

Index

Introduction

The Unit Testing feature allows you to validate the accuracy of your conversational flow's responses, whether deterministic or based on an AI agent.

The test works by setting up interaction pairs: you define the input message, which can be text or a public URL, and the expected response, which can be text and/or file validation. During execution, the test sends each interaction and stops upon detecting the first failure.

This process ensures that the system responds correctly and allows quick adjustments in case of issues, making it an essential tool to verify if the expected behaviors are correct before deployment to production. The feature facilitates maintenance and continuous evolution of your Intelligent Contact.

In addition to creating and running tests, it is now also possible to track the recent execution history of each test and view the details of a specific execution.

With this, builders can better analyze the validation cycle of the flow: run a test, identify failures, adjust the flow, and run again, tracking recent attempts more traceably.

How to Access Unit Tests

You can open Unit Tests as follows:

1. On the Studio screen:

In the top right corner of the screen, click the icon .
In the menu that opens, select Unit Tests.

The following screen will open:

Managing Unit Tests

The unit tests management screen is the starting point to view, create, and run tests for your bot or AI agent.

Create Test

Click the Create Test button in the top right corner or, if no tests have been created, click the centered button in the list Create New Test. You can fill in all test parameters manually or import a configuration file by clicking the button:

Import Configurations: load a file in the Blip format with pre-made tests.

Other Features

Search: Search field to find specific tests by name.
Test List: Displays the unit tests created for the bot. In the listing, you can track:
- Test: name of the configured test.
- Versions: number of executions performed for that test. Each execution generates a new version in the history.
- Last Update: date and time of the last edit made to the test. This information does not necessarily represent the last execution.
- Status: summarized result of the execution, indicating whether there was success, failure, or another state related to the test.
  - Waiting for Execution: The test was created but has not yet been run.
  - Success: All test interactions passed successfully (e.g., "Success 18/18").
  - Metric Failure: The test was run, but some interactions failed (e.g., the message "Metric Failure 25/57" indicates that 25 interactions failed out of 57 configured interactions).
  - Error on Start: The test could not be executed due to an error at the start (e.g., "Error on Start 2/3").
  - Interrupted: The test may be interrupted during execution.
- To view recent executions of a test, click the expansion arrow next to the desired item.
Delete Tests: removes one or more tests.
Run Tests: runs one or more selected tests.

It is also possible to filter tests by status to more quickly locate tests with success, failure, or waiting for execution.

Configuring a Unit Test

When creating or editing a test, you will have access to three configuration tabs: Definitions, Variables, and Interactions.

Interactions

This tab is where you define the sequence of questions and answers to validate your bot's or AI agent's behavior.

Order: The order in which the interactions will be executed. You can reorder interactions by dragging the grid icons.
Description: The text input that will be sent to the bot.
Result: The status of the interaction after the test execution, which can be:
- Waiting for Execution: The interaction has not yet been tested.
- Success: The bot's response matched the expected.
- Error on Start: The interaction could not be started.
- Metric Failure: The bot's response did not match the expected.
- Interrupted: The test was interrupted by the user during execution.

The order of interactions defines the sequence in which the test will be executed. If an interaction fails, execution is interrupted and the following interactions will not be executed.

Configuring an Interaction:

By clicking on an interaction, you can expand the section to configure it in detail.

Input Type: Defines the type of input you are sending.
Input Message: The user input can be plain text or a public URL pointing to a file.
Expected Response:
- Text Blocks: The expected response can be one or more text snippets.

When it comes to structured formats, such as JSON menus, it is advisable to include the JSON directly, ensuring the system understands and compares as expected.

File Type: Additionally, the response may require the presence of specific files, such as documents, images, audio, or videos. The configuration must specify not only the type but also the expected quantity. For example, if the interaction should return two documents, the configuration must reflect this. The test will fail if the response does not exactly match the expected number and type of files.
Text: The text the bot will receive (e.g., "What are the operating hours?").

Textual Comparison Metric:

Similarity:

The similarity metric evaluates how closely a generated response matches the expected response in terms of content and structure. It allows variations while still considering the response valid.

Usage Recommendation:

Ideal for flexible systems, such as intelligent agents, which may generate responses with some variation.
Set the similarity threshold to establish the acceptable degree of variation. For example, a threshold of 6.5 indicates the response must have at least 65% similarity to the expected.

Exact Match:

This metric requires the generated response to be completely identical to the expected response, without any deviation or variation, including punctuation and special characters.

Usage Recommendation:

Ideal for deterministic systems where accuracy is crucial.
Ensures the response is exactly as expected, guaranteeing consistency and precision.
Considers formatting differences, such as line breaks within a block or separation into distinct blocks, indicating messages sent separately.

Variables

In this tab, you can manage the context variables that will be used in the test flow. Add, edit, or remove variables your bot or AI agent may need to start the flow correctly.

Type: context or contact
Name: Variable name (e.g., numbercpf).
Value: The value the variable will have (e.g., 129.452.875-06).
New Variable: Adds a new variable.

Definitions

In this tab, you define the timeout for the test execution.

Response Timeout: Use the slider to set the timeout for each interaction of your test. If the bot's response takes longer than the set time, the interaction will be considered a failure.

Running and Analyzing Tests

After configuring the test:

Click Save.
In the test list, select the desired test.
Click Run Tests.
Monitor the status displayed in the listing.
To view the execution history, click the expansion arrow of the test.
To analyze a specific execution, click the view icon next to the desired version.
In the side panel, check the executed interactions, their statuses, the input message, the expected response, the received response, and the returned JSON.
When an interaction fails, execution is interrupted. Therefore, subsequent interactions may appear as Not Executed.

How to View Token Consumption

When running a unit test that uses AI agents, you can view the token consumption generated in each interaction. This information helps understand how much content was processed by the model, identify agents with higher consumption, and find opportunities to optimize prompts, contexts, and settings.

In the execution history, the Consumption column shows the total tokens used in each interaction. This value represents the sum of the consumption of all calls made by AI agents during that interaction.

While data is being processed, the interface may display the status Calculating Tokens. If it is not possible to obtain the information, the message Data Unavailable will be shown.

To view detailed information:

Run the unit test.
Open the execution history.
Expand an interaction.
Click View Results.
Access the Tokens tab.

The tab will only be displayed when the interaction has calls to language models. At the top of the tab, you can view the consolidated consumption of the interaction:

Input: tokens sent to the model;
Input cached: part of the input content reused from cache;
Output: tokens generated by the model;
Total: sum of tokens processed during the interaction.

Consumption will also be organized by agent used in the flow. For each agent, the following information will be presented:

agent name;
model used;
configured temperature;
maximum token limit;
input tokens;
cached input tokens;
output tokens;
total tokens processed.

When the same agent makes more than one call to the model during the interaction, values will be summed and presented in a consolidated way. If the agent name is not available, the technical identifier of the block will be used.

Regarding data interpretation, it is noted that high consumption may be related to:

very long prompts;
large amount of instructions;
accumulated conversation history;
extensive context or knowledge base;
multiple calls to the model;
very long responses;
token limit set above what is necessary.

Compare different executions of the same test to understand how changes in the prompt, flow, or agent settings impact consumption.

Execution History

Each time a Unit Test is run, a new version is recorded in that test's history. In this context, version means an execution of the test. For example, if a test shows 3 versions, it means it was run 3 times.

To view the history:

On the Unit Tests screen, locate the desired test.
Click the expansion arrow next to the test.
See recent executions displayed below the test.

Each execution shows information such as version, date, and status. Currently, the interface displays the 5 most recent executions of each test.

Viewing Execution Details

To analyze a specific execution, click the view icon next to the desired version. A side panel will open with the details of that execution. In it, you can check:

the execution's interactions;
the status of each interaction;
the input message used;
the expected response;
the received response;
the returned JSON.

To see more information about an interaction, expand the desired item within the panel. This view helps identify at which point the flow diverged from the expected behavior.

Using Unit Tests with Logs and Events

Unit Tests help validate whether an input generated the expected response.

The Logs and Events screen helps investigate more deeply what happened during the flow execution, including messages exchanged, events, context, and technical details.

Both tools can be used together during the validation process. When running a Unit Test, the interactions generated by the test can also be monitored in Logs and Events, allowing investigation of the conversation path within the flow.

A recommended way to use them is:

Open the Logs and Events screen.
In another tab or window, open Unit Tests.
Run the desired unit test.
Return to Logs and Events to monitor the records generated during execution.

Use this combination when the Unit Test shows that the received response was different from expected and you need to understand what happened in the flow up to that result. To learn more, access the article Logs and Events.

Known Limitations

Currently, Unit Tests have some important limitations:

The interface displays only the 5 most recent executions of each test.
Unit Tests continue to work in the context of individual chatbots. Unit tests on routers are not yet supported.
Unit Tests help validate expected behaviors but do not replace manual tests and other validation practices.

Need more help? Explore our content at Blip Academy or Blip Community, watch tutorials on our YouTube channel, or ask your questions in our support channel 😃