Unit Tests – Blip | Blip Help

Index

Introduction

The Unit Tests functionality allows you to validate the accuracy of your conversational flow responses, whether deterministic or based on an AI agent.

The test works by configuring interaction pairs: you define the input message, which can be text or a public URL, and the expected response, which can be text and/or file validation. During execution, the test sends each interaction and stops upon detecting the first failure.

This process ensures that the system responds correctly and allows for quick adjustments in case of problems, being an essential tool to verify if the expected behaviors are correct before being deployed to production. The functionality facilitates the maintenance and continuous evolution of your Smart Contact.

In addition to creating and executing tests, it is now also possible to monitor the recent execution history of each test and view the details of a specific execution.

With this, builders are able to better analyze the flow validation cycle: execute a test, identify failures, adjust the flow, and execute again, monitoring recent attempts in a more traceable way.

How to access Unit Tests

You can open the Unit Tests as follows:

On the Studio screen:

In the upper right corner of the screen, click the icon .
In the menu that opens, select Unit tests.

The following screen will open:

Managing Unit Tests

The unit test management screen is the starting point for viewing, creating, and running tests for your bot or AI agent.

Create test

Click the Create test button in the upper right corner or, if no tests have been created, click the button centered in the list Create new test. It is possible to fill in all the parameters for the test manually or import a settings file by clicking the button:

Import settings: load a file in Blip format with ready-made tests.

Other resources

Search: Search field to find specific tests by name.
Test list: displays the unit tests created for the bot. In the listing, you can monitor:
- Test: name of the configured test.
- Versions: amount of executions performed for that test. Each execution generates a new version in the history.
- Last update: date and time of the last edit made to the test. This information does not necessarily represent the last execution.
- Status: summarized result of the execution, indicating whether there was success, failure, or another state related to the test.
  - Awaiting execution: The test was created, but has not yet been executed.
  - Success: All test interactions passed successfully (e.g., "Success 18/18").
  - Metric failure: The test was executed, but some interactions failed (e.g., the message "Metric failure 25/57" indicates that 25 interactions failed out of a total of 57 configured interactions).
  - Error starting: The test could not be executed due to an error at the start (e.g., "Error starting 2/3").
  - Interrupted: The test can be interrupted during execution.
- To view recent executions of a test, click the expansion arrow next to the desired item.
Delete tests: removes one or more tests.
Run tests: runs one or more selected tests.

It is also possible to filter tests by status to more quickly locate tests with success, failure, or awaiting execution.

Configuring a Unit Test

When creating or editing a test, you will have access to three configuration tabs: Definitions, Variables, and Interactions.

Interactions

This tab is where you define the sequence of questions and answers to validate the behavior of your bot or AI agent.

Order: The order in which the interactions will be executed. You can reorder interactions by dragging the grid icons.
Description: The text input that will be sent to the bot.
Result: The status of the interaction after the test execution, which can be:
- Awaiting execution: The interaction has not yet been tested.
- Success: The bot's response matched what was expected.
- Error starting: The interaction could not be started.
- Metric failure: The bot's response did not match what was expected.
- Interrupted: The test was interrupted by the user during execution.

The order of the interactions defines the sequence in which the test will be executed. If an interaction fails, the execution is interrupted and the next interactions will not be executed.

Configuring an Interaction:

By clicking on an interaction, you can expand the section to configure it in detail.

Input type: Defines the type of input you are sending.
Input Message: The user input can be simple text or a public URL pointing to a file.
Expected Response:
- Text Blocks: The expected response can be one or more snippets of text.

When dealing with structured formats, such as JSON menus, it is advisable to include the JSON directly, ensuring that the system understands and compares as expected.

File Type: Additionally, the response may require the presence of specific files, such as documents, images, audio, or video. The configuration must specify not only the type but also the expected quantity. For example, if the interaction should return two documents, the configuration must reflect this. The test will fail if the response does not exactly match the number and type of expected files.
Text: The text the bot will receive (e.g., "What are the opening hours?").

Textual Comparison Metric:

Similarity:

The similarity metric evaluates how close a generated response is to the expected response in terms of content and structure. It allows for variations while still considering the response valid.

Usage Recommendation:

Ideal for flexible systems, such as intelligent agents, which can generate responses with some variation.
Define the similarity threshold to establish the acceptable degree of variation. For example, a threshold of 6.5 indicates that the response must have at least 65% similarity with what is expected.

Exact Match:

This metric requires the generated response to be completely identical to the expected response, without any deviation or variation, including punctuation and special characters.

Usage Recommendation:

Ideal for deterministic systems where precision is crucial.
Ensures the response is exactly as expected, ensuring consistency and accuracy.
It considers formatting differences, such as line breaks within a block or separation into distinct blocks, indicating messages sent separately.

Variables

In this tab, you can manage the context variables that will be used in the test flow. Add, edit, or remove the variables that your bot or AI agent may need to start the flow correctly.

Type: context or contact
Name: Variable name (e.g., numbercpf).
Value: Value the variable will have (e.g., 129.452.875-06).
New variable: Adds a new variable.

Definitions

In this tab, you define the timeout for the test to run.

Response timeout: Use the slider to set the time limit for each interaction of your test. If the bot's response takes longer than the stipulated time, the interaction will be considered a failure.

Running and Analyzing Tests

After configuring the test:

Click Save.
In the test list, select the desired test.
Click Run tests.
Monitor the status displayed in the listing.
To view the execution history, click the test expansion arrow.
To analyze a specific execution, click the view icon next to the desired version.
In the side panel, check the executed interactions, their statuses, the input message, the expected response, the received response, and the returned JSON.

When an interaction fails, the execution is interrupted. Therefore, the following interactions may appear as Not executed.

Execution history

Each time a Unit Test is executed, a new version is recorded in the history of that test.

In this context, version means an execution of the test. For example: if a test shows 3 versions, this means it was executed 3 times.

To view the history:

On the Unit Tests screen, locate the desired test.
Click the expansion arrow next to the test.
See the recent executions displayed below the test.

Each execution presents information such as version, date, and status.

At the moment, the interface displays the 5 most recent executions of each test.

Viewing the details of an execution

To analyze a specific execution, click the view icon next to the desired version.

A side panel will open with the details of that execution. In it, you can check:

the interactions of the execution;
the status of each interaction;
the input message used;
the expected response;
the received response;
the returned JSON.

To see more information about an interaction, expand the desired item inside the panel. This view helps to identify at which point the flow diverged from the expected behavior.

Using Unit Tests with Logs and Events

Unit Tests help validate whether an input generated the expected response.

On the other hand, the Logs and Events screen helps to investigate in more depth what happened during the flow execution, including trafficked messages, events, context, and technical details.

Both tools can be used together during the validation process. When running a Unit Test, the interactions generated by the test can also be monitored in Logs and Events, allowing you to investigate the path taken by the conversation in the flow. A recommended way to use them is:

Open the Logs and Events screen.
In another tab or window, open Unit Tests.
Run the desired unit test.
Go back to Logs and Events to monitor the logs generated during the execution.

Use this combination when the Unit Test shows that the received response was different from the expected one and you need to understand what happened in the flow up to that result. To learn more, access the Logs and Events article.

Known limitations

Currently, Unit Tests have some important limitations:
The interface displays only the 5 most recent executions of each test.
Token consumption is not yet displayed in the history or in the execution details, but it can be monitored through the Logs and Events screen.
Unit Tests continue to work in the context of individual bots. Unit tests in routers are not yet supported.
Unit Tests help validate expected behaviors, but they do not replace manual tests and other validation practices.

Need more help? Explore our content at Blip Academy or Blip Community, watch tutorials on our YouTube channel, or clear your doubts in our service channel 😃