SIPssert – an OpenSIPS Testing Framework

A SIP system/platform consists of multiple components, that interact in various ways to provide complex features (such as call routing, call pickup, registration and authentication, call parking, conferencing, IVRs and others). Each of the component that takes part of the system consists of both a software and its configuration, therefore in order to verify the correctness of the overall system, you need to:

• check the conformity of the used software
• validate the correctness of the configurations
• be sure that all components inter-operate as expected

This process is called conformance testing, and it is used to ensure that an entire system operates correctly as a whole, according to the defined specifications.

SIPssert is a conformance testing framework that provides various instruments for building and running conformity tests for VoIP-based platforms. It provides a very simple but generic way of provisioning tests for various types of components and scenarios within a system, orchestrate their execution and provide easy means of troubleshooting in case of failure. Its goal is to ensure that OpenSIPS, as well as the other components within a VoIP system perform correctly as a whole by running a set of tests (consisting of all its steps: initialization/provisioning, running the components, testing and finally cleanup) and validating they behave properly.

It relies on Docker to run different components (OpenSIPS, database, SIP UAs, provisioning tools, etc) in lightweight containers, manage their execution (start/stop) and checking their status at the end of scenarios, verifying that all components behaved correctly. You can read more about its internals in the project’s documentation.

Although it was initially designed and built for testing OpenSIPS VoIP-based platforms, it can be easily extended and used for testing any other type of system.

Features

A non-exhaustive list of the features the SIPssert framework provides are:

• execute multiple sets of tests consisting of different scenarios
• orchestrate the execution of a scenario and determine the status of the tests
• provide a simple way of defining the components that are part of a scenario (through YAML files) and their interaction
• dependencies/timing management between different tasks in order to ensure the correct order of execution
• gather troubleshooting information (logs, pcaps, statuses)
• run tests sets with different networking layouts
• auto-mounting configuration directories

Usage modes

On a higher level, SIPssert can perform conformance tests for an OpenSIPS platform in two different modes:

• running OpenSIPS as an internal entity of the test, providing the configuration script (either different or the same) for each scenario executed
• or having OpenSIPS as an external entity, part of a remote platform that is seen as a black-box system

The first mode can be useful in the development process of the OpenSIPS code, ensuring that new features or changes behave correctly and (most important) do not generate regressions. For every existing or new feature or modification, a set of scenarios with different configurations and tuning should be written, to validate that OpenSIPS has the desired behavior. This is what we have done in the SIPssert OpenSIPS tests project.

The second mode, testing the platform as a black-box, is suitable for running pre-production tests, in order to ensure that new features behave correctly and do not have any regressions. This mode requires a mechanism to provision the platform before (and during) the execution of the tests, and ensuring that after the execution the platform remains in the same state as before. We will provide more information on this mode later, in a different post :).

Running tests

The goal of running a test is to verify that the system you are testing behaves as expected under a certain SIP scenario. But before actually being able to run it, a set of prerequisites need to be satisfied. Thus, running a tests consists of the following steps:

• initialization: at this step, you need to prepare the system to run the scenario; this means creating tests user, grant permissions, and any other required provisioning (through API calls, or database updates) of the system
• execution: the next step is starting the components of the system (i.e. starting OpenSIPS, SIP UAs, etc)
• testing: in this step, you can run perform several checks related to the setup, by querying different components (i.e. use OpenSIPS CLI to check OpenSIPS cache, or querying the MySQL database);
• cleanup: the final step, after all the previous steps have completed, you may need to perform some cleanup (i.e. remove created users during initialization, revoke permissions, remove generated data such as CDRs, etc)

Note that the execution and testing steps might easily overlap, as you may start a first set of components, test they run correctly, then start another set, verify again, and so forth, and so on.

For example, consider running a local test that verifies a user can successfully authenticate and register to an OpenSIPS instance. This scenario would presume the following steps:

• initialization: start a MySQL database and provision the credentials in the database, either using the mysql client, either using the opensips-cli tool
• execution: start OpenSIPS with the proper configuration that accepts registrations and authentication; as soon as it is up and running, you may start a SIPP client that sends a REGISTER message, gets a digest challenge, then registers again with the provided credentials
• testing: you may want to use opensips-cli and/or mysql client to verify the correct contacts, with the expected expiration time, etc.
• cleanup: finally, after the test completes, you may need to remove the created users and clear the database

A more complex example, may be testing a call parking scenario on an external (black-box) setup. In this case, the steps can be:

• initialization: provision the testing users through an API call and grant them permissions to do call parking
• execution: start a SIP UA server and client, establishing a call between them; then, park the call on a parking slot, wait a few moments, then start another SIP UA client that retrieves the call
• testing: at any point of the execution, you may verify (through API calls) that the call is properly established, that the parking slot is being occupied and at the end of the call, you may also verify the CDRs (either through DB or using API calls)
• cleanup: clear the call parking permissions and remove the testing users

Execution of the SIPssert framework consists of running multiple tests sets, each containing multiple scenarios. In our SIPssert OpenSIPS tests project, we’ve organized each feature (auth, dialog, b2b) in a tests set. A scenario is the equivalent of a test and consists of multiple tasks that describe the components that should be executed. Going back to our example, a scenario consists of a certain configuration of that feature (i.e. b2b topology hiding, or b2b prepaid scenario). Each scenario is independent, and can not (and should not) affect the execution of a different scenario.

Scenarios

Scenarios/tests are defined in YAML files, and they consist of all the tasks/components that need to be executed in order to validate the test. For example, to test a basic call from a UAC to an UAS, we need (at least) three components: a client UA, a server UA, and of course OpenSIPS in the middle. An optional MySQL database might be required as well, depending on what we need to test.

Scenarios should also provide the dependencies between the executed tasks/containers: For example, for running the basic call, the MySQL database should first be started; then OpenSIPS should be started, but not before the database is provisioned (otherwise it will fail to startup); then the UAS should be started, and last, the UAC. All these requirements, usually with explicit delays between them, are provided in the scenarios.

Init/cleanup tasks and other high-level settings, such as scenario networking layout, or execution timeout, can be provided as well. Check out the scenarios documentation for more information.

Tasks

Tasks are the SIPssert abstraction of the components needed to run a scenario – they describe what is the component to run and how. In terms of implementation, tasks describe the execution of the underlying Docker container: what are the startup parameters, networking devices available, needed configuration files, etc. Tasks can be anything that can be run as a docker container:

• An OpenSIPS server
• A MySQL server
• A bash script that runs OpenSIPS CLI commands, or MySQL queries to provision or check the status of the platform
• A SIPP instance acting either as a UAC and/or UAS
• An API call to a remote endpoint

They require a docker image that has to be executed, and a set of parameters to run. In order to facilitate and simplify the effort for developing tests, we have already created a set of images that you can use:

• OpenSIPS SIPssert image – a Docker image that contains the packages currently needed by the SIPssert OpenSIPS tests
• OpenSIPS CLI – a Docker image that can run the OpenSIPS CLI tool, either as a command, either as a python module
• MySQL client – a Docker image that can execute MySQL commands to a remote MySQL server

But you can basically use any image you want – for example, we are using images that are already available for MySQL servers or running SIPP containers. As long as it provides the component you need, it will simply work.

SIPssert simplifies the specification and configuration of the underlying docker images by providing an extra layer of abstraction in the scenario. For example, instead of providing the plethora number of parameters SIPP needs to run, we have “masked” them under a certain type of task, called uac-sipp. All you need to do is to provide the proxy URI, and it will take care of passing all the underlying parameters to the container.

You can read more about available tasks, as well as their behavior in the task documentation page. You can also create new tasks you need yourself and push them into the project (as pull requests)!

Testing

The framework itself has only information about the components it runs and the interaction between them, but is not aware of the underlying logic of each component, hence it cannot decide by itself whether a task runs correctly or not – it is up to the task itself (as he is aware of its logic) to determine whether it executed correctly or not, and return the feedback in the status code. For example, a SIPP client task knows exactly what is the messages it needs to receive in order to successfully complete the call, so he should be returning 0 if all (mandatory) messages have been received, or a failure otherwise – SIPssert just inspects the return code of the container. If all tasks return success, the scenario is considered successful, otherwise failed.

Troubleshooting

SIPssert facilitates troubleshooting of failed scenarios as well, by gathering all the component’s logs and exit codes. It also starts a pcap trace for every scenario. Having all these available, you can always go back to a certain execution in time and investigate the PCAP and logs to find out why the call has failed.

Automation

You can easily integrate the SIPssert framework in your CI/CD pipeline to periodically run your tests. You can find an example of how to do this in GitHub Actions in our SIPssert OpenSIPS tests project.

Future work

Testing is never complete – even if there are no new features, there are always some bugs hidden out there. So there is always room for writing new tests and/or improving the existing ones. That is why we are planning to continuously maintain and add new features to our existing SIPssert OpenSIPS tests project. You can do that too – if you find a bug in OpenSIPS, providing a SIPssert scenario to prove it would definitely help us understand the issue and easier replicate it.

But other features should be added to the SIPssert framework as well – just give it a try and let us now what is missing – for any ideas, feel free to open a feature request for it.

Conclusions

SIPssert framework can be used to perform conformity checks to your platform, simplifying and comforting roll-out of new features – make sure you give it a try on your own platform.

We can discuss more about the SIPssert project during our anual OpenSIPS Summit 2023, in Houston, US, 23rd – 26th May, so make sure you don’t miss it!