SIPssert and the journey of testing OpenSIPS

Our previous post presented SIPssert – an OpenSIPS Testing framework that does conformance testing for VoIP platforms (and not only). In this post we will describe how the OpenSIPS project has already started to benefit from the SIPssert capabilities in order to detect and prevent bugs as well as to avoid regressions and ensure that new features are compliant with their requirements. To this end, we have developed a set of conformity tests that are being run and checked using the SIPssert framework.

Not only these tests are open-source available on GitHub, but we strongly encourage anyone that is using OpenSIPS to browse, check and extend them in order to ensure that our beloved project is behaving correctly in all the possible scenarios.

But first, let’s see what we have done until now and how or whether these tests were useful so far. Note that the initial tests have been developed for the OpenSIPS 3.3 version and then ported to OpenSIPS 3.4 with slight changes, that we will read about later on.

Initial Testing

We clearly had a late start on this – by the time we started writing the tests, OpenSIPS was already a mature project with 23+ years of development behind, so a full/complete testing would definitely not be possible. However, we focused our tests on key components that are known to be widely used, such as: authentication, registration, record-route mechanisms, dialog support and topology hiding.

Fortunately, all these components/modules were stable enough, and thus we have not identified any behavioral bugs. We did, however, found a special scenario when using topology hiding without dialog but with username passing in contact which was not passing it properly for in-dialog requests (defect #1). But overall, our tests did not identify any major flows in the scenarios we have tested.

B2B Testing

B2B is a very powerful and flexible tool in OpenSIPS’ portfolio, therefore it deserves its own test suite. Hence, we started with a set of 20 different B2B setups, testing various combinations of scenarios, triggers and functions flags. All these were behaving correctly and tests were successfully passed.

UAC Auth Testing

The necessity of these tests came from a different direction – there were several reports on GitHub that the uac_auth module was not sending the correct CSeq number for in-dialog messages challenged, or that the wrong algorithm was being used for qop parameters. Therefore a set 27 tests were performed to test the behavior of the uac_auth() function, and actually the two bugs have been confirmed, and in addition a limitation was discovered:

• CSeq number was not properly bumped for in-dialog requests (defect #2, detected in this test)
• when the challenging server was offering both auth and auth-int qop, the module was doing auth-int, but was advertising auth, hence breaking the authentication (defect #3, detected in this test)
• authentication for in-dialog requests without dialog support is not working correctly, as the CSeq change cannot be persistent throughout the call (defect #4)

Porting tests to 3.4

In order to determine whether our 3.4/master branch is behaving correctly after all the enhancements and new features we have added, we decided to run the tests for this branch as well. Although most of the tests have successfully passed the checks, to our surprise, almost all of the B2B tests were failing. Troubleshooting this revealed a crash in B2B while handling a bridging (defect #5).

Once the crash was fixed and tests were re-run, there was still a large amount of tests that were still failing. Digging into this we identified a regression related to the way the B2B module was expecting to build the contact header for new entities (defect #6). This has been quickly fixed to ensure backwards compatibility.

A last step towards completing the porting was to re-run all the tests once again. At this point, there were still a number of tasks that were failing. Fortunately, we quickly realized that they were actually not bugs, but rather the B2B behavior has been changed (due to latest B2B improvements), therefore the tests had to be adapted to the new SIP flow. Moreover, in order to ensure correctness of the new flows, we’ve added a set of 5 new tests for the main branch.

Conclusions

The more testing we do, the better it is for the project – therefore writing tests, for both old functionality, as well as for new ones, should be a continuous task.

Tests can be developed with different purposes – to prove something works OK (i.e. our initial tests), or to prove the contrary (UAC auth tests) – but all of them bring value to the project. And you can help making OpenSIPS better as well by writing tests yourself and open a PR request for them to get merged in the main repository.

By the time of this blog post, we have developed 81 tests that revealed 6 defects: 2 broken flows (topology hiding without dialog and uac-auth cseq), 1 limitation (uac-auth), 1 crash (in B2B master branch) and 2 regressions (B2B contact build and uac-auth algorithm choosing). All these have been fixed, or are in the process of fixing. But this does not mean that the tests are useless now – by contrary, the more they stay, the higher chance it is to detect possible regressions in the future, so their value increases in time.

And if you want to find out more about SIPssert and conformity tests, make sure you don’t miss our  annual OpenSIPS Summit, the 2023 edition taking place in Houston, US, 23rd – 26th May!!