Stress-Testing OpenSIPS 3.4 LTS

Quite a few years have passed since the last set of stress tests were performed on OpenSIPS, with the performance numbers achieved at that time becoming less and less relevant with each new major release. As the number of features has grown, so has the complexity of the codebase — making the alpha version of OpenSIPS 3.4 an ideal candidate for a revisit of those infamous stress tests!

What Was Tested

Similar to the previous set of tests, we continued to run through the codebases which receive the most usage in the open world:

• the UDP transport layer
• the transaction ™ module
• the dialog module
• SIP authentication
• accounting
• opportunities for caching (since caching always leads to some nice before/after results!)

Additionally, since the OpenSIPS has much richer use-cases nowadays, we included some new codebases as part of the load tests, which have slowly become de-facto use-cases for OpenSIPS (often single-purpose!):

• topology hiding
• the flatstore (db_flatstore) module, for writing data (typically CDRs) directly on disk
• the B2B modules, with some key scenarios
• the TCP transport layer, including the recent parallel reading enhancements

Profiling OpenSIPS

Stress testing often goes hand-in-hand with profiling. The two processes form a feedback loop as follows:

1. load-test the application until the performance limit is found
2. redo the test as part of a profiling run at peak performance level
3. analyze the resource usage distribution for potential bottlenecks
4. if possible: optimize the code and go to Step 1.

For our OpenSIPS profiling purposes, we found out that the Gperftools project (originally, Google Performance Tools) worked quite well and produced insightful overviews as to how OpenSIPS was spending the available resources.

The library was fully integrated with OpenSIPS on master branch, as part of the new -DPROFILING flag under Makefile.conf. Just install libgoogle-perftools-dev and libgoogle-perftools4, enable the flag & recompile and you should be already performing profiling on your next OpenSIPS run!

Short Summary

Overall, the tests were a success, as they initially confirmed some of our fears, then they helped us uncover several bugs and finally, they helped us identify bottlenecks and optimize performance. And although the full test results and configuration files are available in a dedicated wiki page, here are some quick ideas you can get out of them:

• nowadays, OpenSIPS 3.4 will perform 13K CPS by using up 77% avg. CPU across 4 workers on a standard build, compared to OpenSIPS 1.6, which apparently was pushed to 13K CPS using only 19% CPU at the time. This is explicable, as there have been numerous features added across the years which may reduce throughput, to name a few:
  • major overhaul of the transport layer, to accomodate more protocols (WS, WSS, SMPP, to name a few). While it is an amazing and well-structured addition from a maintainability PoV, it does come with some amount of overhead compared to a naked UDP listener.
  • various additional file descriptors added to each UDP reactor (e.g. new fds for Internal-Process Communication)
  • file descriptor timeouts, enabling features such as async statement timeouts
  • various transaction module enhancements (reply AVPs, additional locking, etc.)
• the B2B modules directly benefited from stress-testing, with several crashes or erroneous logs / internal states specific to high-volume SIP traffic being addressed (1, 2, 3)
• the TCP engine has been confirmed to perform on-par with the UDP engine, when working with both a small amount of high-throughput TCP connections and with a large amount of low-throughput connections (the difference is relatively small). All in all, there is not that much of a penalty after switching to TCP.
  • still, the stress-tests have uncovered some issues with the new parallel reading feature specific to high loads, which we expect to address ahead of the 3.4 beta release

Again, full conclusions, numbers, graphs and configuration files for both OpenSIPS and sipp are available in the dedicated wiki page for the OpenSIPS 3.4 stress tests. And if you find this topic fascinating, make sure to attend my talk at the OpenSIPS Summit 2023 Houston (in-person or on YouTube), in just a few weeks!