Containing Memory-Related Issues in an Easy Way with OpenSIPS 3.0

OpenSIPS 3.0 includes major changes to the way in which the available memory allocators can be embedded into a build, as well as to the manner in which they can be selected at runtime. The end goal of this rework is to simplify and speed up many existing workflows (debugging, fine tuning, development) by avoiding unnecessary full rebuilds from source whenever possible.

Memory Sensitive Scenarios

Using the same memory allocator combination in all deployment scenarios is by no means optimal. Let’s take a look at some possible use cases along with requirements and solutions for each of them.

Debugging in Production

We are in production — calls are flowing in and out, millions of CDRs are being produced daily, and we want our uptime to stay as close to 100% as possible. Yet, every few days, some strange memory corruption appears to take place — it always crashes the server, which quickly restarts. Although calls survive thanks to the DB and the service is barely affected, we would want to address the issue. The corefile indicates an irrelevant stack trace, which is useless for the developers as it is just a side-effect, not actually pointing to the actual source of the corruption. What to do next?

The answer is to run with a memory allocator which has additional sanity checks and corruption detection logic built into it. We will happily trade off a bit of performance (due to the extra checks) in return for the ability of being able to catch the bug. The only current allocator which has this property is Q_MALLOC.

Detecting Memory Leaks in Production

Let’s assume a similar scenario as above except that, this time, the bug consists of a slowly accumulating memory leak, which puts us in the position of having to restart the service once every few days in order for the leaked memory to be restored. Is Q_MALLOC still the answer? Surprisingly, the answer is: no!

Although equipped with sanity checks, Q_MALLOC alone is not capable of detecting memory leaks. The good news is that all allocators have optional “memory leak debugging” support. For example, if we were running F_MALLOC, we would now enable F_MALLOC_DBG in order to have it also keep track of the source of all allocations, so we can narrow down the leak. No unnecessary sanity checks in production, just the exact added support we need. The same goes for HP_MALLOC, Q_MALLOC and their memory leak debugging flavors: HP_MALLOC_DBG, Q_MALLOC_DBG.

Once the memory leak troubleshooting support is enabled, we only have to instruct the concerned process (using its PID) to print out its usage map to the opensips log file (e.g. /var/log/opensips.log) with:

opensips-cli -x mi mem_pkg_dump 11774

Tuning for Performance

We want to build a high-end switching service capable of handling calls-per-second amounts in the “tens of thousands” range and we’ve acquired appropriate hardware for this — powerful 16-core servers, with hyper-threading. However, preliminary tests show that the publicly available build seems to cap out at ~2000 CPS even on such a server. Can we do anything more?

Most likely, the bottleneck is due to the global shared memory lock. The more cores you throw at OpenSIPS, the higher the contention on that lock, hence a reduced performance gain as opposed to what may be expected. The answer is to use an allocator tailored for multi-core environments: HP_MALLOC.

OpenSIPS Development

We’re developing a new OpenSIPS module, and we want to trade off as much performance as possible in return for the highest degree of assistance / debuggability. What are the key tweaks to make?

Since we don’t care about performance during development, we should immediately switch to using Q_MALLOC_DBG. It has both the extra checks and the memory leak troubleshooting support. Ideal for our purpose.

As additional helpers, we could also dramatically speed up compilation by enabling the CC_O0 compile flag under Makefile.conf, as well as benefit from extra debug logging via the EXTRA_DEBUG flag.

How Does It All Fit With 3.0?

Until now, users with package-based OpenSIPS installs did not have the luxury of a selectable memory allocator, as this was decided through a series of flags, during compilation — this is not the case anymore.

Starting with 3.0, not only are we able to assign one of multiple allocators at startup (not just at compile time!), but it is also possible to individually assign an allocator for shared memory management and another one for private memory management! This allows users to use the optimal allocator for each type of memory, as well as to enable a debugging allocator only for the memory segment which may be reporting some issues.

As of now, we have added three new command line options:

• -a <allocator>, the global allocator to be used (shared and packaged memory)
• -k <pkg_allocator>, the allocator to be used for packaged memory management (overrides the -a option)
• -s <shm_allocator>, the allocator to be used for shared memory management (overrides the -a option)

A Glimpse Into OpenSIPS Allocators

Since we talked about using the allocators so much, I will end the post with a detailed rundown through each of them — we’ll look at their intended purpose, strengths and weaknesses.

F_MALLOC (fast memory allocator)

The default allocator shipped with the public OpenSIPS builds. It is designed to be lightweight and fast. The metadata occupied by each allocation is kept down to a bare minimum. Recommended for production use.

• overhead for each malloc(): 24 bytes
• sanity checks: no
• memory leak troubleshooting support: yes (using its F_MALLOC_DBG flavor)

Q_MALLOC (quality assurance allocator)

This memory allocator is designed with a “quality assurance” philosophy in mind. When using it, you are happily giving away a bit of memory and some CPU cycles in return for additional sanity checks and buffer overflow/underflow checks. These checks are sometimes essential, as they may detect memory corruptions when they ensue, rather than letting them undetected, causing the system to crash later on, in some unfortunate, random place which would be nearly impossible to debug.

• overhead for each malloc(): 96 bytes
• sanity checks: yes
• memory leak troubleshooting support: yes (using its Q_MALLOC_DBG flavor)

HP_MALLOC (high performance allocator)

This memory allocator was specially designed in order to minimize contention on the global shared memory lock on powerful, multi-core servers (16+ cores). It retains the same ideas behind F_MALLOC (lightweight & fast), while still offering optional support for troubleshooting memory leaks on such high-end systems.

• overhead for each malloc(): 24 bytes
• sanity checks: no
• memory leak troubleshooting support: yes (using its HP_MALLOC_DBG flavor)

And that wraps up this post on OpenSIPS 3.0! To learn more about what OpenSIPS 3.0 can do for you, as well as discuss all things VoIP, you can get a great return on your investment by joining us in the Amsterdam 2019 OpenSIPS Summit and Training!