Big scale SIP forking

SIP forking is an important feature. It helps with parallel ringing, with class V features (like Hunt Groups), but also with failover (like multi-gateway/carrier failover).

SIP parallel and serial forking means creating more and more branches (destinations) for a transaction, as the following real life scenarios demonstrate:

• Multi-level hunt groups in class V VPBX systems – A hunt group targets 6-9 users, each user with multiple devices registered, eventually with Push -Notification enabled – such a Hunt Group may easily translate into like 25 branches. If such a Hunt Group falls over to another similar group, we are looking like to ~50 branches.
• Cascading gateway failover in trunking systems – For a prefix/destination, you may use multiple carriers, each carrier with multiple gateways. This can translate into set of 40 to 60 gateways to be tried. In combination with channel limitations, quality constraints or failover, it may happen quite often to iterate thru all the gateways, having INVITEs with 20-40 branches

Problem 1 – current limited branching capability

With a 25 years old design, OpenSIPS 3.6 is constraint to a maximum of 31 branches per transaction, which is far from enough in the mentioned case.

This limitation comes from the usage of an internal branch bitmask over an uint32 data.

In order to draft the solution, we need to take into consideration the following facts:

• this forking is specific to INVITE transactions only
• the number of forks may heavily vary with the scenario
• the size of a single branch is significant

Solution Step 1 – handling large number of branches

The OpenSIPS 4.0 bring a complete rework of the branch management code, including the mentioned branch bitmask. This was migrated to an array data structure, removing any kind of limitation to future increase of maximum number of branches.

Nevertheless, there is a second problem with the old 3.6 approach – the branches are kept within the transaction as a built-in static array with the size of the max supported branches. And this may negatively impact the memory usage.

Problem 2 – the memory usage

Let’s do a quick math. One branch structure has ~0.5KB . With the current 12 branches setting, a transaction has 6920 bytes. With 64 built-in branches you get 34320 bytes and with 256 branches almost 134KB. Per transaction.

If we consider a 5000 calls per second system, it means 10.000 transactions (minimum) per second. Each transaction may leave for 10 seconds (ringing time summed with the time-to-delete). So, at any time, we end up with 100.000 transactions in memory.

And if one transaction has 134Kb, then 100000 will require 13.4 GB of memory – this is a lot, considering that most of the memory is wasted. Why?

• because only INVITEs do forking, so multi-branches. BYE requests have one branch only.
• not all INVITE requests may require such a high level of forking/branches.

So, the solution here is a dynamic branch allocation mechanism.

Solution Step 2 – dynamic branch allocation

So, instead of allocating all the time the maximum number of branches into the transaction, OpenSIPS 4.0 is doing dynamic progressive allocation for the branches.

In order to keep the array-like approach for the branches, we opted for allocating 4-branches size chunks whenever is needed more. The branch’s array cannot be reallocated as the branches get linked in different timer lists – so it is highly sensitive to changing the memory addresses.

Conclusions

OpenSIPS 4.0 finally gets rid of the limit on number of branches per transaction, but without any memory or processing penalty. On contrary, memory footprint has decreased.

The default code comes with a setting of 256 maximum number of branches, but even this can be easily raised by editing a single definition in the code.