Failover licenses

Symptoms

I can see messages like the following in my log files.

[VoiceObjects.LicenseManager ] [ ] : Failed to retrieve a license from ServerManager '192.168.1.2:7800' within 6s 8ms seconds. Checking failover license availability. [VoiceObjects.LicenseManager] [] [] [] [] [] []: LicenseManager not available, returning failover license (71h 59m 30s 30ms of the grace period remaining, '198' sessions left).

• Where do they come from?
• Does this affect my productive system?
• Can I avoid getting these messages?

Resolution

These messages show a default failover behavior of VoiceObjects. They indicate that the server instance processing a call was unable to communicate with the cluster master and therefore could not obtain a "normal" license for the call session.

The process behind it is as follows: License information is loaded from the Metadata Repository database to all VoiceObjects instances' caches at startup, but only the respective master instance is responsible for the administration of licenses across the entire VoiceObjects cluster at any point in time. Therefore, when a new call comes in, each instance will contact the master instance for a license. By default, the instance will wait for up to 6 seconds for the license. When this request fails (times out), as shown in the first part of the log message above, the instance itself enters the grace period mode and for 72 hours issues up to 200 simultaneous failover licenses (second part of log message). The number of sessions left will decrease until the instance either gets the next regular license from the master or another call ends so that this failover license can be used.The time counter is reset every time a regular license could be retrieved from the master.

This failover ability of VoiceObjects was created to process calls even when there are e.g. network problems preventing the communication between server instances, or the master server instance should fail (e.g. because a machine goes down). Failover licenses are also used when the master instance cannot answer the license request within 6 seconds. This can happen when either the machine or the master instance itself are busy - for example when the system is under high CPU load.

The messages may appear under normal working conditions of a cluster from time to time. They are not a clear sign for a failure, but can be indicators for unwanted behavior in a cluster:

• In case the messages appear only seldomly and without regular pattern they can be ignored.
• If the failover messages always appear when there is a high call volume, there may be an issue with the network capacity or the machine hosting VoiceObjects may be running at its limits (either CPU or RAM) under certain conditions.
• If the time counter for the remaining failover licenses time drops below 60 hours, this instance was not able to reach the master instance for more than 12 hours.If there is only one instance affected, this instance should be restarted. If there is more than one instance affected the master instance should be restarted.

If SNMP is enabled, failover messages will be sent as SNMP trap.

It might be required to configure the timeout of 6 seconds, e.g. to avoid the collision of different timeouts in the system.

To configure the timeout, change the setting

<component id="ServerManagerLicenseHandler"

   <licenseTimeout>6000</licenseTimeout>

...

in the file/WEB-INF/config/system-components.xml to the desired value in milliseconds. The default is 6000 ms = 6s. Note that changing this setting to a lower value might lead to more messages concerning failover licenses. Setting this to a quite high value might lead to a too long initial silence before the call starts.