Reason for Outage (RFO): 16 November 2026
16 November 2026
Incident Overview
On 16 November, our monitoring service began detecting multiple 3CX instances reporting as offline.
The first automated action performed by our monitoring service is to attempt to reboot any instance identified as offline. While this successfully restored the majority of affected instances, a small number failed to recover and were subsequently failed over to alternative infrastructure.
As the incident progressed, it became apparent that the issue was affecting more instances than would typically be expected from an isolated infrastructure incident. Furthermore, our monitoring system was reporting recurring failures affecting the same instances.
Root Cause
Following further investigation, we identified that the underlying Debian operating system had automatically applied an update between 06:00 and 07:00 on 16 November. This update included a new Linux kernel version which was subsequently identified as containing a bug.
The issue is documented in the following Debian bug report:
In summary, the kernel bug causes the operating system to freeze after an unpredictable period of time, ranging from a few seconds to several hours. Once this occurs, the affected instance becomes unresponsive and can no longer be managed normally.
Our investigation established that the previous kernel version was unaffected by this issue.
Resolution
Having identified the problematic kernel update, we began developing a patch to revert affected instances to the previous, stable kernel version.
Once developed and tested, this patch was deployed to all instances identified as suffering from the issue.
As applying the patch required a system reboot, and the majority of instances were not experiencing the problem, we scheduled the deployment for 18:00 on 16 November to minimise disruption to customers.
This planned maintenance was communicated in advance through our Yellowgrid Status Page.
Following deployment, all affected instances were restored. We subsequently continued monitoring the platform to ensure stability and confirm that the issue had been resolved.
Lessons Learnt and Planned Improvements
Although our monitoring, automated recovery and failover processes operated as designed, this incident highlighted some limitations in how these systems respond to operating system and kernel-level failures.
Our automated failover mechanism is primarily designed to protect against infrastructure failures, rather than underlying operating system or application-level bugs. Consequently, while the automated recovery process successfully restored many instances, in some cases this provided only a temporary solution, with the underlying issue causing the instances to fail again.
As a result of this incident, we will be reviewing and improving our monitoring and failover processes to better identify and respond to recurring failures of this nature.
Additionally, with the introduction of 3CX Update 10, an option is now available to allow 3CX to automatically manage operating system security hotfixes.
Following a review of the available options, we have decided to move towards using 3CX-managed security hotfixes. This will allow operating system patch management to be handled directly by 3CX, providing a more controlled and integrated approach to future updates.
Our objective is to reduce the likelihood of similar incidents occurring and further improve the resilience and reliability of the Yellowgrid platform.