{"id":11147,"date":"2026-11-18T15:47:12","date_gmt":"2026-11-18T15:47:12","guid":{"rendered":"https:\/\/www.yellowgrid.co.uk\/blog\/?p=11147"},"modified":"2026-11-18T15:47:13","modified_gmt":"2026-11-18T15:47:13","slug":"reason-for-outage-rfo-16-november-2026","status":"publish","type":"post","link":"https:\/\/www.yellowgrid.co.uk\/blog\/reason-for-outage-rfo-16-november-2026\/","title":{"rendered":"Reason for Outage (RFO): 16 November 2026"},"content":{"rendered":"<p><!--\n  Reason for Outage (RFO): 16 November 2026 \u2014 middle version. Paste into a WordPress \"Custom HTML\" block.\n  Body text is taken word for word from RFO-Nov-2026.docx; the design is styling only, with no added wording.\n  Post title (H1, set by the theme): Reason for Outage (RFO): 16 November 2026\n  The banner at the top is built in HTML\/CSS, so no image upload is needed.\n  The banner's summary wording is not from the document; everything below it is word for word.\n-->\n<link rel=\"stylesheet\" href=\"https:\/\/fonts.googleapis.com\/css2?family=JetBrains+Mono:wght@500&#038;display=swap\">\n<style>\n.ygm{--y:#ffcc00;--ink:#0b1416;--ink2:#12262a;--teal:#3fd0d0;--teal-d:#1d6f6f;--tx:#1d2224;--mu:#5d6462;--ln:#e3e6e3;--soft:#f5f6f3;font-family:inherit;color:var(--tx);font-size:18px;line-height:1.7;max-width:820px;margin:0 auto;counter-reset:sec}\n.ygm *{box-sizing:border-box}\n.ygm p{margin:0 0 1.1em}\n.ygm a{color:var(--teal-d)!important;font-weight:600;text-decoration:underline;text-underline-offset:3px}\n.ygm p.intro{font-size:22px;line-height:1.55;font-weight:500;border-left:4px solid var(--y);padding-left:22px;margin:0 0 1.3em!important}\n.ygm h2{counter-increment:sec;display:flex;align-items:center;gap:16px;font-size:clamp(24px,3vw,30px);line-height:1.25;font-weight:600;letter-spacing:-.01em;color:var(--tx);margin:64px 0 20px!important}\n.ygm h2:before{content:counter(sec,decimal-leading-zero);flex:none;font-family:\"JetBrains Mono\",ui-monospace,monospace;font-size:15px;font-weight:500;width:42px;height:42px;border-radius:10px;background:var(--ink);color:var(--y);display:flex;align-items:center;justify-content:center;letter-spacing:0}\n.ygm h2:first-child{margin-top:0!important}\n\/* Hero banner (from the first version) *\/\n.ygm .hero{background:radial-gradient(ellipse 70% 80% at 95% 110%,rgba(63,208,208,.35),rgba(63,208,208,0) 70%),linear-gradient(160deg,#0a1012,var(--ink));border-radius:22px;padding:52px 48px;margin:0 0 48px!important}\n.ygm .hero .eyebrow{font-family:\"JetBrains Mono\",ui-monospace,monospace;font-size:13px;font-weight:500;letter-spacing:.2em;text-transform:uppercase;color:var(--teal)!important;display:block;margin-bottom:10px}\n.ygm .hero .h-t{font-size:clamp(44px,8vw,76px);font-weight:600;line-height:1;letter-spacing:-.03em;color:#fff!important;margin:0 0 20px}\n.ygm .hero .h-t span{display:inline-block;margin-top:10px;font-size:.5em;letter-spacing:-.01em;color:var(--y)!important}\n.ygm .hero .h-s{font-size:clamp(18px,2.4vw,22px);line-height:1.45;color:#c9ccc7!important;margin:0 0 28px;max-width:640px}\n.ygm .facts{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px;margin:0}\n.ygm .fact{background:rgba(255,255,255,.06);border:1px solid rgba(255,255,255,.12);border-radius:14px;padding:16px 18px}\n.ygm .fact small{display:block;font-family:\"JetBrains Mono\",ui-monospace,monospace;font-size:11px;letter-spacing:.18em;text-transform:uppercase;color:var(--teal)!important;margin-bottom:6px}\n.ygm .fact b{display:block;font-size:17px;font-weight:600;line-height:1.35;color:#fff!important}\n.ygm .fact.ok b{color:var(--y)!important}\n.ygm .hero + h2{margin-top:0!important}\n.ygm .tm{font-family:\"JetBrains Mono\",ui-monospace,monospace;font-size:.86em;font-weight:500;background:var(--ink);color:var(--y);border-radius:6px;padding:2px 8px;white-space:nowrap}\n.ygm .ref{display:inline-flex;align-items:center;gap:12px;border:1px solid var(--ln);background:#fff;border-radius:10px;padding:12px 18px;margin:0 0 1.2em}\n.ygm .ref:before{content:\"\";width:10px;height:10px;border-radius:50%;background:var(--y);flex:none}\n.ygm .note{border-left:4px solid var(--y);background:#fffbe8;padding:20px 24px;border-radius:0 12px 12px 0;margin:0!important;font-weight:500}\n\/* Resolution as a vertical sequence *\/\n.ygm .seq{position:relative;margin:0;padding:0 0 0 34px}\n.ygm .seq:before{content:\"\";position:absolute;left:8px;top:10px;bottom:14px;border-left:2px dashed var(--ln)}\n.ygm .seq p{position:relative;margin:0 0 18px!important}\n.ygm .seq p:last-child{margin-bottom:0!important}\n.ygm .seq p:before{content:\"\";position:absolute;left:-32px;top:8px;width:14px;height:14px;border-radius:50%;background:#fff;border:3px solid var(--ink)}\n.ygm .seq p.last:before{background:var(--y);border-color:var(--ink)}\n\/* Lessons panel *\/\n.ygm .lessons{background:radial-gradient(ellipse 60% 70% at 100% 100%,rgba(63,208,208,.22),rgba(63,208,208,0) 70%),linear-gradient(160deg,#0a1012,var(--ink2));border-radius:18px;padding:36px 38px;margin:64px 0 0!important}\n.ygm .lessons h2{margin:0 0 20px!important;color:#fff!important}\n.ygm .lessons h2:before{background:var(--y);color:var(--ink)}\n.ygm .lessons p{color:#c9ccc7!important}\n.ygm .lessons p.goal{color:#fff!important;font-weight:500;border-top:1px solid rgba(255,255,255,.14);padding-top:18px;margin:22px 0 0!important}\n@media (max-width:720px){\n.ygm{font-size:17px}\n.ygm p.intro{font-size:19px;padding-left:16px}\n.ygm h2{margin:48px 0 16px!important;gap:12px}\n.ygm .hero{padding:34px 24px}\n.ygm h2:before{width:36px;height:36px;font-size:13px}\n.ygm .lessons{padding:26px 22px;margin-top:48px!important}\n.ygm .ref{display:flex}\n}\n<\/style>\n<div class=\"ygm\">\n<div class=\"hero\">\n<span class=\"eyebrow\">RFO \u00b7 Incident report<\/span><\/p>\n<div class=\"h-t\">Reason for Outage<br \/><span>16 November 2026<\/span><\/div>\n<div class=\"h-s\">A Debian kernel update caused some hosted 3CX instances to freeze. Here\u2019s how we found the cause, how we fixed it and what we\u2019re changing as a result.<\/div>\n<div class=\"facts\">\n<div class=\"fact\"><small>Date<\/small><b>16 November 2026<\/b><\/div>\n<div class=\"fact\"><small>Root cause<\/small><b>Linux kernel bug in a Debian update<\/b><\/div>\n<div class=\"fact\"><small>Fix deployed<\/small><b>18:00, 16 November<\/b><\/div>\n<div class=\"fact ok\"><small>Status<\/small><b>Resolved<\/b><\/div>\n<\/div>\n<\/div>\n<h2>Incident Overview<\/h2>\n<p class=\"intro\">On 16 November, our monitoring service began detecting multiple 3CX instances reporting as offline.<\/p>\n<p>The first automated action performed by our monitoring service is to attempt to reboot any instance identified as offline. While this successfully restored the majority of affected instances, a small number failed to recover and were subsequently failed over to alternative infrastructure.<\/p>\n<p>As the incident progressed, it became apparent that the issue was affecting more instances than would typically be expected from an isolated infrastructure incident. Furthermore, our monitoring system was reporting recurring failures affecting the same instances.<\/p>\n<h2>Root Cause<\/h2>\n<p>Following further investigation, we identified that the underlying Debian operating system had automatically applied an update between <span class=\"tm\">06:00<\/span> and <span class=\"tm\">07:00<\/span> on 16 November. This update included a new Linux kernel version which was subsequently identified as containing a bug.<\/p>\n<p>The issue is documented in the following Debian bug report:<\/p>\n<div class=\"ref\"><a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=1154101\" target=\"_blank\" rel=\"noopener\">Debian Bug Report \u2013 1154101<\/a><\/div>\n<p>In summary, the kernel bug causes the operating system to freeze after an unpredictable period of time, ranging from a few seconds to several hours. Once this occurs, the affected instance becomes unresponsive and can no longer be managed normally.<\/p>\n<p class=\"note\">Our investigation established that the previous kernel version was unaffected by this issue.<\/p>\n<h2>Resolution<\/h2>\n<div class=\"seq\">\n<p>Having identified the problematic kernel update, we began developing a patch to revert affected instances to the previous, stable kernel version.<\/p>\n<p>Once developed and tested, this patch was deployed to all instances identified as suffering from the issue.<\/p>\n<p>As applying the patch required a system reboot, and the majority of instances were not experiencing the problem, we scheduled the deployment for <span class=\"tm\">18:00<\/span> on 16 November to minimise disruption to customers.<\/p>\n<p>This planned maintenance was communicated in advance through our <a href=\"https:\/\/status.yellowgrid.co.uk\/\" target=\"_blank\" rel=\"noopener\">Yellowgrid Status Page<\/a>.<\/p>\n<p class=\"last\">Following deployment, all affected instances were restored. We subsequently continued monitoring the platform to ensure stability and confirm that the issue had been resolved.<\/p>\n<\/div>\n<div class=\"lessons\">\n<h2>Lessons Learnt and Planned Improvements<\/h2>\n<p>Although our monitoring, automated recovery and failover processes operated as designed, this incident highlighted some limitations in how these systems respond to operating system and kernel-level failures.<\/p>\n<p>Our automated failover mechanism is primarily designed to protect against infrastructure failures, rather than underlying operating system or application-level bugs. Consequently, while the automated recovery process successfully restored many instances, in some cases this provided only a temporary solution, with the underlying issue causing the instances to fail again.<\/p>\n<p>As a result of this incident, we will be reviewing and improving our monitoring and failover processes to better identify and respond to recurring failures of this nature.<\/p>\n<p>Additionally, with the introduction of 3CX Update 10, an option is now available to allow 3CX to automatically manage operating system security hotfixes.<\/p>\n<p>Following a review of the available options, we have decided to move towards using 3CX-managed security hotfixes. This will allow operating system patch management to be handled directly by 3CX, providing a more controlled and integrated approach to future updates.<\/p>\n<p class=\"goal\">Our objective is to reduce the likelihood of similar incidents occurring and further improve the resilience and reliability of the Yellowgrid platform.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>RFO \u00b7 Incident report Reason for Outage16 November 2026 A Debian kernel update caused some hosted 3CX instances to freeze. Here\u2019s how we found the cause, how we fixed it and what we\u2019re changing as a result. Date16 November 2026 Root causeLinux kernel bug in a Debian update Fix deployed18:00, 16 November StatusResolved Incident Overview [&hellip;]<\/p>\n","protected":false},"author":6,"featured_media":11145,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11147","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-other"],"_links":{"self":[{"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11147","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=11147"}],"version-history":[{"count":4,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11147\/revisions"}],"predecessor-version":[{"id":11151,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11147\/revisions\/11151"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/media\/11145"}],"wp:attachment":[{"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=11147"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=11147"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yellowgrid.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=11147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}