The Day the Kernel Panicked: How a Tiny Update Broke the Internet (and Everything Else)

Imagine a world where airports grind to a standstill, Starbucks baristas face chaotic cash-only lines, 911 services go dark, bets hang in limbo, border crossings slam shut, Swedish soccer stops playing ball (literally!), and supermarkets from Manila to Tokyo refuse credit cards. Sounds like a futuristic dystopian thriller? Nope, it happened last Friday.

When many woke up on Friday morning, they thought Netflix’s Leave the World Behind came true. Instead, CrowdStrike — a U.S.-based cybersecurity company specifically created to stop such a scenario — stumbled, sending Windows servers and global industries into a tailspin.

So, where did it all go wrong?

Before we play the blame game, let’s first decode the outage. And it’s not one outage but a tale of two.

Thursday night, Microsoft’s Azure cloud platform experienced an outage. It affected some U.S. airports (but not air traffic control). The tech titan said the “underlying issue is fixed,” but “residual impact is continuing to affect some Microsoft 365 apps and services.”

Then, early Friday, CrowdStrike released a flawed kernel driver update that made Windows servers sputter and stop. Mac and Linux systems, and many Windows PC systems, escaped unscathed. The flawed update was too much for the airline's systems, which were still recovering. Then, the outage spilled over borders and across industries.

Microsoft, responding to Wired Magazine, said the two incidents were unrelated (of course!). Yes, it’s too early to correlate the incidents, even though the timing does raise eyebrows. But it is clear that CrowdStrike’s now infamous update resulted from the company’s Falcon monitoring product that refreshes endpoints against malware and suspicious activities.

Still, Microsoft had to vet and digitally sign the code. So, it’s incredible that given the deep access to Windows systems, a single kernel flawed update brought global industries to its knees.

“From a business perspective, it is astonishing that one company was allowed to cause such a massive global IT outage. From a technical standpoint, this was possible because a kernel driver was used, which carries a risk of causing a major issue, such as the infamous BSOD (Blue Screen Of Death) seen ‘everywhere’ yesterday,” said Michael Gazeley, managing director at Hong Kong-headquartered Network Box Corporation Limited.

Are we building fragility into our IT infra?

Security experts often point out single points of failure (SPOFs). These are the Achilles heels of IT design. Regarding hardware design and hard infrastructure, we’ve done well in embracing redundancy.

Yet, we seem to be blindsided by processes and software. CrowdStrike outage is a prime example. It is not the only company that had update issues nor the only one that had deep-level access. But this incident has had the most considerable impact to date — significant enough for cybersecurity consultant Troy Hunt, founder and chief executive officer of Have I Been Pwned, to compare it to the potential impact of the Y2K bug “except it’s actually happened this time.”

As of the writing of this article, we have yet to determine why there was a flawed update. “The range of possibilities ranges from human error — for instance, a developer who downloaded an update without sufficient quality control — to the complex and intriguing scenario of a deep cyberattack, prepared ahead of time and involving an attacker activating a ‘doomsday command’ or ‘kill switch’,” Omer Grossman, the chief information officer at CyberArk, pondered.

One thing’s for sure: the outage not only showed how integrated and sophisticated our global IT is, but it also highlighted its fragility.

There are other technical questions as well. For example, why run a kernel update natively? “It is far less risky to run natively in the User Space than in the Kernel Space. That is a significantly safer approach to endpoint cyber security monitoring and would have avoided the disaster seen [last Friday],” added Gazeley.

Grossman highlighted another issue this outage highlighted — our unpreparedness for fast remediation. “It turns out that because the endpoints have crashed — the BSOD — they cannot be updated remotely, and this problem must be solved manually, endpoint by endpoint. This is expected to be a process that will take days,” he wrote.

The real work and danger begin now

A more profoundly unsettling issue is what will happen in the coming weeks and days.

Many of the systems will need manual reboots. “Because of the way that the update has been deployed, recovery options for affected machines are manual and thus limited: Administrators must attach a physical keyboard to each affected system, boot into safe mode, remove the compromised CrowdStrike update, and then reboot,” the Forrester analysts said in a blog. They also noted some administrators having trouble accessing BitLocker hard-drive encryption keys to perform remediation steps.

But more than slow remediation time and customer unhappiness, having systems down will put smiles across many bad actors’ faces. During this time, companies will be vulnerable to bad actors. You can bet that some would be social engineering themselves to inject malware while others would seek access to valuable data.

To stop such outages, businesses must take IT risks and resilience more seriously.

“Better quality control, multi-layered testing, and indeed the use of a secure operating system for key infrastructure such as Linux, could have limited the damage globally too,” Gazeley observed. “Interestingly, that was the case in China, where key and critical systems were not impacted because they did not use either CrowdStrike or Windows.”

Forrester analysts also called business and legal leaders to dig into their contracts' business interruption indemnification clauses. Currently, CrowdStrike offers a warranty that is specific to security breaches. Companies also need to reevaluate their third-party risk strategies to ensure they are not overly focused on compliance and scrutinize for vendor concentration risks.

Essentially, Forrester analysts asked businesses to see contracts as risk mitigation tools and include new security and risk clauses that assign accountability and remediation timelines during disruptive events. “If vendors push back, you’ll need to consider whether the price you negotiated still makes sense and, possibly, whether to do business with them at all,” they wrote.

Yay to human ingenuity!

Nevertheless, we survived the outage. No planes dropped out of the sky, or ocean liners ran onshore (unlike the Netflix movie). Many current technologies (especially on the Operational Technology front) are still thankfully siloed and separated, so no lives were lost on operating tables or air controller losing sight of their planes. Currently, we have inconveniences and black faces.

On the lighter side, the outage showed human ingenuity and resilience. One Indian airline quickly issued handwritten boarding passes. Waitrose in the U.K. noted it was in business, but cash is now king.

Makes you wonder what the world would have looked like last Friday morning if the world was fully automated, interdependent and run by interconnected AI algorithms. As we rebuild and recover, let's remember that sometimes, the most sophisticated solution is the simplest one: a human in the loop.

Image credit: iStockphoto/mesh cube