When Smurfs Invaded Earth: How Regulators Made CrowdStrike Outage Possible

Remember July 19, 2024? It was the day many computer screens resembled a Smurf convention.

Yes, that was CrowdStrike's doing. A faulty update, a cascade of blue screens of death (BSODs), and suddenly, millions of Windows users were thrown back to the dial-up era. Even pen and paper briefly ruled social media.

Airlines were grounded, hospitals scrambled, and even the office coffee machine probably took a coffee break. It was digital chaos and the cost? A cool USD5.4 million for Fortune 500 companies alone. Ouch.

Sure, CrowdStrike messed up. Twice, actually. First, a flawed channel file 291 update (C-00000291.sys with timestamp 2024-07-19 0409 UTC, to be specific) went rogue. Then, their Content Validator, the digital bouncer meant to keep the bad code out, was caught napping on the job.

But this wasn't just a DevSecOps snafu. This was a systemic issue, a brittleness baked into the very core of our software ecosystem, thanks to some well-meaning but ultimately misguided regulations.

The unintentional architects of chaos

When the CrowdStrike outage metastasized, people were shocked at how a third-party security vendor like CrowdStrike could get so close to the kernel.

Turns out, the roots of this debacle might lie in a 2009 agreement between Microsoft and the European Commission.

It’s general knowledge that Windows OS is usually the target of attacks. Microsoft naturally wanted to harden its OS further, and the best way to do this quickly is at or near the kernel. The problem was that it tried to do it its way.

The European Commission of the E.U. was not having it. In the name of fair competition, they forced Microsoft to open its precious Windows kernel to third-party security vendors like CrowdStrike.

In the agreement, the EC mandated that the IT behemoth provide third-party security companies API access. It also asked to document these APIs, which may sound all nice for third-party software developers but does open up questions on potential risks from those with ill intentions. In addition, the agreement allowed federated access to Microsoft’s PC productivity applications, SharePoint, Outlook, Exchange and the .NET framework.

Apple was not asked to do the same. Part of the reason could have been that its macOS had a lower penetration. So, Apple got to keep its closed integration model. Google did not have to follow this as its ChromeOS was based on open source software.

Microsoft, understandably, wasn't thrilled. Their Windows Defender probably would have dominated the market if not for this regulatory intervention. History was against it. After all, the company faced the DOJ Antitrust case.

But hey, who needs market dominance when you can have a global outage, right?

Meet the security giant with a “small” glitch

Of course, regulators aren't the only ones in the hot seat. CrowdStrike somehow let a flawed update slip through its defenses.

In its preliminary Post Incident Review (PIR), the company pointed to a flaw in CrowdStrike's Content Validator component, used to check the integrity of rapid response content updates. That flaw enabled the faulty version of channel file 291 to pass validation, even though it had an error.

That this outage came from CrowdStrike, which isn’t some fly-by-night operation, is in itself an outrage. It is, after all, a company that's practically a superhero in the cybersecurity world.

It was CrowdStrike who uncovered the North Korean hack of Sony Pictures. They, along with Mandiant and ThreatConnect, helped investigate the Democratic National Committee cyberattacks. They also called out the Fancy Bearhack of Ukrainian artillery units (although the Ukrainian government rejected the report).

Their prowess helped SolarWinds to recover from its Orion breach and meet the security needs of U.S. Federal and State agencies. And this was after CrowdStrike stopped Russian hackers from doing a supply chain attack via a reseller’s Microsoft Azure account.

This raises a critical question: should security vendors be held to the same standards as, say, brain surgeons or nuclear engineers? After all, their code can have life-or-death consequences. Maybe it's time for some serious accountability in the Wild West of software development.

Why we should talk about OS resiliency

To be fair, the CrowdStrike event was not a hack and the remedy was issued quickly — except the reversion still came too late in today’s world that moves in real-time.

But Microsoft wasn't entirely innocent of blame either. While their hands were tied by the E.U., they could have built a more resilient operating system. One that doesn't crumble like a stale cookie at the first sign of trouble.

J.J. Guy, chief executive officer of Sevco Security, made an interesting point on LinkedIn: “Any software causing repeated failures on boot should not be automatically reloaded. We’ve got to stop crucifying CrowdStrike for one bug when it is the OS’s behavior that is causing the repeated, systemic failures.”

Why create a product that needs manual intervention in this age of automation and self-healing software? Maybe additional intelligence could have helped to mitigate the situation and not just shut down at every turn. It's like your car refusing to start every time you hit a pothole — not exactly ideal.

The future of fail

We got lucky this time. The cybercriminals were apparently as surprised by the outage as everyone else. But next time? Who knows? Maybe they'll use the chaos to launch their attacks, or perhaps they'll just sit back and enjoy the show.

More importantly, regulators need to wake up and smell the burnt silicon. Opening up operating systems in the name of competition is all well and good, but not at the expense of security. We need a more balanced approach that recognizes security's critical role in our increasingly interconnected world.

The CrowdStrike outage was a loud, obnoxious, blue-screened wake-up call. It's time to rethink our security approach, hold vendors accountable, and build more resilient systems. More importantly, we also need regulators to understand the full consequences of their legislation and set engagement frameworks. Because the next time the world turns blue, we might not be able to reboot our way out of it.

Image credit: iStockphoto/Michael Mulkens