In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here?
A Crowdstrike update being able to blue-screen Windows Desktops and Servers.
Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug).
There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production?
Un-audited dependence on an increasingly long chain of third-parties.
All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).
> Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system.
It doesn't operate in user space, they install a kernel driver.
It's a design decision. People want the antivirus to protect them even if an attacker exploits a local privilege escalation vulnerability or if an attacker that compromised an admin account (which happens all the time in Windows environments) wants to load malicious software. That's kind of the point of these things. Somebody exploits a memory vulnerability of one of the hundreds of services on a system, the antivirus is supposed to prevent that, and to their benefit, Crowdstrike is very good at this. If it didn't run in the kernel, an attacker with root can deactivate the antivirus. Since it's a kernel module, the attacker needs to load a signed kernel module, which is much harder to achieve.
Presumably Crowdstrikes driver also has the ELAM flag which guarantees it will be loaded before any other third party drivers, so even if a malicious driver is already installed they have the opportunity to preempt it at boot.
If we are being pedantic then an ELAM driver can't be guaranteed to load before another ELAM driver of course, but only a small list of vetted vendors are able to sign ELAM drivers so it is very unlikely that malware would be able to gain that privilege. That's the whole point.
Yep. We can't migrate our workstations to Ubuntu 24.04 because Crowdstrikes falcon kernel modules don't support the kernel version yet. Presumably they wanted to move to EBPF but I'm guessing that hasn't happened yet. Also: I can't find the source code of those kernel modules - they likely use GPL-only symbols, wouldn't that be a GPL violation?
I was given to understand that Crowdstrike provided some protection from unvetted export of data. I'm not sure that data would be useful without the rare domain expertise to use it, but I wasn't shown the risk analysis. And then someone else demands and gets ssh access to GitHub. Sigh.
I think "compliance" would be a better word to use that "safety" when it comes to a lot of "security" software on computers.
And I bring up the distinction because while compliance is "sometimes" about safety, it's also very often about KPIs of particular individuals or due to imaginary liability for having not researched every possible "compliance" checkbox conceivable and making sure it's been checked.
Some computer security software is completely out of hand because its primary purpose is to have the appearance of effectiveness for the exec whose job is to tick off as many safety checkboxes as they can find, as opposed to being actually pragmatically effective.
If the same methodologies were applied to car safety, cars would be so weighed down by safety features, that they wouldn't be able to go faster than 40km/h.
They mean distributing Linux + the module together. Like e.g. shipping the Nvidia kernel module alone is fine, but shipping a Linux distro with that module preinstalled is not fine.
Two different "it". As an analogy: selling pizza Hawaii is dicey, but you can sell pineapple slices and customers can add those to their pizza themselves.
Last time I dealt with HP, I had to use their fakeraid proprietary kernel module which "tainted" the kernel. Of course they never open-sourced it. I guess it's not necessary.
GPL exported symbols are the ones that are thought to be so tightly coupled to the kernel implementation that if you are using them, you are writing a derivative work of the kernel.
Yeah that was also my understanding, and I can't imagine a av module able to intercept filesystem and syscalls to be only using non-core symbols. But of course you never know without decompiling the module
Are they? Apple has pretty much banned kernel drivers (kexts) in macOS on Apple Silicon. When they were still used, they were a common cause of crashes and instability, not to mention potential gaping security holes.
Most things that third-party kernel drivers used to do (device drivers, file systems, etc) are now done just as well, and much more safely, in userspace. I'm surprised if Microsoft isn't heading in this direction too?
Presumably, Crowdstrike runs on macOS without a kernel extension?
> Presumably, Crowdstrike runs on macOS without a kernel extension?
That's correct: CrowdStrike now only installs an "Endpoint Security" system extension and a "Network" system extension on macOS, but no kernel extension anymore.
One would hope that Crowdstrike does a similar thing on Linux and relies on fanotify and/or ebpf instead of using a kernel module. The other upside to this would be not having to wait for Crowdstrike to be constantly updating their code for newer kernels.
I believe so but would like better details. We used to use another provider that depended on exact kernel versions whereas the falcon-sensor seems quite happy with kernel updates.
Whatever protection is implemented in user-land can be removed from user-land too. This is why most EDR vendors are now gradually relying on kernel based mechanisms rather than doing stuff like injecting their DLL in a process, hooking syscalls, etc...
First, we were talking about EDR in Windows usermode.
Second, still, that doesn't change anything. You can make your malware jmp to anywhere so that the syscall actually comes from an authorized page.
In fact, in windows environment, this is actively done ("indirect syscalls"), because indeed, having a random executable directly calling syscalls is a clear indicator that something is malicious. So they take a detour and have a legitimate piece of code (in ntdll) do the syscall for them.
The original Windows NT had microkernel architecture, where a driver/server could not crash the OS. So no, Crowdstrike didn't have an option really, but Microsoft did.
As PC got faster, Microsoft could have returned to the microkernel architecture, or at least focused on isolating drivers better.
They've done it to a degree but only for graphics drivers, Windows is (AFAIK) unique amongst the major OSes in that it can nearly always recover from a GPU driver or hardware crash without having to reboot. It makes sense that they would focus on that since graphics drivers are by far the most complex ones on most systems and there are only 3 vendors to coordinate API changes with, but it would be nice if they broadened it to other drivers over time.
NT was never a true microkernel. Most drivers are loaded into the kernel. Display drivers being a huge pain point, subsequently rolled back to user space in 2000, and printer drivers being the next pain point, but primarily with security -- hence moving to a Microsoft-supplied universal print driver, finally in Windows 11.
There's a grey area between "kernel drivers are required for crowdstrike" and "windows is not modular enough to expose necessary functionality to userspace". It could be solved differently given enough motivation.
My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.
He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.
I was just a contractor there, and don’t work with them at the moment. But I’m a customer of theirs and they’re definitely having an outage right now, so I’m guessing it’s all still in place.
I don’t work there any more. But they were having an outage, so I’m guessing they never got fired as a client (guessing that they’re still using Crowdstrike) and could still take that offer (of being fired as a client) if they wanted to.
> I hope this sort of thing humbles them a little.
What I hope, is that they stop to exist as a product and as a company. They have caused inconvenience, economic damage in global scale and probably also loss of life, given that many hospitals, ER units had outages. It has been proven that their whole way of working is wrong, from the very foundation to the top.
He was pretty senior for his role, but really I have no idea whether he was representative of the wider company culture.
We had a buggy client release during the rollout which consumed all the CPU in one of our test environments (something he assured us could never happen), and he calmed down a bit after that. Prior to that though he was doing stuff like finding our CISO on LinkedIn to let him know how worried he was about our rollout pace, and that without CS protection a major breach could be imminent.
At the end of the day, if you give an application a deep set of permissions, that's on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.
Administrators are to blame because management (and a lot of 'cybersecurity policies') demand there's a virus scanner on the machines?
While virus scanners might pick up some threats not addressed by OS updates yet every one of them I've seen is a rootkit in disguise wanting full system privileges. There are numerous incidents with security holes and crashes caused by these security products. They also aren't that clever: repeatedly scanning the same files 'on access' over and over again wasting CPU and IO is not going to give you any extra security.
I often watch Crowdstrike thrash my laptop's resources, making it slow to do compiles. Cybersecurity won't let me disable it either, so I just set it to lower priority process.
As someone who worked for a company, who's a Crowdstrike partner, I assure you that Crowdstrike does not sell to administrators. It is very much a product sold to management and company auditors.
Where you're correct is that it's on the administrators to rollout the updates, but I'm not sure that's how Crowdstrike works. It's a managed solution and updates are done for you, maybe that can be disabled, but I honestly don't know.
CS is not sold to SA or technical types. It's sold to management as a risk reduction.
The whole point is that if you are technical, you are so untrusted that management is willing to require circumvention of known good practices and force installation of this software against technical advice.
I have worked in Finance for 25 years, and the amount of pressure I had to stand from Auditing on "Why do we have a 20-day-window on applying most updates as we get them from suppliers? We are not best practice!" is gruelling.
These people report to the Board Chairman, don't understand any real implication of their work, and believe the world is a simplistic Red - Amber - Green grid.
I understand most CIOs / CTOs / CISOs in Corporate would buckle.
It's actually worse than phone updates. Ever looked at your phone and noticed it hasn't updated to the new OS despite it having been out for a few days already? This is why.
Wading out my depth here, so forgive any stupidity following.
And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).
You're taking about how things are, the comment you're replying to is talking about how things could be. There's not a contradiction there.
Originally, x86 processors had 4 levels of hardware protection, from ring 0 up to ring 3 (if I remember right). The idea was indeed that non-OS drivers could operate at the intermediate levels. But no one used them and they're effectively abandoned now. (There's "level -1" now for hypervisors and maybe other stuff but that's besides the point.)
Whether those x86 were really suitable or not is not exactly important. The point is, it's possible to imagine a world where device drivers could have less than 100% permissions.
The problem I have with this is that anti-virus software has never felt like the most reliable, well-written, trustworthy software that's deserving of it's place in Ring 0.
I understand I'm yelling into the storm here, because anti-virus also requires that level of system access due to the nature of what it's trying to detect. But then again, does it only need Ring 0 access for the worst of the worst? Can it run 99% of the time in Ring 1, or user space, and only instantiate it's Ring 0 privileges for regular but infrequent scans or if it detects something else may be 'off'?
Default Ring 0? Earn it.
This turns into a "what's your threat level" discussion.
Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.
Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.
There's a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...
I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest?
Mustangs have a reputation as being 'crowd (or streetlight) seeking' missiles.
This is due to their price making them relatively more available to the enthusiasts than say Hellcats, enthusiasts who may not be experienced enough to deal with having that much power available to them in a RWD car. This confluence of power, confidence and lack of skill often comes to a head when the enthusiast goes to a car meet to show off and meet with like minded folks. At the conclusion of the meet, or during a group drive, they'll often pull a sick burnout as they pull out of the parking lot on to a street.
A sick burnout they haven't practiced, and will often cause them to lose the back end sending the car into the curb, a tree, or a crowd of like minded attendees at the car meet. Therefore, the reputation.
Mustangs are famous for their high power and poor handling - there are lots of videos showing drivers doing burnouts, losing control, and striking the crowd they are showing off to.
Maybe it's time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question.
Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days.
Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance.
Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows.
Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.
The discussed issue is not related to any meaningful difference between Windows and Linux – Crowdstrike used a kernel driver, apparently containing a serious bug, which took down the system, which is something any kernel driver can do, no matter which kernel you use. At least Windows have a well-developed framework for writing userspace drivers, unlike Linux.
> Linux being the most advanced operating system in the world without question.
Very strong and mostly unfounded claim; there are specific aspects where Linux is "more advanced", and others where Windows come out ahead (e.g. almost anything related to hardware-based security and virtualization).
> your system will be supported for many years
Windows Server 2008 was supported until earlier this year, longer than any RHEL release.
> you won't suffer BSoDs in 2024
Until you install a shitty driver for a dubious (anti)malware service.
I don't understand this sort of blindness? Linux fails all the time, with rather terrible nobody to root vulns because some idiot failed to use the right bounds check. Ye gods, XZ utils was barely a few months ago!
Hmm? It was released for two plus months? 5.6.0 and 5.6.1
I'd also say this wasn't a good example of 'linux handling it better': usually when a mess like this occurs on windows all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news. In XZ's case, the backdoor was published before the team knew about it, huge mess.
You’re right that it went noticed for a long time, just one clarification
> all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news
AFAIK, distros were notified and released a patched version of xz like a week before it hit the news, so at least a lot of machines received it via automatic updates.
Depends which news you're talking about. MS guy who discovered it found it March 29th, published to oss. It was in infosec news same day as redhat, others pushed out critical advisories. Patch didn't come til a day or two later.
You're half right - people who compiled it from source could theoretically get those releases, but no, it wasn't released in any distros. So in practice since no linux distro released it, no-one relying on linux distros was exposed to it.
> Maybe it's time that critical systems switch to Linux.
I switched critical systems to illumos and BSD years ago and it's been smooth sailing ever since. Nowadays there really is no need to contribute to linux monoculturization whatsoever.
I too want to see Linux more widely adopted, but it won't prevent this from happening. People will install corrupted kernel modules on Linux too for anti-virus purposes.
All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn't have that despites its clear technical advantages.
It is an incredibly frustrated battle akin to Sisyphus.
Crowdstrike has a linux version. It is mandatory in our linux servers in my company so that is not the solution.
I would say issue 1 is management/compliance forcing admins to install malwares like crowdstrike. But issue 1 is because of issue 2 which is about admins / app devs / users aren't smart enough to not have their machines compromised on a regular basis in the first place. And issue 2 is because issue 3 of the software industry not focusing on quality and making bug free software.
All in all this should be mitigated by more diversity in OS, software and "said security solution". Standardization and monopolies works well until they don't and you get this kind of shit.
I think we don't do enough to fight back this requests in a language that is understood by management. Ask them to sign a security waiver assuming risks for installing software techs would classify as a malware and RCE risk.
Companies like CS live on reputation, it should be dragged down.
One place I'm at recently required us to install it in our Kubernetes cluster which powers a bunch of typical web apps.
Falcon sensor is the most CPU intensive app running in the cluster and produces a constant stream of disk activity (more so than any of our apps).
It hasn't crashed anything yet but it definitely leaves me feeling iffy about running it.
I don't like CrowdStrike at all. I got contacted by our security department because I used curl to download a file from GitHub on my dev box and it prompted a severe enough security warning that it required me to explain my intent. That was the day I learned I guess every command or maybe even keystroke I type is being logged and analyzed.
We were also forced to run that until the agent had introduced a memory leak that ate almost all the memory on all the hosts. Thankfully we managed to convince our compliance people that we could run an immutable OS rather than deploy this ~~malware~~ XDR agent.
Windows actually runs a lot of drivers in user-mode, even GPU drivers. largely this is because third-party drivers were responsible for the vast majority of blue screens, but the users would blame Microsoft. which makes sense; Windows crashes so they blame Windows, but I doubt anyone blamed Linux for the kernel panic.
I think windows can be blamed on how badly you can fix that kind of issues. I mean on linux or any bsd admins would build an iso image that would automatically run a script that would take care of optionnally decrypting the system drive, then remove crowdstrike. Or alternatively simply building a live system that take an address via dhcp and start an ssh server. and admins would remotely and automatically run a playbook that mount that iso on the hypervisor, boot it, remotely apply the fix, then boot back the system on the system drive.
Maybe this is just my ignorance about windows and its ecosystem but it seems most admins this morning were clueless on how to fix that automatically and remotely on n machines and would resort to boot in safe mode and remove a file manually on each single server. This is just insane to think that supposed windows sysadmins / cloudops have no idea how to deploy a fix automatically on that platform.
It can kill process based on memory scanning. Imagine systemd was getting killed at every boot?
An issue might not be as universal as on windows, because some distros do things differently like not using glibc, or systemd, or whatever. Yet there are some baselines common to the most popular ones.
Well, Microsoft tried to lock down its kernel with Windows Vista and then antivirus vendors cried that they won't be able to protect Windows, anticompetetive etc.
I could never get smooth scrolling to work on Linux in any mainstream web browser, most people don’t seem to see it, but I’m sensitive to things like that.
Like with a laptop trackpad? I'm smooth-scrolling through these comments right now, and don't remember when scrolling wasn't smooth by default on any trackpad.
It’s smooth to a point, but not smooth like OS X is. It might have improved (I think I last tried desktop Linux a year ago). I do enjoy using Linux as my default headless OS.
A Crowdstrike update being able to blue-screen Windows Desktops and Servers.
Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug).
There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production?
Un-audited dependence on an increasingly long chain of third-parties.
All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).