525
CrowdStrike offers a $10 apology gift card to say sorry for outage
(techcrunch.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 24 Jul 2024
525 points (98.7% liked)
Technology
34781 readers
243 users here now
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
founded 5 years ago
MODERATORS
I lost a day's holiday, and our team spent 8 man days on this entirely preventable mistake.
$10? Try extending our licence by another year for free, that might start going towards it.
Why would you want another year of their software for free? This is their second screw up (apparently they sent out a bad update that affected some Debian and RHEL machines a couple years ago). I'd be transitioning to a competitor at the first opportunity. It seems they aren't testing releases before pushing them out to customers, which is about as crazy to me as running alpha software on a production system.
I'm sure you have reasons, and this isn't really meant to be directed at you personally, it's just boggling to me that the IT sector as a whole hasn't looked at this situation and collectively said "fuck that."
Because AV, like everything else, costs a fortune at enterprise scale.
And yeah, I do understand your real point, but it's really hard to choose good software. Every purchasing decision is a gamble and pretty much every time you choose something it'll go bad sooner or later. (We didn't imagine Vmware would turn into an extortion racket, for example. And we were only saying a few months ago how good value and reliable PRTG was, and they've just quadrupled their costs)
It doesn't matter how much due diligence and testing you put into software, it's really hard to choose good stuff. Crowdstrike was the choice a year ago (the Linux thing was more recent than that), and its detection methods remain world class. Do we trust it? Hell no, but if we change to something else, there are risks and costs to that too.
Unfortunate reality for lot for medium to big size businesses.
Maybe AV, at an enterprise scale, is actually a horrible idea that reduces security, availability, and reliability and should be abolished through policy.
Maybe, but it's not going to happen soon. Any malware type insurance requires effective AV on all devices, and C-levels do love their insurance.
Not just Crowdstrike - any vendor that does automatic updates, which is more and more each day. Microsoft too big for a bad actor to do as you describe? Nope. Anything relying on free software? Supply chain vulnerabilities are huge and well documented - its only a matter of time.
Nah, I don't buy that. When you're in critical infrastructure like that it's your job to anticipate things like people being above or below versions. This isn't the latest version of flappy bird, this is kernel level code that needs to be space station level accurate, that they're pushing remotely to massive amounts of critical infrastructure.
I won't say this was one guy, and I definitely don't think it was malicious. This is just standard corporate software engineering, where deadlines are pushed to the max and QA is seen as an expense, not an investment. They're learning the harsh realities of cutting QA processes right now, and I say good. There is zero reason a bit of this magnitude should have gone out. I mean, it was an empty file of zeroes. How did they not have any pipelines to check that file, code in the kernel itself to validate the file, or anyone put eyes on the file before pushing it.
This is a massive company wide fuckup they had, and it's going to end up with them reporting to Congress and many, many courts on what happened.
Even an AI is good enough to avoid (or let someone avoid) pushing a similar bug 🫣