Ask HN: What is in C-00000291*.sys?
103 comments
Context: CrowdStrike (https://news.ycombinator.com/item?id=41002195)
https://cyberplace.social/@GossiTheDog/112812260542179660
> I've obtained copies of the .sys driver files Crowdstrike customers have. They're garbage. Each customer appears to have a different one.
https://cyberplace.social/@GossiTheDog/112812454405913406
> The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
> I've obtained copies of the .sys driver files Crowdstrike customers have. They're garbage. Each customer appears to have a different one.
https://cyberplace.social/@GossiTheDog/112812454405913406
> The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
They might just be encrypted with a customer-specific key. That wouldn't surprise me.
They weren't last time I looked. They seem to contain a bunch of different things, but you can absolutely download and parse them without needing to decrypt anything.
If you have a Crowdstrike customer ID (CID) — which you can pull from any device that has the implant — you can request any channel file you want from their file server.
Ask for metahash+/cfs/channelfiles/0000000291/<YOUR CUSTOMER ID>/C-00000291-00000000-00000001.sys and you should get something that starts with:
If you have a Crowdstrike customer ID (CID) — which you can pull from any device that has the implant — you can request any channel file you want from their file server.
Ask for metahash+/cfs/channelfiles/0000000291/<YOUR CUSTOMER ID>/C-00000291-00000000-00000001.sys and you should get something that starts with:
00000000: aaaa aaaa 0100 2301 0000 0500 0000 0000 ......#.........
00000010: 0100 0000 4808 0000 2c08 0000 0600 0004 ....H...,.......
Could you please upload it to some place and share a link. Curious to examine the contents.
I woke up after they'd already pulled the bad update, and I don't have an affected system.
If I look at the C-00000291-00000000-00000032.sys version that Crowdstrike LFO serves me, I get something that looks superficially reasonable (not random garbage or full of zeroes).
I would share it, but my understanding is that channel files specifically can have different contents for different customers (as opposed to other files like their Linux kernel drivers, which is definitely the same giant .xz blob for everyone). So I'd rather not upload something that's potentially tied to a specific customer/company without asking for permission
But if you have a valid customer ID from Crowdstrike, I'm happy to point you the little tool I use. You can request old versions of channel files from LFO and look at the diff as much as you like. But I can't guarantee you'll actually be getting a file that's obviously broken or full of zeroes like some people are describing. My C291 0.32 looks superficially normal.
If I look at the C-00000291-00000000-00000032.sys version that Crowdstrike LFO serves me, I get something that looks superficially reasonable (not random garbage or full of zeroes).
I would share it, but my understanding is that channel files specifically can have different contents for different customers (as opposed to other files like their Linux kernel drivers, which is definitely the same giant .xz blob for everyone). So I'd rather not upload something that's potentially tied to a specific customer/company without asking for permission
But if you have a valid customer ID from Crowdstrike, I'm happy to point you the little tool I use. You can request old versions of channel files from LFO and look at the diff as much as you like. But I can't guarantee you'll actually be getting a file that's obviously broken or full of zeroes like some people are describing. My C291 0.32 looks superficially normal.
No the hashes are the same.
There are different files that can differ between customers.
9ca565965198cf1a5073ce974c5826d14aa3ac3cf574b3d2d2e7f80624b9ed90 = C-00000291-00000000-00000033.sys = https://www.virustotal.com/gui/file/9ca565965198cf1a5073ce97...
61486fe1aa690260bd846245f0fac204b26f41ed83f66e962cb548837f3ceb98 = C-00000291-00000000-00000057.sys = https://www.virustotal.com/gui/search/61486fe1aa690260bd8462...
There are different files that can differ between customers.
9ca565965198cf1a5073ce974c5826d14aa3ac3cf574b3d2d2e7f80624b9ed90 = C-00000291-00000000-00000033.sys = https://www.virustotal.com/gui/file/9ca565965198cf1a5073ce97...
61486fe1aa690260bd846245f0fac204b26f41ed83f66e962cb548837f3ceb98 = C-00000291-00000000-00000057.sys = https://www.virustotal.com/gui/search/61486fe1aa690260bd8462...
most of your comments on HN are being marked as dead
The most successful malware of 2024, even though it only does denial of service.
So far.
[deleted]
It's a crowdstrike update file with a bug in it, from what I gather. This makes your Windows machine go blue screen and stop working as it starts up. If you manage to remove it by various methods, it doesn't run and you're fine.
More informed people will give you more details, but this kind of AV software often has privileged access to the OS, so it can scan your files. The same privileged access also means it can really mess things up if it's not well tested.
By contrast your ordinary python or VBA script should not be able to blue screen your machine, especially not during startup.
More informed people will give you more details, but this kind of AV software often has privileged access to the OS, so it can scan your files. The same privileged access also means it can really mess things up if it's not well tested.
By contrast your ordinary python or VBA script should not be able to blue screen your machine, especially not during startup.
> By contrast your ordinary python or VBA script should not be able to blue screen your machine, especially not during startup.
This is misleading. If you run a python or VBA script on startup with elevated permissions to e.g. setup a driver (with a bug) or do hardware configurations (in a way which doesn't work) then it will as much blue screen as a sys file.
The issue is not that it's a `.sys` file, but that a high privilege administrative task run at startup fails so hard that it blue screens.
(In windows the sys file extension is basically just a marker that the file is part of the "system" without being clear what it is, it can be anything from a data blob,over a data base, over python/VBA files to a device driver or a blob combining multiple of such things. They mostly contain device drivers and hardware configuration, but there are some which are basically just the equivalent of /etc/hostname)
This is misleading. If you run a python or VBA script on startup with elevated permissions to e.g. setup a driver (with a bug) or do hardware configurations (in a way which doesn't work) then it will as much blue screen as a sys file.
The issue is not that it's a `.sys` file, but that a high privilege administrative task run at startup fails so hard that it blue screens.
(In windows the sys file extension is basically just a marker that the file is part of the "system" without being clear what it is, it can be anything from a data blob,over a data base, over python/VBA files to a device driver or a blob combining multiple of such things. They mostly contain device drivers and hardware configuration, but there are some which are basically just the equivalent of /etc/hostname)
Every Windows executable can crash a Windows system with the SE_SHUTDOWN_NAME privilege (which, contrary to the name, not only allows safe normal shutdowns) if they want to, though, which most of the time every user has.
The "if they want to" part is the difference, kernel drivers can easily cause BSoDs on accident, e.g. by writing to an invalid address, normal processes can't, they have to jump through hoops by choice.
The "if they want to" part is the difference, kernel drivers can easily cause BSoDs on accident, e.g. by writing to an invalid address, normal processes can't, they have to jump through hoops by choice.
A malware delivery platform sponsored by the US security state. All customers get customised versions nothing to see here.
It provides the electrolytes Windows craves.
On another note, I know nothing about cybersec, is there a reason for which antivirus on windows run at ring 0 while I read that on Linux and Mac they don't have kernel level access?
It's not an anti-virus, it's intended to monitor all and everything on the machine. You^WAn attacker might want to hide what you're doing and thus it runs at that level.
You may be right but they do market it as "Next-Generation Antivirus (NGAV)"/"Antivirus with Threat Intelligence" probably because it's a word people are familiar with
[CrowdStrike Falcon® Pro: Antivirus with Threat Intelligence](https://www.crowdstrike.com/products/bundles/falcon-pro/)
[CrowdStrike Falcon® Pro: Antivirus with Threat Intelligence](https://www.crowdstrike.com/products/bundles/falcon-pro/)
I understand that these "NGAV" must be in ring 0 (device driver) because they want to inspect more things directly. And be more protected there, avoiding being attacked. I'm not sure they can achieve this.
The Linux version of CS sensor defaults to being installed as a kernel module as well.
via Reddit, don't know where so can't credit:
Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process -- https://access.redhat.com/solutions/7068083 -- 1 month ago
Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process -- https://access.redhat.com/solutions/7068083 -- 1 month ago
Who would trust that?
your employer's IT department
The cyber unit within IT is more likely, those ones are besotted with ticking compliance checkboxes, the delegation of responsibility and a game of musical chairs at any cost.
It is even more likely that IT was at loggerheads with cyber, but nowadays cyber seems to be able to trump everything and everyone.
It is even more likely that IT was at loggerheads with cyber, but nowadays cyber seems to be able to trump everything and everyone.
This is a management move. IT probably wasn't in the loop, since this effectively reduce the responsibilities of IT in terms of compliance.
And especially their auditors
I work in a customer-facing role for a similar product.
The handful of customers that asked about kernel modules / drivers saw it as a plus not a con.
I complained specifically about that and they bounced it back to me like this was safe and cannot use the machine as I want.
A business that is party to a contract that requires the use of such software.
Does anyone understand what a channel file is? Some sort of patch/dll that a driver loads?
It's a binary format specific to Crowdstrike for delivering updates. It's not a DLL or a plain patch, it's their own file format. And they have a lot of different things in there (config, regularly updated data, instructions to download the next driver version, etc etc)
Just kernel drivers. To know what is inside, you can disassemble them with https://github.com/NationalSecurityAgency/ghidra
These files are apparently just definitions/input for the actual CrowdStroke driver, that presumably someone named .sys so it sounds more scary and important.
Peak irony then that you can just delete them. Surely an antivirus should recognize when its being rollbacked?
Peak irony then that you can just delete them. Surely an antivirus should recognize when its being rollbacked?
> that you can just delete them
Don't you have to be in recovery mode, with it switched off, to do that?
Don't you have to be in recovery mode, with it switched off, to do that?
Specifically, if the file is corrupted, in what way is it corrupt? I’m fascinated by how this issue occurred.
It likely contains a bug but is an uncorrupted file. Corrupted to me means the file was changed or modified in a way from it's intended state and likely won't run anymore.
Crowdstrike has published some technical details:
https://www.crowdstrike.com/blog/falcon-update-for-windows-h...
> I’m fascinated by how this issue occurred.
NGO, i think, discovered that parsing random data can lead to interesting results.l
NGO, i think, discovered that parsing random data can lead to interesting results.l
I wonder if the * (asterisk / star) character in its name isnt causing the issue.
Star is part of a command to delete any file matching a wildcard `C-00000291*.sys` meaning "starts with 'C-00000291', has any kind of text after that and ends with '.sys'". Actual files appear to differ for every machine
This exchange reminds me of a time many years ago where I was reading a computer magazine and they talked about “Windows 9x” and I thought they were referring to some version of Windows that I had never heard of.
Only a couple of years later did I understand that it was simply a shorthand for talking about Windows 95 and Windows 98.
Edit: And looking now at Wikipedia apparently it also covers Windows Me. (The Millennium Edition that came between Windows 98 and Windows XP.) https://en.wikipedia.org/wiki/Windows_9x
Only a couple of years later did I understand that it was simply a shorthand for talking about Windows 95 and Windows 98.
Edit: And looking now at Wikipedia apparently it also covers Windows Me. (The Millennium Edition that came between Windows 98 and Windows XP.) https://en.wikipedia.org/wiki/Windows_9x
Also the reason that Microsoft skipped from Windows 8 to Windows 10. Too much risk of apps testing the first 8 characters of the Windows version against "Windows 9" and concluding they were running on 95/98 rather than 9.
Hm, is there any website that explains why C-00000291*.sys caused the widespread BSODs? For example, was it some kind of definition file that was accessing invalid memory locations?
This is just generic speculation
New to IT, but this sure seems like a huge security risk. Even though the CEO and others have said otherwise. People who are more experienced, please let me know if I am wrong.
Does anyone have the actual file, I have a copy but it seems to be a good version unfortunately. Really appreciate if anyone can upload it here
Can someone share a link to a copy of the offending channel file. Now a crowd strike customer, but interested in poking at its contents. Thanks!
[deleted]
Does anyone have the BSoD dump file when it crashed? our a C-00000291-00000000-00000032.sys output de-identified ?
Does anyone have the BSoD dump file our the C-00000291-00000000-00000001.sys de-identified to analyze both
[deleted]
[deleted]
[deleted]
Wouldn't want to be the guy who pushed this particular commit. It's ironic that the company that is supposed to prevent this sort of thing causes the biggest worldwide outage ever. Crowdstrike is finished. Let's hope this will result in at least a small increase in desktop Linux market share.
Just a small reminder that's it's never "the guy" and always "the process", or lack thereof.
So it's "the guy" who's job was to make and enforce "the process", got it.
Yeah, but heads will have to roll for this one, the world will be calling for blood, so who better if not "the guy"?
When the world calls for blood against your organization, it's a test of the organization's character: will they throw a scapegoat under the bus (even if there is a directly responsible person) or will they defend their staff, accept fault, and demonstratively improve process?
The answer is yes
the management that enabled the process. And follow the chain to the top, they are paid very well to own the risks
More importantly, the companies that enabled auto update from a vendor to production rather than having a validation process. This sort of issue can happen with any vendor, penalising the vendor won't help with the next time this happens.
Was there a way to not enable these channel updates? If so, would you still check all the mandatory security measures when being audited?
The way is to not install third party software with kernel level access that you can't stop pulling remote updates.
How does that pass a security audit in the first place?
How does that pass a security audit in the first place?
It’s both. If you’re an engineer and you push out shitty code that takes down 911 systems and ambulances, you f’ed up. Push back against processes that cause harm, or have the potential to cause harm. You are ultimately responsible for your actions. No one else. The excuse of “I was just following orders” has been dead and buried since WW2.
Yeah, ideally management should know better. But management aren’t usually engineers. Even when they are, they don’t deal with the code on a day to day basis. They usually know much less about the actual processes and risks than the engineers on the ground.
Yeah, ideally management should know better. But management aren’t usually engineers. Even when they are, they don’t deal with the code on a day to day basis. They usually know much less about the actual processes and risks than the engineers on the ground.
if one of the people i manage is not up to the task the fault is mine. I've hired them. I should setup a system of hard gained trust and automation to avoid or at least minimize them fucking up. When fuckups happen, they are my fuckups. Critical systems don't survive only on trust, obviously. If I don't setup the teams and the systems properly, my bosses will also take the blame for having put me in that position.
I'm not advocating for lower layers to avoid responsabilities. But if an head needs to roll you should look above. That said, peole are hardened by fuckups, so there are better solutions than rolling heads, usually.
Right. In one sense, what we're talking about is different ideas on how companies / teams work. There's a wonderful book called "Reinventing Organizations" by Laloux that I recommend to basically everyone. In the book, the authors lay out a series of different organisational structures which have been invented and used throughout the ages. The book talks about early tribes where the big man tells everyone what to do (eg mobsters), to rigid hierarchies + fixed roles (the church, schools) to modern corporations with a flexible hierarchy, and some organisation structures beyond that.
The question of "who is ultimately responsible" changes based on how we see the organisation. In organisations where the chief decides everything, its up to the chief to decide if they should place blame on someone or not. In a modern corporation, people at the bottom of the hierarchy are shielded from the consequences of their actions by the corporation. But there's also a weird form of infantilisation that goes along with that. We don't actually trust people on the ground to take responsibility for the work they do. All responsibility goes up the management hierarchy, along with control, power and pay. Its sort of assumed that people who haven't been promoted are too incompetent to make important choices.
I don't think thats the final form of how high functioning teams should work. Its noble that you're willing to put your head on the chopping block, but I think its also really important to give maximal agency to your employees. And that includes making people feel responsible and empowered to fix problems when they see them. You get more out of people by treating them like adults, not children. And they learn more, and I think that's usually, in the long run, better for everyone.
I agree that if a company has a bad process, employees shouldn't be fired over it. But I also think if you're an employee in a company with a bad process, you should fight to make the process better. Never let yourself be complicit in a mistake like this.
The question of "who is ultimately responsible" changes based on how we see the organisation. In organisations where the chief decides everything, its up to the chief to decide if they should place blame on someone or not. In a modern corporation, people at the bottom of the hierarchy are shielded from the consequences of their actions by the corporation. But there's also a weird form of infantilisation that goes along with that. We don't actually trust people on the ground to take responsibility for the work they do. All responsibility goes up the management hierarchy, along with control, power and pay. Its sort of assumed that people who haven't been promoted are too incompetent to make important choices.
I don't think thats the final form of how high functioning teams should work. Its noble that you're willing to put your head on the chopping block, but I think its also really important to give maximal agency to your employees. And that includes making people feel responsible and empowered to fix problems when they see them. You get more out of people by treating them like adults, not children. And they learn more, and I think that's usually, in the long run, better for everyone.
I agree that if a company has a bad process, employees shouldn't be fired over it. But I also think if you're an employee in a company with a bad process, you should fight to make the process better. Never let yourself be complicit in a mistake like this.
Thank you for the reading suggestion!
> It’s both. If you’re an engineer and you push out shitty code that takes down 911 systems and ambulances, you f’ed up.
This is wrong. If a company is developing that kind of software is the responsibility of the company to provide a certain level of QA before they release software. And no, it's not that "engineers are pushing out shitty code", but that the shitty company allows shitty code to be deployed in customers' machines.
This is wrong. If a company is developing that kind of software is the responsibility of the company to provide a certain level of QA before they release software. And no, it's not that "engineers are pushing out shitty code", but that the shitty company allows shitty code to be deployed in customers' machines.
Yeah, agreed. Who cares whose semicolon it was?
What matters is how this was deployed without any testing.
What matters is how this was deployed without any testing.
> The excuse of “I was just following orders” has been dead and buried since WW2.
Only for the loosers.
Only for the loosers.
Many major companies have post-mortem reviews for this kind of thing. Most of the big failures we see is a mix of people being rushed, detection processes failing, a miscommunication/misunderstanding of the effects of a small change.
One analogy is rounding - one rounding makes no difference to a transaction, but multiple systems rounding the same direction can have a large scale impact. It's not always rounding money - it can be error handling. A stops at the error, B goes on, turns out they're not in sync.
Which guy is it? The person who pressed the button? The manager who gave that person more than one task that day? The people who didn't sufficiently test the detection process? The people who wrote the specs without sufficient understanding of the full impact? The person who decided to layoff the people who knew the impact three months ago?
One analogy is rounding - one rounding makes no difference to a transaction, but multiple systems rounding the same direction can have a large scale impact. It's not always rounding money - it can be error handling. A stops at the error, B goes on, turns out they're not in sync.
Which guy is it? The person who pressed the button? The manager who gave that person more than one task that day? The people who didn't sufficiently test the detection process? The people who wrote the specs without sufficient understanding of the full impact? The person who decided to layoff the people who knew the impact three months ago?
> Crowdstrike is finished
Unlikely, just as Solarwinds wasn't finished when they distributed malware that got government agencies hacked. You underestimate the slow turning radius of giant company purchasing departments.
Unlikely, just as Solarwinds wasn't finished when they distributed malware that got government agencies hacked. You underestimate the slow turning radius of giant company purchasing departments.
As I posted elsewhere, the SolarWinds stock has never recovered from its high before the hack. And it is on a downward trend.
Crowdstrike is finished? Ha!
SolarWinds got the US government hacked by the Russians and they still exist.
SolarWinds got the US government hacked by the Russians and they still exist.
Interestingly SolarWinds is headquartered in Austin and CrowdStrike recently moved there too.
The SolarWinds stock has never recovered from its high before the hack. And it is on a downward trend.
Enterprise Linuxes also employ Crowdstrike or similar "security" products as mandatory part of their IT deployments. Often (always?) this is due to companies wanting certification for their secure processes, in order to sell to government or large corporations that require them.
Crowdstrike in their official statement said "Linux and MacOS not affected". Are there any reports stating otherwise?
not affected because the bug is in the windows specific code, not because it works so much different on linux
Why the fuck didn't MSFT just do blue/green canarying? No update should be rolled out to a billion devices at once until it's baked in a million devices for a bit, and that only after baking in 10,000 devices for a bit.
CrowdStrike is not MSFT. This also affected Linux installations with CrowdStrike installed, from what I've read.
Thanks, sorry, I commented before getting my facts in order. Comment still stands as applied to CrowdStrike.
> Crowdstrike is finished
Boeing is still there... we'll see
Boeing is still there... we'll see
Yes indeed. That's kind of how Chernobyl happened.
> Crowdstrike is finished.
We thought about Microsoft the same way, some 15 years ago. /s
We thought about Microsoft the same way, some 15 years ago. /s
How is the * (star / asterisk) character allowed in the file name?
I thought such characters are forbidden by Windows.
https://learn.microsoft.com/en-us/windows/win32/fileio/namin...
How did the tool even manage to create such a file?
I thought such characters are forbidden by Windows.
https://learn.microsoft.com/en-us/windows/win32/fileio/namin...
How did the tool even manage to create such a file?
echo lol > 'l*l';
ls 'l*l'
I bet it's a glob pattern and indicates "anything"
I bet it's a glob pattern and indicates "anything"