CISA Director on Log4j Vulnerability(cisa.gov)
cisa.gov
CISA Director on Log4j Vulnerability
https://www.cisa.gov/news/2021/12/11/statement-cisa-director-easterly-log4j-vulnerability
84 comments
One of Sovereigntechfunds goal is to support maintainers of critical open source infrastructure with public money.
https://sovereigntechfund.de/en
Germany for now, but plans are to expand "up" to Europe as far as I know (we are all in the same boat, aren't we)
Germany for now, but plans are to expand "up" to Europe as far as I know (we are all in the same boat, aren't we)
Spring Boot apps aren't affected by this unless you switched logging frameworks. Nor any JBoss frameworks. How many people actually use Log4j2? It's probably in 4th or 5th place among Java logging frameworks, and likely still has less uptake than the original Log4j.
I’m sure this is the only community that might pay attention to a software BOM as mentioned in the article, but this is a great idea and makes a lot of sense (to me as a consumer at least).
In the production of electronic things I've been pointing out that software needs to be on the BOM for other reasons. It is so often overlooked and considered zero cost even though companies pay people to develop it.
Making people look at a BOM would also discourage the mess that is npm.
Making people look at a BOM would also discourage the mess that is npm.
Most systems use package management now; npm is only the poster child.
Reviewing a stack of BOMs is going to be a challenge for any organization. Say your production Linux has 1000 packages. Each of those might have hundreds or thousands of deps in varying versions, in their respective package managers (BOMs).
Business needs to step up its process game. How are BOMS (dep lists) reviewed? Do we expect zero CVEs? How do you filter out false positives, or irrelevant ones? Do you dump everything with that dep or help the maintainer fix it? Many questions.
Reviewing a stack of BOMs is going to be a challenge for any organization. Say your production Linux has 1000 packages. Each of those might have hundreds or thousands of deps in varying versions, in their respective package managers (BOMs).
Business needs to step up its process game. How are BOMS (dep lists) reviewed? Do we expect zero CVEs? How do you filter out false positives, or irrelevant ones? Do you dump everything with that dep or help the maintainer fix it? Many questions.
One outcome of reviewing a BOM will be people asking "why do we have 5000 little dependencies? I thought we were paying YOU to write software."
We do something along this axis with our products. We have to be able to provide B2B software that will be stable on timelines measured in half-decades (per contractual requirements), so the specific vendors we decide to depend upon are a huge part of our decision making process. I will happily admit we probably wrote a little bit too much stuff in-house, but the number of clear wins easily outweighs the "no DIY allowed" concerns.
Getting us to vendor out something like logging/tracing/telemetry would take an act of god at this point. We explicitly spent a week ripping out Microsoft's byzantine logging from AspNetCore in favor of something we could trust and understand. Our entire logging framework now lives in 1 class file and consists of maybe 30 useful lines of source code. None of them have the capability to reach out to a remote host, download a DLL and then execute it in the current context. This sort of problem we are seeing with Log4j today is precisely the sort of experience we hope to avoid by doing a lot of our tooling in-house.
I think those who parroted "don't reinvent the wheel" over and over like it's some doomsday cult should accept some shame for the situation many developer ecosystems find themselves in today.
Getting us to vendor out something like logging/tracing/telemetry would take an act of god at this point. We explicitly spent a week ripping out Microsoft's byzantine logging from AspNetCore in favor of something we could trust and understand. Our entire logging framework now lives in 1 class file and consists of maybe 30 useful lines of source code. None of them have the capability to reach out to a remote host, download a DLL and then execute it in the current context. This sort of problem we are seeing with Log4j today is precisely the sort of experience we hope to avoid by doing a lot of our tooling in-house.
I think those who parroted "don't reinvent the wheel" over and over like it's some doomsday cult should accept some shame for the situation many developer ecosystems find themselves in today.
Developers are relying more and more on automated scanners and the likes to manage this. Your modern python or javaScript stack just has way too many packages and they change daily. Just look at your dependency lockfile balloon when a random dependency updates a point and brings in a few more packages. It’s really a horrible thing.
I’d like to say the scanners were fast but not fast enough because the first wave of attacks was nearly instant. This was definitely a nightmare scenario where a simple unauthenticated GET could pull in a kit that was already live and ready to go.
I’d like to say the scanners were fast but not fast enough because the first wave of attacks was nearly instant. This was definitely a nightmare scenario where a simple unauthenticated GET could pull in a kit that was already live and ready to go.
I don't think anyone in the Java community is surprised to find they have a dependency on Log4j. It's one of those libraries used so widely that it's practically stdlib.
Unless they spell out how they proactively support the one man hobbyist maintainer I don't believe them.
https://xkcd.com/2347
Indeed. The statement is so much disconnected from the reality that it is embarrasing to read.
There are a few initiatives to fix that, though:
https://www.coreinfrastructure.org/
http://www.loadsharers.org/
https://www.coreinfrastructure.org/
http://www.loadsharers.org/
I think CycloneDX [1] is the go-to SBOM specification these days.
[1] https://cyclonedx.org/
[1] https://cyclonedx.org/
I was just digging into Cloud Native Buildpacks (buildpacks.io) as an alternative to Dockerfiles yesterday and realised that they actually have SBOM generation built into them for popular languages which is a really nice easy security infrastructure upgrade for everyone using them.
“we are also convening a national call with critical infrastructure stakeholders on Monday afternoon”
Shouldn’t this be happening, like, yesterday?
Shouldn’t this be happening, like, yesterday?
No. The stakeholders are busy patching their shit. Pulling folks into meetings shouldn't be the priority when teams around the world in virtually every tech organizations are in firefighting mode.
This type of a call would likely be focused on assessing current state at that point.
This type of a call would likely be focused on assessing current state at that point.
Could you share links to any threads or accounts people are sharing regarding tech organisations in firefighting mode against this?
I work for a large non-US based tech company and have not heard anything about this beyond HN.
I work for a large non-US based tech company and have not heard anything about this beyond HN.
Work in one of the largest financial org in the world as a Java dev for critical system (albeit not internet facing), learned of this just now on this thread...
Edit: upon checking, we're safe, it doesn't impact log4j1, only the second version. We're not cowboys using versions as young as 2012 lol.
Edit: upon checking, we're safe, it doesn't impact log4j1, only the second version. We're not cowboys using versions as young as 2012 lol.
It’s definitely happening. People are just too busy to post about it. I’m on a few hours sleep over the last couple days myself.
I'm sure many folks here spent their Friday, Saturday, and possibly even Sunday patching, and won't speak up in case their profile connects to their company.
Friday mid-afternoon a Google search for the exploit showed there were many websites in several languages giving instruction on how to exploit the vulnerability. This is hitting hard and fast.
Friday mid-afternoon a Google search for the exploit showed there were many websites in several languages giving instruction on how to exploit the vulnerability. This is hitting hard and fast.
I work for a bank in APAC region and we had at least one big change out on Friday evening to address this.
It was on the news in my country. There have been several notable ransomware attacks in the last few years, it's become an issue for a country and government that's gone all in on digital.
I had an ex colleague at Google talking about it on Facebook...if he had to bring an internal stuff of Facebook, it must be really bad.
The people that would be in that meeting are not the individuals actually patching servers.
No, they are managing the teams patching things and helping with prioritization.
If the stakeholders participating in the call on Monday are not the people working on patching, managing, or prioritizing, then having the call earlier than that is pointless, because those stakeholders are too removed from the situation, and getting that information will detract from getting work done.
It's pretty simple really, having a status or coordination call while the work is being done is roughly analogous to a photo-op by a politician during an emergency. It looks good for the voters, but it takes away resources from the folks who are actually doing the work.
If the stakeholders participating in the call on Monday are not the people working on patching, managing, or prioritizing, then having the call earlier than that is pointless, because those stakeholders are too removed from the situation, and getting that information will detract from getting work done.
It's pretty simple really, having a status or coordination call while the work is being done is roughly analogous to a photo-op by a politician during an emergency. It looks good for the voters, but it takes away resources from the folks who are actually doing the work.
Its the weekend! Security can wait. /s
For some company's that's true, in one i worked we shutdown all systems (including router and firewall) every Friday at 18:00 ;)
I recall some US gov agency, possibly the IRS of all things, that would only accept forms being submitted during US working hours. Because the computers also need time off I guess?
In some cases I know (with smaller orgs) it happens because someone put a requirement in that they need to react to incidents with the service within X hours, but didn't provide the budget for the employees or external service needed to have that 24/7.
I've heard of similar things being done for ADA compliance. Something about requiring a phone line to be up at all the same times as the website so instead of a 24/7 call center you just turn the site off.
Do not worry, because:
> We have established a JCDC senior leadership group to coordinate collective action and ensure shared visibility into both the prevalence of this vulnerability and threat activity
Very clearly they know what they will be doing.
> We have established a JCDC senior leadership group to coordinate collective action and ensure shared visibility into both the prevalence of this vulnerability and threat activity
Very clearly they know what they will be doing.
Security vulnerabilities aren’t exploited on weekends out of courtesy
I always deeply appreciated this strict code of honour in the 'industry'. But one should take care of what constitutes the workweek here: https://en.wikipedia.org/wiki/Workweek_and_weekend
Unfortunately NSO group likely has a different definition of weekend.
do I need to worry about this as a frontend web developer?
No, the kitchen is on fire but the front door looks nice and fresh.
Do you have any java in your application stack (i.e. elastic search)?
If there’s no java, then there’s no log4j and you may worry about other issues.
If there’s no java, then there’s no log4j and you may worry about other issues.
*are you using ElasticSearch, flink, spark, prest,…etc. do they read in user data supplied by your front end. Could this user data end up being logged intentionally or part of an error log output.
If so, you might be vulnerable and should update those systems.
those don't run on the front end
It doesn’t matter. An nginx web server logging an odd user agent, elasticsearch picks it up. And then, due to some error triggered by the attacker, the content of the webserver log line gets logged on the ES side (for example if it violates a constraint). Frontend bypassed, attack owns ES. I’ve spent my weekend celebrating that I don’t have Java software in a stack that I’m responsible for right now - and feeling sorry for my ex-colleagues that do.
Edit: I haven’t tested or checked whether ES is vulnerable or not - but given the severity of this issue I’d default to the pessimistic stance of assuming it is, until proven otherwise.
Edit: I haven’t tested or checked whether ES is vulnerable or not - but given the severity of this issue I’d default to the pessimistic stance of assuming it is, until proven otherwise.
It does matter. To quote the original parent: "do I need to worry about this as a frontend web developer?"
A frontend web developer doesn't manage nginx, nor ES.
The backend should be patched. It is not safe to just mitigate the bug by using code in the frontend.
A frontend web developer doesn't manage nginx, nor ES.
The backend should be patched. It is not safe to just mitigate the bug by using code in the frontend.
As a frontend dev shouldn't you worry about your clients, rather than the boundaries of your organisation ? Harass whomever is putting the log4j dependency in the backend until they patch, don't expect them to know by default.
Sure, but it's not your job to know every single vulnerability that can be in your backend.
I'm seeing on 7.15, logstash and elasticsearch both ship log4j in the vulnerable range, but in my case, I'm running a new enough java that it shouldn't be an issue.
As has been commented several times on other threads here on HN, a new enough Java only protects against one kind of exploit (directly loading arbitrary bytecode) but not others (serialization tricks to execute arbitrary function calls, or data exfiltration).
Someone should tell them that any system that it logging user supplied data can be affected.
Not only "user facing" systems. Not even a WAF can protect these.
Not only "user facing" systems. Not even a WAF can protect these.
Sadly every WAF vendor is falling over themselves to claim otherwise. I've already had arguments with senior leadership suggesting there's no need to worry about patching because multiple vendors have promised their solutions are better.
The further I get into security the more embarrassed I am for some of the offerings.
The further I get into security the more embarrassed I am for some of the offerings.
I work as a software engineer in cybersecurity.
> embarrassed... for some of the offerings
Welcome to the club.
> embarrassed... for some of the offerings
Welcome to the club.
I am half convinced you can build a successful cyber security business putting a box in a network that does absolutely nothing. I think there's a requirement to at least show a blinking led and have a, not necessarily patched, cable plugged in. But that's about it.
My thinking is, that if you show a cool enough interface (not connected to the box), with lots of widgets and stats, and they don't detect a hack in the time span of 2 years, you can probably walk away with pretty penny! If they do get hacked, just pretend you technically _did_ see the alert, but a junior employee on your side failed to act on it. Sack them, and then re-hire them later. They are the 'fall' person whose job is basically getting fired. Give your customer a discount and try to stay on for 2 more years!
My thinking is, that if you show a cool enough interface (not connected to the box), with lots of widgets and stats, and they don't detect a hack in the time span of 2 years, you can probably walk away with pretty penny! If they do get hacked, just pretend you technically _did_ see the alert, but a junior employee on your side failed to act on it. Sack them, and then re-hire them later. They are the 'fall' person whose job is basically getting fired. Give your customer a discount and try to stay on for 2 more years!
You pretty much can:
> Case in point: the Air Gap. Levy set up a website showcasing a magic amulet of his own creation. Like many cyber defences, his piece of hardware promised to defend against all known and unknown viruses, and stop zero day exploits. His product? An empty box with a blue blinking light on it. Levy had to take his website offline when he started getting sales enquiries by email.
https://www.wired.co.uk/article/ian-levy-national-centre-cyb...
> Case in point: the Air Gap. Levy set up a website showcasing a magic amulet of his own creation. Like many cyber defences, his piece of hardware promised to defend against all known and unknown viruses, and stop zero day exploits. His product? An empty box with a blue blinking light on it. Levy had to take his website offline when he started getting sales enquiries by email.
https://www.wired.co.uk/article/ian-levy-national-centre-cyb...
That's pretty much what I've experienced myself at one point.
We had a client sending extremely sensitive data around by email. One day I was told we should all relax, the problem had been solved. You see he'd been sold a PGP hardware appliance.
As the person running the mail system, I could attest that mail wasn't flowing through it. It was literally in a rack. I don't even think it was given an IP address on their actual network. Multiple auditors came in to review the safety of the sensitive data that we had. They were all shown pictures of the rack with the PGP appliance in it, and that always was considered sufficient.
We had a client sending extremely sensitive data around by email. One day I was told we should all relax, the problem had been solved. You see he'd been sold a PGP hardware appliance.
As the person running the mail system, I could attest that mail wasn't flowing through it. It was literally in a rack. I don't even think it was given an IP address on their actual network. Multiple auditors came in to review the safety of the sensitive data that we had. They were all shown pictures of the rack with the PGP appliance in it, and that always was considered sufficient.
This reminds me a lot of audiophile woo as well. It seems like this sort of grift could be applied to just about anything where the technology is indistinguishable from magic.
Got any examples of audiophile scams?
This comes to mind
http://audiophile.rocks/index.html
http://audiophile.rocks/index.html
I once saw a spray for CDs that would "absorb stray laser light" or something like that. The bit that gets me is that according to the instructions you're supposed to spray it only on the label side of the disc.
Gold plated audio cables come to mind.
I'm sure you would make money, though the law might catch up with you eventually. A similar tale:
https://www.bbc.com/news/uk-29459896
https://www.bbc.com/news/uk-29459896
If the WAF blocks any request containing "{", that would be fairly safe, right?
Yes, some attacks could still get through (e.g. a backend that receives requests base64-encoded), but that's the case generally with WAFs I think.
Yes, some attacks could still get through (e.g. a backend that receives requests base64-encoded), but that's the case generally with WAFs I think.
Thag would also mean blocking any apis that use json. This would not end well in the short term, or long term.
You could probably write a more advanced check than just "{" that would let most JSON through while still blocking the attack.
The devil is in the details. As a random example, you might have a process where all the pre-WAF requests (or explicitly the requests blocked by WAF) get forwarded to a log analysis system that itself uses log4j and is vulnerable, allowing the attacker to gain RCE in your monitoring infrastructure.
Also, blocking any request containing "{" is tricky - like, that's so generic that it's time-consuming to verify that it won't break anything, and it's very likely that JSON is used somewhere in that application traffic, so you can't simply do that.
Also, blocking any request containing "{" is tricky - like, that's so generic that it's time-consuming to verify that it won't break anything, and it's very likely that JSON is used somewhere in that application traffic, so you can't simply do that.
You could also have a log analysis system that displays logs in the browser and has an XSS vulnerability. So that bypass isn't unique to log4j.
> JSON
You could probably write a more advanced check than just "{" that would let most JSON through while still blocking the attack.
> JSON
You could probably write a more advanced check than just "{" that would let most JSON through while still blocking the attack.
Cloudflare did something along these lines:
https://blog.cloudflare.com/how-cloudflare-security-responde...
https://blog.cloudflare.com/how-cloudflare-security-responde...
There's a lot of talk now about a number of things including web application firewalls and how we must fund open source and I hope something good comes out of it.
However, the thing that I wonder is why we still are in a position where every single application and every single dependency can attempt to load dlls (ref npm hacks lately) or reach out to the network?
Why is Deno the only one who seems to have a good solution for this, and why doesn't Deno have more traction?
I know one can do a lot more using outbound firewalls and selinux but after seeing how brilliantly Deno solves this I wonder why not every program lets me do this.
However, the thing that I wonder is why we still are in a position where every single application and every single dependency can attempt to load dlls (ref npm hacks lately) or reach out to the network?
Why is Deno the only one who seems to have a good solution for this, and why doesn't Deno have more traction?
I know one can do a lot more using outbound firewalls and selinux but after seeing how brilliantly Deno solves this I wonder why not every program lets me do this.
The JVM running with a security policy can do this.
It's pretty rarely done though.
It's pretty rarely done though.
The idea of having different declarative security realms is fine but it's not what the Java Security Manager is.
The Java Security Manager is an API that allows to intercept and run codes, so devs use it as a Trojan Horse to patch code instead of fixing the root of the issue.
The Java Security Manager should die.
The Java Security Manager is an API that allows to intercept and run codes, so devs use it as a Trojan Horse to patch code instead of fixing the root of the issue.
The Java Security Manager should die.
> The idea of having different declarative security realms is fine but it's not what the Java Security Manager is.
But.. it is?
The JVM tracks where bytecode was loaded from, and then you can define a policy to limit what that code can do.
Here's an example giving read-only access to /etc
https://access.redhat.com/documentation/en-us/red_hat_jboss_...
But.. it is?
The JVM tracks where bytecode was loaded from, and then you can define a policy to limit what that code can do.
Here's an example giving read-only access to /etc
https://access.redhat.com/documentation/en-us/red_hat_jboss_...
Good news, it is going to die.
https://openjdk.java.net/jeps/411
https://openjdk.java.net/jeps/411
> It's pretty rarely done though.
Exactly. That is why I like the Deno solution even as a long time Java user: it is on by default and so simple that there is no reason to disable it.
Exactly. That is why I like the Deno solution even as a long time Java user: it is on by default and so simple that there is no reason to disable it.
I would like to use Deno but I am not in a position of power even if I was, it's not always easy to switch, especially with a full backlog.
Anyway, how does Deno solve this?
Anyway, how does Deno solve this?
I'll start with the question:
Deno solves this by making you specify on the command line or in a config file what an application should have access to, for instance for a simple web application you can specify that it only has read acesss to one folder of static files, write access to the log folder and can only connect to the postgres server. It is also very simple, see:
https://deno.land/[email protected]/getting_started/permissions
As for the first part, I'm in the lucky position where nobody can just rewrite everything to todays flavor of js, but where we can decide in the team to test out new technology when we have a chance to, for example on a new small project. There's another guy on the team that is enthusiastic about it too and I guess he'll throw together a demo soon, then we'll discuss it.
Deno solves this by making you specify on the command line or in a config file what an application should have access to, for instance for a simple web application you can specify that it only has read acesss to one folder of static files, write access to the log folder and can only connect to the postgres server. It is also very simple, see:
https://deno.land/[email protected]/getting_started/permissions
As for the first part, I'm in the lucky position where nobody can just rewrite everything to todays flavor of js, but where we can decide in the team to test out new technology when we have a chance to, for example on a new small project. There's another guy on the team that is enthusiastic about it too and I guess he'll throw together a demo soon, then we'll discuss it.
Thanks for your time. I am about to start a new personal project and your post, combined with some curiosity, is all I needed to hear. I've reached a point where doing the same thing with the same tech is making me feel uneasy.
Full disclosure: I've only toyed with it so far.
That said: What I have seen seems extremely interesting, both the audited standard library and the permission model, the built in support for TypeScript, single binary runtime and the avoidance of a single package repo.
Good luck with your next personal project. I guess now might be a perfect time to invest in learning Deno. It might not take of, but if it doesn't it will probably make me a little sad when I realize it, and I am a Java man (although I enjoy a number of other languages as well).
That said: What I have seen seems extremely interesting, both the audited standard library and the permission model, the built in support for TypeScript, single binary runtime and the avoidance of a single package repo.
Good luck with your next personal project. I guess now might be a perfect time to invest in learning Deno. It might not take of, but if it doesn't it will probably make me a little sad when I realize it, and I am a Java man (although I enjoy a number of other languages as well).
I quite liked the exploit that made a Minecraft server serve Doom to its clients: https://twitter.com/gegy1000/status/1469714451716882434?s=21
[deleted]
This is pretty awful. Seems easy to fuzz every input form and API param to see which websites are vulnerable just by seeing which sites get a response. Once a site is found to be vulnerable, a malicious actor can try to funnel all logs to an external server, add a remote shell, and potentially scan the production network of whatever was running log4j. Once in the internal network, they can again scan for log4j exploits. Too many groups blanket whitelist cloud IPs like AWS lambda. Seems like there will be a cascade of experian-level data leaks coming. Even if things are somewhat locked down we've seen time and time again, there's internal sprawl where access to an internal bucket or git repo, or an escalation in the CI/CD pipelines leads to full access, then data dumps/leaks.
What do you think is happening? I’ve already found a few requests for / with a referrer that starts with ${jdni:
The french announcement mentions this:
"In general, it seems that the use of a Java runtime environment in version 8u121 or later makes it possible to guard against the main attack vector mentioned by the researchers behind the discovery."
Is it correct?
https://www.cert.ssi.gouv.fr/alerte/CERTFR-2021-ALE-022/
Is it correct?
https://www.cert.ssi.gouv.fr/alerte/CERTFR-2021-ALE-022/
As has been commented several times on other threads here on HN, a new enough Java only protects against one kind of exploit (directly loading arbitrary bytecode) but not others (serialization tricks to execute arbitrary function calls, or data exfiltration).
1. Step up and shoulder part of the finding for maintainers 2. Close their double quotes on press releases