Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent(twitter.com)
twitter.com
Cisco Fixes RV320/RV325 Vulnerability by Banning “curl” in User-Agent
https://twitter.com/RedTeamPT/status/1110843396657238016
321 comments
Of all the security post-mortems I’ve ever wanted to read, it’s sad I’ll probably never get to read this one and its tale of how a team of well-paid comfortable engineers got together and decided this patch was a good idea.
Having been involved in meetings where "stop ship" was the phrase of the day, I'd bet money that the following at least vaguely resembles a real conversation:
Engineer Alice: We should really fix this properly.
Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?
Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.
Manager: And we can just prevent it with an extra regex rule in the front end code?
Engineer Alice: ... Yes. irrepressibly existential sigh
Manager: How much QA effort to call this good?
QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours.
Manager: And for the proper fix?
QA Engineer: Oh, hell, at least a week of stress testing to really be sure.
Manager: Add the regex rule.
Engineer Alice: We should really fix this properly.
Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue?
Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk.
Manager: And we can just prevent it with an extra regex rule in the front end code?
Engineer Alice: ... Yes. irrepressibly existential sigh
Manager: How much QA effort to call this good?
QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours.
Manager: And for the proper fix?
QA Engineer: Oh, hell, at least a week of stress testing to really be sure.
Manager: Add the regex rule.
On HN its always big bad management who is the cause of every security problem or shoddy piece of engineering. If only that pesky management would screw off then we could do things "properly".
You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
"Engineer" conveys image of middle class white-collar job with relatively high status, good education and responsibilities. That word now used for everyone doing programming related jobs inside office space for no good reason.
I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.
https://en.wikipedia.org/wiki/Grey-collar
https://en.wikipedia.org/wiki/Technician
I think the word "tehnician" should be used to describe most grey-collar ICT jobs, including most programmers. Their responsibility and scope of their work is limited. Many programming jobs are closer to blue collar factory level mechanic than engineer.
https://en.wikipedia.org/wiki/Grey-collar
https://en.wikipedia.org/wiki/Technician
Yup. The reality of this industry is that most programmers are closer to the handymen you call to fix your faucet or lay floor tiles than to actual engineers. And we all have first-hand or second-hand horror stories related to handymen.
I've come to the conclusion recently that we're all just barbers, bloodletting and burring holes. Until we have proper certifiable and reproducible competency, we'll never become surgeons.
Don't you need a license to be a barber?
In modern times yes, here's some interesting reading on the guilds licensing originated from - https://en.wikipedia.org/wiki/Worshipful_Company_of_Barbers
And yet the US has consistently been in the vanguard of software development worldwide. Do we need surgeons? Do you need a four year postgraduate degree and three more years of apprenticeship before slinging together a web app? Certainly the industry has its problems, and some applications demand more rigor, but I don’t see this as much of a general solution.
The market didn't demand a high survival rate or efficacy for barbering either, but do you think that the (conscious or unconscious) that software engineers are some sort of magical wizards will last forever?
There may be explanations beyond wizardry - e.g. maybe barbers were historically often self-employed, small business owners who realized they stood to gain from creating barriers to entry such as licensing requirements.
Thankfully there are countries where Engineer is still a proper word, not something that you are allowed to call yourself after a 6 month bootcamp.
Yup, in the UK it bugs me when I keep meeting people who introduce themselves as an 'engineer'. When I ask them if they do mechanical or civil engineering, then I usually get to say "ah, so you're a programmer, just like me".
One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.
One thing I did sometimes like about being in Austria is the obsession with academic titles they have there... if somebody is a qualified engineer, they invariably use "Ing" as a title in place of "Mr" or "Ms"; people with two doctorates (not uncommon) really do call themselves Dr Dr So-and-so.
Austrian here. "Ing" (Ingenieur, so basically Engineer) is a protected title (not an academic one, though) but to be honest, it does not signal status or qualification to me. The actual academic titles are worth more.
Still doesn’t really mean anything. Due to a loophole in major assignment I can legally call myself Ing., but what I’ve really studied is more or less computer science.
What is qualified in Austria? In the USA, in most states, you must be licensed to have engineering in your business name. Your name is usually suffixed with PE.
To use the "Ing" title you have to have a certain education and a few years of practice. To be honest, it doesn't mean much.
But in the USA you can refer to yourself generically as a Software Engineer without any formal education.
Yes. I don’t think there is even a PE licensing path for software, so even with a CS degree you could never become a licensed engineer. You have to work under a PE for 5 years, then take the PE exam, then CE credits.
I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.
I have an MSEE, but can’t put engineer in my title. Though Ing. sounds kind of cool.
There actually was a PE for Software Engineering, but it is being discontinued, because almost no one took it.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
Wow, didn’t know they had one. I wonder how they handle the licensing as there would be no one to complete the 5 year understudy; chicken and the egg.
It bothers me to no end when I hear programmers here in Ontario call themselves engineers when they neither graduated from an engineering program nor completed their PEO exam. It's very much illegal, and the PEO _will_ go after companies that allow their employees to title themselves as engineers.
How dare you leave out electrical engineering!
That said, there's many variants that are perfect valid: network engineer, sales engineer, systems engineer, etc.
And most states in the US too. Just rarely enforced. Only a licensed PE can do business as an “Engineer”. (For example have a company with the word Engineer or Engineering in the name).
I believe the city of Portland OR recently sued a Danish man for "practicing engineering without a license". They were upset at him for generating strong proof of shortened yellow light times at intersections with red light cameras. He had a degree (and I believe a certificate) from Europe, but it wasn't recognized by the Authorities.
Protected term in Canada as well, but I don't know the last time anyone that was used against in any level of court. During my education there was a case study of some guy digging basements calling himself an engineer when he wasn't but that's about it.
And still others have, though it may not be appreciated in all industries, _Chartered_ Engineers.
And how many 'engineers' would tack on 1000x too much complexity to justify their 'engineering' effort?
software in many cases is more art than science.
software in many cases is more art than science.
I think this talk is relevant https://youtu.be/lLnsi522LS8
Please bitch "associate professional"
If you're a company the size of Cisco, and you have two widely publicised vulnerabilities like this, then this sort of failure is _always_ management's responsibility.
That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.
That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers, but it can take other shapes too. It might be poor oversight and QA processes, or a failure to hire and/or assign competent engineers for the task.
> That responsibility might come in the form of "big bad management" chasing the cheapest possible "fix" against the advice of the the hero engineers
This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.
This is the narrative that always shows up in HN. Big bad management vs the hero engineer. It makes us feel good about ourselves while also absolving us of all responsibility. Its a bullshit narrative.
You live in a nice world, because usually the actual power (and duty) of a dev team member is to advise, not to agree or refuse.
You can always quit. Or force them to fire you for refusing to implement a non fix.
In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.
Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix.
Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
At a large organisation I worked at years ago we used to call it Coder Shopping - when a manager would do the rounds trying to find someone who would say "yes" in a situation like this.
Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...
Invariably there would be someone who would, and if you were the person who had initially refused you probably wouldn't even be aware that it had happened until the bug reports came in...
If there were actual accountability, software engineers would have a much better lever against management. For some software this is case. If you write safety critical software you can be held personally responsible for accidents. In those industries engineer pushback is much more effective.
My background is safety-critical and -- speaking from first-hand experience -- it was truly incredible how much push-back we'd get when raising safety issues.
System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!
In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".
System tick timer rolled over after 2^32 1ms clock ticks (a little over seven weeks) and the software mishandled it by rebooting and losing control of the process. It wasn't until QA got involved (it turned out they saw it on a long-term test) that we were cleared to actually fix it. It was a two-character change in an inline function!
In some places "cover it up, don't discuss it in email" is the natural response. It really depends on the management and the organisational culture. You can have management who support the engineering team, or you can have management who micromanage and overrule and end up turning a two-day hotfix cycle into ten months of "try soldering a resistor to every single pin on the PCB".
Ha ha, a resistor on every pin is a signal integrity hack.
Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.
You might be surprised, but if you move to Europe, it is actually so! Move to Berlin of Frankfurt. There is always work for Software Developer here, it is hard not to get one and you get spammed by recruiters all the time.
You don't have to take shit from management, you can just go elsewhere. Everything is also very near.
> Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.
Quitting assumes the possibility of finding a job after spending 4 months doing leetcode 6 hours a day and not having people that depend on you.
Quitting assumes the possibility of finding a job after spending 4 months doing leetcode 6 hours a day and not having people that depend on you.
I like your spirit, but there's another conclusion we can draw: very few of us outside of a few special areas in embedded are "engineers". We tinker with bits and bytes to make business systems. Sometimes we bump up against some general engineering fundamentals. But we're just technicians meeting business goals.
Yep, and the same can be said for “real engineers”. Most EEs, civil engineers, etc work on boring stuff as well with negligible risk of doing any harm.
You suggest it was just one lone engineer being incompetent. Then you respond to downvotes with the non-sequitur "take some pride in your work" - are you suggesting pride is a fix for incompetence? What?
admax88q(3)
Absolutely: Cisco made this happen, not Engineer Alice. If your boss asks you to do something within my ethical boundaries and you have 5 tired co-workers in favour of shipping it, eventually you'll break and say "fine". It doesn't make you responsible or the one that pulled the trigger.
The fact that this fix made it to prod tells you that something is deeply deeply broken with cisco’s processes that goes well beyond any individual’s responsibility.
Exactly. It's not like somebody just did this like in a three people startup. There are code reviews and quality management and many more, and this lousy "fix" has passed all those stages and instances. If nobody declined this, there's a serious problem within Cisco.
na85(2)
We can "No True Scotsman" this all day long.
It is bad management to hire incompetent engineers.
It is bad management to fire competent engineers that push back.
It is bad management to make employees feel as though they would be fired for standing up in any way to management.
Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.
We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.
The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.
Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.
It is bad management to hire incompetent engineers.
It is bad management to fire competent engineers that push back.
It is bad management to make employees feel as though they would be fired for standing up in any way to management.
Take away the rhetorical shortcuts, and responsibility still always falls at the feet of the ones that have the power to make the decision. As long as engineers are hierarchically bound to obey managers upon pain of sudden and unplanned unemployment, the managers are ultimately responsible for everything the company does. If the company does not have tenured engineers that have stop work authority, the managers are fully to blame. And if they do, management still might be partially at fault.
We have this headcanon where a competent engineer could stand up, sweep away the slide deck on screen with a wave of the hand, and offer ironclad mathematical proof that the quick fix is no fix at all, and that the long, painful, but correct fix is what is Right for the Company (tm). The manager would then do the slow clap and tell that engineer to fix it right this time. It is a fantasy that is a great comfort to those of us doing maintenance work on a massive crock of shit, making the payments on technical debt that we are expressly forbidden from making any attempts to retire. But it is a fantasy. A real manager would cut the engineer off in mid-proof and say that the quick fix is happening, and that would be the end of the story.
The only way for an engineer that is "at will" and without a contract to push back is by falling on their own sword. They can quit, at great cost to themselves, to create a very minor inconvenience to the company, who will end up doing what it wanted anyway.
Being in this situation myself, you're damned right I have no responsibility for the crap code that goes out, because I have almost no autonomy in the manner in which I do my job. I cannot work outside the scope of the maintenance tickets I am assigned. And my company rents me out by the hour to the customer, so it has negative incentive to allow me to build up infrastructural capital that would allow me to do the same work in less time. I will blame management all day long, and then a bit more on evenings and weekends.
There are plenty of poor engineers, but this “fix” has to go through a process defined and enforced by managers. In a good process, multiple engineers would review the code, and if the office culture is good, then objections will be brought up and listened to, and if a fix is insufficient, management will understand and assign resources to make it right. None of those things depend on any one engineer or manager. They’re a culture issue. But the only people with the power to fix a broken culture is management. Engineer Alice doesn’t have that power herself.
In large orgs there would also be some kind of security team which would have input on the CVEs. Cisco does have them. It sounds like they were not involved at all here for some reason.
The only way, and I mean the only way, this gets blocked is because a manager (not a dev) assesses the risk of improper fixing to be higher than the reward of not doing a shit ton of work justifies.
A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.
It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.
A significant portion of that is making sure the manager is fully cognisant of what the quick fix means. Alice's job is to ensure that happens. The managers job is to advocate to his bosses that this has to be done and is a net win to do it properly.
It is as likely manager person failed in advocacy as it is she failed in information transfer, but there are plenty of gradients here.
And I might add: especially at big companies where being yes men is a higher valued characteristic than being technical
Oh, so much this.
"I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
"I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
It really is amazing the kind of mental gymnastics HN commenters will display in order to shift the blame to everybody else, isn't it?
Yes, but this is software engineering. Where there is no such thing as malpractice, and where managers can not be prosecuted for criminal negligence even when their overriding of the engineers professional position gets dozens of people killed. In this delightful world, managers rule the roost and engineers do not have the ability to refuse to do anything.
If management let this important security fix get executed and deployed without a competent engineer in the loop, that's a horrible mistake in itself, but it's far less likely than management choosing this alternative based on accurate information from engineers.
[deleted]
Big bad management gets to keep their jobs when they make bad product decisions, so let's call it a wash.
If you read the parent post and think "bad management" I don't think the message was clear.
Stereotypes are valid first order approximations.
Who hired the incompetent engineers?
I've been Engineer Alice before (not with a security issue, but a pretty bad systems issue). In case anyone is ever in that situation, here are some more productive replacements for that existential sigh (in that they sometimes work). Choose the one most suited to manager's biases. The cliches are important: think of it as giving manager an easy way to explain it to their manager.
> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.
> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.
> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
> Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle.
> Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run.
> Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
No! The answer is an unequivocal “No“, without any „but“s or anything like that. The fix does not fix the problem, it is not even a fix, just a wrong code change that sets out to do something but does not achieve it.
Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
In this specific case (I have never encountered a proposal quite this bad), the correct answer is probably "Give me the proof of concept code and about ten minutes, and I'll give you a version that works despite the patch." You're right about that. "No" is likely to bog down the discussion with "but Bob says it will work", and it's better to skip to the end.
I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
Sometimes you need an interim solution. A decent organization will be able to manage them by replacing them with actual resolution.
Cisco has a bad track record in that regard and software in general.
My employer has reached a settlement with Cisco after buying their "firepower" solution which has been plagued with basic software and usability bugs.
Cisco has a bad track record in that regard and software in general.
My employer has reached a settlement with Cisco after buying their "firepower" solution which has been plagued with basic software and usability bugs.
Ten minutes? How long does it take to type a user agent string into the curl command line?
In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)
In my case, a user agent string pretending to be Firefox is already part of my .curlrc, so it's no time at all, just grab the PoC code and fire away. (I've got commented out user-agent lines for a bunch of other browsers that I can uncomment as the need or mood strikes me.)
I'm basically assuming five minutes to read the script, negligible time to change it, and another five to make sure it worked.
You must live in a very nice, ideal world, where simply saying "no, that's not the right way to do it" will convince managers to ignore the pressures placed on them to, at times, value speed over correctness.
What they are saying is that this isn't just "the wrong way to do it". It doesn't actually fix the problem it's supposed to fix, therefore it's actually wrong to say it's a functioning patch.
Of course, but their boss doesn’t know that (or care about it), he just wants to see the fixed bug graph line approach the total bug graphline.
It’ll be totally explainable as engineer failure if something somehow doesn’t work.
It’ll be totally explainable as engineer failure if something somehow doesn’t work.
Apparently I do? Not sure what you want to hear here, but management does listen to me and my engineering peers. Sometimes things are a bit gray and fuzzy, but in this case here where they definitely are not, I am absolutely certain that there would be no overriding of our arguments.
That's awesome, actually. For me, it's not as bad as Cisco seems, but there are definitely times I get overruled and have to take shortcuts that I'm not always comfortable taking.
Indeed, and what's worse is that many managers[1] will stop listening after "Yes, ...".
[1] The incompetent ones, of which there are many.
[1] The incompetent ones, of which there are many.
How about:
No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.
No, it won't actually fix anything, it'll only look like it did for most trivial cases, and 6 hours after releasing it our company will be on the frontpage of Reddit.
I actually said something like that during a meeting. "Yes, you can do it that way, but if someone finds out we will be in the news." This argument worked surprisingly well.
Or you can just do the minimum just like the manager and cruise on the ship until it sinks while surveying for another ship to jump on at the first opportunity.
> Manager: And we can just prevent it with an extra regex rule in the front end code?
> Engineer Alice: ... Yes. irrepressibly existential sigh
The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
I have to agree that the answer is no.
curl -A "anything but Curl" x.x.x.x
the problem has not been fixed by a user agent string.
curl -A "anything but Curl" x.x.x.x
the problem has not been fixed by a user agent string.
Engineer Alice is the only person incompetent at their job in this conversation. If "irrepressibly existential sigh" is how you argue security with your bosses then maybe you're not senior enough to be in meetings like these.
The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
If you answer with the existential sigh, you’ve had at least 20 conversations with a similar outcome before. Nobody arrives at a new company that jaded.
I see your point and I suspect that that's what the author meant.
At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
Ah, yes, I see how that is another valid interpretation.
I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’.
Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P
I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’.
Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P
> The Manager and QA Engineers here depend on the expertise of the engineers.
Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
In similar situations I
1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.
2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient.
2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
This is about all you can do. But you're right. You have to explain it and document it: "It's trivial to change User-Agent, and any decent hacker would probably not be using 'curl' as their User-Agent. We can look at logs from past events to validate this assumption."
Bob is either a brown-noser or a complete turncoat. Don't be like Bob.
Bob is probably an engineer with twenty years experience, who's been thoroughly ground down by management and peer power-plays and has realised that the best solution is to keep your head below the parapets!
Lol, this is too true. Though I think I learned 80% of this in my first couple months at my first real full time job
Ironically, this is the change most likely to break things for customers that use curl for automation.
Ouch, that hits close to home. It's like you've sat in on all the same meetings I have, word for word.
Manager: Bob get to work on the quick fix that will mitigate some of the attacks based on the published exploit to go out within a day, Alice get to work on the correct fix to address the underlying issue to go out in a few weeks when it's fully tested and ready.
Manager: Alice, yeah, I know I said we'd carve out time this sprint for you to work on the long-term fix. But this other hot feature request came in, and there's potentially a lot of money riding on getting it done quickly. So can you put off the long-term fix, maybe a week? Two tops. You can just put the ticket.. yeah that one.. back on the backlog. There you go. Thanks so much!
Then someone realises one of their enterprise clients is using curl for a critical system, and said patch has completely broken it. Or more likely, they realise after customer/company calls complaining about how their systems are now busted...
This reminds me of the old BOFH posts from days past.
The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.
The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.
Wow, You got this one!
>how a team of well-paid comfortable engineers got together and decided this patch was a good idea
Test-driven development
Test-driven development
Only if you are really bad at it. Really, really bad.
It's completely unrelated to any development methodology, this is someone unaware of basic HTTP protocol semantics.
So just bad development, period.
It's completely unrelated to any development methodology, this is someone unaware of basic HTTP protocol semantics.
So just bad development, period.
It looks like this is a low end Cisco device, released over 5 years ago. Patches are probably handled by underpaid interns, not well-paid engineers.
i'll raise you a case where a team of very well paid senior engineers/architects and PMs dismissed a remote execution vulnerability down to a very low "some next release" priority on the grounds that "notepad.exe doesn't seem to do any damage" - as you may have guessed the vulnerability PoC used notepad.exe . After seeing that with my own eyes, this Cisco curl is just "meh" for me :)
Every time I see an examle of this I think "holy crap we're all going to die". The bus drivers are blind.
I've had some visibility to big networking equipment manufacturers... the security or even stability on their web interfaces is terrible. It's like a strange rule of thumb for all the networking equipment manufacturers. If you have choice, disable the web interface.
> how a team of well-paid comfortable engineers got together and decided this patch was a good idea.
That's assuming that the well-paid and comfortable engineers decided on and implemented the patch.
That's assuming that the well-paid and comfortable engineers decided on and implemented the patch.
[deleted]
I don't see what the fuss is about. This is an effective mitigation, given that software can't just arbitrarily lie about its user agent.
I guess the biggest problem is that the mitigation isn't future-proof. Someday, there might be other command-line tools besides cURL that can be used for this kind of attack.
A better fix would be to just drop all incoming IPv4 packets that have the "evil" bit set. (https://www.ietf.org/rfc/rfc3514.txt) Extending this protection to IPv6 is left as an exercise for the reader.
A better fix would be to just drop all incoming IPv4 packets that have the "evil" bit set. (https://www.ietf.org/rfc/rfc3514.txt) Extending this protection to IPv6 is left as an exercise for the reader.
Ah, but what about packets that aren't evil, just gullible and manipulated by some third party?
Well if you’d read the RFC it’s obvious it’s your routers responsibility to flag those bits as evil. And if you’re using heuristic intrusion detection systems you need to do flag each bit as evil 50% of the time. It’s good we have such amazing protections in place.
Then add a "tainted by evil" flag to the ipv7 spec, or better, just add a "pure" flag to future ip spec.
I think I'd almost prefer an "alignment" flag, so we don't mischaracterize all the Chaotic Neutral packets.
It's not like curl can't easily spoof a user agent either.
We can claim the LSB of the sequence number.
I think you should have added a /s tag... some people don't get jokes unless you hit them over the head with it.
I will never stoop so low as to telegraph my own joke.
I couldn't agree more. The joke construction was a bit weak though, when you miss a good part of the audience. Add something like, "I mean, could you imagine the chaos it would cause if IE told websites it was really Mozilla?" and you demonstrate mastery of the subject matter, which should be enough to let other experts know you were facetious rather than ignorant. Unless you have timing issues... or need the comedian's plausible deniability.
I don't find that as funny, I think making the sarcasm more obvious makes the joke less funny as well. I prefer to take my chances than to go overboard and neuter the joke.
I like the original more without your embellishment. The addition sounds like every attempt at follow up humor on reddit.
Yeah but it's not Reddit, it's HN. You have to know your audience, read the crowd. On Reddit it's 90% sarcasm so there's no fixing it. Here it's the reverse and people take things seriously without a tell. You just have to bury the tell in another joke or it will ruin the funny.
"given that software can't [ridiculous thing no one has ever claimed]" is plenty for a tell.
Just the fact that this “fix” exists tells you some people believe differently.
If you can’t discern the joke, the joke isn’t for you to begin with; so it doesn’t matter.
Sir, your assessment of this particular joke is excellent. We should start a joke approval committee to avoid further confusion.
Yes, very well, as first action of the Joke Approval Committee, I propose that this august body instantiate the Recursed Approval Subcommittee, whose role will be to instantiate the Recursed Approval Subcommittee.
That's a perfect example you should post to JAC before. It would not pass, please never post this "joke" again.
Welcome on board :)
Welcome on board :)
As soon as the subcommittees hit their break condition (0), they'll break for coffee and donuts, and I can submit my "joke" to the JAC. And I don't make a habit of repeatedly posting the same joke, but I cannot make the promise in this case.
It puts the motion in the basket.
In textualform, sarcasm and ignorance look the same. otoh, it can sometime be quite funny to see responses from people that take it seriously...
I've never posted sarcasm online without at lease one person (that guy) who takes it seriously and decides to reply. But I think it's worth it because the people who get it will get it. If a small group doesn't then who cares (if everyone doesn't then you did a bad job constructing a joke).
Exactly. Hacking your client to use another user-agent or another IP address is illegal under CFAA. See Craigslist Inc. v. 3Taps Inc. etc...
So when I check desktop site in chrome I'm hacking?
Of course! So is right clicking and selecting inspect element to modify the page!
That's why super secure websites block right-click events on their webpage. Makes them unhackable!
I would call it hacking.. you sometimes run into those websites with 50 different opacity bits for all their web app things, and you don’t want to deal with all that rubbish so you set a “opacity: 1 !important” and the whole website looks off but it’s finally useable. I’d call that a hack.
Or also when you run into those websites designed on a $2000 screen that thinks it’s tres moderne to use 808080 text, so you “hack” it into readable text.
I think those are all happy little hacks :-)
Or also when you run into those websites designed on a $2000 screen that thinks it’s tres moderne to use 808080 text, so you “hack” it into readable text.
I think those are all happy little hacks :-)
Actually you can use -A to change the user agent in Curl itself. For example:
Most programmers should be able to write a small script using Python’s urllib or urllib2 to do a basic PUT or GET request with any User Agent. Heck, just type “change user agent python urllib” into google, click on the first stack overflow link, and copy/paste the answer. You can use Go, ruby, java, nodeJS, or PHP instead as well.
Edit: Maybe your comment was sarcasm :)
curl -A "YourNewUserAgentString” http://url.com
Most programmers should be able to write a small script using Python’s urllib or urllib2 to do a basic PUT or GET request with any User Agent. Heck, just type “change user agent python urllib” into google, click on the first stack overflow link, and copy/paste the answer. You can use Go, ruby, java, nodeJS, or PHP instead as well.
Edit: Maybe your comment was sarcasm :)
> Edit: Maybe your comment was sarcasm :)
I still appreciate you putting in the effort to explain. Have an upvote!
I still appreciate you putting in the effort to explain. Have an upvote!
Exactly, once they roll out the patch with a boolean OR in the conditional as '$http_user_agent ~* "wget"', it'll pretty much be hack proof!
It does seem so obviously simple to sidestep that it makes me wonder if they pushed it because they had an active attack they believed it would mitigate. Were they concerned that the attack was already baked into some automated script in the wild and this could perhaps at least trip up the script long enough for them to engineer a real fix?
Scripted attacks can of course be modified too, but it still takes more than zero human effort to do it.
Scripted attacks can of course be modified too, but it still takes more than zero human effort to do it.
Well, they can, but first they have to flip the RFC3514 "Evil" bit.
Curl can, easily. Curl is not just a client to get a file and pipe it to other programs, it's a diagnostic tool, and it has all sorts of parameters to help it simmulate all kinds of interaction with a webserver, including the -A parameter to change the user-agent, this is why this 'fix' is so stupid.
This is quite funny
I am not laughing one bit let me tell you. Honesty in user agents is one of the last few defenses we have against hackers running rampant on the internet and turning it into some lawless wasteland where people don't even answer truthfully about their A/S/L.
[deleted]
They might be able to fix it by making the only valid user agent “not-curl-honest”.
[deleted]
curl -A "iswearimnotcurl" <exploit>
libcurl is open source, I would imagine it'd not be difficult to make it "lie."
Someone who also did not get the joke is going to read your comment, and suggest that libcurl being closed source somehow fixes the problem.
(Hint: the curl command already has a --user-agent parameter which allows the user to set the user agent string.)
Eh? Software can do exactly that.
How? Do you think developers would be so crazy as to add some sort of "--user-agent" option to software like curl that would cause arbitrary strings to be presented as the user agent? Why would somebody write software to do this, just go on the internet and tell lies?
I thought all this security nonsense ended in the 90s when they introduced the IS-MALICIOUS-REQUEST header. It must be, because I haven't seen any web traffic declaring itself malicious since then.
I believe most devices implemented now 11 years old RFC 3514. So evil packets and non-evil packets can be easily distinguished.
Even without --user-agent, you can still pipe 'echo -e "GET ...\rnHost: ... "' through nc, or even telnet.
Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".
Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".
> Therefore, we obviously need to patch echo (and all echo shell builtins) to refuse to output strings containing "GET", "POST", "HTTP", or "Host:".
Unfortunately it is also possible to do this using file redirection or to write a new program that will make a TCP connection and send arbitrary data through it, making it necessary to do the same for all editors, compilers and interpreters.
That sounds like a lot of work though (there are many such programs), so maybe we ought to just patch the kernel to prohibit any of those strings from being written to a file descriptor. There can't be that many false positives.
Unfortunately it is also possible to do this using file redirection or to write a new program that will make a TCP connection and send arbitrary data through it, making it necessary to do the same for all editors, compilers and interpreters.
That sounds like a lot of work though (there are many such programs), so maybe we ought to just patch the kernel to prohibit any of those strings from being written to a file descriptor. There can't be that many false positives.
Why stop there? The user can easily modify the kernel to disable such prevention measures for malicious usage.
We must have a regulation to require a hardware level detection and prevention features of curl which all hardware vendors must follow.
We must have a regulation to require a hardware level detection and prevention features of curl which all hardware vendors must follow.
oh god stop some politician might see this and take you for being srs
But users have fingers, which can be used to disable or subvert said "features". Thus, those fingers are just gonna have to go. They're a security risk, and security risks are against the regulations.
Ooh boy I'm imagining some bureaucratic version of an operating system that uses this sort of security up and down the entire stack like some rube Goldbergian contraption to whack a mole a particular vulnerability. Like an operating system designed by doctor Seuss.
While that's probably technically feasible, that sounds like something only a terrorist would do anyway. All the more reason to ban the entire program, I say!
[deleted]
Only software developers with no honour would ever do that.
Pretty sure that was sarcasm haha
[deleted]
And it is also very good that regexes are simple, unambiguous, easy to debug and their engines are heavily scrutinized that no code execution/undefined behavior is possible.
Cisco is crumbling under its own weight.
This is a symptom of the rot in their management, and probably also a sign that they have hired too many incompetents.
It probably also is a sign of the current age. After the recovery from the IT-bubble programming got really hot. Thus: Too many of the new programmers wants to be programmers because it pays well - not because they love their craft. So therefore we have a bunch of well-paid but uninterested people seeking jobs at prestigious companies.
Culture matters. I want my socially maladapt terminal junkies back plz.
This is a symptom of the rot in their management, and probably also a sign that they have hired too many incompetents.
It probably also is a sign of the current age. After the recovery from the IT-bubble programming got really hot. Thus: Too many of the new programmers wants to be programmers because it pays well - not because they love their craft. So therefore we have a bunch of well-paid but uninterested people seeking jobs at prestigious companies.
Culture matters. I want my socially maladapt terminal junkies back plz.
> I want my socially maladept neckbeards and terminal junkies back plz.
the MBA types don't like these hacker types - personality clash and whatnots.
But the MBA types control the company from above, and the hacker types don't like to do management work. The result is obvious.
the MBA types don't like these hacker types - personality clash and whatnots.
But the MBA types control the company from above, and the hacker types don't like to do management work. The result is obvious.
Some time ago I went through the list of all the major router manufacturers and rated them on 1) security, and 2) long term usability, and 3) culture.
My conclusion was that I would buy my infrastructure from Allied Telesis. It's pretty much a Japanese version of Cisco, but it's still healthy.
Ubiquity was number 2. I refrain from buying from them only because of their glossy UI.
Mikrotik was on that list. Until I saw how horrible their winbox protocol was. And their implementation of SMB.. I must assume there are still plenty of unknown RCEs there.
My conclusion was that I would buy my infrastructure from Allied Telesis. It's pretty much a Japanese version of Cisco, but it's still healthy.
Ubiquity was number 2. I refrain from buying from them only because of their glossy UI.
Mikrotik was on that list. Until I saw how horrible their winbox protocol was. And their implementation of SMB.. I must assume there are still plenty of unknown RCEs there.
https://threatpost.com/hardware-vendor-offers-backdoor-every...
At least Allied Telesis documents their backdoors :)
At least Allied Telesis documents their backdoors :)
LOL
I guess there is nothing good.. what is wrong with people :(
I guess there is nothing good.. what is wrong with people :(
Oh god...
I had several Allied Telesis switches at a previous job. The hardware was fine but resetting the configuration password was a remarkably painful process.
Can you elaborate on Mikrotik? I've only heard good things about them and am very satisfied with the one hAP I bought from them.
They have had several remote code execution vulnerabilities lately (summer 2018). While they were very quick to patch them they did not notify their customers in any way. There was nothing on their website that said anything about the urgency.
Instead of reusing functionality that exists in the router already (ssh?), the authentication for winbox is something they built themselves. It was in the winbox auth that the main security flaw was. It just looked really bad to me.
The winbox client also downloads and runs any DLL that is sent by the winbox server. The winbox client has a windows certificate so all it's code is trusted. So own the router and you get the admins workstation too.
It just feels like maybe they hired some random guy without much appreciation for security for doing winbox.
The SMB server also had a rce a while ago.
That said, I guess that if you disable winbox and stuff that should not face the internet, you are probably safe?
Too much for me though. I would not feel safe.
Instead of reusing functionality that exists in the router already (ssh?), the authentication for winbox is something they built themselves. It was in the winbox auth that the main security flaw was. It just looked really bad to me.
The winbox client also downloads and runs any DLL that is sent by the winbox server. The winbox client has a windows certificate so all it's code is trusted. So own the router and you get the admins workstation too.
It just feels like maybe they hired some random guy without much appreciation for security for doing winbox.
The SMB server also had a rce a while ago.
That said, I guess that if you disable winbox and stuff that should not face the internet, you are probably safe?
Too much for me though. I would not feel safe.
WinBox is not that simple and cannot be replaced with SSH. WinBox works on Ethernet level, so one can connect to router by MAC address and recover when IP level configuration is invalid.
disable winbox and use their webui?
also, while not perfect, what is so horrible about their winbox protocol?
also, while not perfect, what is so horrible about their winbox protocol?
Culture matters. I want my socially maladapt terminal junkies back plz.
I know some fastidiously groomed VSCode front-end hipsters whose entire bodies involuntarily tense up if you suggest a lazy technical workaround to them. It's funny and makes me feel good about the future.
But I get you; I agree completely. You do need people who really care. They're just not as easy to stereotype any more (if they ever were.)
I know some fastidiously groomed VSCode front-end hipsters whose entire bodies involuntarily tense up if you suggest a lazy technical workaround to them. It's funny and makes me feel good about the future.
But I get you; I agree completely. You do need people who really care. They're just not as easy to stereotype any more (if they ever were.)
Your first point is just blabber. Cisco is a large company and this is a $100 product that they don't really care much about. It's the type of product that is produced at the cheapest possible cost, by interns. The way this product has been developed and managed is completely intentional and not a sign of mismanagement. Come back in 3 and then 6 months and let's see what effect this has had on their revenue and market cap, and then you can start talking about mismanagement.
As a sign of the quality of the current median developer, I think you are spot on. But I don't know that it's worth griping about. That is the world in which we live ... so live in it. In fact, all the better for those of us (if I may dare to put myself in a more elite group) that are highly skilled ... we don't have to work on some me-too unimportant SOHO router, at correspondingly low me-too unimportant wages.
As a sign of the quality of the current median developer, I think you are spot on. But I don't know that it's worth griping about. That is the world in which we live ... so live in it. In fact, all the better for those of us (if I may dare to put myself in a more elite group) that are highly skilled ... we don't have to work on some me-too unimportant SOHO router, at correspondingly low me-too unimportant wages.
Bwahaha, hired too many incompetents?
They have outsourced the fuck out of their projects to uh, questionable means.
They have outsourced the fuck out of their projects to uh, questionable means.
This "fix" seriously hurts Cisco's credibility. How can you trust their products?
Perhaps they are thinking that noone gives a damn anyway after no less than five backdoors² were found in their products in 2018 alone? Just incredible.
² https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a...
² https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a...
Cisco have a loong history of things like the. I mean go back to 2005 when they actually sent people to a security conference to rip pages out of the handouts that mentioned a security vulnerability in their products (https://www.computerworld.com/article/2482483/cisco-s-blunde...)
That (and other bad security responses) have not really hurt them, so at this stage, this won't make a lot of difference, I'd expect.
That (and other bad security responses) have not really hurt them, so at this stage, this won't make a lot of difference, I'd expect.
Cisco is in the business of selling big, black, expensive boxes that have a lot of security badges and fancy icons. People who buy such boxes don't care if they actually work, they want a big box so that they can claim they "invested in security".
Actually, they want the Cisco box because it has those 'badges' and that then allows them to check off all the check-boxes on the semi-annual security compliance report paperwork they have to file with some other department.
But you are right, they don't care if the box is secure, they just care that they have a CYA that allows them to "check off their own boxes" on their reports.
But you are right, they don't care if the box is secure, they just care that they have a CYA that allows them to "check off their own boxes" on their reports.
What would you buy instead of Cisco?
HP and Dell are the same thing. From experience a decade ago, the HP usually did not have the enterprise features they advertised.
The other minor brands are very hard to procure if you're not in the US or a primary English speaking country.
HP and Dell are the same thing. From experience a decade ago, the HP usually did not have the enterprise features they advertised.
The other minor brands are very hard to procure if you're not in the US or a primary English speaking country.
Juniper? Extreme? The list is not very long, but doesn't contain just Cisco.
Last I checked, long ago, they were impossible to procure in France.
Pretty sure Extreme is still non-existent in Europe as of today.
For all the flaws of Cisco, well the only flaw is the price, they can deliver in any language anywhere in the world.
Pretty sure Extreme is still non-existent in Europe as of today.
For all the flaws of Cisco, well the only flaw is the price, they can deliver in any language anywhere in the world.
Juniper has offices and partners all over EU, for example in Paris [0]. Similar to Extreme [1].
I don't know what problems you encountered in France, but I can assure you both of them are very well represented across the whole EU, even less developed parts of it.
The reason managers pick Cisco is mostly that "nobody gets fired for picking Cisco" imho.
[0] https://www.juniper.net/us/en/contact-us/sales-offices/paris...
[1] https://fr.extremenetworks.com/
I don't know what problems you encountered in France, but I can assure you both of them are very well represented across the whole EU, even less developed parts of it.
The reason managers pick Cisco is mostly that "nobody gets fired for picking Cisco" imho.
[0] https://www.juniper.net/us/en/contact-us/sales-offices/paris...
[1] https://fr.extremenetworks.com/
Do people trust their products?
Meanwhile the US goes around telling other countries not to use Huawei because they can't guarantee security. [1]
[1] https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-fo...
[1] https://www.forbes.com/sites/zakdoffman/2019/02/19/huawei-fo...
I doubt they are referring to their security bugs.
There can be more than one company that's a security risk.
User agents shouldn't exist any more. They serve only to help unsuspecting users be fingerprinted.
Considering recent chrome, firefox user agent contains id string of almost every browser in existence, this is so true:
Mozilla/5.0 (...) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Mozilla/5.0 (...) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
You just wait for Edge-based-on-Chromium's User-Agent :D
Per a buddy who is using said Edgium (Chromium-based Edge): https://i.imgur.com/p6ZMoJY.png
Oh ffs
I vote we change all user agents to just "curl".
“Apache” would be ironic.
Might as well save 3 bytes and just call use “c”.
Well, the next best thing is what webkit is doing (https://webkit.org/blog/8042/release-notes-for-safari-techno...) by freezing the user agent.
So as time goes by the user agent becomes increasingly homogenous and so, less useful for tracking.
So as time goes by the user agent becomes increasingly homogenous and so, less useful for tracking.
I believe this change was rolled back, since my User Agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.2 Safari/605.1.15".
Rubbish. They are incredibly useful for debugging.
They would be a lot more useful if they were more straighforward and honest.
Just give me "Chrome 68" instead of "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
Just give me "Chrome 68" instead of "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
firefox has a sane user agent. mine is:
>Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
>Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
edit: sorry, accidentally submitted early
So what. Removing one of the largest sources entropy available for fingerprinting users is important. We shouldn't maintain the terrible long-term effects[1] of tracking everything just to help you have an easier time debugging.
[1] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
So what. Removing one of the largest sources entropy available for fingerprinting users is important. We shouldn't maintain the terrible long-term effects[1] of tracking everything just to help you have an easier time debugging.
[1] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
Does it really provide that much entropy? Maybe back in the days when browsers didn’t auto-update, but intuitively it should only provide about four bits at most (os: Android iOS macOS windows, browser: chrome safari edge ie ff) except for a few rare users who don’t stay up to date or choose a weird browser (and those folks probably are a little confused about how privacy works).
ohhhhh you betcha.
https://amiunique.org/
i ran this on my work laptop (windows 10 / firefox - mostly default settings). the "details" tab shows a full breakdown of each fingerprinting component with a "similarity ratio" (percentage of fingerprints that share the same value).
every entry is 20% or greater, most of them around 50% except for timezone (4%), canvas (1%), and user agent (0.12%).
so, yeah, user agents are still the largest source of entropy by a wide margin for fingerprinting scripts.
https://amiunique.org/
i ran this on my work laptop (windows 10 / firefox - mostly default settings). the "details" tab shows a full breakdown of each fingerprinting component with a "similarity ratio" (percentage of fingerprints that share the same value).
every entry is 20% or greater, most of them around 50% except for timezone (4%), canvas (1%), and user agent (0.12%).
so, yeah, user agents are still the largest source of entropy by a wide margin for fingerprinting scripts.
I suspect this is just stale data. I can’t think why the most recent version of a browser would put me at one in a thousand.
There are better solutions available than hanging functionality on unreliable vestigial bits and pieces that shouldn't be there anyway which, because others abuse the functionality, you can't trust to be correct for debugging purposes.
99% of the traffic has the correct user-agent which is useful in tracking down issues that are browser specific. The other 1% will just get ignored as noise. And it's not like we can't tell what type of browser they are using with browser specific objects that we can pull from javascript. We just can't infer the version of the browser which is critical for debugging issues.
I use a randomized user agent and every once in a while get rejected by a site that has decided my browser can't be supported. Usually when I'm on some Safari version. I doubt much effort is being done to identify fake user agents since there are better fruit to pick in the fingerprinting game.
That might be true, although you don't list the solutions, but it doesn't refute my point in any case.
Per https://webkit.org/blog/8042/release-notes-for-safari-techno... the user agent is frozen in webkit now.
You're writing your websites wrong.
I've had to debug around a short-lived bug in the Firefox rendering engine that only showed up in the FF rendering engine before. The site itself? Standards-compliant; there was flexibility in the standard (ironically, for flexboxes ;) ) about performance constraints and our site just fell down Gecko's slowness tree and hit every branch on the way.
Given the engines are allowed to vary, this experience is inevitable from time to time. The user-agent is really valuable in helping us figure out when that's what we're seeing.
Given the engines are allowed to vary, this experience is inevitable from time to time. The user-agent is really valuable in helping us figure out when that's what we're seeing.
If there’s a way to write them without the occasional browser-specific bug, I’d sure like to know what it is.
They help you figure out if a user agent wants a mobile view of a page without Javascript... which is pretty useful.
We don't need user agents for that. There's an easy way to tell whether a visitor wants a cleaner view of a page, without Javascript: Yes.
I said "mobile", not "cleaner". Sizes and aspect ratios are different on phones than PCs.
> I said "mobile", not "cleaner". Sizes and aspect ratios are different on phones than PCs.
User agent doesn't tell you size or aspect ratio.
Even if you know I'm running Chrome on Android on a Galaxy Note 8, you don't know the size (physical or pixel) or aspect ratio of the viewport, because: (1) I may or may not be viewing in side by side windows, (2) the screen can be in any of three resolutions, (3) the screen can be portrait or landscape, (4) the browser may or may not be set to use the full space on the long dimension of the screen.
OTOH, CSS Media Queries answer these questions.
User agent doesn't tell you size or aspect ratio.
Even if you know I'm running Chrome on Android on a Galaxy Note 8, you don't know the size (physical or pixel) or aspect ratio of the viewport, because: (1) I may or may not be viewing in side by side windows, (2) the screen can be in any of three resolutions, (3) the screen can be portrait or landscape, (4) the browser may or may not be set to use the full space on the long dimension of the screen.
OTOH, CSS Media Queries answer these questions.
Not necessarily. Websites should use CSS media queries to work at all aspect ratios and sizes without needing server-side code or JavaScript.
the could be accomplished just by having User-Agent: mobile, all the other data in the user-agent are useless for this purpose.
There is presumably some way, at least hypothetically, to fix the mess that User-Agent has become. (Everybody is 'Mozilla' now! And yet somehow a unique snowflake 'Mozilla' that lets you be tracked...)
But it's not quite that. It's not that a device is "mobile" instead of "stationary" that matters. It's maybe that it's a touch screen. It's maybe that it's a small screen. (With an especially uneven aspect ratio?) Maybe that it's handheld. It's maybe the combination of all of those things. But it's not really that it's "mobile" exactly. And there are, and possibly will be increasing proportions of, devices that have some but not all of these qualities we package together in our brains as 'typical mobile'
But it's not quite that. It's not that a device is "mobile" instead of "stationary" that matters. It's maybe that it's a touch screen. It's maybe that it's a small screen. (With an especially uneven aspect ratio?) Maybe that it's handheld. It's maybe the combination of all of those things. But it's not really that it's "mobile" exactly. And there are, and possibly will be increasing proportions of, devices that have some but not all of these qualities we package together in our brains as 'typical mobile'
All of those small differences of mobile devices are better served by media queries I personally think. You could replace "mobile" in my example by "small-screen" if you prefer.
Agreed. Media queries need to be improved though. There are no good media queries to detect "touch screen" and/or "no keyboard or mouse/pointer, only touch screen", only JS hacks that attempt it. This can matter. But yes, improving media query (or input method query?) APIs for both CSS and JS is, I agree, the correct solution to User-Agent madness. It's just not entirely straightforward or obvious or easy, and as always getting all browsers on the same page is an issue.
Well at least we acknowledged there's some utility.
Now consider that mobile devices are different as well..
Now consider that mobile devices are different as well..
The current user agent does not give you much insight about what type of mobile device you have anyway, especially on Android.
I use Firefox for Android on ChromeOS -- it produces interesting results.
So instead of sending a user agent, send only the page ratio!!
This is not too helpful -- you can resize the window in such a way that the ratio changes...
I think CSS is probably good enough that we can live without UA sniffing for this.
We sometimes render different HTML to handle browser and device behavioural variations, including FOUT-management variations in how they load assets that therefore can't be handled in JS even if we wanted to. Historically we also redirected to a pure HTML/CSS version of our tools when IE9 or earlier was detected.
I'm pretty sure deleting User agents field will introduce another non-standard X-prefixed header.
That is not correct. They also serve to exclude users of browsers you dont know or like.
Don't blame the manager, the PO, the CEO. This is ABSURD engineering incompetence. The fellow that did that _fix_ probably had no idea how to properly solve the issue.
If it is then it's management's responsibility for allowing that incompetence to exist. These sort of issues all come from the culture which is driven from the top.
So having an absurdly incompetent engineer implementing this fix isn't a management problem?
Managers, POs, and CEOs are responsible for preventing this by:
- hiring people who'd know not to do it
- creating processes for more than one security expert to review security patches
- requiring the testing of patches against real-world workarounds
- allotting the time and budget necessary for all of the above
- hiring people who'd know not to do it
- creating processes for more than one security expert to review security patches
- requiring the testing of patches against real-world workarounds
- allotting the time and budget necessary for all of the above
Here is the original vulnerability report: https://www.redteam-pentesting.de/en/advisories/rt-sa-2019-0...
... which has
I also note the timeline
* 2019-01-22 Firmware 1.4.2.20 released by vendor
...
* 2019-02-07 Incomplete mitigation of vulnerability identified
...
* 2019-03-25 Vendor requests postponed disclosure
So this is apparently a bad fix that Cisco has known about since February, and asked for an extension in order to fix again.
-A kurl
I also note the timeline
* 2019-01-22 Firmware 1.4.2.20 released by vendor
...
* 2019-02-07 Incomplete mitigation of vulnerability identified
...
* 2019-03-25 Vendor requests postponed disclosure
So this is apparently a bad fix that Cisco has known about since February, and asked for an extension in order to fix again.
"That's not how this works. That's not how any of this works."
Did they hire someone off freelancer.com to fix this ?
(in reference to : https://news.ycombinator.com/item?id=19318498#19329754)
(in reference to : https://news.ycombinator.com/item?id=19318498#19329754)
I posted something related to this a couple weeks ago.
Chase banned my web client because its not Windows/OSX.
It works fine if I change my user agent...
FWIW the RV series is the ex-linksys stuff so it should all be thrown in the trash anyways.
I... do they know about -A? Someone should tell them.
Okay, so the fix is bad. Now, you have to wonder how this "fix" made it through code review, QA, release... You can blame the engineer. Perhaps they were rushed, inexperienced, or both, but this is a failure on all levels.
How do you know it wasn't fixed by the offshore maintenance team with an offshore manager controlled by a VP wanting a quick promotion for doing things fast? That would be where I work the likely scenario.
I think this is called "Test Driven Development". /s
Actually, this sort of superficial fix can be a result of (overzealous) TDD.
Too busy chasing the green.
This would be more of an integration test right?
Tests must not be robust enough, then.
yes, they need to add a test for wget (and a comparable fix) - that will teach them how to do tdd right.
They'll just add another user agent check.
The updated PoC command from the exploit page[1]:
[1] https://www.redteam-pentesting.de/en/advisories/rt-sa-2019-0...
$ curl -s -k -A kurl -X POST -b "$COOKIE" \
--data "page=self_generator.htm&totalRules=1&OpenVPNRules=30"\
"&submitStatus=1&log_ch=1&type=4&Country=A&state=A&locality=A"\
"&organization=A&organization_unit=A&email=ab%40example.com"\
"&KeySize=512&KeyLength=1024&valid_days=30&SelectSubject_c=1&"\
"SelectSubject_s=1" \
--data-urlencode "common_name='a\$(ping -c 4 192.168.1.2)'b" \
"https://192.168.1.1/certificate_handle2.htm?type=4"
[1] https://www.redteam-pentesting.de/en/advisories/rt-sa-2019-0...
That's hilariously reasonable if someone didn't read the news latetly and suddenly got a 403 while executing this payload using curl would probably quit and be like "meh it got patched". Look on the bright side guys.
curl -A "UserAgentString" http://example.com
I have one of these routers :\ So far I'm unable to get the PoCs to work but that doesn't make me confident.
I'm curious how pen testers reverse the .bin firmware download into source code, anyone have any insight there?
I'm curious how pen testers reverse the .bin firmware download into source code, anyone have any insight there?
That's not source code, that's just a config file embedded into the firmware. For most firmware, it's a large binary file that often has different sections containing stuff like an OS kernel at one spot, a compressed archive somewhere else, etc. For the Cisco firmware in question in the middle of the firmware is a compressed CPIO archive (think of it like a zip file) that contains the root file system. The router is actually a small linux computer, inside that archive they have a config file for nginx (a web server) that tells nginx to return an error if it sees the user agent string for curl. There's no programming required here, just a trivial configuration change that masks the actual issue.
Ida64/Binary analysis tooling, I presume.
I feel like this tweet sums up the situation: https://twitter.com/dogetard/status/1111110061768822784?s=20
Without actually knowing the truth, my guess is outsourcing: Cisco thought it would be a great idea to save some cash on technical staff so they now do a lot of the grunt work through company X in country Y.
It's not like anyone could ever change their user agent in their curl config.
Oh, wait...
cat ~/.curlrc user-agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
Oh, wait...
cat ~/.curlrc user-agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
They do know that you can spoof the user agent I regularly do this to crawl sites.
I wish it was Curly. We could then make jokes with Moe and Larry.
Ouch. Thankfully, I get to work on a more interesting router...
Source says they also did some input sanitizing along with blocking curl, and they had to make a new PoC to get around that. If I'm reading that right then this isn't really an issue, nothing wrong with defense in depth.
Edit:
>The update adds several filters to handle single quotes in user input. However, these filters can be evaded by specially crafted inputs. By providing the following string for the certificate's common name, a "ping" command can be injected:
Title is misleading, implying the only patch was blacklisting curl.
Edit:
>The update adds several filters to handle single quotes in user input. However, these filters can be evaded by specially crafted inputs. By providing the following string for the certificate's common name, a "ping" command can be injected:
Title is misleading, implying the only patch was blacklisting curl.
Speaking of filtering, the diff between the two exploits is:
Does their fix specifically check for injection starting with `a'` ? And only works for GET requests? Mind-boggling...
Edit: The new exploit also targets https instead of http. I would've said that surely that would not make a difference, but given what's already happened I'm not sure.
-"common_name=a'\$(ping -c 4 192.168.1.2)'b"
+"common_name='a\$(ping -c 4 192.168.1.2)'b"
Does their fix specifically check for injection starting with `a'` ? And only works for GET requests? Mind-boggling...
Edit: The new exploit also targets https instead of http. I would've said that surely that would not make a difference, but given what's already happened I'm not sure.
"As a user, when I send a request like below I want it to not pwn the router."
The equivalent of a "pls dont hack" sign is not defense in depth.
Good to know they at least half fixed the problem, I guess. But that's not enough, and they should be capable of testing this.
Good to know they at least half fixed the problem, I guess. But that's not enough, and they should be capable of testing this.
I would expect them to check for any RFC3514 bits as well. Defense in Depth.
The real defense is the attacker needs to access and authenticate with the router's web interface. A more honest patch would be to legitimize the bug as a new feature since it must be too amateur-hour over there to actually address any webshit security issues. "Dear Admin, here's a textarea to run arbitrary commands as root, don't hurt yourself!"
Can anything without a "please don't hack" sign considered defense in depth?
Probably not, hence an appropriate first patch.
Probably not, hence an appropriate first patch.
Yes it can. If you see a product that claims to be secure, with multiple layers of security, certain exceptionally-fragile measures should decrease how much you believe that claim. Layers of security have to at least be mildly effective to count as layers.
If you're fighting an active attack you can stall by filtering on some arbitrary parameter unrelated to the actual problem. For anything that's supposed to last more than an hour, it's worse than useless. It makes your system more complex for no security benefit. An idea like that should never make it into a product release.
If you're fighting an active attack you can stall by filtering on some arbitrary parameter unrelated to the actual problem. For anything that's supposed to last more than an hour, it's worse than useless. It makes your system more complex for no security benefit. An idea like that should never make it into a product release.
I agree, it's just good sense. It's even encapsulated in the bit of folk wisdom about throwing the baby out with the bathwater.
Your humor is too sophisticated for us...