Beware of Base64 Encoded Strings(garrit.xyz)
garrit.xyz
Beware of Base64 Encoded Strings
https://garrit.xyz/posts/2024-04-15-beware-of-base64-encoded-strings
50 comments
Title sounds like click bait :) . There is nothing wrong with base64 strings. The base64 command, on the other hand, will line-split its output if it’s very long. Using the -w0 param as the article discovered and shared is the way to go - but this is dependent on the specific b64 tooling that was used.
I’ve never encountered this issue because macOS/BSD `base64` doesn’t have this default. So I agree, a bad title for an issue with one tool.
Another prime example on the typical arrogance of this trade, of computer scientists:
My brain immediately goes like: You nitwit, base64 comes back from E-Mails, where you would work with Mail-Server's conception that your mails were all written, legible characters. Of course you should expect newlines, potentially whitespaces and tabs in it.
But that would be stupid of me. There's so much old knowledge and know-how we carry along for many decades - around 50 years in case of mail and SMTP. Unix and Linux are a specially egregious example, with so many conventions.... just ran into a long standing bug a few days ago, with some GECOS implementation (that's 62 years old stuff). We do a lot of good in throwing out the now unnecessary old stuff and separating the wheat from the chaff. Again and again.
In so far: Thanks for the pointer on that Base64 appears strange (also please do not let us talk about groff).
My brain immediately goes like: You nitwit, base64 comes back from E-Mails, where you would work with Mail-Server's conception that your mails were all written, legible characters. Of course you should expect newlines, potentially whitespaces and tabs in it.
But that would be stupid of me. There's so much old knowledge and know-how we carry along for many decades - around 50 years in case of mail and SMTP. Unix and Linux are a specially egregious example, with so many conventions.... just ran into a long standing bug a few days ago, with some GECOS implementation (that's 62 years old stuff). We do a lot of good in throwing out the now unnecessary old stuff and separating the wheat from the chaff. Again and again.
In so far: Thanks for the pointer on that Base64 appears strange (also please do not let us talk about groff).
RTFM, from `man base64`:
-w, --wrap=COLS
wrap encoded lines after COLS character (default 76). Use 0 to disable line wrapping
Sure, but it’s still a bad default IMO, the expected behavior would be an unformatted string without any linebreaks
For old farts like me who used to use News groups where binary data was first encoded in uuencode and later with the innovative base64, it is totally expected behavior.
Also look into the text files für the public/private keys. It is always this, because the screen has only 80 characters in width.
Also look into the text files für the public/private keys. It is always this, because the screen has only 80 characters in width.
So you pass in a 1gig binary you want a very, very long line of text that many text editors would choke on?
The purpose of base64 is to turn binary into text that can be embedded in normal text messages such as email and Usenet. A right margin of about 80 characters is very much what one should expect such a tool to default to.
The purpose of base64 is to turn binary into text that can be embedded in normal text messages such as email and Usenet. A right margin of about 80 characters is very much what one should expect such a tool to default to.
If you're passing a 1GB binary to base64 you're ... incredibly unlikely to be dealing with the resulting base64 manually (copypaste, text editor, whatever). So, yes, the expected default would still be no linebreaks.
The primary purpose of these tools have changed over time as the environment has changed. You know nobody uses usenet anymore, and nobody copypastes base64 into emails anymore, right?
The primary purpose of these tools have changed over time as the environment has changed. You know nobody uses usenet anymore, and nobody copypastes base64 into emails anymore, right?
The default will not be changed because it breaks the original use cases. Which some people still have even if you do not.
[deleted]
Right, but it seems they did RTFM once they discovered this quirk in the behaviour of the command.
Maybe it would behoove people to read the manual before they "discover a quirk"? I always at least give the manual a good skim before running the command at all...
This surely isn't the first time they've run the base64 command. If they have used it before, I wouldn't expect them to read the manual again unless they are trying to get different behavior out of the tool. Instead they had to read it again to get the same behavior out of the tool
How did they figure out to use the tool in the first place? Copied from Stackoverflow? Guessed?
I can't answer that because I'm not them, but maybe man...? My point is that:
- base64 is a stupid easy command to use the default behavior (guessing is easily possible)
- they probably would not have retained flags from the man page unless they needed to use them the first time they used base64
- they probably didn't need to use any flags with base64 in the past
- it's perfectly reasonable to expect base64 to base64 long inputs the same way it encodes short ones. In fact I would say any other suggestion is unreasonable
- it is perfectly reasonable to not read the man page again when you have a concrete expectation of how a program should behave
- base64 is a stupid easy command to use the default behavior (guessing is easily possible)
- they probably would not have retained flags from the man page unless they needed to use them the first time they used base64
- they probably didn't need to use any flags with base64 in the past
- it's perfectly reasonable to expect base64 to base64 long inputs the same way it encodes short ones. In fact I would say any other suggestion is unreasonable
- it is perfectly reasonable to not read the man page again when you have a concrete expectation of how a program should behave
IMHO this is completely unexpected behaviour and why would you read the manual if you don't expect this? base64 is not a difficult program to use. I certainly don't read the manual for every command that just takes an input - like sha256sum etc.
[deleted]
It's "discussions" like this that ensure I will never write up the small mistakes I occasionally make, even if people might benefit from knowing about them.
If ever you make a mistake, never, ever write it up. It might get submitted here for everyone to tell you how stupid you were.
If ever you make a mistake, never, ever write it up. It might get submitted here for everyone to tell you how stupid you were.
> If ever you make a mistake, never, ever write it up.
Sure you can write it up! But post it to maybe, Reddit?
The HN audience expects certain standard of quality and insight in the article if the article is on the front page.
Here the community has voted that this article does not meet the bar. I can't speak for others but for me this article is too obvious and uninteresting.
Instead of complaining about something that didn't happen here (nobody said the OP was stupid), maybe learn from the community feedback and refrain from posting articles that don't meet the community expectations?
Sure you can write it up! But post it to maybe, Reddit?
The HN audience expects certain standard of quality and insight in the article if the article is on the front page.
Here the community has voted that this article does not meet the bar. I can't speak for others but for me this article is too obvious and uninteresting.
Instead of complaining about something that didn't happen here (nobody said the OP was stupid), maybe learn from the community feedback and refrain from posting articles that don't meet the community expectations?
> The HN audience expects certain standard of quality and insight in the article if the article is on the front page.
It was on the front page ... it hit the top spot. Various people in the community clearly did think it met the bar.
> ... nobody said the OP was stupid ...
Reading between the lines, several people effectively said exactly that.
I know, I'm in "Old man yells at cloud" territory here, but I was kinda hoping people would find a more positive way of interpreting the situation, rather than "Can't the author just RTFM?!?" types of responses.
My agèd and fading memory of what HN was like when I first joined suggests that people used to be more interested in being constructive and learning from other people's mistakes, even the mistakes that, when they didn't make them, seemed obvious in retrospect. Perhaps especially those ones.
Since this didn't meet your bar, since it's all too obvious for you, I guess there's very little for you to learn. I'm sorry you (and others) felt you had to dump on it, instead of leaving it for people who can still learn from things as "obvious" as this. After all, the votes that got it to the front page show that not everyone knows as much as you do.
It was on the front page ... it hit the top spot. Various people in the community clearly did think it met the bar.
> ... nobody said the OP was stupid ...
Reading between the lines, several people effectively said exactly that.
I know, I'm in "Old man yells at cloud" territory here, but I was kinda hoping people would find a more positive way of interpreting the situation, rather than "Can't the author just RTFM?!?" types of responses.
My agèd and fading memory of what HN was like when I first joined suggests that people used to be more interested in being constructive and learning from other people's mistakes, even the mistakes that, when they didn't make them, seemed obvious in retrospect. Perhaps especially those ones.
Since this didn't meet your bar, since it's all too obvious for you, I guess there's very little for you to learn. I'm sorry you (and others) felt you had to dump on it, instead of leaving it for people who can still learn from things as "obvious" as this. After all, the votes that got it to the front page show that not everyone knows as much as you do.
> It was on the front page ... it hit the top spot. Various people in the community clearly did think it met the bar.
It was on the front page. It hit the top spot. Then it got flagged. Many people in the community clearly did think it was clickbait and did not meet the bar!
It was on the front page. It hit the top spot. Then it got flagged. Many people in the community clearly did think it was clickbait and did not meet the bar!
Yeah, as I say, people are a lot less tolerant now about people sharing simple mistakes they make than they were when I first started posting here. People used to grimace in sympathy, acknowledge that they've also made simple mistakes, and move on.
Now things like that get flagged, even though there's clearly a sizeable proportion of people who do still find it useful[0].
The demographics have changed[1].
So I'll remember, on your advice, that if ever I write up any simple mistakes I make, I won't post them here.
[0] FWIW, I think flags required to kill something is far fewer than votes required to get to the front page. So even if a vast majority of people think something is interesting, a far smaller collection of people can get it killed.
[1] Or my memory is faulty. I've been here 15 years now ... maybe it was always like this.
Now things like that get flagged, even though there's clearly a sizeable proportion of people who do still find it useful[0].
The demographics have changed[1].
So I'll remember, on your advice, that if ever I write up any simple mistakes I make, I won't post them here.
[0] FWIW, I think flags required to kill something is far fewer than votes required to get to the front page. So even if a vast majority of people think something is interesting, a far smaller collection of people can get it killed.
[1] Or my memory is faulty. I've been here 15 years now ... maybe it was always like this.
The base64 command wraps its output every 76 characters by default.
This is because of its original use which was to encode binary files to include in emails. Very long lines in emails don't work well.
See the precursor to base64 which is the uuencode command which you probably have installed too if you have the base64 command. base64 is a better, standardised version of uuencode.
Use --wrap=0 to disable.
This is because of its original use which was to encode binary files to include in emails. Very long lines in emails don't work well.
See the precursor to base64 which is the uuencode command which you probably have installed too if you have the base64 command. base64 is a better, standardised version of uuencode.
Use --wrap=0 to disable.
Or better : use the --wrap option so it gets clearer that no wrapping on long lines happens .
This is exactly what the article suggests.
I think they mean use "--wrap=0" instead of "-w0" as it's a little better self-documenting for those who come across your script.
In general I always try to use the long-form options in scripts as readability is more important that conciseness.
In general I always try to use the long-form options in scripts as readability is more important that conciseness.
The article notes this was not possible in their specific context, as only the short form is implemented in busybox.
The commit that they have a screenshot of notes this, but it isn't in the article text. FWIW I can't read that screenshot with zoom level 100% on my desktop, but I can if I zoom further or open it in a new window (it isn't clickable though).
That's what I meant indeed. The article never mentions busybox in the text, does it?
> $(echo -n $BASIC_AUTH_USERNAME:$BASIC_AUTH_PASSWORD | base64)
I would really like some quoting of that argument. Otherwise a weird password can really break things.
I would really like some quoting of that argument. Otherwise a weird password can really break things.
Any good reason to still use base64 compared to base32 ?
Since base64 seems to be not much more compact in practice (?), but also fails at transmission through the lossy channel that is human communication.
(Arguably, there's also hexadecimal, but compactness starts becoming an issue there ?)
Since base64 seems to be not much more compact in practice (?), but also fails at transmission through the lossy channel that is human communication.
(Arguably, there's also hexadecimal, but compactness starts becoming an issue there ?)
Specially useful when you encode ssh key to base64 in github to use it in github actions.
I spent hour to figure it why the key is not working and the answer is -w0
[deleted]
Does it still do this when piping into things?
I regularly use base64 to {en,de}code binary files that I want to copy and paste - laziest rsync imaginable
Surprised I haven't run into this wrapping or worse, lol
Surprised I haven't run into this wrapping or worse, lol
Beware of !rtfm
Don't people RTFM anymore?
Is this another "changeme is valid base64" moment?
Come on! Base64 has been around for 30 years!
Is this another "changeme is valid base64" moment?
Come on! Base64 has been around for 30 years!
If you've used a command for the past 10[0] years, and it's always worked, and you go to use it again, and you're using it in a similar way to how you always have, do you read the man page?
No, of course you don't. You expect things to behave the way they always have.
Then when something goes wrong you don't immediately suspect the command you've successfully been using for the past 10 years and read the man page for all the commands you've been using for the past 10[1] years.
No, you suspect the new code.
[0] For some value of 10[1].
[1] In my case significantly longer than 10.
No, of course you don't. You expect things to behave the way they always have.
Then when something goes wrong you don't immediately suspect the command you've successfully been using for the past 10 years and read the man page for all the commands you've been using for the past 10[1] years.
No, you suspect the new code.
[0] For some value of 10[1].
[1] In my case significantly longer than 10.
"If you've used a command for the past 10[0] years, and it's always worked, and you go to use it again, and you're using it in a similar way to how you always have, do you read the man page?"
Yes, that's the first thing I do.
Yes, that's the first thing I do.
I'm trying to find where the cut-off point might be.
Do you read the man page every time you use "ls"?
Do you read the man page every time you invoke the compiler for your favourite language?
If you use someone else's machine, do you read the man page for every command you type? The distro could be different, so you can't be sure it will be the same.
I suspect the answer here in each case is "no", so I'm curious as to where you draw the line. If you used a command yesterday, and the day before and the day before that, do you read the man page today?
Again, I suspect the answer is "no", but again, I'm curious as to where that line really is.
Do you read the man page every time you use "ls"?
Do you read the man page every time you invoke the compiler for your favourite language?
If you use someone else's machine, do you read the man page for every command you type? The distro could be different, so you can't be sure it will be the same.
I suspect the answer here in each case is "no", so I'm curious as to where you draw the line. If you used a command yesterday, and the day before and the day before that, do you read the man page today?
Again, I suspect the answer is "no", but again, I'm curious as to where that line really is.
TL;DR: the base64 util breaks lines after a certain number of columns. Use a parameter to specify "don't break". [0]
[0] https://www.man7.org/linux/man-pages/man1/base64.1.html
[0] https://www.man7.org/linux/man-pages/man1/base64.1.html
[deleted]