Myths about /dev/urandom (2014)(2uo.de)
2uo.de
Myths about /dev/urandom (2014)
https://www.2uo.de/myths-about-urandom
66 comments
While I agree with what you’re saying, I think it should be pointed out that the author clearly states that it’s a simplistic model and even mentions the multiple pools briefly (though I’m unsure if that was add post-comment, as author is on hn also).
No, the page hasn‘t been changed in quite some time. The last real change was the 4.8 change, long after 4.8 was out.
Does increasing the rate of external entropy risk swamping the system generated entropy? I.e. if the external source is compromised could this also compromise urandom such that the results could become predictable?
No! This is the thing everyone needs to understand: once you have achieved "entropy" with a modern CSPRNG (including T'so's), you never lose that property. Think of the RNG like a stream cipher like AES-CTR. How much data do you have to encrypt with AES-CTR to "lose" or "leak" the underlying AES key?
Here I have to disagree (in the small) essentially the problem of 'breaking' a RNG is discovering that hidden state (the entropy pool) behind the CSPRNG - if you believe that the NSA, and their competitors, don't each have a team of PhDs pouring over the linux random code looking for ways to do that you're dreaming. After all if you know its state you can predict the keys it will make in the future
My intuition is that every operation we do (start TCP connections, generate keys, backoff after a failed login, ....) reveals a tiny fraction of a bit about that hidden state, web assembly calling random over and over again probably reveals much much more, over time it builds up.
(anyone want to guess if there's a quantum computer attack here, it kind of looks like the sort of problem where there might be)
So I think that with a sophisticated enough attacker entropy does degrade, the trick is to replace that entropy at a faster rate that the rate at which you guess the worst case attacker might be able to guess.
For most people it probably doesn't matter, you're exposing so little state over time you're likely effectively as safe as the parent poster claims, it kind of depends on whether, in this age when we know that nation state actors are watching every packet that passes thru their hands, watching every thing we're doing online, are you interesting enough for them to spend the time and effort? does the US/5Is think you're not a Trump supporter? are the Chinese interested in your trade secrets? whi knows what the Russians think
As I mentioned above using /dev/urandom doesn't have a mechanism you can use to signal this guess about how much entropy it needs over time to be safe, prior to 4.8 we used to have a mechanism that said "give it some more entropy every N seconds, 60 by default" but that's now broken
(provided here a python applet that periodically drains a little bit of /dev/random entropy if it has plenty available to stir the input pool that /dev/urandom uses for its entropy source)
My intuition is that every operation we do (start TCP connections, generate keys, backoff after a failed login, ....) reveals a tiny fraction of a bit about that hidden state, web assembly calling random over and over again probably reveals much much more, over time it builds up.
(anyone want to guess if there's a quantum computer attack here, it kind of looks like the sort of problem where there might be)
So I think that with a sophisticated enough attacker entropy does degrade, the trick is to replace that entropy at a faster rate that the rate at which you guess the worst case attacker might be able to guess.
For most people it probably doesn't matter, you're exposing so little state over time you're likely effectively as safe as the parent poster claims, it kind of depends on whether, in this age when we know that nation state actors are watching every packet that passes thru their hands, watching every thing we're doing online, are you interesting enough for them to spend the time and effort? does the US/5Is think you're not a Trump supporter? are the Chinese interested in your trade secrets? whi knows what the Russians think
As I mentioned above using /dev/urandom doesn't have a mechanism you can use to signal this guess about how much entropy it needs over time to be safe, prior to 4.8 we used to have a mechanism that said "give it some more entropy every N seconds, 60 by default" but that's now broken
(provided here a python applet that periodically drains a little bit of /dev/random entropy if it has plenty available to stir the input pool that /dev/urandom uses for its entropy source)
The point is that the LRNG isn't unknowable, and if the NSA can break it, they can break a lot of other stuff. The LRNG isn't built on AES-CTR (but it just as easily could have been); it's (last I checked) essentially a sort of PRF built on SHA1. All the machinations in the LRNG with multiple pools and the elaborate mixing function and the counters are just plumbing around what is essentially a keystream generated from SHA1 "keyed" by "entropy". The "entropy updates" the kernel rekey that PRF, not because the PRF is weakening but because a vulnerability (orthogonal to the LRNG) could expose kernel memory and thus the LRNG's state, and the updates allow it to recover from that kind of compromise.
That's a lot of words because the LRNG is overly complicated. But let's make it clearer: what you're saying about every TCP sequence number "revealing" part of the "hidden state" is exactly equivalent to the argument that every byte of AES-CTR ciphertext is revealing the hidden AES key. If that were the case, AES would be grievously broken.
If you don't believe that AES-CTR and the LRNG are roughly equivalent, try this thought experiment: retain the initialization functionality from the current LRNG --- building a pool of mixed and hashed unpredictable data from kernel events --- and scrap the rest of the LRNG, replacing it with... AES-CTR, using a hash of the entropy pool as its key. Reads from random/urandom (or kernel random reads, as with your sequence numbers) just tap bytes from the AES-CTR keystream. Is every TCP sequence number still revealing a little bit of the RNG state?
The Python script seems entirely silly to me; once the first read from /dev/random succeeds, you're guaranteed to have a fully initialized LRNG, and every subsequent read from urandom will be unpredictable and indistinguishable from random. Nothing is served by updating the LRNG after that point.
That's a lot of words because the LRNG is overly complicated. But let's make it clearer: what you're saying about every TCP sequence number "revealing" part of the "hidden state" is exactly equivalent to the argument that every byte of AES-CTR ciphertext is revealing the hidden AES key. If that were the case, AES would be grievously broken.
If you don't believe that AES-CTR and the LRNG are roughly equivalent, try this thought experiment: retain the initialization functionality from the current LRNG --- building a pool of mixed and hashed unpredictable data from kernel events --- and scrap the rest of the LRNG, replacing it with... AES-CTR, using a hash of the entropy pool as its key. Reads from random/urandom (or kernel random reads, as with your sequence numbers) just tap bytes from the AES-CTR keystream. Is every TCP sequence number still revealing a little bit of the RNG state?
The Python script seems entirely silly to me; once the first read from /dev/random succeeds, you're guaranteed to have a fully initialized LRNG, and every subsequent read from urandom will be unpredictable and indistinguishable from random. Nothing is served by updating the LRNG after that point.
I think you misunderstand my point, we're continually revealing tiny fractions of a bit of the internal state = hopefully really tiny fractions, an AES stream doesn't reveal the key, but other places reveal the direct output of the random number generator (TCP sessions are a great example, various nonces), all those tiny fractions can add up, but provided we add entropy faster than we expose all those little bits we're probably OK
If the output of the random number generator used in tcp sessions is aes-ctr keyed by hash of entropy pool and aes does not reveal the key, then the output of the random number generator used in tcp sessions does not reveal the entropy pool.
What you wrote is magical thinking.
What you wrote is magical thinking.
You have to be root to add entropy to /dev/*random so you have to trust root - mind you many distros (ubuntu for example) use the CPU's RNG to provide external entropy by default, they trust Intel, you have to decide if you do too.
However kernel entropy is real, adding more entropy to the entropy pool is always is always adding more entropy. If the pool had 1000 bits in it and you add 5 bits from the kernel and a fake 1000 bits from a bogus external source it still has 1005 bits of entropy (however /dev/random thinks it will be able to extract 2005 bits)
We actually encourage our customers who are particularly paranoid to lie to the kernel about the quality of the entropy our RNG generates - saying the entropy quality is lower than it really is causes more good quality data to be mixed into the kernel pool (of course the defaults don't do this)
The kernel pools are actually quite small (4k bits) so there's a real practical limit to how much you can play this sort of game
However kernel entropy is real, adding more entropy to the entropy pool is always is always adding more entropy. If the pool had 1000 bits in it and you add 5 bits from the kernel and a fake 1000 bits from a bogus external source it still has 1005 bits of entropy (however /dev/random thinks it will be able to extract 2005 bits)
We actually encourage our customers who are particularly paranoid to lie to the kernel about the quality of the entropy our RNG generates - saying the entropy quality is lower than it really is causes more good quality data to be mixed into the kernel pool (of course the defaults don't do this)
The kernel pools are actually quite small (4k bits) so there's a real practical limit to how much you can play this sort of game
BTW - a tiny bit of python to keep /dev/urandom fed post 4.8 - useful if you have a hardware RNG (either an external one or the one in your CPU):
#! /usr/bin/python
#
# OneRNG /dev/urandom tickler
# (C) Copyright Paul Campbell Moonbase Otago 2018
# Released under a GPL 3.0 license
#
import time
with open("/dev/random", 'rb') as r:
while True:
with
open("/proc/sys/kernel/random/entropy_avail", 'r') as e:
iv = e.read()
print(iv)
if iv > 1500:
x= r.read(1000)
e.close()
time.sleep(1)GPL 3.0 for a few obvious lines seems to be really an unfitting license.
Non-licensed English language version, containing the same information:
Once per second, read /proc/sys/kernel/random/entropy_avail and if the resulting number is more than 1500 perform a read of 1000 bytes from /dev/random.
Non-licensed English language version, containing the same information:
Once per second, read /proc/sys/kernel/random/entropy_avail and if the resulting number is more than 1500 perform a read of 1000 bytes from /dev/random.
sorry, just grabbed my standard disclaimer, can't release code without copyrighting it first. As others have pointed out it's trivial to reimplement in the language of your choice go for it
(this is more of a tangential licensing PSA, since we're on the subject)
Not licensing something is, ironically, restrictive. If you want to release to public domain (or equivalent), you have to explicitly mention that. E.g. by licensing under CC0.
Otherwise, default copyright and "all rights reserved" applies.
https://creativecommons.org/share-your-work/public-domain/cc...
(But, IMO, nothing wrong with publicizing use of GPL 3! :) thanks either way to both)
Not licensing something is, ironically, restrictive. If you want to release to public domain (or equivalent), you have to explicitly mention that. E.g. by licensing under CC0.
Otherwise, default copyright and "all rights reserved" applies.
https://creativecommons.org/share-your-work/public-domain/cc...
(But, IMO, nothing wrong with publicizing use of GPL 3! :) thanks either way to both)
I think GP is getting at something else. As the length/complexity of the code goes to zero, the copyright and "all rights reserved" vanishes; this is well established, or else we'd all have to license "Hello world" examples from Kernighan and Ritchie's book.
If someone writes the same one-sentence pseudocode as GP in a book or blogpost, nobody could reasonably claim that an implementation would fall under the copyright of that book. And I think you'd be very hard pressed to argue it's copyrightable just because someone typed it out into real code.
If someone writes the same one-sentence pseudocode as GP in a book or blogpost, nobody could reasonably claim that an implementation would fall under the copyright of that book. And I think you'd be very hard pressed to argue it's copyrightable just because someone typed it out into real code.
Ruby version:
#!/usr/bin/env ruby
source = File.open('/dev/random', 'rb')
loop do
source.read(1000) unless File.read('/proc/sys/kernel/random/entropy_avail').to_i < 1500
sleep 1
endDo you have a gist or similar link for this code? Copy & paste doesn't work due to indentation issues.
What problem does this solve?
entropy_avail returns bits of entropy to a maximum of 4096(?), but this reads 1000 bytes. Why check if it's greater than 1500 if you're going to use it all anyway? Am I missing something here?
remember that this is for people with external entropy sources, the kernel entropy will refill fast. 1500 is more a measure of "is /dev/random busy at the moment" if it is then it's pulling data from from the external entropy source and /dev/urandom is essentially already being fed with more entropy
The missing mechanism we're trying to emulate used to essentially read N bytes (whatever user space happened to want at the time) every M seconds our experience has been that that can be way more than 1000 bytes, but reduce it if you like
The missing mechanism we're trying to emulate used to essentially read N bytes (whatever user space happened to want at the time) every M seconds our experience has been that that can be way more than 1000 bytes, but reduce it if you like
This article naturally leads to a question that I didn’t see addressed, perhaps you can answer it here:
If it’s unnecessary to ever prefer /dev/random to urandom, then why does /dev/random’s current blocking implementation still exist? Surely between all of the kernel developers, between all of the Unix-ish OS’s, over the last couple of decades would have realized that /dev/random should simply act like /dev/urandom?
If it’s unnecessary to ever prefer /dev/random to urandom, then why does /dev/random’s current blocking implementation still exist? Surely between all of the kernel developers, between all of the Unix-ish OS’s, over the last couple of decades would have realized that /dev/random should simply act like /dev/urandom?
Because Theodore Ts'o, a major Linux contributor and maintainer, created and maintains /dev/random and blocks attempts to make it nonblocking like every other cryptographically secure random source. It's a political problem, not a technical one.
See https://news.ycombinator.com/item?id=6550256 for a much longer argument about it.
See https://news.ycombinator.com/item?id=6550256 for a much longer argument about it.
I don't see how this is political. In that link he presents actual problems that have arisen from non-blocking versions. I'd call that a technical justification.
I don't necessarily agree with his justification, but that's also a difference in technical opinion. The only way I'd fault him is if I can come up with a an approach that addresses his concerns yet he still refuses to accept them.
Full disclosure: I've actually interacted with him on a few occasions. IMO calling him risk/change-averse would be an understatement, which partly explains this stance. While at times frustrating for us, this is exactly the quality you want in someone maintaining such a critical part of the kernel.
I don't necessarily agree with his justification, but that's also a difference in technical opinion. The only way I'd fault him is if I can come up with a an approach that addresses his concerns yet he still refuses to accept them.
Full disclosure: I've actually interacted with him on a few occasions. IMO calling him risk/change-averse would be an understatement, which partly explains this stance. While at times frustrating for us, this is exactly the quality you want in someone maintaining such a critical part of the kernel.
I suspected that “politics” was going to end up being the answer to my question. Simply because this seems to be a very contentious issue, even if it doesn’t need to be.
Edit: Thanks for the link!
Edit: Thanks for the link!
And what is the current state of /dev/urandom on Linux? Does it finally block on cold start with too little entropy? If it does, then effectively everybody can finally just use /dev/urandom, if I understand it correctly?
If it's still not blocking, it seems that still the best thing to do is using getrandom( ..., 0 )?
If it's still not blocking, it seems that still the best thing to do is using getrandom( ..., 0 )?
Why should it be made non-blocking? If you want a large string with little entropy just use /dev/urandom.
The political/dogmatic argument is that is should be made non-blocking. The process of acquiring entropy in typical computers is definitely blocking.
In FreeBSD, /dev/random = /dev/urandom.
But in Linux, there's a difference.
128 bits can be expanded into an infinite stream of unpredictable output. If those 128 bits are not known, then it is guaranteed to be secure.
But you can't do this "those 128 bits are used but are kept not known" thing with software. Sure, it is being tried again and again, and it fails every time.
You need separate hardware to safely prevent these 128bits from leaking.. but then why don't we ditch whole idea and implement true HW RNG instead?
You need separate hardware to safely prevent these 128bits from leaking.. but then why don't we ditch whole idea and implement true HW RNG instead?
I don't think you understood what he was saying, because if you think "you can't protect 128 bits in software" rebuts it, you also don't believe that any software cryptography works, in any setting; virtually all of it involves managing secret or unpredictable 128 bit values.
His point is that the mechanism used to generate random bytes by a modern CSPRNG is virtually identical to the mechanism that encrypts the application_data in a TLS session.
His point is that the mechanism used to generate random bytes by a modern CSPRNG is virtually identical to the mechanism that encrypts the application_data in a TLS session.
So you see absolutely no difference between leaking /dev/urandom PRNG seed from which all random numbers are generated for days or weeks, and leaking just one key for one TLS session?
Mechanism can be identical, but the stakes are definitely not.
Mechanism can be identical, but the stakes are definitely not.
Because we don't generate random numbers to look at them and marvel at their philosophical pureness, we use them. In software.
So those bits will be known to software, no matter where the generation happens.
Your argument would make a bit of sense if you were arguing for doing all cryptographic operations in hardware. Like on a smartcard. But that's not what we're talking about.
So those bits will be known to software, no matter where the generation happens.
Your argument would make a bit of sense if you were arguing for doing all cryptographic operations in hardware. Like on a smartcard. But that's not what we're talking about.
/dev/random is more secure than /dev/urandom because it blocks if there's not enough entropy. So if you want to generate private key, you should use /dev/random or you risk getting weak keys.
I'm so confused. I feel like literally every single point in the actual article is saying the exact opposite?
I find /dev/random frustratingly confusing.
I find /dev/random frustratingly confusing.
In low-entropy conditions (e.g. right after OS in VM has been installed) /dev/random is better but it might block (for minutes or hours). /dev/urandom is always issues some data, so your application won't hang because of it, but it might issue not very crypto strong bytes in those conditions. Best solution is to use hardware crypto or pass random data from host to VM or something like that. Also you might consider to use getrandom syscall. It's non-blocking if entropy pool has been initialized, otherwise it blocks (AFAIK the same as FreeBSD /dev/random). Probably the best of both worlds. That's how I understand the whole topic anyway.
> That's how I understand the whole topic anyway.
This suggests that you're not an expert. Therefore, it is wise to just stick to what the experts write, in articles such as this. That's what I do. Cryptography is a very complex subject; non-experts who give advice without knowing exactly why they are saying what they're saying are contributing to the problem, so please avoid doing that.
This suggests that you're not an expert. Therefore, it is wise to just stick to what the experts write, in articles such as this. That's what I do. Cryptography is a very complex subject; non-experts who give advice without knowing exactly why they are saying what they're saying are contributing to the problem, so please avoid doing that.
Nota bene: I‘m not an expert, just quoting them. :-)
I used to be very confused, too, and after learning something about it, I wrote that page. Almost didn‘t finish and publish it when tptacek put his page up, but by that time I had already invested most of the effort, and I wanted to be less terse.
I used to be very confused, too, and after learning something about it, I wrote that page. Almost didn‘t finish and publish it when tptacek put his page up, but by that time I had already invested most of the effort, and I wanted to be less terse.
True, maybe I should have added ...and non-experts who try to faithfully summarize them for us mere mortals.
Edit: actually, I think I would recommend everyone interested in this subject to read at least a few texts by actual experts (professors in cryptography and people who work for high-stakes security businesses), to get a taste of what it's all about.
Edit: actually, I think I would recommend everyone interested in this subject to read at least a few texts by actual experts (professors in cryptography and people who work for high-stakes security businesses), to get a taste of what it's all about.
After the system has been running for a few minutes, there is no difference in security between the two.
[deleted]
This document is very informative for sure, but I don't really think it's dispositive for many of today's use cases for randomness. In fact, I think this document is pretty dangerous.
I referenced this document here in this stackexchange question, which I think has a good and lively discussion about the real stakes of /dev/urandom today:
https://security.stackexchange.com/questions/186086/is-alway...
I referenced this document here in this stackexchange question, which I think has a good and lively discussion about the real stakes of /dev/urandom today:
https://security.stackexchange.com/questions/186086/is-alway...
I don't think you're going to find too many cryptography engineers recommending that you use /dev/random, or, for that matter, the blocking variant of getrandom(2).
This might be a live debate on message boards, or on the wrong stack overflow boards --- though Pornin's answer[1] on this question is one of the all-time good Stack Overflow answers --- but among practitioners, this is long settled.
[1]: https://security.stackexchange.com/questions/3936/is-a-rand-...
This might be a live debate on message boards, or on the wrong stack overflow boards --- though Pornin's answer[1] on this question is one of the all-time good Stack Overflow answers --- but among practitioners, this is long settled.
[1]: https://security.stackexchange.com/questions/3936/is-a-rand-...
Well, perhaps you have a better answer to my question than forest's answer, to which I awarded the bounty. For my money, I think that's one of the finest and most balanced pieces of writing on this topic available on the internet today.
> I don't think you're going to find too many cryptography engineers recommending that you use...the blocking variant of getrandom(2).
It depends on what you mean by "the blocking variant". To quote Cory Benfield:
> Can we please try to be clear about what kind of blocking we mean? getrandom(flags=0) absolutely can block: that's what the original issue was all about. To ensure it never blocks you need to call getrandom(GRND_NONBLOCK): that's why the flag exists.
It may (or may not) surprise you to learn that PEP 524, which uses this (sometimes blocking) variant of getrandom, has been accepted.
Here's a discussion of the matter featuring several reputable, kickass security-minded python people discussing the matter and indeed endorsing the sometimes-blocking (flags=0) variant of getrandom:
https://bugs.python.org/issue27266
For full transparency: I read your post on this topic before I posted my question and found that, as far as I was able to read, it added nothing new to the public discourse on this topic. It does not seem to overcome the very real and well-sourced concerns raised by forest in his answer to my question.
Anyway, I'm not sure what you mean by "the wrong stack overflow boards" - can you tell me which are right and which are wrong?
The link which you've provided, which of course I had already read thoroughly, as I pointed out in my question, is not from Stack Overflow, but from Security StackExchange, just like my question.
I haven't yet marked an answer correct, so if you actually have something substantive to add that you think is better than forest's answer, it's possible that you can grab the check.
> I don't think you're going to find too many cryptography engineers recommending that you use...the blocking variant of getrandom(2).
It depends on what you mean by "the blocking variant". To quote Cory Benfield:
> Can we please try to be clear about what kind of blocking we mean? getrandom(flags=0) absolutely can block: that's what the original issue was all about. To ensure it never blocks you need to call getrandom(GRND_NONBLOCK): that's why the flag exists.
It may (or may not) surprise you to learn that PEP 524, which uses this (sometimes blocking) variant of getrandom, has been accepted.
Here's a discussion of the matter featuring several reputable, kickass security-minded python people discussing the matter and indeed endorsing the sometimes-blocking (flags=0) variant of getrandom:
https://bugs.python.org/issue27266
For full transparency: I read your post on this topic before I posted my question and found that, as far as I was able to read, it added nothing new to the public discourse on this topic. It does not seem to overcome the very real and well-sourced concerns raised by forest in his answer to my question.
Anyway, I'm not sure what you mean by "the wrong stack overflow boards" - can you tell me which are right and which are wrong?
The link which you've provided, which of course I had already read thoroughly, as I pointed out in my question, is not from Stack Overflow, but from Security StackExchange, just like my question.
I haven't yet marked an answer correct, so if you actually have something substantive to add that you think is better than forest's answer, it's possible that you can grab the check.
Regarding that Stack Exchange answer: avoiding the Linux KRNG in favor of alternate RNG's like OpenSSL's is literally the opposite of what cryptographic software security engineers will tell you. For the past decade and a half, virtually all notable cryptographic failures owing to bad randomness came from systems that weren't using the kernel RNG.
Yeah, as you can see, I pushed back on that.
But, as far as I can tell - and I do indeed write with this in mind - the movement to create applications for very fresh, naive installations such as containers or IoT devices, does indeed present a problem for purely-blocking random.
My sense is that you and others in the community encouraging blind trust of /dev/urandom need to adjust your language when consulting on this matter in the future.
Since I mostly write python, I'm happy that this has been worked out and that python now uses blocking random (via getrandom(flags=0)) in these very rare situations.
But, as far as I can tell - and I do indeed write with this in mind - the movement to create applications for very fresh, naive installations such as containers or IoT devices, does indeed present a problem for purely-blocking random.
My sense is that you and others in the community encouraging blind trust of /dev/urandom need to adjust your language when consulting on this matter in the future.
Since I mostly write python, I'm happy that this has been worked out and that python now uses blocking random (via getrandom(flags=0)) in these very rare situations.
Once again: there is exactly one time in the lifecycle of an entire system when you want a blocking read from the Linux KRNG: when the system is first booting, to ensure it's seeded (FreeBSD is smarter about this and handles that automatically). Apart from that one time, every other read on your system should be from urandom.
No, this doesn't need to updated for IoT or "containers" (Linux containers share the KRNG).
No, this doesn't need to updated for IoT or "containers" (Linux containers share the KRNG).
> there is exactly one time in the lifecycle of an entire system when you want a blocking read from the Linux KRNG
Right. I'm glad to hear you phrasing it this way.
And listen: I don't mean to be flamy here, and it's possible that my sense of these things is tainted by our many... colorful discussions here on HN, but the unradom orthodoxy, to which getrandom and python's os.urandom are in part a response, is something for which you are arguably a tiny bit responsible, having actually written "Use urandom. Use urandom. Use urandom. Use urandom. Use urandom. Use urandom" on your blog. That kind of punditry has no place in security engineering.
There is a need for a degree of sobriety with these sorts of security considerations. Sometimes edge cases move toward the mainstream, and that's exactly what has happened with this odd little phenomenon of programs being executed much, much earlier in the OS lifecycle in the case of containerization and IoT devices.
Yes, some containers are deployed in settings where the host KRNG can be and is used. But not all. And many IoT devices suffer from a sufficiently weak set of entropy sources as to need to block.
These are real concerns, as identified by the study I linked from the University of Michigan. The researchers there were able to replicate over 1% of the SSH keys that they found in the wild because the devices that created them had insufficient entropy at the time of the key generation.
All I'm saying is: don't treat these sorts of security issues as subject to orthodoxy. Clear thinking and solid engineering are the orders of the day.
Right. I'm glad to hear you phrasing it this way.
And listen: I don't mean to be flamy here, and it's possible that my sense of these things is tainted by our many... colorful discussions here on HN, but the unradom orthodoxy, to which getrandom and python's os.urandom are in part a response, is something for which you are arguably a tiny bit responsible, having actually written "Use urandom. Use urandom. Use urandom. Use urandom. Use urandom. Use urandom" on your blog. That kind of punditry has no place in security engineering.
There is a need for a degree of sobriety with these sorts of security considerations. Sometimes edge cases move toward the mainstream, and that's exactly what has happened with this odd little phenomenon of programs being executed much, much earlier in the OS lifecycle in the case of containerization and IoT devices.
Yes, some containers are deployed in settings where the host KRNG can be and is used. But not all. And many IoT devices suffer from a sufficiently weak set of entropy sources as to need to block.
These are real concerns, as identified by the study I linked from the University of Michigan. The researchers there were able to replicate over 1% of the SSH keys that they found in the wild because the devices that created them had insufficient entropy at the time of the key generation.
All I'm saying is: don't treat these sorts of security issues as subject to orthodoxy. Clear thinking and solid engineering are the orders of the day.
The blog post you're referring to contains exactly the advice I just gave in the preceding comment.
I don't see a link to a Michigan study in this thread. If you post it again, I'll be happy to explain why it doesn't say that you should use /dev/random, or a userspace RNG.
I don't see a link to a Michigan study in this thread. If you post it again, I'll be happy to explain why it doesn't say that you should use /dev/random, or a userspace RNG.
Well, man, that's not the debate. As so often in these conversations, you're running away with the goalposts. If I may advise: these are exactly the antics that are costing you credibility with those of us out here in the field actually building cryptographic systems.
Good day.
Good day.
I don't understand your objections, like, at all. Every "concession" you think you're extracting is right there, in a blog post I wrote 4 years ago. Those are the goalposts. They haven't moved since 2014.
Everything else you have to say just reads like word salad. I should be concerned about container RNGs, but not Linux containers, unless they're the Linux containers that don't share the KRNG? Or the IOT devices where blocking somehow solves the cold-start entropy problem, and at the same time can't be done in the init scripts? Even the Python issue thread you posted --- yes, I read it --- disagrees with you. I have no idea what you're talking about and, candidly, I don't know that you do either.
I've written (until now) perhaps 1 word on this thread for every 30 of yours, and I think my comments are straightforward and make obvious points. You respond to each of them with a wall of text. Can I suggest that maybe instead of a grand disagreement, you've instead just done a poor job communicating whatever your concern might be?
Regarding that last sentence of yours (er, the one before your "good day"): I am extremely comfortable with who does and doesn't find me credible. Thanks anyways.
Everything else you have to say just reads like word salad. I should be concerned about container RNGs, but not Linux containers, unless they're the Linux containers that don't share the KRNG? Or the IOT devices where blocking somehow solves the cold-start entropy problem, and at the same time can't be done in the init scripts? Even the Python issue thread you posted --- yes, I read it --- disagrees with you. I have no idea what you're talking about and, candidly, I don't know that you do either.
I've written (until now) perhaps 1 word on this thread for every 30 of yours, and I think my comments are straightforward and make obvious points. You respond to each of them with a wall of text. Can I suggest that maybe instead of a grand disagreement, you've instead just done a poor job communicating whatever your concern might be?
Regarding that last sentence of yours (er, the one before your "good day"): I am extremely comfortable with who does and doesn't find me credible. Thanks anyways.
Well look, the problem is solved, at least for people using python. So there's really no need on my end to discuss it further.
As for community communication and understanding: I really don't think there's any good reason that we all have to lock horns all the time. I do think that you'll find more joy in these sorts of discussions if you can be less caustic and more deliberative. It's no more difficult to be nice than it is to be mean; give it a try. And if you get to the edge of your knowledge (and, it seems to me that you do, fairly frequently, which is great - you are obviously interested in a wide variety of societal topics just like I am), don't just keep marching. Stop and acknowledge your short-comings and recognize that there are others who might have had some original / compelling / correct thoughts on these matters also.
Again, this is no skin off my nose. My current project is python, and python has wisely decided to stop using urandom in favor of getrandom(flags=0), so this is a non-issue.
But won't it be nice if the next non-issue can be handled with greater ease and less toxicity?
As for community communication and understanding: I really don't think there's any good reason that we all have to lock horns all the time. I do think that you'll find more joy in these sorts of discussions if you can be less caustic and more deliberative. It's no more difficult to be nice than it is to be mean; give it a try. And if you get to the edge of your knowledge (and, it seems to me that you do, fairly frequently, which is great - you are obviously interested in a wide variety of societal topics just like I am), don't just keep marching. Stop and acknowledge your short-comings and recognize that there are others who might have had some original / compelling / correct thoughts on these matters also.
Again, this is no skin off my nose. My current project is python, and python has wisely decided to stop using urandom in favor of getrandom(flags=0), so this is a non-issue.
But won't it be nice if the next non-issue can be handled with greater ease and less toxicity?
and less toxicity?
Wait a minute, you have made nasty insinuation after nasty insinuation in just about every single comment you've written on this thread and then sprinkled them liberally with finely ground condescension. That's where the toxicity is coming from.
Wait a minute, you have made nasty insinuation after nasty insinuation in just about every single comment you've written on this thread and then sprinkled them liberally with finely ground condescension. That's where the toxicity is coming from.
getrandom(...,0) isn't the blocking version of getrandom, nor is it equivalent to /dev/random. I'm not sure you're even clear what you've been arguing about.
> I just claim that it's bad to block when the entropy estimate is low.
As always when it comes to engineering, good and bad can only be defined when you are specific enough about what it is you're trying to accomplish.
Is it a single-user client system with an intelligent user who won't try to subvert security (maybe because you've defaulted to /dev/urandom but given them a choice to override and use /dev/random), and you're trying to do something of crucial importance like generate a long-lived crypto key? Then blocking might be fine or even the better way.
Is it a multi-user and/or server system where randomness is a scarce resource whose exhaustion could bring down the system like any other scarce resource (disk, network, RAM, CPU, etc.)? Then blocking is probably a terrible idea and could even open you up to denial of service attacks, thus causing you to achieve less security.
As always when it comes to engineering, good and bad can only be defined when you are specific enough about what it is you're trying to accomplish.
Is it a single-user client system with an intelligent user who won't try to subvert security (maybe because you've defaulted to /dev/urandom but given them a choice to override and use /dev/random), and you're trying to do something of crucial importance like generate a long-lived crypto key? Then blocking might be fine or even the better way.
Is it a multi-user and/or server system where randomness is a scarce resource whose exhaustion could bring down the system like any other scarce resource (disk, network, RAM, CPU, etc.)? Then blocking is probably a terrible idea and could even open you up to denial of service attacks, thus causing you to achieve less security.
First problem with this response: programmers have a terrible understanding of what "crucial importance" is. In fact, most things in cryptography that want random values need unpredictable values. The old Linux man page creates the impression that you want "good" random for keys, and "mediocre" random for everything else, which is false.
Second problem is, there is no such thing as "good" and "poor" random. There are two states: unpredictable (seeded) and predictable (not yet seeded). Once you read seeded state, you're done; blocking serves literally no purpose.
Second problem is, there is no such thing as "good" and "poor" random. There are two states: unpredictable (seeded) and predictable (not yet seeded). Once you read seeded state, you're done; blocking serves literally no purpose.
> programmers have a terrible understanding of what "crucial importance" is.
If you cannot understand requirements, you cannot produce satisfactory software.
Anyway, it was intentional that I gave an example where an educated end user is able to make the choice rather than a programmer having to guess.
> The old Linux man page creates the impression that you want "good" random for keys, and "mediocre" random for everything else, which is false.
It certainly is false, and it is a myth that many people believe, but it isn't what I was saying.
> Once you read seeded state, you're done; blocking serves literally no purpose.
As the article explains under the last section ("Not everything is perfect"), /dev/urandom does not guarantee that you are reading seeded state.
Also, those numbers' unpredictability rests on the crypto being really good. Which I'm sure it is, but crypto weaknesses have been discovered in the past. It would be defensible to seek additional protection against that possibility for peace of mind if peace of mind is more important to you than running time and/or availability.
If you cannot understand requirements, you cannot produce satisfactory software.
Anyway, it was intentional that I gave an example where an educated end user is able to make the choice rather than a programmer having to guess.
> The old Linux man page creates the impression that you want "good" random for keys, and "mediocre" random for everything else, which is false.
It certainly is false, and it is a myth that many people believe, but it isn't what I was saying.
> Once you read seeded state, you're done; blocking serves literally no purpose.
As the article explains under the last section ("Not everything is perfect"), /dev/urandom does not guarantee that you are reading seeded state.
Also, those numbers' unpredictability rests on the crypto being really good. Which I'm sure it is, but crypto weaknesses have been discovered in the past. It would be defensible to seek additional protection against that possibility for peace of mind if peace of mind is more important to you than running time and/or availability.
Well written post, I enjoyed it, it makes sense.
The part tearing into the man page was especially excellent!
Following your point a little further.. the thought of updating the man pages with warnings like the following made me laugh:
“AES, SHA-3, RSA, Diffie-Hellman, Elliptic curves, and all those crypto packages you're using, OpenSSL, GnuTLS, Keyczar, your operating system's crypto API, are all theoretically vulnerable to a cryptographic attack. Knowledge of how to do this is not available in the current unclassified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, do not use them.”
The part tearing into the man page was especially excellent!
Following your point a little further.. the thought of updating the man pages with warnings like the following made me laugh:
“AES, SHA-3, RSA, Diffie-Hellman, Elliptic curves, and all those crypto packages you're using, OpenSSL, GnuTLS, Keyczar, your operating system's crypto API, are all theoretically vulnerable to a cryptographic attack. Knowledge of how to do this is not available in the current unclassified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, do not use them.”
This seems to just muddy the waters.
https://sockpuppet.org/blog/2014/02/25/safely-generate-rando... is shorter, more succinct, and to the point.
https://sockpuppet.org/blog/2014/02/25/safely-generate-rando... is shorter, more succinct, and to the point.
On a VM that someone else runs (i.e. Google or AWS), is either /dev/random or /dev/urandom cryptographically secure?
If no, are there any "safe" choices? (other than an external source like an HSM or hardware random number generator)
If no, are there any "safe" choices? (other than an external source like an HSM or hardware random number generator)
Most vms / cloud providers can provide a virtualized source of entropy to be mixed into the pool (google "virtio-rng"). Good OSs (like linux with the right virtio modules) will use that to have a high amount of entropy very early into boot. If that's what you're asking, then the answer is "yes, clouds have good urandom available, usually earlier than on real machines".
If instead you're asking "but can the cloud compromise me?" then the answer is "you're wrecked".
If you do not trust the hardware your vm is running on (be it real or virtual hardware), then you have no hope.
Any source of random numbers you read from will read those numbers into memory. The hypervisor or sufficiently-malicious-hardware will then be able to read your memory and get the data, even if it came from an hsm.
The "safe" choice is to trust that your hardware/hypervisor is not stealing such data.
If you can't trust that, you may have to start fabricating your own processors and running your own data-centres.
Be sure to hire better physical security than AWS has or else it's all for naught when someone comes in and yanks your memory to read it.
If instead you're asking "but can the cloud compromise me?" then the answer is "you're wrecked".
If you do not trust the hardware your vm is running on (be it real or virtual hardware), then you have no hope.
Any source of random numbers you read from will read those numbers into memory. The hypervisor or sufficiently-malicious-hardware will then be able to read your memory and get the data, even if it came from an hsm.
The "safe" choice is to trust that your hardware/hypervisor is not stealing such data.
If you can't trust that, you may have to start fabricating your own processors and running your own data-centres.
Be sure to hire better physical security than AWS has or else it's all for naught when someone comes in and yanks your memory to read it.
What do you mean cryptographically secure? Do you mean secure against Google or AWS attacking you by poisoning or recording the randomness? That type of security doesn't sound possible to me. There is the field of homomorphic encryption that has the goal of protecting you against attacks by your cloud provider, but from what I've heard it's extremely slow and requires to you rearchitect all your jobs.
I think both the diagrams here are simplistic - in reality pre 4.8 there were 3 pools (an input pool and 2 output pools, one of which blocked) and post 4.8 where there are 2 pools, an input pool and a blocking pool (urandom now pulls from the input pool thru a CSPRNG).
One big downside of the new (post 4.8) architecture is that urandom_min_reseed_secs is ignored - pre-4.8 you could ask the kernel to periodically pull a little bit of real entropy from the input to refresh the urandom output (non-blocking) pool, by default that was once every 60 seconds, but in our case with lots of entropy available we'd set it to happen once a second - now post 4.8 it looks like urandom_min_reseed_secs is ignored and the input pool (where /dev/urandom gets its data from) is not regularly updated unless data is also pulled from /dev/random - which may make sense if entropy is rare, but is not so useful when back pressure from /dev/random is left to control the rate at which external entropy flows into the system - I suspect it's time to present a patch that makes urandom_min_reseed_secs work again
Nevertheless the conclusion "use /dev/urandom" is still valid