You Don't Need UUID(henvic.dev)
henvic.dev
You Don't Need UUID
https://henvic.dev/posts/uuid/
199 comments
I think a better argument is that that the identifier needs to be unambiguous in cases where e.g. an ID needs to be communicated over the phone between an end user and a customer service rep. But that is easily solved with encodings like Base58 that exclude ambiguous characters.
When the encoding relies on differentiating between lowercase and capitalized characters, you’re hitting a high level of ambiguity. Hexadecimal UUIDs don’t have this problem.
Once you get over about 10 characters the chance that the user will skip or repeat a character or misread one goes up.
Sometimes when I order parts for my car the site wants the VIN to verify that the parts are correct for that vehicle. I often have to make two or three attempts at entering it to get it right.
Sometimes when I order parts for my car the site wants the VIN to verify that the parts are correct for that vehicle. I often have to make two or three attempts at entering it to get it right.
Eventually, everyone converges to babble[0], BIP-39[1], mnemonic[2], diceware, pgp-word list or equivalent.
[0] https://bohwaz.net/archives/web/Bubble_Babble.html
[1] https://iancoleman.io/bip39/
[2] https://web.archive.org/web/20090918202746/http://tothink.co...
[0] https://bohwaz.net/archives/web/Bubble_Babble.html
[1] https://iancoleman.io/bip39/
[2] https://web.archive.org/web/20090918202746/http://tothink.co...
I don't know of anyone who has converged on that.
Although after reading that Babble link it sounds like a great idea for product id's. Scenarios where it is unlikely for people to have to read it out but when they do it's easy enough to convey.
I regret that there are scenarios that we should have converged on it but didn't.
Although after reading that Babble link it sounds like a great idea for product id's. Scenarios where it is unlikely for people to have to read it out but when they do it's easy enough to convey.
I regret that there are scenarios that we should have converged on it but didn't.
you might want to consider an alternative with a base32 encoding with a luhn checksum
https://github.com/tttp/dxid
https://github.com/tttp/dxid
[deleted]
Plus I could easily represent UUID in base64, such as JlEt5BSYe0enwB7nxl5V6g, which makes it shorter if I need that.
Maybe use a smaller alphabet though, base64’s can be quite hard to read / spell out / reproduce as it includes confusing pairs.
While somewhat less dense, base58 or rfc 4648 base32 mitigate these issues.
While somewhat less dense, base58 or rfc 4648 base32 mitigate these issues.
And for UUIDs, Base58 and Base64 both require 22 characters, so no density loss! Even Base32 only takes you up to 26 characters.
Some alphabets have the disadvantage of generating swears, which might be a problem for your use. Applying filters is a difficult approach. Hex has the advantage there.
> You can't find the product sir? Try typing into the search bar DEADBEEF
If you think of the worse ID you can spell with the full alphabet next, please don't write it here :)
Base32 or Base36 have the advantage of using single case which is easier to read out. And no symbols.
The official Base32 doesn't work for encoding UUID7, which is time-ordered, because its symbols for 0-31 are not in ascending ASCII order.
My own preferred format for UUID7 is one I call "id25". It's really Base35 because alphabets of 35 and 36 characters both need 25 characters to represent the 128 bits in a UUID. So I can start with Base36 and take out one of the next most ambiguous character pairs. The result looks something like '0pydgw5pifvapk5zyhmpso5tx'.
The two other advantages of using this id25 format rather than UUID7 are:
- The id25 format is quite distinct from UUID4. So if you have UUID7s being generated distributedly (rather than centrally) and the UUID7 time-ordering feature is important, it's nice to have a format that a trivial "if '-' in uuid_string..." check will spot.
- Because there are no hyphens or symbols, the whole id25 uuid can be selected in web page with a double click. Whereas a uuid will need a mouse movement to get all 5 parts.
My own preferred format for UUID7 is one I call "id25". It's really Base35 because alphabets of 35 and 36 characters both need 25 characters to represent the 128 bits in a UUID. So I can start with Base36 and take out one of the next most ambiguous character pairs. The result looks something like '0pydgw5pifvapk5zyhmpso5tx'.
The two other advantages of using this id25 format rather than UUID7 are:
- The id25 format is quite distinct from UUID4. So if you have UUID7s being generated distributedly (rather than centrally) and the UUID7 time-ordering feature is important, it's nice to have a format that a trivial "if '-' in uuid_string..." check will spot.
- Because there are no hyphens or symbols, the whole id25 uuid can be selected in web page with a double click. Whereas a uuid will need a mouse movement to get all 5 parts.
Do many use cases need lexical ordering for the encoding of the UUID? (The UUIDs themself would still be ordered; it's just the human/URL/wire encoding that isn't.)
Most of the use cases I can think of where I'd want an ordered UUID, I probably wouldn't need to lexically compare the encoding of multiple such IDs (where I can see plenty of utility in the ability to compare the underlying IDs).
Most of the use cases I can think of where I'd want an ordered UUID, I probably wouldn't need to lexically compare the encoding of multiple such IDs (where I can see plenty of utility in the ability to compare the underlying IDs).
For sure yes! There are tons of situations where you need to use the string form of an ID, e.g. filenames, and you definitely want the lexical ordering of files in a directory to be correct.
I've always seen the human encoding as the main point, so you can infer things about them while working with them. A database, for example, would have a datetime column that can be used instead most of the time.
Crockford base32 works, doesn't it?
The hazard of base64 is the use of '+', '/', and '=' characters, which most HTTP engines treat as special.
Luckily the spec provides for a url safe variant (sometimes called base64url): https://datatracker.ietf.org/doc/html/rfc4648#section-5
It exists, but I wouldn’t call that “lucky”, or a selling point.
I like your thinking, but somehow fear that at some point in time in the future I'll get an ID like "S5o0QlI1" and my brain will automatically eject itself from my skull in disgust.
> that the average end user cares to be dealing with any random string of numbers and digits.
A developer, which I took to be "the user" for the purposes of this writeup, cares. My small concern is that the rand is insufficient to generate a unique enough string (just use a lib like snowflake to get overkill entropy), but I'm sick of having the format at all for inconsequential ids (eg https://www.uuidgenerator.net/).
A developer, which I took to be "the user" for the purposes of this writeup, cares. My small concern is that the rand is insufficient to generate a unique enough string (just use a lib like snowflake to get overkill entropy), but I'm sick of having the format at all for inconsequential ids (eg https://www.uuidgenerator.net/).
Also as a developer I really dislike UUIDs because the hyphens make it impossible to double-click copy.
It’s a small gripe, but I need to copy IDs multiple times per day and it adds up. Other ID formats like cuid or KSUID don’t have hyphens in their canonical representation, and it makes them far more pleasant to work with
It’s a small gripe, but I need to copy IDs multiple times per day and it adds up. Other ID formats like cuid or KSUID don’t have hyphens in their canonical representation, and it makes them far more pleasant to work with
Thats typically something you can configure in your terminals settings. Though avoiding requiring everyone know how to do this and actually do this is an improvement in developer experience so there's some merit here.. but is there any real standards around what characters get included in copying like this, or any best practices based on common defaults?
if not it might just become a situation of optimizing for your own workflow. and considering how ubiquitous uuids are and how common dash delimiters are outside of that(jira ticketscome to mind) maybe it would be better to fix places where double click selections do not include dashes instead of trying to avoid dashes in identifiers?
if not it might just become a situation of optimizing for your own workflow. and considering how ubiquitous uuids are and how common dash delimiters are outside of that(jira ticketscome to mind) maybe it would be better to fix places where double click selections do not include dashes instead of trying to avoid dashes in identifiers?
If that’s your issue passing the formatting option to your GUID library to get the dash-free solution seems like it would resolve it.
Yes! This might seem silly, but if you're working with a lot of data, it can be a pain and time drain...
I don't think it's a small gripe, IMO it's a critical UX issue!
ULIDs don't have this problem, for the record.
ULIDs don't have this problem, for the record.
I read the article, scratched my head, pondered, came here to see if anyone else was confused, and I'm glad I did.
The crux of this argument seems to be that UUIDs are too long? Which I disagree with. I can't memorize them, no, and it would be cumbersome to try to say one aloud, but these aren't situations I've ever found myself in.
Does it make the URL in the URL bar longer? Yeah, but does that matter?
Does it make the URL in the URL bar longer? Yeah, but does that matter?
> Does it make the URL in the URL bar longer? Yeah, but does that matter?
No, it doesn't.
If you're using a central system to generate id's (UUID's or other), then you are indeed missing a lot of the benefits of UUID's and might as well use some other ID scheme.
The benefits of UUID are many systems can generate them, and your database can eventually accept them, all with a very high guarantee of uniqueness/no-collisions. Not to mention the other nice bits like non-sequential and non-enumerability.
Most modern databases have UUID/Binary types/functions to allow for more efficient storage/handling anyway.
No, it doesn't.
If you're using a central system to generate id's (UUID's or other), then you are indeed missing a lot of the benefits of UUID's and might as well use some other ID scheme.
The benefits of UUID are many systems can generate them, and your database can eventually accept them, all with a very high guarantee of uniqueness/no-collisions. Not to mention the other nice bits like non-sequential and non-enumerability.
Most modern databases have UUID/Binary types/functions to allow for more efficient storage/handling anyway.
For a simple practical example, it's awfully useful for a lot of systems to be able to generate UUIDs in an app while offline (think document creation, etc) and reconcile things later by shooting the blob of content off to the server, while knowing that you only need to handle the dance of getting IDs and updating stuff locally again in your error handling case.
That’s an excellent point; that can end up being a bottleneck across many systems.
What can?
Unique ID generation.
A service responsible for handing out UIDs.
> Does it make the URL in the URL bar longer? Yeah, but does that matter?
I appreciate shorter URLs any time I copy and paste them, which always involves looking at them and sometimes involves scrolling to the end to remove tracking- and search-related fluff.
128 bits is an absurd amount for a unique ID within a single system. Even 48 bits is very, very large -- enough to provide a unique ID (MAC address) to every Ethernet device made for something like 100 years.
The purpose of a UUID is to provide a negligible probability of collisions between randomly-generated IDs in a global space forever. Almost no applications require that, so most of the time using a UUID is just needlessly taking up space.
I appreciate shorter URLs any time I copy and paste them, which always involves looking at them and sometimes involves scrolling to the end to remove tracking- and search-related fluff.
128 bits is an absurd amount for a unique ID within a single system. Even 48 bits is very, very large -- enough to provide a unique ID (MAC address) to every Ethernet device made for something like 100 years.
The purpose of a UUID is to provide a negligible probability of collisions between randomly-generated IDs in a global space forever. Almost no applications require that, so most of the time using a UUID is just needlessly taking up space.
> 128 bits is an absurd amount for a unique ID within a single system. Even 48 bits is very, very large -- enough to provide a unique ID (MAC address) to every Ethernet device made for something like 100 years.
What you want is that the number of possible items to enumerate is significantly less than the square root of the cardinality of the ID range, means you can relatively safely randomly generate IDs with few collissions.
What you want is that the number of possible items to enumerate is significantly less than the square root of the cardinality of the ID range, means you can relatively safely randomly generate IDs with few collissions.
Basically, you want UUID's 122 bits (128 minus 6 wasted on variant + version) if you can't afford collisions. If you have some DB enforcing unique IDs at creation time, you can manage with a lot fewer bits if you want shorter URLs or something.
That’s not the only major purpose for a UUID and even if it was, it still would be flat out wrong to claim that almost no application requires that.
Maybe I'm missing something, then? I'm having trouble thinking of applications where it seems like a good idea to generate a permanent ID number completely randomly at all, much less applications that need to let the whole world do so. I can see how Windows device drivers have that requirement, but the point of the article is that even a massive site like YouTube (which doesn't want videos to be randomly discoverable via URL manipulation) does not need anywhere close to 128 bits. What are these applications you speak of?
The purpose of a universally unique identifier was indeed to be universally unique.
>Yeah, but does that matter?
I'm really reaching here, but an ID that omits special characters is easier to extract from a url, since it can be double-clicked in a URL bar to select it, whereas a GUID with hyphens forces the user to select the beginning and the end of the string.
Realistically I think this matters to developers more than users (I hope your application doesn't force users to interact with GUIDs in the address bar) but that's the only "waste" I can identify.
EDIT: Someone elsewhere noted that they can base64 encode their GUIDs to achieve this and I'm seething that I'd never thought to do that
I'm really reaching here, but an ID that omits special characters is easier to extract from a url, since it can be double-clicked in a URL bar to select it, whereas a GUID with hyphens forces the user to select the beginning and the end of the string.
Realistically I think this matters to developers more than users (I hope your application doesn't force users to interact with GUIDs in the address bar) but that's the only "waste" I can identify.
EDIT: Someone elsewhere noted that they can base64 encode their GUIDs to achieve this and I'm seething that I'd never thought to do that
As you said in your edit, you can of course base64 the GUID, but also don't most GUID libraries have an option to just omit the hyphens? I know .NET allows you to do that, and even correctly parse GUIDs that don't have any hyphens.
I usually did that in .NET simply because I thought it was prettier without the hyphens.
I usually did that in .NET simply because I thought it was prettier without the hyphens.
To select an UUID, double click the first segment and drag the selection till the last segment before releasing. No need to aim for the exact beginning and end characters.
(Maybe that's what you meant, amd yes, it's still clunkier than simply double clicking, but not much)
(Maybe that's what you meant, amd yes, it's still clunkier than simply double clicking, but not much)
> I'm really reaching here, but an ID that omits special characters is easier to extract from a url, since it can be double-clicked in a URL bar to select it, whereas a GUID with hyphens forces the user to select the beginning and the end of the string.
The is-hyphen-part-of-a-word issue depends on the text control (browser, UI widget library, …) / terminal you use, which is not universal.
In any case, the hyphens/dashes are not required part of the representation, and one can use the “dashless” representation (exactly 32 hex digits) in URLs and in-code without any issues whatsoever, as far as I know; for example, Postgres accepts either “dashful” or “dashless” UUIDs the same.
The is-hyphen-part-of-a-word issue depends on the text control (browser, UI widget library, …) / terminal you use, which is not universal.
In any case, the hyphens/dashes are not required part of the representation, and one can use the “dashless” representation (exactly 32 hex digits) in URLs and in-code without any issues whatsoever, as far as I know; for example, Postgres accepts either “dashful” or “dashless” UUIDs the same.
> easier to extract from a url, since it can be double-clicked
Maybe the double-click logic needs to be less stupid then. Or have it progressive, e.g. double-click = alphanumeric, triple-click = stop at symbols, quadruple-click = stop at slashes only, quintiple-click = the whole URL.
Use the device microphone to listen for swear words and adjust algorithm with machine learning accordingly
Maybe the double-click logic needs to be less stupid then. Or have it progressive, e.g. double-click = alphanumeric, triple-click = stop at symbols, quadruple-click = stop at slashes only, quintiple-click = the whole URL.
Use the device microphone to listen for swear words and adjust algorithm with machine learning accordingly
It matters aesthetically. A huge site like Amazon might care about that, otherwise it's not a high priority.
It somewhat matters aesthetically, but mostly people are going to ignore long random strings.
UUIDs are at least visually kind of white noise, companies have no problem+with+links?that=look&like=this, so why do they care about UUIDs?
UUIDs are at least visually kind of white noise, companies have no problem+with+links?that=look&like=this, so why do they care about UUIDs?
Cause some links are more likely to be shared, meaning they'll appear in a comment or SMS rather than just in your URL bar. Like if I copy a link from YouTube into HN: https://www.youtube.com/watch?v=00Uw9kgZ4QM
I'm sure they put some thought into not making it ?v=f6dde15a-3485-4f36-b0b9-a3b56fbc12ae.
I'm sure they put some thought into not making it ?v=f6dde15a-3485-4f36-b0b9-a3b56fbc12ae.
apple.com/iphone
You can't beat that. Or would you rather receive texts that include the following?
Let's not match the lowest common denominator, shall we?
You can't beat that. Or would you rather receive texts that include the following?
https://www.amazon.com/dp/B09V3HZ8B5/?pd_rd_w=DAycy&content-id=amzn1.sym.04e31e32-5e01-4048-8b59-a557891d595f:amzn1.sym.04e31e32-5e01-4048-8b59-a557891d595f&pf_rd_p=04e31e32-5e01-4048-8b59-a557891d595f&pf_rd_r=M1KXM468KNZJV189JBE7&pd_rd_wg=3O1uC&pd_rd_r=e51c1edc-213f-4c6d-b081-e23afa6bf4ab&ref_=vn_s_iwp&qid=1694453553
Let's not match the lowest common denominator, shall we?
Sure, if Amazon sold about 10 products, such a URL scheme would work great. Additionally, this constantly redirects to the latest version of the iPhone, whereas Amazon's URLs are pointing to a specific version.
A more "apples to apples" comparison (pun intended) would be the MagSafe charger:
https://www.apple.com/shop/product/MHXH3AM/A/magsafe-charger...
A more "apples to apples" comparison (pun intended) would be the MagSafe charger:
https://www.apple.com/shop/product/MHXH3AM/A/magsafe-charger...
You only really need this part:
https://amazon.com/dp/B09V3HZ8B5
https://amazon.com/dp/B09V3HZ8B5
Yes but that's irrelevant, I used a real URL that you will receive in your texts. It's real life, it happens and it sucks. Let's not normalize it.
No. It's not irrelevant.
As you have been told apple.com/iphone = www.amazon.com/dp/B09V3HZ8B5
The rest of what you posted on the Amazon link is tracking bullshit. Now, THAT is irrelevant !
As you have been told apple.com/iphone = www.amazon.com/dp/B09V3HZ8B5
The rest of what you posted on the Amazon link is tracking bullshit. Now, THAT is irrelevant !
Users aren't expected to manipulate URLs manually. If your URLs have extra stuff, that's what people are going to copy/paste into messages.
Amazon product URLs are looong, but maybe people don't share them enough for Amazon to care. YouTube and Twitter put more effort into making them short.
Amazon product URLs are looong, but maybe people don't share them enough for Amazon to care. YouTube and Twitter put more effort into making them short.
I wonder if this is something Unicode could address. You could have something like quote marks that mean "Anything between these is data that shouldn't be seen in digital media but should be printed"
So you could have urls like:
example.com?<a single icon>&productname=somethinghumanreadable
So you could have urls like:
example.com?<a single icon>&productname=somethinghumanreadable
The browser would also have to understand not to copy that part by default. But HTTP disallows unicode chars in URLs. There could instead be some "sharable URL" web API, but it seems like a small enough problem that sites are content just providing a custom share button if needed.
[deleted]
Apple doesn't need to create pages for millions of arbitrary items for sale.
The point is that you can always always always do better than a 100-character URL.
Some of us will strip off the crud to get the base URL, to avoid being tracked.
> Does it make the URL in the URL bar longer? Yeah, but does that matter?
Yes. That's one of the author's arguments.
) A simple ID like 3c6n63N is more than enough to represent any product while keeping it readable and making communication easier. A UUID alternative like a73ba12d-1d8b-2516-3aee-4b15e563a835 is just wasteful from an user’s perspective.
Yes. That's one of the author's arguments.
) A simple ID like 3c6n63N is more than enough to represent any product while keeping it readable and making communication easier. A UUID alternative like a73ba12d-1d8b-2516-3aee-4b15e563a835 is just wasteful from an user’s perspective.
The point of a UUID is to have a unique machine ID that can be generated in parallel by separate, non-synchronized processes.
You don't need to make it a user exposed identifier, and you can have a centralized (but out-of-the-critical-path) process assigning friendly IDs if you can afford visibility delays (for either the whole item or at least the friendly ID, which if it is the only user-facing locator may mean that the individual item can’t be directly accessed even if it is visible in aggregates, until that is assigned; if you allow access by both the UUID and the friendly ID, with the latter presented as preferred once available, this is resolved but this may appear messier.)
You don't need to make it a user exposed identifier, and you can have a centralized (but out-of-the-critical-path) process assigning friendly IDs if you can afford visibility delays (for either the whole item or at least the friendly ID, which if it is the only user-facing locator may mean that the individual item can’t be directly accessed even if it is visible in aggregates, until that is assigned; if you allow access by both the UUID and the friendly ID, with the latter presented as preferred once available, this is resolved but this may appear messier.)
Exactly. Also, not everything is going to be an URL, or human-visible at all.
Others have noted the usefulness of generating ids for user-created data without involving the server, while being able to post that data at a later time without worrying too much about collisions.
Suspenders-and-belt practice would of course dictate that the server do some type of collision detection and mitigation before storing the received data.
Personally, I'd look on home-brewed solutions for generating unique "friendly ids" with the same deep suspicion I look at home-brewed crypto, particularly if this were done client-side for multiple clients without involving a server round-trip. Getting it right is a Hard Problem. The perceived complexity of UUIDs is there precisely because it is a Hard Problem.
Others have noted the usefulness of generating ids for user-created data without involving the server, while being able to post that data at a later time without worrying too much about collisions.
Suspenders-and-belt practice would of course dictate that the server do some type of collision detection and mitigation before storing the received data.
If UUID exists in database and user ID is different, tell client "Yo, we've got a collision here. Your new ID for this data is blah".
Generate new UUID. If I've ever used that UUID before, generate a new one.
If the server rejects my UUID, switch to the new one provided by the server
Personally, I'd look on home-brewed solutions for generating unique "friendly ids" with the same deep suspicion I look at home-brewed crypto, particularly if this were done client-side for multiple clients without involving a server round-trip. Getting it right is a Hard Problem. The perceived complexity of UUIDs is there precisely because it is a Hard Problem.
Maybe it’s one if the author’s arguments, but that doesn’t mean the author is right.
Sure — it's just a weird question to pose when the author specifically provides their opinion on it. It'd be different to say "I don't think it matters".
You're just nitpicking.
"Does it matter" is a very reasonable response to someone expressing their opinion.
"Does it matter" is a very reasonable response to someone expressing their opinion.
It's not nitpicking.
If I say "x matters and here's why" and you respond "yeah but does x matter?", you're either not paying attention or being rude and dismissive.
If I say "x matters and here's why" and you respond "yeah but does x matter?", you're either not paying attention or being rude and dismissive.
I guess in the rare instance where you're literally reading a URL out loud it would help. Most of the time you're sending a URL, at which point length is mostly irrelevant as you're "copy/paste" on both sides.
Must be oral communication, otherwise copy and paste the link.
And I bet that for most users 3c6n63N is as confusing as a73ba12d-1d8b-2516-3aee-4b15e563a835
And I bet that for most users 3c6n63N is as confusing as a73ba12d-1d8b-2516-3aee-4b15e563a835
There is the rub. Readable for what purpose? Aesthetics? And why? It is meaningless in their example.
And what do they mean with communication? Reading it aloud? If the URL is too long to stick in a tweet or message, there are plenty of URL shorteners out there.
And what do they mean with communication? Reading it aloud? If the URL is too long to stick in a tweet or message, there are plenty of URL shorteners out there.
Copying very long URL’s into a Markdown document or a social media post is pretty annoying. URL shorteners are opaque and add a point of failure.
On link-sharing sites like this one, we care about the URL’s we share and will often strip off the tracking stuff.
On link-sharing sites like this one, we care about the URL’s we share and will often strip off the tracking stuff.
Agreed. Decorative URL slugs have been successfully used by everything from blogs to Amazon itself for over a decade. If anything, that should be the focus here. Short URLs aren't much of a help, especially when short is relative on something like a mobile device.
> Yeah, but does that matter?
It might, e.g. with embedded systems. I can neither confirm nor deny that we once BOFed our own devices because we assumed that 1200 bytes was a big enough buffer to hold a URL.
It might, e.g. with embedded systems. I can neither confirm nor deny that we once BOFed our own devices because we assumed that 1200 bytes was a big enough buffer to hold a URL.
>Does it make the URL in the URL bar longer? Yeah, but does that matter?
Nowadays many links are much longer because of referrer information and tracking parameters. So the UUID doesn't add much.
Nowadays many links are much longer because of referrer information and tracking parameters. So the UUID doesn't add much.
I noticed a recent change on YouTube where the "share" button now includes extra tracking stuff, making it longer than the actual URL in the address bar. Which is even funnier considering that they bought the youtu.be domain just to make shared URLs shorter.
Address bar: https://www.youtube.com/watch?v=SHWhKRKlG98
Share button: https://youtu.be/SHWhKRKlG98?si=gwQHHze3UnjTobJg
Address bar: https://www.youtube.com/watch?v=SHWhKRKlG98
Share button: https://youtu.be/SHWhKRKlG98?si=gwQHHze3UnjTobJg
They might if several sets of tracking information end up there, because people are blocking the cookies they might otherwise be in for instance, and each set of tracking info has an identifier or few. The length of UUIDs will multiply up quite quickly there.
Not that people will care. Once a URL is longer than a few words plus a code most users won't even look, even if they bother that far. You'll be in danger of starting to hit some browser/server/proxy limits though.
Not that I mind that sort of information being lost, I'm happy for the incessant tracking of everyone's everything these days needs to go to hell.
Not that people will care. Once a URL is longer than a few words plus a code most users won't even look, even if they bother that far. You'll be in danger of starting to hit some browser/server/proxy limits though.
Not that I mind that sort of information being lost, I'm happy for the incessant tracking of everyone's everything these days needs to go to hell.
> these aren't situations I've ever found myself in
You've never needed to give an order number, member ID, account ID, invoice number, etc?
You've never needed to give an order number, member ID, account ID, invoice number, etc?
When it comes to random identifiers, the advantage of uuid4 is mainly that it's a standard thing that everyone understands well, and finding a lib that generates one securely won't require any thought. It's never the ideal solution, but it's often good enough.
I would love to use short UUIDs, but every part of my stack can get a long one with a single, trusted import.
I agree. If there was a shorter standard, I would use that for most cases.
I agree. If there was a shorter standard, I would use that for most cases.
There are so many nuances and edge cases not mentioned in the article. The author needs to do several projects in real life before making global pronouncements like this.
Someone else implied this in another comment but this line appears at the end of the article:
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
When one wonders why that was included they might discover that the "short ID" example given near the top of the post is an Amazon link. One might then further wonder what the purpose of this article is.
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
When one wonders why that was included they might discover that the "short ID" example given near the top of the post is an Amazon link. One might then further wonder what the purpose of this article is.
The lack of humbleness and respect for devs who have came before you and solved all these problems is coming through strong. But I myself was that way 15 years ago. It takes a lot of hard lessons to realize everything new is old and we truly stand on the shoulders of 7000 giants all standing on each others’ shoulders.
Did you even read beyond the title? To say that because of this choice of title, I lack humbleness or respect for peers who came before me is despicable.
My title is too presumptive. Yet, the reality for almost everyone who doesn't interface with other systems that already require UUID is that.
The brief article gives some points – backed by trustworthy sources – and shows some alternatives without further ado. I didn't spend days writing it. I could cover many more scenarios but didn't invest my time in it. So what?
My title is too presumptive. Yet, the reality for almost everyone who doesn't interface with other systems that already require UUID is that.
The brief article gives some points – backed by trustworthy sources – and shows some alternatives without further ado. I didn't spend days writing it. I could cover many more scenarios but didn't invest my time in it. So what?
> Did you even read beyond the title?
Not the person you're replying to, but the end of the first paragraph in your article is
> and I want you to understand why you *certainly* don’t need it
(emphasis mine)
Not the person you're replying to, but the end of the first paragraph in your article is
> and I want you to understand why you *certainly* don’t need it
(emphasis mine)
It's good to get a fresh perspective sometimes
Oh yeah? Indulge me, please.
IMO, a good middleground is using schemes like TypeID[0], ulid[1], or KSUID[2] that provides a more compact and readable (base32) representation and provides better database locality (K-sortable).
[0] https://github.com/jetpack-io/typeid [1] https://github.com/ulid/spec [2] https://github.com/segmentio/ksuid
[0] https://github.com/jetpack-io/typeid [1] https://github.com/ulid/spec [2] https://github.com/segmentio/ksuid
ulids are beautiful little ids without many of the drawbacks and risks of other id systems. They are sufficiently unique for most work (anywhere that you don't need to actually uniquely engineer your ids) and their built in ordering and slick encoding are just icing on the cake.
As noted on an earlier HN submission about typeid, K-sortable only matters to distributed DBs, and things get tricky there. Spanner for example says to use deliberately unsorted primary keys.
I really think when you're picking PKs, you should simply use whatever the DBMS recommends for performance. It's not the PK's job to be typed, sequenced, human-readable, or anything like that; that can be handled by other cols and logging rules. Its job is to be fast in those joins you'll constantly be making against it.
I really think when you're picking PKs, you should simply use whatever the DBMS recommends for performance. It's not the PK's job to be typed, sequenced, human-readable, or anything like that; that can be handled by other cols and logging rules. Its job is to be fast in those joins you'll constantly be making against it.
Sortable primarily keys matters very much for any DB clustering on them, e.g. MySQL/InnoDB and SQL Server.
It also matters, albeit to a much lesser extent, to others. Postgres stores tuples in a heap, but the PK is still a B+tree (ish), so an INSERT or UPDATE heavy workload will suffer somewhat.
It also matters, albeit to a much lesser extent, to others. Postgres stores tuples in a heap, but the PK is still a B+tree (ish), so an INSERT or UPDATE heavy workload will suffer somewhat.
Yeah, you're right. But for those, the typical PK recommendation of bigserial is already time-ordered and mostly packed.
I'm recently finding Cuid2 to be the best of these alternative GUIDs. They seem to have all of the benefits for what you would want to use a GUID for, but none of the drawbacks of existing implementations.[1]
[1]: https://github.com/paralleldrive/cuid2#comparisons
[1]: https://github.com/paralleldrive/cuid2#comparisons
Cuid2 is explicitly not K-sortable, no?
Isn't UUIDv7 K-sortable?
Author here. I posted this because I've witnessed many systems in companies I've worked for where our end-users needed UUID to communicate with it (technical support, customer ID, etc.) in a way that makes communication harder. We could've used another shorter ID scheme, which would be fine.
The good thing about UUID is that it's omnipresent. From what I've heard, it's this lengthy (2^32) because it was hard to guarantee uniqueness when it was conceived in the telecom industry. The length is overkill, and per se, that's fine, but the fact that it dampers communication is awful.
That all said, since posting this, I've come to terms with accepting that it's part of life ¯\_(ツ)_/¯
P.S. Using a second human-friendly ID to end-users is an alternative adopted by some projects. However, most projects don't bother, and also, most good IDs you might want to share with people would make UUID unnecessary anyways (in practice).
The good thing about UUID is that it's omnipresent. From what I've heard, it's this lengthy (2^32) because it was hard to guarantee uniqueness when it was conceived in the telecom industry. The length is overkill, and per se, that's fine, but the fact that it dampers communication is awful.
That all said, since posting this, I've come to terms with accepting that it's part of life ¯\_(ツ)_/¯
P.S. Using a second human-friendly ID to end-users is an alternative adopted by some projects. However, most projects don't bother, and also, most good IDs you might want to share with people would make UUID unnecessary anyways (in practice).
Just use the first N characters of the UUID if you want a shorter version of the UUID.
And if you want to be really human friendly, use one of those silly name generators.
And if you want to be really human friendly, use one of those silly name generators.
> it's this lengthy (2^32)
UUIDs are 128 bits, so it's 2^128 rather than 2^32
UUIDs are 128 bits, so it's 2^128 rather than 2^32
There is no fundamental tech advance that caa make the birthday paradox irrelevant or obsolete, and there never will be.
all of cryptocurrency relies on sha256 the fact that the space is huge, and just ignores collisions. seems to work fine.
There have been so many incidents in crypto that exploited broken assumptions about hash collisions.
Factually, X != hash(X). Sometimes you can make the simplifying assumption that X == hash(X), but only in well-defined contexts, subject to proper risk analysis; never in general, or as a presumption of a system that needs to be correct.
Factually, X != hash(X). Sometimes you can make the simplifying assumption that X == hash(X), but only in well-defined contexts, subject to proper risk analysis; never in general, or as a presumption of a system that needs to be correct.
SHA-256 has, well, 256 bits of entropy. What I took issue with was the claim that UUID's 128 bits (well, almost, anyway) are overkill with modern advances in randomness.
It's amusing that the UUIDs are considered 'ugly' while the Amazon and YouTube IDs are not. Beauty is in the eye of the beholder. I don't find UUIDs pretty, at all, but the others are even uglier to my taste.
Yes, UUIDs are overkill for most applications. But CPUs and hard drives are, relatively speaking, cheap. Using an existing, battle-tested unique ID library implementation has advantages. The 8 bytes per record you're saving over bigserial is, for most use cases, negligible. 1,000,000 rows? You'll save 8 MB by switching away from UUIDs.
Most databases won't be that large. Use a UUID if you want; pretend you'll have Really Big Data some day if it makes you happy. Render it using a special function if the hyphens are too ugly.
Yes, UUIDs are overkill for most applications. But CPUs and hard drives are, relatively speaking, cheap. Using an existing, battle-tested unique ID library implementation has advantages. The 8 bytes per record you're saving over bigserial is, for most use cases, negligible. 1,000,000 rows? You'll save 8 MB by switching away from UUIDs.
Most databases won't be that large. Use a UUID if you want; pretend you'll have Really Big Data some day if it makes you happy. Render it using a special function if the hyphens are too ugly.
> A UUID alternative like a73ba12d-1d8b-2516-3aee-4b15e563a835 is just wasteful from an user’s perspective.
The argumentation in this article is pretty poor from my experience. A UUID isn’t meant to be handled by the non-technical end user. The end user usually doesn’t and shouldn’t care about the URL. I can assure there are bigger architectural problems in your design if your user has to care about accessible internal ids.
The argumentation in this article is pretty poor from my experience. A UUID isn’t meant to be handled by the non-technical end user. The end user usually doesn’t and shouldn’t care about the URL. I can assure there are bigger architectural problems in your design if your user has to care about accessible internal ids.
Does this help see what's wrong with UUIDs in URLs? A GitHub issue URL in a parallel universe:
https://github.com/a73ba12d-1d8b-2516-3aee-4b15e563a835/e16957bf-7d7f-41f1-97f4-98c93a6c3540/issues/4e4e9133-bcdb-45b9-8dfe-9e951846e3c6I see bad url design for other reasons but nothing wrong with UUIDs, so no it doesn’t help me see the argument.
What do you mean by bad url design? Because it's using UUIDs? That was my point.
Not OP but it's obvious to me that by bad url design, they mean that you could have used usernames and issue numbers in the URL but you chose to go with random IDs where there was no need for it.
A fairer comparison would be that in that universe the url would be:
https://github.com/4e4e9133-bcdb-45b9-8dfe-9e951846e3c6
urn:uuid:4e4e9133-bcdb-45b9-8dfe-9e951846e3c6
Absolutely. The issue uuid should then be a UUIDv5 namespaced to the context and you only have one globally unique id. I think people just don't understand uuids properly and treat them as a "gives me some random thingy".
An issue that is not solved by either UUIDv4 or the proposed solution (random base58 strings) is indexing performance.
Both of those solutions typically make it hard for a DB if you write new entries, assuming you have an index on the ID.
In addition it might be more calming to actually be sure that a particular ID is not in use without doing a round-trip.
Is it practical to pre-allocate empty entries and reserve a set of them?
Both of those solutions typically make it hard for a DB if you write new entries, assuming you have an index on the ID.
In addition it might be more calming to actually be sure that a particular ID is not in use without doing a round-trip.
Is it practical to pre-allocate empty entries and reserve a set of them?
> An issue that is not solved by either UUIDv4 or the proposed solution (random base58 strings) is indexing performance.
Lots of languages are catching up to UUIDv7, which solves the indexing issue.
Lots of languages are catching up to UUIDv7, which solves the indexing issue.
Are there modern databases that can’t readily index on a 128-bit value?
I don't think it's a matter of not being able to index on it, but that mostly random UUIDs (like v4) can lead to some interesting index fragmentation you have to stay on top of somehow.
If you use sequential ids, the IDs most in use are the higher ones. So your index is “hot at the top” and you can keep that part in memory.
This is only a problem with extremely big indexes.
This is only a problem with extremely big indexes.
Sort orders are complicated for UUIDs because of the interesting defined structure to them, and of course endian issues.
One fun and useful reference: https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=10...
It's particularly interesting that Microsoft SQL Server was designed to optimize indexing for UUIDv1 machine IDs. That makes a certain amount of sense for a database cluster if IDs are sorted by machine. Of course, developers don't let developers use UUIDv1 in 2023 because those machine IDs are not secure in a general sense and can be a privacy/data leak in the worst cases.
Other databases sort/index UUIDs differently. There's no real "standard" and optimizing the storage of a UUID key is a game of playing to the strengths of your specific database.
(On one project I put some work into matching the much-better-defined ULID sort order to MS SQL Server uniqueidentifier columns for better database locality.)
One fun and useful reference: https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=10...
It's particularly interesting that Microsoft SQL Server was designed to optimize indexing for UUIDv1 machine IDs. That makes a certain amount of sense for a database cluster if IDs are sorted by machine. Of course, developers don't let developers use UUIDv1 in 2023 because those machine IDs are not secure in a general sense and can be a privacy/data leak in the worst cases.
Other databases sort/index UUIDs differently. There's no real "standard" and optimizing the storage of a UUID key is a game of playing to the strengths of your specific database.
(On one project I put some work into matching the much-better-defined ULID sort order to MS SQL Server uniqueidentifier columns for better database locality.)
Random ids require random inserts and that is bad for performance. It isn't a deal-breaker in most situations but it is a real cost that you do pay.
You already got multiple answers explaining the performance issue.
Now in many, typical applications you'd have to scale up quite significantly before this becomes a problem.
But if your application requires you to store small, new entries very quickly, you'll start to notice this even with moderate scale. Disk persistence is often the bottleneck already, and this might make it worse.
Now in many, typical applications you'd have to scale up quite significantly before this becomes a problem.
But if your application requires you to store small, new entries very quickly, you'll start to notice this even with moderate scale. Disk persistence is often the bottleneck already, and this might make it worse.
> As Tom Scott shows in his video, 11 base58-encoded characters are enough for YouTube to serve content even when considering that private videos should be undiscoverable.
Nitpick: Google isn't concerned about the discoverability of private videos; those can only be viewed when granted access. You're thinking of unlisted videos.
Nitpick: Google isn't concerned about the discoverability of private videos; those can only be viewed when granted access. You're thinking of unlisted videos.
Fixed! Thanks!
I think is an “engineer thinks harder about UX” kind of problem. UUIDs are fine for what they’re good at. Try to avoid making them user-facing because, yeah, they’re ugly. I also wouldn’t hand 128 bit integers to the user if I could avoid it.
Don’t forget: UUIDs are not dash-separated strings. They’re integers. You can render them differently if those dashes are sucky.
Don’t forget: UUIDs are not dash-separated strings. They’re integers. You can render them differently if those dashes are sucky.
I recently found ULID.
I like its simplicity. It is sequential. It has a very low probability of collision. And it is of a more reasonable length.
Because it encodes the time, theoretically you could use it to grab the CreatedDate of a record without a need for another field.
https://github.com/ulid/spec
I like its simplicity. It is sequential. It has a very low probability of collision. And it is of a more reasonable length.
Because it encodes the time, theoretically you could use it to grab the CreatedDate of a record without a need for another field.
https://github.com/ulid/spec
Sequential isn't always simple. If your system is distributed in the slightest way (even just multiple threads), it's a bit of a luxury to find sequential guarantees.
ULIDs can get you rough ordering guarantees even in distributed systems. Precision, of course, requires synchronization, which is usually a non-starter.
Isn't auto generating UUIDs "fast enough" in most situations that its negligible?
In particular databases (where UUIDs taking space was a big concern) they have largely switched to a packed binary format that makes the size of UUIDs over time a non issue for all practical purposes.
In particular databases (where UUIDs taking space was a big concern) they have largely switched to a packed binary format that makes the size of UUIDs over time a non issue for all practical purposes.
UUIDs are packed in DBs. But they tend to get misused as PKs. In common relational DBs, your PK for every table should probably be bigserial.
No, that shouldn’t be the criteria (common relational db) when deciding to use bigserial v UUID but the overall architecture and requirements of your system.
Whatever your overall requirements are, if you've already decided on a relational DB to solve them, you probably want bigserial PKs in said DB. The DBMS is optimized for that (as I've noticed in Postgres), and you don't gain anything by using UUIDs instead there. UUID in a secondary col or elsewhere in your system, that's fine.
If you have to look up by UUID from outside and have no foreign keys, UUID PKs can be faster. But that's the classic use case for a non-relational DB.
If you have to look up by UUID from outside and have no foreign keys, UUID PKs can be faster. But that's the classic use case for a non-relational DB.
I completely agree. UUIDs are almost always evidence of overengineering, especially when user-facing. 64-bits is "enough" of an address space for almost any purpose, even global. And while it seems like speaking an ID over the phone or having to scan it manually is something you'd never have to do, in practice it happens all the time. Cut-and-paste is not always an option on all platforms.
128 bits versus 64 bits is a deal breaker?
Yes. Even 64 bits is too many, really: at 6 bits per human-recognizable token, it's more than 10 tokens, which is outside the range of what almost any human can keep in their working memory. You can't even hold it in your head long enough to type it in a different window. 128 bits is completely beyond that; when confronted with 128-bits like a UUID, people just give up. Seriously, try actually typing in a UUID sometime. Even with the grouping it's incredibly difficult. Between 64-bits and 128-bits you discover that you absolutely need an automated way to transmit the data, and then how do you get it to that airgapped computer?
The more important requirement is that the id should not be guessable and not easily be brute-forced (example: YouTube private video listings). Now, what's the minimum required entropy to make it unguessable but also make it short enough to have people tell the id over the phone or something?
No comments on this page about Snowflake IDs.
https://en.wikipedia.org/wiki/Snowflake_ID
They fit within 64 bits, allow for more than enough processes to handle 10k+ transactions per second, give enough of a timestamp headroom for decades into the future, and where ID generation can be made isolated to each process.
They don't work well for anything related to archival work, but you might as well use a regular ID for that anyway, unless you're also actively scraping terabytes of data off of the Internet every second, in which case UUIDv5's good enough for your extreme edge case.
...But at that point you might as well just roll your own 128-bit version of a Snowflake ID.
https://en.wikipedia.org/wiki/Snowflake_ID
They fit within 64 bits, allow for more than enough processes to handle 10k+ transactions per second, give enough of a timestamp headroom for decades into the future, and where ID generation can be made isolated to each process.
They don't work well for anything related to archival work, but you might as well use a regular ID for that anyway, unless you're also actively scraping terabytes of data off of the Internet every second, in which case UUIDv5's good enough for your extreme edge case.
...But at that point you might as well just roll your own 128-bit version of a Snowflake ID.
>...But at that point you might as well just roll your own 128-bit version of a Snowflake ID.
Or UUIDv7, which is designed to solve the same problem
Or UUIDv7, which is designed to solve the same problem
Use sequential IDs, but run them through a maximal linear feedback shift register.
They will be in a deterministic order, but will appear semi-random to the end user.
For things like product IDs or user IDs, etc you don't actually need them to be random. But perhaps you don't want them to simply start counting sequentially.
They will be in a deterministic order, but will appear semi-random to the end user.
For things like product IDs or user IDs, etc you don't actually need them to be random. But perhaps you don't want them to simply start counting sequentially.
I'm building an object database. In different places, it uses four different categories of IDs, depending on requirements:
1. Compact sequential is used, obviously, where order matters. It has the drawback of requiring a coordination with a singleton. (This can be sharded/vectorized, of course.) Aside from that, it also leaks the number of objects/transactions, just by looking for the highest number available. Can be varint-encoded very nicely.
2. Compact non-sequential is used where I need a small identifier, but not leak the number of objects. Since it's compact, I must still guarantee uniqueness as in (1). This is currently implemented using a block cipher on top of the compact sequential ID generator. The drawback of this is that the domain may still be in guessable territory, depending on how much is generated. A 64-bit integer filled with 4B only requires 4B guesses to hit a collision. I don't use this much. The key can never be rotated: a key number would eat up precious bits.
3. Sparse random, used where non-guessability is important, aside from not leaking rates/counts. Take a Google Docs sharable link as an example. I doubt YouTube cares about this. This is where something like a 128-bit number like UUID or ULID shines. The space is large enough that uniqueness is assumed, given a decent PRNG.
Sure, I try to use the nicest one at any given point (e.g. using a compact sequential instead of non-sequential during debugging.) But fact is that sparse random just tick more boxes.
4) Sparse, human readable. For "vouchers". They are bearer tokens that give requests more powers, e.g. to create an account or act as admin. These should be reasonably human readable, so they can be spoken. They obviously need to be sparse and hard to guess, which requires a trade-off in length.
I present them in three ways: base32, english words and QR-code. Pick one; they all do the same. For copy-pasting, base32 might be best (or base58, by all means.)
For shouting to a colleague, the sequence of english words might be better. I'd add other languages as needed: it's just a fixed list of words. The nice thing is it can encode the sequence in base-500 or base-1000 without being obnoxious. (The Matrix protocol and others use emoji lists, but it's the same idea. [1]) Finally, if you have a phone in your pocket or camera on your computer, perhaps the QR-code is the easiest way to use the voucher code.
[1] Actually, IIRC, Matrix only uses 64 emojis, which feels a bit wasteful.
1. Compact sequential is used, obviously, where order matters. It has the drawback of requiring a coordination with a singleton. (This can be sharded/vectorized, of course.) Aside from that, it also leaks the number of objects/transactions, just by looking for the highest number available. Can be varint-encoded very nicely.
2. Compact non-sequential is used where I need a small identifier, but not leak the number of objects. Since it's compact, I must still guarantee uniqueness as in (1). This is currently implemented using a block cipher on top of the compact sequential ID generator. The drawback of this is that the domain may still be in guessable territory, depending on how much is generated. A 64-bit integer filled with 4B only requires 4B guesses to hit a collision. I don't use this much. The key can never be rotated: a key number would eat up precious bits.
3. Sparse random, used where non-guessability is important, aside from not leaking rates/counts. Take a Google Docs sharable link as an example. I doubt YouTube cares about this. This is where something like a 128-bit number like UUID or ULID shines. The space is large enough that uniqueness is assumed, given a decent PRNG.
Sure, I try to use the nicest one at any given point (e.g. using a compact sequential instead of non-sequential during debugging.) But fact is that sparse random just tick more boxes.
4) Sparse, human readable. For "vouchers". They are bearer tokens that give requests more powers, e.g. to create an account or act as admin. These should be reasonably human readable, so they can be spoken. They obviously need to be sparse and hard to guess, which requires a trade-off in length.
I present them in three ways: base32, english words and QR-code. Pick one; they all do the same. For copy-pasting, base32 might be best (or base58, by all means.)
For shouting to a colleague, the sequence of english words might be better. I'd add other languages as needed: it's just a fixed list of words. The nice thing is it can encode the sequence in base-500 or base-1000 without being obnoxious. (The Matrix protocol and others use emoji lists, but it's the same idea. [1]) Finally, if you have a phone in your pocket or camera on your computer, perhaps the QR-code is the easiest way to use the voucher code.
[1] Actually, IIRC, Matrix only uses 64 emojis, which feels a bit wasteful.
I usually go for Nano Id for new projects https://github.com/ai/nanoid
I generate uuidv7 as PK and have a publicId "slug" with a 13 character nanoId to display in the URL
not sure if this is duplicative but down the road i figure I can navigate whichever direction i need to go in
not sure if this is duplicative but down the road i figure I can navigate whichever direction i need to go in
+1 for nanoid, which matches what I used to implement myself as a lib function in new projects
In case it helps anyone, I just went down the rabbit hole of calculating the amount of random bits required to avoid collisions. The traditional Birthday Paradox formula gives us a probability but it's not very intuitive to understand what the probability means in terms we are used to as developers, so I tried a different approach: expected time period for a collision to happen.
https://colab.research.google.com/drive/1ec4n7Ex9bnkl_c45EUl...
https://colab.research.google.com/drive/1ec4n7Ex9bnkl_c45EUl...
You Don't Need UUID, But You Probably Want It:
- Don't invent your own ID datatype (especially 11 byte one), this is almost guaranteed to cause dangerous bugs, because each integration will have to carefully implement/hack it.
- Use UUID (preferably the new v7). 128-bit UUID is implemented, for you, pretty much everywhere.
- Serial integers still work too, but you should choose them consciously to fit the data model.
- Implement "natural keys" if you want pretty/memorable/Cool URLs. Never use non-standard PKs to store custom semantic data, because inevitably you will get garbage PKs that need to be fixed, and migrating a PK value is extremely risky.
- Don't invent your own ID datatype (especially 11 byte one), this is almost guaranteed to cause dangerous bugs, because each integration will have to carefully implement/hack it.
- Use UUID (preferably the new v7). 128-bit UUID is implemented, for you, pretty much everywhere.
- Serial integers still work too, but you should choose them consciously to fit the data model.
- Implement "natural keys" if you want pretty/memorable/Cool URLs. Never use non-standard PKs to store custom semantic data, because inevitably you will get garbage PKs that need to be fixed, and migrating a PK value is extremely risky.
> This solution uses the human-readable base58 encoding scheme.
We could encode the 128 bit UUID integer in base58 as well if needed. Its textual representation won't conform to the standard but we'd get the same number of bits. Which would be 2^122 or 2^121 bits of uniqueness not 2^128 if it's a proper UUID.
11base58 character certainly doesn't have the 2^122 bits. So we could decided separately if we could either reduce the number of bits needed and/or use a different encoding.
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
:-)
We could encode the 128 bit UUID integer in base58 as well if needed. Its textual representation won't conform to the standard but we'd get the same number of bits. Which would be 2^122 or 2^121 bits of uniqueness not 2^128 if it's a proper UUID.
11base58 character certainly doesn't have the 2^122 bits. So we could decided separately if we could either reduce the number of bits needed and/or use a different encoding.
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
:-)
A UUID is 128 bits, or 16 bytes. This code suggests IDs which are 11 bytes. That's not really comparable.
A UUID can also be encoded in any form, it doesn't need to be represented as the dashed string notation which is common, you can just as easily use the base58 alphabet suggested in the post.
But the code in the post doesn't encode to base58 correctly. You need to map 58 bits of the input, sequentially, to one character in the alphabet as output. You can't just mod each byte of the input by the alphabet length and use the corresponding alphabet element.
A UUID can also be encoded in any form, it doesn't need to be represented as the dashed string notation which is common, you can just as easily use the base58 alphabet suggested in the post.
But the code in the post doesn't encode to base58 correctly. You need to map 58 bits of the input, sequentially, to one character in the alphabet as output. You can't just mod each byte of the input by the alphabet length and use the corresponding alphabet element.
The base58 code in the post is wrong, but your version isn't right either.
Since the alphabet is 58 characters, each character contains log2(58) bits. Since that's not an integer, the encoding process is a bit more complex than just mapping bits to characters in a table.
https://digitalbazaar.github.io/base58-spec/#encode
Since the alphabet is 58 characters, each character contains log2(58) bits. Since that's not an integer, the encoding process is a bit more complex than just mapping bits to characters in a table.
https://digitalbazaar.github.io/base58-spec/#encode
Hi. I'm the author of the post here.
I agree lines 12-14 doesn't do a proper encoding, but that's not what I was after. Would you still say it's still wrong if you consider that my function is not really encoding the string per se, but using rand.Read to generate entropy for what I want to be the final output (random string with the base58 alphabet)?
I agree lines 12-14 doesn't do a proper encoding, but that's not what I was after. Would you still say it's still wrong if you consider that my function is not really encoding the string per se, but using rand.Read to generate entropy for what I want to be the final output (random string with the base58 alphabet)?
Your IDs are []byte of len=11. Those bytes can be represented in many ways. You can represent them as hex strings via encoding/hex.EncodeToString(id), or base64 strings via encoding/base64.StdEncoding.EncodeToString(id), or base32 strings via encoding/base32.StdEncoding.EncodeToString(id), or etc.
Looks like the most used base58 package is https://pkg.go.dev/github.com/btcsuite/btcutil/base58, but looking at the implementation [0] I'm not impressed, there's definitely a much better approach.
[0] https://github.com/btcsuite/btcutil/blob/v1.0.2/base58/base5...
But how you encode 11 bytes of data is kind of orthogonal to the important thing, which is that you have 11 bytes of data. Those bytes should be always be stored in memory (in your application, or in your DB, or wherever) as the actual 11 bytes of the ID, and not as a base58 or base64 or JSON or whatever other kind of string that can be decoded to the actual 11 bytes they represent.
Likewise, a UUID shouldn't be stored as a string like "64d3f2e0-a4dc-48d3-98ad-7f09eb3b082f", that's a specific encoding of the actual 16 UUID bytes, you should store, process, etc. those bytes directly.
Looks like the most used base58 package is https://pkg.go.dev/github.com/btcsuite/btcutil/base58, but looking at the implementation [0] I'm not impressed, there's definitely a much better approach.
[0] https://github.com/btcsuite/btcutil/blob/v1.0.2/base58/base5...
But how you encode 11 bytes of data is kind of orthogonal to the important thing, which is that you have 11 bytes of data. Those bytes should be always be stored in memory (in your application, or in your DB, or wherever) as the actual 11 bytes of the ID, and not as a base58 or base64 or JSON or whatever other kind of string that can be decoded to the actual 11 bytes they represent.
Likewise, a UUID shouldn't be stored as a string like "64d3f2e0-a4dc-48d3-98ad-7f09eb3b082f", that's a specific encoding of the actual 16 UUID bytes, you should store, process, etc. those bytes directly.
pedantry: uuidv4 is only 122 bits of randomness.
> 6 predetermined variant and version bits, leaving 122 bits for the randomly generated part
> --wikipediaIndeed. That's why I mentioned it's a 128-bit identifier format in the first line of the article, rather than 128-bit identifier. I probably spent 10min writing something to explain this, but then it wouldn't really be helpful so I removed it.
Pivoting to the related subject of compact textual representations of _typed_ data, check out CESR[0]. Through some careful choices of code prefixes and payload lengths that won't require Base64 padding, it provides a novel encoding scheme for JSON plus cryptographic signatures (all while enabling true concatenative composability). It is at the heart of the KERI[1][2][3], a decentralized identity management scheme, and ACDC[4], a mechanism for verified credentials.
[0] https://trustoverip.github.io/tswg-cesr-specification/draft-... [1] https://keri.one/keri-resources/ [2] https://trustoverip.github.io/tswg-keri-specification/draft-... [3] https://github.com/WebOfTrust/keripy/ [3] https://trustoverip.github.io/tswg-acdc-specification/draft-...
[0] https://trustoverip.github.io/tswg-cesr-specification/draft-... [1] https://keri.one/keri-resources/ [2] https://trustoverip.github.io/tswg-keri-specification/draft-... [3] https://github.com/WebOfTrust/keripy/ [3] https://trustoverip.github.io/tswg-acdc-specification/draft-...
I have created a custom unique identifier scheme before. This post doesn't address some of the challenges you come across.
1. When dealing with user friendly IDs, it's often important to make sure there are no ambiguous characters. This is a UX requirement for anyone that may need to read an ID and type it in at any point. For example, this means removing certain confusing characters like "0oIi1l5sS", etc. You end up with a much smaller set of characters. In my case, young children were required to type in teacher codes. Needless to say, I had to limit a lot of possible confusing characters.
2. You will have collisions. How do you handle them (ie, retry N times until you get a valid one)? What happens if you can't get a valid one? This happened in a product of mine. We had ~8 character codes that were human readable and type-able and we... ran out of codes.
3. How can you embed more information into the code such as versioning? It's often useful to prefix a code with some info in the event you need to modify it later (such as expand the character set or length)?
1. When dealing with user friendly IDs, it's often important to make sure there are no ambiguous characters. This is a UX requirement for anyone that may need to read an ID and type it in at any point. For example, this means removing certain confusing characters like "0oIi1l5sS", etc. You end up with a much smaller set of characters. In my case, young children were required to type in teacher codes. Needless to say, I had to limit a lot of possible confusing characters.
2. You will have collisions. How do you handle them (ie, retry N times until you get a valid one)? What happens if you can't get a valid one? This happened in a product of mine. We had ~8 character codes that were human readable and type-able and we... ran out of codes.
3. How can you embed more information into the code such as versioning? It's often useful to prefix a code with some info in the event you need to modify it later (such as expand the character set or length)?
Number two can be addressed with something like this:
https://preshing.com/20121224/how-to-generate-a-sequence-of-...
You can shrink the range by using a smaller prime number and throwing on out-of-range input. And then you can offset the output to make sure the base-N encoded value is always D digits long.
You can shrink the range by using a smaller prime number and throwing on out-of-range input. And then you can offset the output to make sure the base-N encoded value is always D digits long.
If your users needs to communicate an ID orally to you, you already failed in numerous ways.
Is there any existing standardized middle ground between UUIDs, random bits of length n, and serial IDs? I totally buy that UUIDs are unpleasant in URLs in many cases. I can also imagine that in many cases UUIDs for small objects might be a huge % of on the wire payload size.
But besides human friendly slugs, which usually have an ID mapping behind the scenes in my experience, it seems like there might be more work than value for startups in many cases.
But besides human friendly slugs, which usually have an ID mapping behind the scenes in my experience, it seems like there might be more work than value for startups in many cases.
I like UUIDs aesthetically. They're also an instant queue that something is an "entity" and familiar to people for that purpose.
Databases and lots of other things also have native support for them which makes them even more appealing to use.
Of course, there are plenty of reasons to avoid them like the article discusses. I'd almost certainly not criticize someone for choosing smaller IDs as long as security isn't an issue.
Databases and lots of other things also have native support for them which makes them even more appealing to use.
Of course, there are plenty of reasons to avoid them like the article discusses. I'd almost certainly not criticize someone for choosing smaller IDs as long as security isn't an issue.
I don't know that I really agree with the logic here; certainly for some cases having a pretty URL matters (e.g. URL shorteners or even something like YouTube), but if your goal is to minimize URL size, and if you're centrally synchronizing anyway to avoid conflicts, why not just use integers or something? If you use a GMP-style bigint, I can virtually guarantee it'll be faster than any random string concat glue you come up with, and you're never going to run out of integers.
I like using GUID specifically because there's usually a built-in implementation everywhere, and because it's so stupidly huge, the likelihood of a conflict is statistically zero, meaning I don't need to bother synchronizing against a server.
Yes, it's technically possible I could save a few bytes of bandwidth or something by using some kind of base encoding, and maybe in some kind of embedded case that might matter, but for a vast majority of cases, GUID is absolutely fine, and since it's used everywhere, it's also very thoroughly tested by multi-billion-dollar corporations meaning that I don't have to worry much about any issues.
So no, you don't need GUID, you don't need a lot of stuff in the software world. I could poll bytes from /dev/urandom and probably get something that works well enough, I could turn off garbage collection and just pre-allocate all my memory before hand, I could avoid an OS entirely and write straight to the metal, I could do a lot of things, but I don't because I value my time . GUID solves a specific problem pretty well.
EDIT:
The reason I'm specifying this is because I think the title would be better if it was something like "You (probably) don't need UUID".
I like using GUID specifically because there's usually a built-in implementation everywhere, and because it's so stupidly huge, the likelihood of a conflict is statistically zero, meaning I don't need to bother synchronizing against a server.
Yes, it's technically possible I could save a few bytes of bandwidth or something by using some kind of base encoding, and maybe in some kind of embedded case that might matter, but for a vast majority of cases, GUID is absolutely fine, and since it's used everywhere, it's also very thoroughly tested by multi-billion-dollar corporations meaning that I don't have to worry much about any issues.
So no, you don't need GUID, you don't need a lot of stuff in the software world. I could poll bytes from /dev/urandom and probably get something that works well enough, I could turn off garbage collection and just pre-allocate all my memory before hand, I could avoid an OS entirely and write straight to the metal, I could do a lot of things, but I don't because I value my time . GUID solves a specific problem pretty well.
EDIT:
The reason I'm specifying this is because I think the title would be better if it was something like "You (probably) don't need UUID".
I mean, don't use UUID for user-facing things sure, but that doesn't mean it's not useful in of itself.
That's a terrible way to use it is all.
That's a terrible way to use it is all.
> If you’re talking about a web system composed of microservice architecture all running on the same datacenter, perhaps sharing the same database
We have different ideas what 'microservice architecture' means, I guess.
One of the key points of UUIDs is to be able to generate (probably) non-conflicting values without coordination
We have different ideas what 'microservice architecture' means, I guess.
One of the key points of UUIDs is to be able to generate (probably) non-conflicting values without coordination
I think there's a better "You Don't Need UUID", which is that you probably aren't going to generate so many integers per second that you need to do so in a way that's perfectly collision free.
For example, you can update a postgres integer 1000s of times per second, sequentially, and 100s of 1000s of times per second if you allow for gaps/ unordered sequencing. The reason to choose a UUID is because it doesn't require a db. Totally fair. But you probably already have a db lying around if you're generating UUIDs to begin with?
Or because you need an unguessable token, which a UUID is great for.
But if you have a postgres db around somewhere consider just having a counters table and using that. It'll be fast enough for almost anyone, it'll return smaller tokens, and it'll return tokens that are sequentially allocated (in range. These are nice properties to have!
For example, you can update a postgres integer 1000s of times per second, sequentially, and 100s of 1000s of times per second if you allow for gaps/ unordered sequencing. The reason to choose a UUID is because it doesn't require a db. Totally fair. But you probably already have a db lying around if you're generating UUIDs to begin with?
Or because you need an unguessable token, which a UUID is great for.
But if you have a postgres db around somewhere consider just having a counters table and using that. It'll be fast enough for almost anyone, it'll return smaller tokens, and it'll return tokens that are sequentially allocated (in range. These are nice properties to have!
Database "sequence" types that are used to create integer IDs often no not make any guarantees about being gap-free. This can be a problem for people who don't realize this, and use them for something that is legally required to be gap-free, such as invoice numbers in some jurisdictions.
Unguessability is a nice layer to have in your security strategy, especially if you're passing IDs to and from a web page, but if that's all you rely on you're not doing enough.
Unguessability is a nice layer to have in your security strategy, especially if you're passing IDs to and from a web page, but if that's all you rely on you're not doing enough.
> Database "sequence" types that are used to create integer IDs often no not make any guarantees about being gap-free.
Yep, the docs from Postgres are very clear that `serial` has gaps. But you can also just create a counter table and manually manage the sequence if you have strict serial requirements.
Of course, if you have serial requirements a UUID would already not be viable.
Yep, the docs from Postgres are very clear that `serial` has gaps. But you can also just create a counter table and manually manage the sequence if you have strict serial requirements.
Of course, if you have serial requirements a UUID would already not be viable.
For the web, sequential ID attacks represent a significant problem in terms of a Resource Enumeration Attack. It gets worse when auth isn’t even in play yet (the user isn’t logged in, or doesn’t even have an account yet, or this info is meant to be available outside of auth but to only one specific user), but you need to display only the current item, and not any others. Having a sequential ID of any kind allows the user to trivially hack their way into any other item they wish by just incrementing or decrementing that ID.
UUIDs represent a trivially easy way of implementing a non-incrementing ID with a ridiculously unwieldy address space that makes it supremely unrealistic for the vast majority of users to mess with. They’re just going to give up long before the first successful hit.
UUIDs represent a trivially easy way of implementing a non-incrementing ID with a ridiculously unwieldy address space that makes it supremely unrealistic for the vast majority of users to mess with. They’re just going to give up long before the first successful hit.
Best part is watching the video and seeing the guy dance at the end over doing it in a single take.
Doesn’t UUID have mechanisms to prevent collisions even if you start two processes simultaneously with the same initial seed? There are human readable tweaks you can make to them that preserve all of the good properties without limiting the pool much though.
v4 is. v1 relied on the MAC address and the time.
https://devblogs.microsoft.com/oldnewthing/20040211-00/?p=40...
https://devblogs.microsoft.com/oldnewthing/20040211-00/?p=40...
I’m not sure if this is accurate any longer, but it was bad practice to use UUID’s as PK’s in PostgreSQL for example due to indexing and performance. Is this still true? It’s really nice in PostgreSQL to just do:
id UUID DEFAULT gen_random_uuid()We need them tho. Especially UUIDv4 and UUIDv5. We have a distributed system and we manage permission identities with that. Each item that's supposed to be restricted has either of these ids. UUIDv4 for global things like workspaces and users, UUIDv5 for workspace-specific stuff (the workspace UUIDv4 is given as a namespace for the v5) and it's quite beautiful and prevents clashes. So it really depends on your usage. We still have long ids that are exposed to the client tho.
My experience: If you're storing data in a database keyed by a GUID, you get a nicer random distribution in an index, avoiding hot partitions when your workload is seemingly random.
You don't "need" them, but they're almost universally understood and supported which makes them super convenient and a good starting point for many use cases. The point in which the "wasteful" nature of a UUUID really bites you is so far down the road that it is almost not even worth thinking about until you get there. Chances are you won't be dealing in petabytes at all and the 64 bit savings you are fretting over doesn't really matter.
Just use UUID.
If you need something user friendly, use the first 6 or 8 characters.
If you need to worry about index performance due to enormous index sizes, you probably know it.
If you need something user friendly, use the first 6 or 8 characters.
If you need to worry about index performance due to enormous index sizes, you probably know it.
I see ulid and nanoid being recommended here. I like cuid2 based on input from its README [0] and discussion [1]
[0] https://github.com/paralleldrive/cuid2
[1] https://github.com/paralleldrive/cuid2/issues/7
[0] https://github.com/paralleldrive/cuid2
[1] https://github.com/paralleldrive/cuid2/issues/7
cuid [0] seems to be a random letter, concatenated with a sha3 hash of:
- the system time
- a process-global singleton counter (incrementing with each ID)
- a "salt" derived from some number of bytes taken from an entropy source
- a "fingerprint" of some number of bytes taken from the same entropy source
this is an awful lot of work to be doing for each ID, to what end? well issue 7 [1] says that
> Cuid2 is good when you need something secure, extremely collision resistant, and your system is distributed or decentralized (e.g. you want to be able to create records with ids on the client side), or you are building software that may need to scale horizontally).
but I'm not sure how cuid as implemented provides a better answer in any of these dimensions to a plain old UUID
maybe I'm just being cynical, I dunno
[0] https://github.com/paralleldrive/cuid2/blob/fb07094487ba5ad0...
[1] https://github.com/paralleldrive/cuid2/issues/7
- the system time
- a process-global singleton counter (incrementing with each ID)
- a "salt" derived from some number of bytes taken from an entropy source
- a "fingerprint" of some number of bytes taken from the same entropy source
this is an awful lot of work to be doing for each ID, to what end? well issue 7 [1] says that
> Cuid2 is good when you need something secure, extremely collision resistant, and your system is distributed or decentralized (e.g. you want to be able to create records with ids on the client side), or you are building software that may need to scale horizontally).
but I'm not sure how cuid as implemented provides a better answer in any of these dimensions to a plain old UUID
maybe I'm just being cynical, I dunno
[0] https://github.com/paralleldrive/cuid2/blob/fb07094487ba5ad0...
[1] https://github.com/paralleldrive/cuid2/issues/7
> Version 4 is completely randomly generated (hence, it has more entropy) and is what most web systems seem to use. It has 16^32 = 2^128 bits that guarantee uniqueness and has an insignificant risk of collision.
Small correction here but it is not 128 random bits entirely: 6 bits are reserved for the version marker.
Small correction here but it is not 128 random bits entirely: 6 bits are reserved for the version marker.
> This solution uses the human-readable base58 encoding scheme.
How readable is base58 in Arabic or Chinese? I've concluded that only the standard numbers 0123456789, algebra symbols +-/*= and phone symbols #* are universal enough for a global encoding. Any other insights are welcomed!
How readable is base58 in Arabic or Chinese? I've concluded that only the standard numbers 0123456789, algebra symbols +-/*= and phone symbols #* are universal enough for a global encoding. Any other insights are welcomed!
UUIDs are great for when you have a distributed system where connectivity between the nodes is intermittent, and every node needs to be able to create new entities without ever having ID collisions.
For instance, UUIDs allow mobile apps to create entities while offline.
For instance, UUIDs allow mobile apps to create entities while offline.
It’s a sales pitch for base58 that I’m not buying at all. According to the code segment, they include numeral “1” but exclude lowercase “l”?
0-9 and a-f are unambiguous and widely understood. They’re more “human readable”.
0-9 and a-f are unambiguous and widely understood. They’re more “human readable”.
I wonder how Stripe generates their IDs. They look quite nice.
They use a type safe extension of UUIDv7
Here is a go implementation and explanation: https://github.com/jetpack-io/typeid
uuids are ubiquitous and easy and implemented in many languages in the standard library - therefore a dev can get near-guaranteed uniqueness without thinking too hard about it.
Until another standard similar to what the article is suggesting becomes widely implemented in standard libraries then uuid isn't going anywhere, although in principle I agree with many of the arguments presented.
Until another standard similar to what the article is suggesting becomes widely implemented in standard libraries then uuid isn't going anywhere, although in principle I agree with many of the arguments presented.
Everyone wanted UUIDs and we had IRIs all along...
YoU dOnT nEeD _____
AKA I will make broad sweeping stupid headlines
AKA I will make broad sweeping stupid headlines
This is such a HN article.
You don't need to tell me I don't need UUID
...because I am going to keep using uuid
...because I am going to keep using uuid
I would challenge the premise we appear to be starting from, that the average end user cares to be dealing with any random string of numbers and digits. GUIDs work well, they’re implemented everywhere, and you won’t find out long after you go into production you made some mistake that is going to make it so you have to migrate away from them.