Storing State in the URL(antonz.org)
antonz.org
Storing State in the URL
https://antonz.org/storing-state/
58 comments
"Building for Users: Designing URLs", Aaron Swartz in "A Programmable Web" (2013) https://doi.org/10.2200/S00481ED1V01Y201302WBE005
badly neglected but the more recommended read.
badly neglected but the more recommended read.
I agree however very few users even see or read the URL.
If you deal with tech-illiterate users by putting everything in black boxes, you're contributing to perpetuating tech illiteracy.
They do when it’s shared them as a link
Not always no. smartphone software often abstract away that process.
Plus even in those instances where links are manually copied and pasted, you're still overestimating the competency of the average user to identify the risk of state in URLs even when it seems bloody obvious to us technical folk.
Plus even in those instances where links are manually copied and pasted, you're still overestimating the competency of the average user to identify the risk of state in URLs even when it seems bloody obvious to us technical folk.
do you have studies showing this? I think WE do when it's shared as a link, but I think most users get a message - here is that link: long blue text starting with https: for them to click on. And they click on the blue text but do not read.
It's a question of security and privacy: what matters is what can be read from the URL.
I'm surprised that hash parameters are not mentioned to store state. They have many advantages that (for the kind of apps I do) work pretty well:
* they are not sent to the server
* hence, the URL size limit is that of the browser, not that of the backend handling the request
* hence, they disclose less info to the middlemen
* they can be shared easily from person to person
One drawback is that they require more processing on the frontend side.
As always, trade offs.
* they are not sent to the server
* hence, the URL size limit is that of the browser, not that of the backend handling the request
* hence, they disclose less info to the middlemen
* they can be shared easily from person to person
One drawback is that they require more processing on the frontend side.
As always, trade offs.
especially neat when a crypto key like at https://sebsauvage.net/wiki/doku.php?id=php:zerobin
The correct term for these is "fragment identifiers". They're not parameters. They are meant to identify, e.g., anchors in documents identified by the rest of the URI.
And, yes, you can use those too, especially for SPAs!
And, yes, you can use those too, especially for SPAs!
> they are not sent to the server
The fact that the url hash-fragment isn't sent to the web server is a mere default. There's no controlling what plugins or scripts do.
> hence, they disclose less info to the middlemen
Care to elaborate? I don't see how a url (over tls) exposes more information unless tls is MiTM'd.
> they can be shared easily from person to person
That's a property of the url itself, and not unique to the hash-fragment, per se?
The fact that the url hash-fragment isn't sent to the web server is a mere default. There's no controlling what plugins or scripts do.
> hence, they disclose less info to the middlemen
Care to elaborate? I don't see how a url (over tls) exposes more information unless tls is MiTM'd.
> they can be shared easily from person to person
That's a property of the url itself, and not unique to the hash-fragment, per se?
Fair point.
> I don't see how a url (over tls) exposes more information unless tls is MiTM'd.
What is not sent cannot be hacked. I was not suggesting an explicit security feature, rather a state of things.
> That's a property of the url itself, and not unique to the hash-fragment, per se?
Yes, but it makes for a nice combo with the previous point, as the URL now contains 2 things: the Locator itself that your peer and the server will access and (resp.) serve, plus the context that only your peer will have access to.
> I don't see how a url (over tls) exposes more information unless tls is MiTM'd.
What is not sent cannot be hacked. I was not suggesting an explicit security feature, rather a state of things.
> That's a property of the url itself, and not unique to the hash-fragment, per se?
Yes, but it makes for a nice combo with the previous point, as the URL now contains 2 things: the Locator itself that your peer and the server will access and (resp.) serve, plus the context that only your peer will have access to.
Thank you for bringing this up. This is a very good point. I have thought of hash URLs being obsolete outside of some neat CSS tricks and section anchoring. But they still serve a purpose outside of that and can even simplify server side handling in some cases.
I'm by no means an anti-SPA zealot, I think they have their place.
But, it's pretty humorous at times to see people put all this effort into replicating what was once the default behavior of the web.
HATEOAS is new again?
But, it's pretty humorous at times to see people put all this effort into replicating what was once the default behavior of the web.
HATEOAS is new again?
Exactly. This post was like a blast from the past about things that have been working fine for ages. Curious if young developers read this and thought it was a modern invention.
HATEOAS is a lot better when you can make the browser super smart by putting making it execute... the application.
Not sure I understand the point of this... did the HTTP query string just got reinvented, or is there something more subtle?
I think you're talking about the same thing but use a different term compared to the original post ("URL parameters" vs. "query string").
>> Store serialized state
Learnt recently that Base64 encoding and decoding in JS doesn't quite handle Unicode strings. One solution[1] is to URI encode/decode the text (e.g. JSON):
Learnt recently that Base64 encoding and decoding in JS doesn't quite handle Unicode strings. One solution[1] is to URI encode/decode the text (e.g. JSON):
base64 = btoa(encodeURIComponent( str ))
str = decodeURIComponent(atob( base64 ))
Basically btoa was probably a quick hack implemented to treat JS strings as byte arrays and then lived on with that behavior. The sensible API approach nowadays would probably be for a Base64 encoding to take a TypedArray as well as functions to convert strings into bytes in a desired encoding.
This is what all web sites did before there were HTTP cookies.
I came across this problem when working with ASP.NET Core. Sending nested dictionaries is very verbose and nontrivial to serialize and deserialize.
So I rolled my own solution to parse JSON parameters and integrated it with the framework for validation and OpenAPI support.
https://abdus.dev/posts/aspnetcore-model-binding-json-query-...
So I rolled my own solution to parse JSON parameters and integrated it with the framework for validation and OpenAPI support.
https://abdus.dev/posts/aspnetcore-model-binding-json-query-...
Thanks! I stumbled upon your post earlier this year when I ran into the exact same issues
Glad it's been a help to you :)
Reminds me of one of my favourite hack projects, storing the whole content in the URL:
https://wgx.github.io/anypage/?eyJoMSI6IiBTdG9yaW5nIHN0YXRlI...
https://wgx.github.io/anypage/?eyJoMSI6IiBTdG9yaW5nIHN0YXRlI...
>They load the third page, open the potato card and freeze in silent admiration for a few seconds. And then accidentally refresh the page. What will your application do?
It will show the same exact page because I think that PWAs are stupid and I don't build them.
It will show the same exact page because I think that PWAs are stupid and I don't build them.
Only UI state should be stored in the URL and navigational UI state should (almost) always be stored in the URL. That way when you refresh or share the page it will look (near) identical.
Don’t store user state in the URL, that may bite them when it’s shared.
Don’t store user state in the URL, that may bite them when it’s shared.
Not all UI state though. If you use the back button, you want something to happen that's similar to going back a page, not undo a tiny change on that page.
I say that navigation state should be in the URL.
I say that navigation state should be in the URL.
What is navigation and what is state is not always clear. Example:
Visit https://www.dpreview.com/
Next, we open this article: https://www.dpreview.com/samples/7583235225/nikon-z-50mm-f1-...
Within the article, we click the main photo, which opens another page with a slideshow: https://www.dpreview.com/sample-galleries/8661039925/nikon-z...
As we're keenly interested in the slideshow, we use the next arrow to cycle through all 47 photos.
Now that we're done with that, the slideshow has no further use, so let's hit BACK to return to the article we came from, and BACK again to return to the homepage.
Doesn't work, we have to hit BACK 47 times first. Because in this case slideshow manipulation is seen as navigation, and thus part of history.
Is navigating a slideshow state or navigation in itself? You tell me, it's not very obvious. Each slide is bookmarkable so it very much looks like a navigation action. And yet it does not lead to a great experience.
Visit https://www.dpreview.com/
Next, we open this article: https://www.dpreview.com/samples/7583235225/nikon-z-50mm-f1-...
Within the article, we click the main photo, which opens another page with a slideshow: https://www.dpreview.com/sample-galleries/8661039925/nikon-z...
As we're keenly interested in the slideshow, we use the next arrow to cycle through all 47 photos.
Now that we're done with that, the slideshow has no further use, so let's hit BACK to return to the article we came from, and BACK again to return to the homepage.
Doesn't work, we have to hit BACK 47 times first. Because in this case slideshow manipulation is seen as navigation, and thus part of history.
Is navigating a slideshow state or navigation in itself? You tell me, it's not very obvious. Each slide is bookmarkable so it very much looks like a navigation action. And yet it does not lead to a great experience.
That depends on what method of storing state in the URL you use; History#replaceState[1] versus #pushState[2], not on storing state in the URL in general.
[1] https://developer.mozilla.org/en-US/docs/Web/API/History/rep...
[2] https://developer.mozilla.org/en-US/docs/Web/API/History/pus...
[1] https://developer.mozilla.org/en-US/docs/Web/API/History/rep...
[2] https://developer.mozilla.org/en-US/docs/Web/API/History/pus...
> I say that navigation state should be in the URL.
That's a truism. You still have to decide what navigation is.
- opening or closing a modal
- switching between representations (example: grid view vs list view)
- collapsing / hiding major parts of the screen
- entering something into a search box
- turning knobs on a form that draws a graph
- drag and drop ui elements without side effects on the wider application
- the above with side effects on the wider application
All of these things and more have been implemented with or without changing the URL.
That's a truism. You still have to decide what navigation is.
- opening or closing a modal
- switching between representations (example: grid view vs list view)
- collapsing / hiding major parts of the screen
- entering something into a search box
- turning knobs on a form that draws a graph
- drag and drop ui elements without side effects on the wider application
- the above with side effects on the wider application
All of these things and more have been implemented with or without changing the URL.
A little while ago I tinkered with data URLs to carry a small 80's home computer program directly in the emulator URL, it worked well apart from the fact that the "payload" is limited to around 1.5 KBytes.
The navbar in browsers may accept more (up to 4KByte in my experiments), but other 'infrastructure' where the URL needs to travel through (for instance URL shorteners) may clamp the source URL to a much shorter length.
It would have been awesome to distribute ready-to-play emulator links for home computer scene demos without hosting the data anywhere, alas it didn't work except for very small demos :)
The navbar in browsers may accept more (up to 4KByte in my experiments), but other 'infrastructure' where the URL needs to travel through (for instance URL shorteners) may clamp the source URL to a much shorter length.
It would have been awesome to distribute ready-to-play emulator links for home computer scene demos without hosting the data anywhere, alas it didn't work except for very small demos :)
You could put the payload into the fragment part of the url (after the '#' symbol). This part is not passed to the server and is not subject to the limitations of intermediate systems you have no control over. I assume that the payload is only needed client-side anyways.
I was surprised to see the suggestion of storing the state in the query string, not the fragment. Two URLs which differ in their query string are supposed (up to isomorphism) to identify different resources on the server, but here they don't, because they are only interpreted client-side. Whereas that is exactly what the fragment is for.
I've used the fragment to store client-side state in a few apps, and it seems fine.
Is there some reason to use the query string rather than the fragment?
I've used the fragment to store client-side state in a few apps, and it seems fine.
Is there some reason to use the query string rather than the fragment?
It's rather the opposite. Using querystring is the correct (intended) way, fragments are a hack. A fragment is a way to jump to a named anchor within the document, which by convention has been "misused" by developers to do client-side routing and URL state changes.
This workaround was needed when history push state did not yet exist or work across browsers. The hash hack is no longer needed, but still popular.
Does it matter at this point, for client-side state?
I think real query strings "look better", which is subjective but a more real world advantage is as follow. When you start out with a fully client-side web app (server sends empty body/div only) and use proper query strings in the client, you then later have the flexibility to add better server side rendering without any URL migration needed. Because the server can read the querystring, unlike hashes.
This workaround was needed when history push state did not yet exist or work across browsers. The hash hack is no longer needed, but still popular.
Does it matter at this point, for client-side state?
I think real query strings "look better", which is subjective but a more real world advantage is as follow. When you start out with a fully client-side web app (server sends empty body/div only) and use proper query strings in the client, you then later have the flexibility to add better server side rendering without any URL migration needed. Because the server can read the querystring, unlike hashes.
Another way to store booleans that can be worth mentioning is to add (or not) a parameter in the url without a value.
To reuse the example from https://antonz.org/storing-state/#boolean-value
{"search": "potatoes", "popular": true} can be serialized as ?search=potatoes&popular
To reuse the example from https://antonz.org/storing-state/#boolean-value
{"search": "potatoes", "popular": true} can be serialized as ?search=potatoes&popular
A long time ago while implementing the checkout page for an e-commerce site that's written as single-page application, we run into problem where, if the user goes to a 3rd party site to authorize payment method, we need to restore the checkout state. One approach that did not work for us is to store the state in session storage, because we do not know exactly when we should unset this state. The solution we took was to base64-serialize the state, gzip this (so that it does not exceed the browser's URL limit), and save it to the URL as query parameter. We used the History API to replace the URL whenever the local state changes, we write it there at URL. That way, if user goes to 3rd party site, we can specify the current full URL as the callback URL after the payment method is authorized.
A side effect of this is that if user decided to abandon our site at checkout page, they are able to go back using browser navigation with state fully restored.
A side effect of this is that if user decided to abandon our site at checkout page, they are able to go back using browser navigation with state fully restored.
Another approach, keep a versioned schema for the filter object and save encoded representation in url.
https://github.com/ananthakumaran/u
https://github.com/ananthakumaran/u
Im a strong proponent for storing everything in the url that makes sense. Before hot module replacement and fancy hacks to improve developer experience, developers were naturally motivated to do this as (auto-)refreshing the page while working would reveal all the state that would be lost. Now this is often just ignored even at bigger sites like quora and creates very annoying user experience when sharing links or apps/ tabs get nuked or when tabs reload after session restore. As a side note it is really surprising how many native apps also lose important UI state after being in the background.
> Deeper nesting is not typically used. It would be very nontrivial to deserialize such URL parameters back into the object.
Did you mean to write “very trivial” here? This is the same formatting as used when submitting forms with array and/or dictionary data, and further nesting commonly adds further brackets `like[foo][bar]=baz`. IME the querystring [0] package is great for this.
[0] https://www.npmjs.com/package/querystring
Did you mean to write “very trivial” here? This is the same formatting as used when submitting forms with array and/or dictionary data, and further nesting commonly adds further brackets `like[foo][bar]=baz`. IME the querystring [0] package is great for this.
[0] https://www.npmjs.com/package/querystring
I'd be inclined to ask a second question in addition to "What happens when you press refresh?":-
What happens when the user presses BACK.
Pagination seems acceptable, but if I go to a list page, then apply 3 filters, do I have to click "back" 3 times to leave the page? Do you just overwrite the history to allow the described 'refresh recovery' without clogging the history? Do you "debounce" sessions of filter changes within a period of time into a single "back"?
What happens when the user presses BACK.
Pagination seems acceptable, but if I go to a list page, then apply 3 filters, do I have to click "back" 3 times to leave the page? Do you just overwrite the history to allow the described 'refresh recovery' without clogging the history? Do you "debounce" sessions of filter changes within a period of time into a single "back"?
A trick I've used on top of serializing JSON in the URL is in-browser compression with a library like pako [0] (zlib in JavaScript). This helps you get more data in before you reach the browser limit for URL length.
[0] https://github.com/nodeca/pako
[0] https://github.com/nodeca/pako
Early php used to store the session id in urls: https://example.com/?PHPSESSID=23d2332d2 All links in the page contained the same id.
I believe it must have been before browser cookies was widely supported?
I believe it must have been before browser cookies was widely supported?
Yes I think so, Java(JavaEE?) did the same with jsessionid.
A few years back I used to store user data in the URL in conjunction with the goo.gl API (now deprecated), sort of a hack but worked great as a serverless key value storage, used it at fretflip.com to create shareable fretboard charts for guitar.
Any more than 2 search params becomes unholy to look at and understand.
I dont think sharing URLs with state is that relevant anymore. If I want to share something I am looking at that has state dependencies, I take a screenshot. I dont, or ever, expected the ability of a URL to take someone anywhere but the home page or top level sub-directory
If I refresh the page and my state is lost, that is expected behavior to me.
If I do not like the state of the page, I can refresh the page and start over.
It is frustrating when I refresh the page and literally nothing happens, and now I need to go into the URL and delete a bunch of shit in order to refresh my state.
It is WAY more important to me to be able to quickly build any state from scratch. I do not ever want to be 10 minutes into building some specific state on your website. I do not care how well you handle it because I dont trust it to persist anyway. I do not want to trust it to persist.
Focus your energy on improving the navigation of your site or build state into the site itself. Let me save search criteria terms into a named list associated with my account if it's important. I want to be dependent on the URL for as little as possible, even indirectly.
If I refresh the page and my state is lost, that is expected behavior to me.
If I do not like the state of the page, I can refresh the page and start over.
It is frustrating when I refresh the page and literally nothing happens, and now I need to go into the URL and delete a bunch of shit in order to refresh my state.
It is WAY more important to me to be able to quickly build any state from scratch. I do not ever want to be 10 minutes into building some specific state on your website. I do not care how well you handle it because I dont trust it to persist anyway. I do not want to trust it to persist.
Focus your energy on improving the navigation of your site or build state into the site itself. Let me save search criteria terms into a named list associated with my account if it's important. I want to be dependent on the URL for as little as possible, even indirectly.
TIL Stockholm Syndrome also exists for terrible UX.
i will emphasize again the importance of being able to build any state on your site quickly from scratch
how is being unbothered by a lack of features others claim they need a form of stockholm syndrome?
how is being unbothered by a lack of features others claim they need a form of stockholm syndrome?
Having the state stored in speaking URLs in a transparent manner allows advanced users to easily strip information they don't want to share.