Defining a new HTTP method: HTTP Search (2021)(httptoolkit.com)
httptoolkit.com
Defining a new HTTP method: HTTP Search (2021)
https://httptoolkit.com/blog/http-search-method/
130 comments
The proposal has changed the name to QUERY, just yesterday.
This is why the RFC suggests you redirect to a GET like /resource/queryABCD
Yes, that’s also lost when you do POST. Which is by design though. A HTTP Search seems like only drawbacks.
SEARCH can have a request body.
Many systems limit the length of the URL, so this is significant.
Many systems limit the length of the URL, so this is significant.
If I send a request body with GET, what modern systems would even have a problem with that? Is there some caching middleware somewhere that I've never heard of or just ignored that will screw it up?
If there was a GET-with-body http verb I'd probably use it at one point or another, but I often wonder where plain GET would start blowing up if I just used it for that.
Honestly, I think rest is a mess, and that everything should just be POST with no values in the url at all.
If there was a GET-with-body http verb I'd probably use it at one point or another, but I often wonder where plain GET would start blowing up if I just used it for that.
Honestly, I think rest is a mess, and that everything should just be POST with no values in the url at all.
GET with a body is pretty useless because the standard doesn’t allow the body to affect the results. So, proxies, browsers, etc. are free to ignore the body when caching results.
even though most servers support it AFAIK, many clients don't. Many will even silently discard the body when sending a GET request.
So can POST. It's entirely redundant
From reading the article, one of the key items addressed by HTTP Search is caching. POST requests are usually not safe to cache
Is search safe to cache though?
It would definitely heavily depend on the dataset you're querying... Very little value in cached search results if you're searching through time-sensitive data such as logs or other live-datasets.
Most datasets I've searched also has a concept of permissions, so person a couldn't be served the same cached result as person b... I think search can't be cached at the http level either, its too heavily dependent on the data you're searching through so you'd have to implement it in the application anyway.
the article does make a good point though: a `get` request that supports a `body` would be nice, and thats pretty much all they're arguing for with the `search` verb.
It would definitely heavily depend on the dataset you're querying... Very little value in cached search results if you're searching through time-sensitive data such as logs or other live-datasets.
Most datasets I've searched also has a concept of permissions, so person a couldn't be served the same cached result as person b... I think search can't be cached at the http level either, its too heavily dependent on the data you're searching through so you'd have to implement it in the application anyway.
the article does make a good point though: a `get` request that supports a `body` would be nice, and thats pretty much all they're arguing for with the `search` verb.
The normal cache controls apply, so you can make it as safe as you need.
Verbs specify safety/idempotency guarantees for API-blind middleware, as well as whether (if either applies) a body needs taken into account; POST is not idempotent, SEARCH/QUERY is safe, and therefore also idempotent, but differs from GET in that that guarantee is body-specific.
Might as well just say "ok now GET can have body" instead of all that noise
Won’t retroactively change things that support GET to support GET-with-body. A new verb makes it much more likely that anything that supports the verb supports the desired semantics.
(Of course, you could instead bump the HTTP version for support of GET-with-body, but given how HTTP/2 and HTTP/3 are defined in terms of HTTP/1.1, you’d need three new versions of HTTP for a change to core verb semantics. A new verb, again, is far simpler.)
(Of course, you could instead bump the HTTP version for support of GET-with-body, but given how HTTP/2 and HTTP/3 are defined in terms of HTTP/1.1, you’d need three new versions of HTTP for a change to core verb semantics. A new verb, again, is far simpler.)
The limit of a GET request's length is at least several K on most systems I've used, so it's rarely an issue.
That's not as much as you might think.
There's a reason elasticsearch accepts POST
There's a reason elasticsearch accepts POST
But what exactly is the difference between POST and SEARCH? Both include the request parameters in the body, so they would be obscured from the user. Unless they aren't, in which case it is a matter of the choices made in the implementation.
Is it implied idempotency and the lack of a confirmation dialog when the user reloads the page?
Is it implied idempotency and the lack of a confirmation dialog when the user reloads the page?
Cachability, perhaps. You cannot cache a post, but may be able to cache a search or query verb.
> One advantage of GET is I can just copy the URL and share it.
no, you cant. if the server requires any headers such as Authorization or Cookie, this method will fail.
no, you cant. if the server requires any headers such as Authorization or Cookie, this method will fail.
It’s not a “no you can’t” just because you know of some exceptions.
And even then, they are still correct while you are not. You can copy the GET url even if it ultimately it requires authentication in a way that you can’t do it all for a POST request.
And even then, they are still correct while you are not. You can copy the GET url even if it ultimately it requires authentication in a way that you can’t do it all for a POST request.
Typical scenario is to redirect to login and after successful authentication return back to the requested URL, so the method doesn’t fail if server is implemented correctly.
Related, I wish there was a more standard way to include things like verb and headers in a URI. I hacked an implementation that parses /something#a:b&c:d to set the headers a and c, I was thinking for verb I could do https+get.
verbs don't identify a resource, so why would they be in a URI?
A separate format to serialize a request spec is a good idea, sure, but it is a distinctly different thing than the URI of the resource referenced by the request.
A separate format to serialize a request spec is a good idea, sure, but it is a distinctly different thing than the URI of the resource referenced by the request.
Is that not just the HTTP 1.1 format? It's a perfectly readable plain text format.
I'm pretty sure you know what the parent means.
Forget about what acronyms stand for. The thing that hyperlinks point to and you can type in your browser bar is called the URL or the link.
And the point is that being able to specify a verb and headers in a link would be super useful in certain situations.
Continue to call it a URL or URI and just change the "R" in those from Resource to Request, semantics problem solved. Or invent a new URA where "A" stands for "action" and it's a valid hyperlink. The naming of it is the least important part here.
Forget about what acronyms stand for. The thing that hyperlinks point to and you can type in your browser bar is called the URL or the link.
And the point is that being able to specify a verb and headers in a link would be super useful in certain situations.
Continue to call it a URL or URI and just change the "R" in those from Resource to Request, semantics problem solved. Or invent a new URA where "A" stands for "action" and it's a valid hyperlink. The naming of it is the least important part here.
We already have a format to identify an entire request including verb and headers. It’s called HTTP.
Yes, but it can't be used as a hyperlink or typed into the browser bar.
HTTP is a two-way messaging protocol. What's being talked about here is the capabilities in hyperlinks. Totally different.
HTTP is a two-way messaging protocol. What's being talked about here is the capabilities in hyperlinks. Totally different.
That sounds like a deficiency of browsers, not of URLs. Browsers can already make arbitrary HTTP requests, and there are some rudimentary ways to expose this in hypertext (such as forms or XHR), but there’s nothing stopping a browser from letting you dump a full HTTP request into a text field and sending it.
You're ignoring such a huge part of the HTTP specification there, like request idempotency, caching, the security model, and surely stuff I'm forgetting right now.
HTTP, at its heart, is a way to compose an action from a verb and a noun - such as "get this", or "update that". The request method, or verb, is intertwined with the URI, or resource, the noun - together, they form the action the user agent intends to carry out. "GET /foo" is entirely distinct from "POST /foo", and there are lots of considerations why it has been implemented like that. I cannot recommend reading the spec (or letting ChatGPT summarise it for you) enough, it will really make more sense.
Having said all that, I know what situations you are referring to - say, issuing a PATCH request with an HTML form, or circumventing some redirect bug with a POST request. Still, all of those problems hint at some other, more general issue, and solving such inconvenience would come at the price of a completely broken HTTP specification. Protocols like email, or HTTP, have only been around for so long because they were designed elegantly and carefully. Let's not break that for convenience' sake :)
HTTP, at its heart, is a way to compose an action from a verb and a noun - such as "get this", or "update that". The request method, or verb, is intertwined with the URI, or resource, the noun - together, they form the action the user agent intends to carry out. "GET /foo" is entirely distinct from "POST /foo", and there are lots of considerations why it has been implemented like that. I cannot recommend reading the spec (or letting ChatGPT summarise it for you) enough, it will really make more sense.
Having said all that, I know what situations you are referring to - say, issuing a PATCH request with an HTML form, or circumventing some redirect bug with a POST request. Still, all of those problems hint at some other, more general issue, and solving such inconvenience would come at the price of a completely broken HTTP specification. Protocols like email, or HTTP, have only been around for so long because they were designed elegantly and carefully. Let's not break that for convenience' sake :)
Believe me, I've read the spec as I've wound up implementing HTTP servers from scratch multiple times a couple decades ago.
None of this suggestion is about incorporating all of HTTP's functionality. It's just the situations that you say you know I'm referring to -- verbs, things like authorization headers, a POST payload.
Expanding the functionality of hyperlinks wouldn't break anything about HTTP. It would just allow more requests to be defined in a single line of text (a hyperlink), rather than requiring lines of JavaScript to define. The browser (or cURL or whatnot) would convert the link to the actual HTTP request. Zero changes to HTTP.
None of this suggestion is about incorporating all of HTTP's functionality. It's just the situations that you say you know I'm referring to -- verbs, things like authorization headers, a POST payload.
Expanding the functionality of hyperlinks wouldn't break anything about HTTP. It would just allow more requests to be defined in a single line of text (a hyperlink), rather than requiring lines of JavaScript to define. The browser (or cURL or whatnot) would convert the link to the actual HTTP request. Zero changes to HTTP.
Well. Hyperlinks are part of the spec though, and if you modify them and expect clients to know how to deal with that, you’ll need to modify the specification, and that implies you need to define how those changes affect caching, Proxies, and the security model. There’s a pretty good reason you send credentials in a header, not the URL. What’s my browser supposed to show in the history?
What you’re looking for is a browser extension, not a haphazard URI change.
What you’re looking for is a browser extension, not a haphazard URI change.
you can call it https+rpc and be formatted like
https+rpc:||news.ycombinator.com|reply?id=36096485&goto=item%3Fid%3D36095032%2336096485#method=PUT#H-Accept=text/html
with | instead of / to workaround this site encoding
https+rpc:||news.ycombinator.com|reply?id=36096485&goto=item%3Fid%3D36095032%2336096485#method=PUT#H-Accept=text/html
with | instead of / to workaround this site encoding
This is an interesting idea for a browser extension. Maybe needs a change in name from URL/URI. Could be a DURI, Discrete Universal Record Interaction. Just spitballing. You could share one-liners similar to how one might share curl command-lines but expect them to work in multiple environments.
Sounds like what curl does. Maybe curl:{curl commandline}
Guess that needs to be different than the ? query params. I wonder if you could use ?key=value as your signal for GET, and ?value as your signal for SEARCH. I think it's up to the service to parse the string that comes after ? in a URI so maybe that's a way to go.
That would unfortunately break the internet as a GET query without ?key=value and just ?value is perfectly reasonable.
The primary use case of SEARCH is programmatic, e.g., making complex requests to a search API in order to render results. Those are API requests, they're not being shared around.
> One advantage of GET is I can just copy the URL and share it.
Often but not always.
The article is wrong when it says message bodies for GET are defined to be meaningless. They are in fact just not defined to be meaningful, which is very much not the same thing.
Nothing in the spec for GET blocks using message bodies with it. Elastic search famously uses(used?) bodies with GET requests.
Often but not always.
The article is wrong when it says message bodies for GET are defined to be meaningless. They are in fact just not defined to be meaningful, which is very much not the same thing.
Nothing in the spec for GET blocks using message bodies with it. Elastic search famously uses(used?) bodies with GET requests.
unless you control every hop between client and server, GET bodies can be arbitrarily dropped, and can't be relied upon
This seems reasonable, but unlikely to happen. GET is already pretty much sufficient, especially given that clients generally support megabytes worth of query strings in these modern times. If we're doing this though, I'll cast my vote for naming it more semantically as QUERY, and forgoing whatever little WebDAV compatibility would be had otherwise.
QUERY would be more compatible with WebDAV as the would be no confusion. The relevant compatibility is with middle boxes like proxy servers that mostly already forward SEARCH but would block QUERY.
Middle boxes can arbitrarily limit the request URL size in certain cases.
Sure, but in the times of encrypted traffic, the only middle boxes capable of limiting the query should be ones explicitly added by the service owner.
In an enterprise setting you can just roll out your own root certificate to every machine's trust store, and have your middle box MitM all traffic with the help of that.
Funny you should say that:
https://httpwg.org/http-extensions/draft-ietf-httpbis-safe-m...
https://httpwg.org/http-extensions/draft-ietf-httpbis-safe-m...
I understand the benefits of QUERY over GET because GET is expected to return a resource in the response. Why not just use POST though? Send a post message to a server asking to conduct a search and sending the results of that query in the response?
QUERY is the one missing method to cover all reasonable combinations of safe/idempotent/neither and whether (if safety or idempotency applies) a body needs taken into account to identify a semantically identical request.
Not idempotent: POST
idempotent/not safe: PUT (body matters), DELETE (no body)
safe: QUERY (body matters)/GET (no body)
That’s why QUERY is needed. POST is not a good substitute.
Not idempotent: POST
idempotent/not safe: PUT (body matters), DELETE (no body)
safe: QUERY (body matters)/GET (no body)
That’s why QUERY is needed. POST is not a good substitute.
The biggest issue I've personally experienced is semantics when wrapping your head a design or debugging a service you aren't familiar with.
"POST" has come to imply a "write" operation, and mixing up the the reads and writes of a system leads to cognitive dissonance.
"POST" has come to imply a "write" operation, and mixing up the the reads and writes of a system leads to cognitive dissonance.
I’ve, amazingly, managed to hit the arbitrary megabyte limit in real production software.
Same, that's configurable though, and 1mb sounds like nginx. It's a lot higher on many other web servers including Apache.
I can report that cloudlfare workers fail on 44kb of url
Agreed.
Could be used for e.g. SQL over HTTP
Could be used for e.g. SQL over HTTP
Agree. I was also thinking about the "Query" naming
See also QUERY, imho a better name than SEARCH which is semantically a GET but with a request body -- it's cacheable!
https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-meth...
https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-meth...
Yep, I’ve already implemented QUERY. Sorry SEARCH.
They're effectively the same I think - this post is from 2021 but a later version of the same draft renamed the concept to QUERY.
I’d like to see this (or something like it) ship. Right now the two most common alternatives are:
1) GET, but base64 encode a json payload as a query param. Main downsides are that you can hit URL size limits, it makes the client and server slightly more complex (base64 encode/decode), and it sucks with browser dev tools - can’t nicely inspect the request in the browser, have to copy and decode it
2) Use POST to search. This confuses both machines and people. On the machines side, it doesn’t play well with caches, retry middleware, etc. On the people side, it confuses developers, ppl monitoring metrics, etc. - you think it’s creating something but it isn’t. Basically goes against all the reasons we have separate GET/POST/PUT/PATCH/DELETE methods in the first place
Yeah, we have workarounds, but they have pretty significant downsides. I’d rather have this new method. Honestly, I think one of the main reasons ppl reach for RPC frameworks over RESTful APIs is the awkwardness around making search queries - this would really fix that issue.
1) GET, but base64 encode a json payload as a query param. Main downsides are that you can hit URL size limits, it makes the client and server slightly more complex (base64 encode/decode), and it sucks with browser dev tools - can’t nicely inspect the request in the browser, have to copy and decode it
2) Use POST to search. This confuses both machines and people. On the machines side, it doesn’t play well with caches, retry middleware, etc. On the people side, it confuses developers, ppl monitoring metrics, etc. - you think it’s creating something but it isn’t. Basically goes against all the reasons we have separate GET/POST/PUT/PATCH/DELETE methods in the first place
Yeah, we have workarounds, but they have pretty significant downsides. I’d rather have this new method. Honestly, I think one of the main reasons ppl reach for RPC frameworks over RESTful APIs is the awkwardness around making search queries - this would really fix that issue.
I would say that people should be more open to using HTTP in non-REST ways.
Sometimes you want to send a request and receive a response, with arbirtrary restrictions and side effects that suit your cases.
There are a lot of good reason for rest to exists, but also sometime you want to POST /open-garage-door?t=5-minutes and call it a day
Sometimes you want to send a request and receive a response, with arbirtrary restrictions and side effects that suit your cases.
There are a lot of good reason for rest to exists, but also sometime you want to POST /open-garage-door?t=5-minutes and call it a day
Sure, the odd RPC style HTTP call is fine - not everything is cleanly modelled as a resource. But the lack of a clean way to make GET requests with a more complex request payload is simply a wart/shortcoming in the HTTP protocol IMO, and one that’s very correctable. Sucks to have to abandon RESTfulness even when I have a very “clean fit for the resource model” call to make, but the request body is too complex/large for HTTP query params.
Right now, you have two main options:
Use a GET, and squeeze all the parameters you need in the URL or headers somewhere
Use a POST, and have the request considered as unsafe & uncacheable
Wanting a special method for search hurts the conceptual integrity of HTTP, where resource representation is the core idea to build on top, it isn’t supposed to be just a request/response protocol.
That's exactly what we do: submitting a search/filter form through a POST request which creates a record in the database representing the search/filter criteria and then redirects to the same endpoint with the ID of that record. The endpoint looks for the ID in the query, attempts to load the same record, and if found, executes and returns the search results.
It works very well, is completely transparent to the end user, has automatic query logging, and provides a clean URL that can be easily shared.
It works very well, is completely transparent to the end user, has automatic query logging, and provides a clean URL that can be easily shared.
I really wish I'd thought of this method when I was building a search based application. It was my first time and I struggled with structuring everything
> POST or PUT to a resource representing the search, then you’re free to redirect and subsequently GETs of this resource can be cached.
But that only means that specific instance of the search query is cacheable, not the search query itself, no? I presume the POST would create a new identifier for the resource, so that yes, that specific resource is cacheable. (Even if the server says "oh, I've just seen this POSTed query, let me return the same resource ID", another client will still have to POST to the server, a non-cacheable action.)
The idea of cacheable isn't just by the server, but by anything in between the client and the resource. By using QUERY, the query itself can be cached.
But that only means that specific instance of the search query is cacheable, not the search query itself, no? I presume the POST would create a new identifier for the resource, so that yes, that specific resource is cacheable. (Even if the server says "oh, I've just seen this POSTed query, let me return the same resource ID", another client will still have to POST to the server, a non-cacheable action.)
The idea of cacheable isn't just by the server, but by anything in between the client and the resource. By using QUERY, the query itself can be cached.
What I like about this proposal is that no one has to use it. There's lots of comments here saying why they prefer GET, and those people are welcome to keep using it- GET will remain unchanged.
But for those applications where having a complex body in the query request is a better choice, this tool can be available. I think that's a win.
But for those applications where having a complex body in the query request is a better choice, this tool can be available. I think that's a win.
TIL sending a POST request to a URL invalidates the cache for GET requests of the same URL.
Going to have to review my REST APIs to make sure that’s not a problem.
Going to have to review my REST APIs to make sure that’s not a problem.
Incredible, I did not know that. Does anybody point out to some resources to learn more about this behavior?
I'm sorry, but isn't that the most basic thing possible about GET and POST?? GET is used to read a resource, POST to write to it... why would you NOT expect writing to a resource to invalidate caches of that resource??
Because usually GET /posts gets you a list of posts, POST /posts create a new one.
I don't know what level of incredulity is appropriate in my opinion (probably not as much as the parent comment), but isn't that another good reason to expect cache invalidation? Surely creating new posts would expectedly invalidate the list of posts cached, it has to now include at least one new one.
I think you’d have to have a fairly awkward API design in order for this to be an issue. For example, if “POST /posts” does something else, like flags a post for a moderator to review.
That being said, I don’t see this feature being particularly beneficial either. Only the caches which observed the request could know to purge the cache. So you can’t actually rely on it for cache invalidation.
That being said, I don’t see this feature being particularly beneficial either. Only the caches which observed the request could know to purge the cache. So you can’t actually rely on it for cache invalidation.
Many people seem to be unenthusiastic about it because the limitation on query strings are usually large enough.
I personally like this addition, because it no longer requires all queries be shoehorned into query strings. You can use any syntax you like, be it SQL or GraphQL etc.
I personally like this addition, because it no longer requires all queries be shoehorned into query strings. You can use any syntax you like, be it SQL or GraphQL etc.
Wouldn’t be enough to “extend” GET so that it supports a body payload as well?
It already does.
The RFCs are clear that a server should ignore any body sent by a client after a GET request header. But RFCs don't matter; real life matters, and in real life there are lots of HTTP clients that can't make a GET request with a body. We had to go into our API and add POST support to every endpoint that was GET + body because there was a long tail of clients that couldn't manage to make the GET request. In practice, support is NOT widespread enough to rely on GET + body requests.
Sort of, but it’s not that solid: https://datatracker.ietf.org/doc/html/rfc9110#name-get
(paragraph starting “Although request message framing is independent of the method used”)
(paragraph starting “Although request message framing is independent of the method used”)
It’s not exhaustive though. Many servers and proxies support it, but not all. For instance, GCP load balancers do not, but ALB does. At least that was case a few years ago.
It sort of does, but support is spotty. I've tried using it only to have things like MITM proxies on my customers' corporate networks drop the body. It'd be a large push to get universal support. It may be more difficult than simply adding a new method.
But if a new method is needed, servers have to support it (e.g., by updating them to the next version that supports the new method). In the same sense, couldn’t that new version “just” not drop the body of GET requests?
In any case we are talking about updating servers.
In any case we are talking about updating servers.
The only software I have seen that does this is Elasticsearch - it works with their client libraries but I'm always worried there may be proxies and suchlike which don't support it.
They support POST as a fallback.
They support POST as a fallback.
No, because that would make GET not-cachable by URL alone, which breaks systems that rely on the fact that it is.
A new method doesn’t break existing systems.
A new method doesn’t break existing systems.
You can actually do that already. It's one of those things where the specification is a bit loose on this. Elasticsearch is an example of a product that supports doing a GET with a body to do search. Works fine but not all http clients allow you to do this so they also support POST.
Nice discussion on HTTP GET with a body here: https://stackoverflow.com/questions/978061/http-get-with-req...
TLDR. the http 1.1 spec was a bit vague on this and allowed people to send a body but specified that servers should ignore that if they did. Later updates of the 1.1 spec removed that sentence and actually allow this. The caveat that the spec mentions that some implementations may not support it.
So an update is not needed; it's already allowed.
I tend to not get too hung up on the meaning of http verbs. I sort of lost interest in the endless debates on this years ago. I like a well designed REST API of course and I try to follow the principle of the least amount of surprise when I implement one myself. Using exotic new experimental http verbs would be surprising. Using GET this way is also a bit surprising. I actually steer clear of using PATCH as well. Just use a PUT or a POST and move on.
Adding new verbs to HTTP is redundant as far as I'm concerned. Not really against it but where does it stop? And what do we get out of it? It doesn't sound like it's worth the trouble.
Nice discussion on HTTP GET with a body here: https://stackoverflow.com/questions/978061/http-get-with-req...
TLDR. the http 1.1 spec was a bit vague on this and allowed people to send a body but specified that servers should ignore that if they did. Later updates of the 1.1 spec removed that sentence and actually allow this. The caveat that the spec mentions that some implementations may not support it.
So an update is not needed; it's already allowed.
I tend to not get too hung up on the meaning of http verbs. I sort of lost interest in the endless debates on this years ago. I like a well designed REST API of course and I try to follow the principle of the least amount of surprise when I implement one myself. Using exotic new experimental http verbs would be surprising. Using GET this way is also a bit surprising. I actually steer clear of using PATCH as well. Just use a PUT or a POST and move on.
Adding new verbs to HTTP is redundant as far as I'm concerned. Not really against it but where does it stop? And what do we get out of it? It doesn't sound like it's worth the trouble.
> GET requests ... can't have a request body. It's not specifically banned, but it is defined as being completely meaningless
This is widely believed but false. And repeating it leads me to believe that the author is not careful enough to create new standards. They are just not defined to be meaningful, which is not the same thing.
In fact the standard says that sending a message body with a GET request should not (specifically SHOULD NOT, not MAY NOT) be done unless you have confirmed that the servers accept it, because servers are not required to accept it, but that if you are talking to a server directly that you know supports it then it's perfectly fine and within the standard.
This is widely believed but false. And repeating it leads me to believe that the author is not careful enough to create new standards. They are just not defined to be meaningful, which is not the same thing.
In fact the standard says that sending a message body with a GET request should not (specifically SHOULD NOT, not MAY NOT) be done unless you have confirmed that the servers accept it, because servers are not required to accept it, but that if you are talking to a server directly that you know supports it then it's perfectly fine and within the standard.
As other commenters pointed out, support is spotty and it's generally a greyzone. A new verb (I'd also support "QUERY") would clarify things.
But it doesn't actually work in many browsers.
Another way to think about it, if you're doing REST as it seems the author of this article seems to strive for is to introduce a "search resource". You'd then POST to e.g. POST /searches with the query in the body, and on the server side a search resource is created. The server performs the search query and returns the result with a search ID. To do the same search again just do a GET /searches/ID.
One of the user cases listed - being able to send a large body of data to a server for it to encrypt (though it could be any sort of transformation, e.g. reformatting, encoding, format conversion, or even operations like "spell check" etc.) strongly suggests to me we don't need a new verb, but should simply redefine the expected behaviour for GET requests for when they do or don't have a body - even if it means there's a expectation and method for clients to first query servers whether they have that ability (as servers that don't support "search" will return an error if that verb is used, whereas servers, esp. proxies/load balancers etc, may well simply ignore the body if GET is used.).
And to not define caching semantics seems somewhat crazy - it's bound to result in implementations inventing their own (the obvious one would be to combine some hash of the body with the URL to uniquely identify the request).
By adding the new verb, it implicitly formalizes that body in GET should not be used. Previously it was allowed, although not always expected.
This may be a good thing and it could also be a bad thing if middleware starts relying on this new expectation and break old applications that previously assumed get bodies to be passed through.
They should clarify the expected behavior of GET bodies more explicit, whether it is allowed or not doesn’t really matter, as long as it is crystal clear.
This may be a good thing and it could also be a bad thing if middleware starts relying on this new expectation and break old applications that previously assumed get bodies to be passed through.
They should clarify the expected behavior of GET bodies more explicit, whether it is allowed or not doesn’t really matter, as long as it is crystal clear.
being from 2021, I guess this one didnt pan out
Its still actively being worked (the most recent workgroup draft, and IIRC several of the internet drafts since the posted one, names it QUERY, SEARCH as a name for thr verb is probably dead) though, yes, the particular 2021 draft didn’t get adopted as anything more.
I'm trying to imagine an example of the authors primary use case: "a complicated data retrieval, sending lots of data but not changing the server state". Is he imagine sending something like a SQL statement SELECT with multiple WHERE and JOIN clauses using a recursive CTE?
If defining the attributes of the thing to return takes that much complexity, maybe that's a sign the endpoint is poorly designed, rather than GET query string length being insufficiently large.
If defining the attributes of the thing to return takes that much complexity, maybe that's a sign the endpoint is poorly designed, rather than GET query string length being insufficiently large.
A great example of this today is GraphQL. Even if you disagree with the premise of GQL, you can understand how you can have complicated filtering and sorting logic, and conditional expansion of nested resources. For a nontrivial data model, you may want to select only a minimal number of fields on each nested resource. Having lots of different variants of the same endpoint to serve the same query with different shapes of output is difficult to maintain, so this makes some sense.
I knew someone would bring up GraphQL, as I consider it a sort of way to present a database schema in JSON, so of course it would result in complex SQL-like queries. I understand why it works for Facebook, but it's not something I've ever reached for to solve any programming problem I've had.
It's absolutely awesome for implementing any kind of faceted search inside of a web application. Generating reports, finding resources, and of course social media do this sort of operation all the time. Gql makes it way cleaner and more efficient than GET this, parse it, POST a query, GET more things.
If you aren't in those spaces it probably seems unnecessary.
If you aren't in those spaces it probably seems unnecessary.
Nothing about GQL in particular is necessary to appreciate the need for this. Consider the Stripe API, where you might fetch invoices and expand the customers for each of those invoices, and then fetch the default payment methods for each of those customers. Stripe does this with a mess of query string parameters over a REST-ish API, but it could be accomplished much better with a request body.
GQL solves this problem in a particular way, but even folks who don't turn to GQL still have this problem.
GQL solves this problem in a particular way, but even folks who don't turn to GQL still have this problem.
We actually run into the query limit problem super often where I work. User-facing search pages for very complicated data models create very complicated URLs quickly.
That reminds me of reporting software like Crystal Reports. A thin veneer over a relation model (or, often, a collection models abstracted). I understand why businesses choose them, but they've always struck me as a bit of a shortcut. I'm always more curious about what the business does with the reports, and why we can't program to those requirements.
I was under the impression that new methods could not be amended into an existing RFC by revision but would require a new major.minor version all together is that not the case? Or is this worked around with the concept of extensions and if so doesn't that mean vendors of middle boxes are not required to recognize it?
Maybe read only query getting larger than the GET length limit indicates it's post-processing the object for you and not a "cachable" resource from a key value retrieval perspective? I.e your POST'ing a read only query since an application has to process the query for your at that point ?
If they do add this I hope they also add support to the Browser History pushState API to set the request body. You can obviously set an arbitrary JS state object, but for server rendered "html over the wire" single page apps it would be nice to be able to do a pushState that is sent to the server on refresh.
I used a SEARCH verb for an REST API years ago. Granted, we only had to support clients that we wrote ourselves. Semantically, however, it made a lot of sense to include a payload in the request body describing the query.
No one else needed to know or care that it existed.
No one else needed to know or care that it existed.
Can anyone steelman this idea for me? Seems like an incredible waste of time to make this.
This sounds promising as a lot of use cases are shoe horned into the current verbs.
A bit surprised EVALUATE isn't discussed as a name since it would make sense both for search queries and other "get using body" scenarios.
One of the arguments for a separate SEARCH verb is to differentiate between the cacheing needs of POST. Aren’t cache headers sufficient for using the existing verbs?
Alternative naive solution: Add a body to GET requests.
As I said, this is totally naive. Is there some obvious reason why this is a terrible idea, or not really possible?
As I said, this is totally naive. Is there some obvious reason why this is a terrible idea, or not really possible?
There are a lot of client implementations that don't support this. You already can add a body to GETs but you'll likely run into issues somewhere due to this. I believe that JavaScript's fetch doesn't allow for this as one example. That's a big one that is probably important to have but there are many others.
Because a lot of existing proxies and other middleware will just drop the body. Or even worse not consider it in caching decisions but still proxy it.
I want this. SEARCH/POST APIs forever.
i suppose you could post and store a query, give it a number and use get.
The missing ones are HTTP OPEN/CLOSE to improve the sessions management and security audits.
Did you know that HTTP is an inherently stateless protocol?
From the very first line in the RFC[1]:
"The Hypertext Transfer Protocol (HTTP) is a stateless application- level protocol for distributed, collaborative, hypertext information systems."
Which is why you need to send a header every time you make a request with authorization data.
Given that: no, stateful methods like OPEN/CLOSE would not fit at all with how HTTP is designed to work.
[1] https://www.rfc-editor.org/rfc/rfc7231
From the very first line in the RFC[1]:
"The Hypertext Transfer Protocol (HTTP) is a stateless application- level protocol for distributed, collaborative, hypertext information systems."
Which is why you need to send a header every time you make a request with authorization data.
Given that: no, stateful methods like OPEN/CLOSE would not fit at all with how HTTP is designed to work.
[1] https://www.rfc-editor.org/rfc/rfc7231
What if modern browser support advanced URL bar with POST method ? (Like Postman UI)
This new method would be idempotent and safe. POST is neither.
But, adding other methods than GET to the address bar in browsers is an interesting idea. It'd affect such a small percent of people it'd most likely be a net time sink than time saver, though.
But, adding other methods than GET to the address bar in browsers is an interesting idea. It'd affect such a small percent of people it'd most likely be a net time sink than time saver, though.
Or just use GET since it already works. I don't see any problems with large query params. You can also split the search into a POST request and a GET request for the result.
> I don't see any problems with large query params.
Most servers will reject URLs after a certain size, 4-8KB.
Most servers will reject URLs after a certain size, 4-8KB.
This is correct, and one only needs to consider the impact of large or unlimited length urls on server memory consumption per request to see that a relatively small limit is desirable and even necessary.
Depends. But moving the payload between the body and the query will not solve anything performance wise if you design for it.
A query of that size will likely not benefit from a GET anyway so use a POST.
Sure except POST has its own set of issues - like not being cachable. But SEARCH or QUERY can be cached safely!
[deleted]
A situational issue that I've run into a number of times with GET is various middle boxes inadvertently logging sensitive data that was in the query string of a GET. By sensitive data, I mean PII and CU information. While we have used POST to side step the issue, it'd be nice if there was a more appropriate method.
But a new HTTP method would require an update to the middle boxes... if you're doing that, you could also consider updating them to avoid the issue you described.
Overall, I'm not necessarily inherently against new HTTP methods. If equipment that is supposed to be transparent to HTTP isn't, should be solved elsewhere, not by a new HTTP method.
Overall, I'm not necessarily inherently against new HTTP methods. If equipment that is supposed to be transparent to HTTP isn't, should be solved elsewhere, not by a new HTTP method.
There's no button you can press to get all middleboxes updated.
It's better to fail on the ones that slip through the cracks than to leak data. And the failure alerts people that updates are needed here.
It's better to fail on the ones that slip through the cracks than to leak data. And the failure alerts people that updates are needed here.
Many logging systems exclude query parameters from the logging by default. It is also fine to include PII data in the logs as long as they are handled correctly and deleted after 30 days. But I have got burned by this problem as well. Like people using PIIs for identifiers.
I’m pretty sure most browsers have fairly low limits for uri length.
The advantage of GET is that you can always link to it. Doesn't matter with POST because you're never going to want to link to a POST.
So, GET URLs with base64 encoded data is definitely ugly, but at least you can save them and share them easily.
So, GET URLs with base64 encoded data is definitely ugly, but at least you can save them and share them easily.
While I love the proposal (apart from the name, I can see the SEARCH verb being used for something that's not search), they should also address the URL share-ablity aspect.
Something like https://google.com/search<some-special-character><query> where query can be arbitrarily large (>2000 URL length restriction) and the browser is smart enough to treat only https://google.com/search as the URL and anything after that as the body. The complete "URL" can be big and shared anywhere else.