Append data to the url query(daniel.haxx.se)
daniel.haxx.se
Append data to the url query
https://daniel.haxx.se/blog/2022/11/10/append-data-to-the-url-query/
32 comments
Daniel Stenberg, curl's lead developer, made a good tutorial video[1] on curl, focusing on advanced use of its HTTP options.
Here are some org-format notes I took on curl, mostly from the contents of that video: [2]
[1] - https://www.youtube.com/watch?v=I6id1Y0YuNk
[2] - https://paste.grml.org/plain/3559
Here are some org-format notes I took on curl, mostly from the contents of that video: [2]
[1] - https://www.youtube.com/watch?v=I6id1Y0YuNk
[2] - https://paste.grml.org/plain/3559
When I use curl, it's mostly when I need to check header or to trace requests in the network, for which I just use -vvv. Simple one-off stuff.
So I wonder how much demand there actually is for advanced features like this. When I need to make POST requests with queries and data, I would just use python because I most likely need to process the response (parse html, store json output to database, etc.).
Adding advanced features are nice, but IMO this might lead to over-engineering and it might be hard to maintain down the line.
So I wonder how much demand there actually is for advanced features like this. When I need to make POST requests with queries and data, I would just use python because I most likely need to process the response (parse html, store json output to database, etc.).
Adding advanced features are nice, but IMO this might lead to over-engineering and it might be hard to maintain down the line.
> Adding advanced features are nice, but IMO this might lead to over-engineering and it might be hard to maintain down the line.
curl's not exactly a new project -- this is 'down the line' for it.
I personally welcome the change. I wont use it often, but when I will, I'll be quite happy that it's there. There are a number of other features I'd give up in it's stead.
curl's not exactly a new project -- this is 'down the line' for it.
I personally welcome the change. I wont use it often, but when I will, I'll be quite happy that it's there. There are a number of other features I'd give up in it's stead.
> Adding advanced features are nice, but IMO this might lead to over-engineering and it might be hard to maintain down the line.
Well, one person [1] has been maintaining it since 1998. It's a good thing that HTTP doesn't change that often.
[1] https://daniel.haxx.se/
Well, one person [1] has been maintaining it since 1998. It's a good thing that HTTP doesn't change that often.
[1] https://daniel.haxx.se/
More than several times, there are data APIs that return raw text/csv, esp taking data from older systems.
Writing a scraper for those, bash & curl comes in handy w/o needing extra cruft of file writing ceremonies (ex: with open…), just pipe straight to file.
Writing a scraper for those, bash & curl comes in handy w/o needing extra cruft of file writing ceremonies (ex: with open…), just pipe straight to file.
I thought this was going to be about base64 encoded json query param(s) - instead of posting base64 or making a bunch of different params, do a GET with one param of base64 encoded json. Easy for the client, easy for the server, easy to share, only downside that I see is that it makes things a bit obtuse for users, but an extension or browser affordance could fix that.
If you're into this stuff, you might enjoy my "Twitter as a CDN tool", here's pong in a single Tweet: https://twitter.com/rafalpast/status/1316836397903474688
> Easy for the client, easy for the server, easy to share, only downside that I see is that it makes things a bit obtuse for users
There are more downsides:
- processing and creating payload can be harder are you're passing plaintext w. ca. 30% size increase on average
- base64-encoded assets (e.g. SVGs) would cause a huge performance drop in WebKit (this probably has improved since the last time I ran benchmarks).
- since you're encoding content and not its location, you can't easily update the underlying resource (hence to update my version of Pong, I'd need to create a new unique URL). IIRC IPFS suffers (used to?) from a similar issue
- not an issue now, but still was 1-2 years ago: some browsers would allow only for 2 or 4kb of text in URL
> Easy for the client, easy for the server, easy to share, only downside that I see is that it makes things a bit obtuse for users
There are more downsides:
- processing and creating payload can be harder are you're passing plaintext w. ca. 30% size increase on average
- base64-encoded assets (e.g. SVGs) would cause a huge performance drop in WebKit (this probably has improved since the last time I ran benchmarks).
- since you're encoding content and not its location, you can't easily update the underlying resource (hence to update my version of Pong, I'd need to create a new unique URL). IIRC IPFS suffers (used to?) from a similar issue
- not an issue now, but still was 1-2 years ago: some browsers would allow only for 2 or 4kb of text in URL
Hah, that Twitter thing is neat.
> not an issue now, but still was 1-2 years ago: some browsers would allow only for 2 or 4kb of text in URL
Yes, I had in mind smaller payloads, things that would have been query params on a GET
> not an issue now, but still was 1-2 years ago: some browsers would allow only for 2 or 4kb of text in URL
Yes, I had in mind smaller payloads, things that would have been query params on a GET
> This is curl’s 249th command line option
This made me smile!
And also
And also
$ rclone help flags . | grep "^ *--" | wc -l
665
Almost quarter of a thousand. Yay!
That does raise an interesting question though. If one wanted to create a command-line tool (or, rather, toolset) as comprehensive as curl, but with a much smaller API surface, how would they go about it? Sure, some of these flags are just shorthands, and some of them could be removed entirely (-o → stdout redirection), but that still leaves a lot of options.
That does raise an interesting question though. If one wanted to create a command-line tool (or, rather, toolset) as comprehensive as curl, but with a much smaller API surface, how would they go about it? Sure, some of these flags are just shorthands, and some of them could be removed entirely (-o → stdout redirection), but that still leaves a lot of options.
Curl is really really big on backwards compatibility, which I don't mean to critisize.
If you were starting over from scratch even trying to make it curl-like, you could probably trim down the options substantially.
The OP is in fact an example of a new option that can more-or-less cover the same functionality as some old options, plus more, but the old options remain.
If you were starting over from scratch even trying to make it curl-like, you could probably trim down the options substantially.
The OP is in fact an example of a new option that can more-or-less cover the same functionality as some old options, plus more, but the old options remain.
How would you keep it there? You'd have to spend a lot of effort discouraging feature requests.
You don't need to discourage them. Just ignore them.
Perhaaps take a look at the most popular ones every now and then, but a free software project owes the world nothing.
Perhaaps take a look at the most popular ones every now and then, but a free software project owes the world nothing.
In theory yes.
In practice I assume (I have no practical experience) that as the project gets larger the number of involved parties grows. And suddenly you need to balance interests of not just you but also the community, contributors, large users, sponsors and possibly more.
In practice I assume (I have no practical experience) that as the project gets larger the number of involved parties grows. And suddenly you need to balance interests of not just you but also the community, contributors, large users, sponsors and possibly more.
Curl supports a lot of crazy stuff that adds command line options, like IMAP/LDAP/POP3/SMTP means adding --login-options, --mail-auth, --mail-from, --mail-rcpt-allowfails, --mail-rcpt. If you read the man page, you'll find other flags that aren't really central to a http(s) fetch tool, like ftp, tftp.
Wow. Every time I've tried to dig into the man pages to find an option that I need it's been overwhelming; and now I know why!
This is one seriously under appreciated piece of code, and I am so grateful that it exists.
This is one seriously under appreciated piece of code, and I am so grateful that it exists.
They changed the curl man page in the last year so that all options have 'see also' recommendations. The documentation is amazing.
https://twitter.com/bagder/status/1460867775690387457
https://twitter.com/bagder/status/1460867775690387457
The article says the new option replaces the -G option. But, of course, for compatibility they cannot remove -G.
curl inferring the HTTP method from options is just bad design. It should have always been explicit via `-X`, but weirdly enough it sometimes warns when using that option.
Also, it seems some curl behaviours live in the past where only GET and POST verbs exists, ignoring others like PUT and PATCH.
Also, it seems some curl behaviours live in the past where only GET and POST verbs exists, ignoring others like PUT and PATCH.
They didn't live in the past when implemented :)
Backwards compatibility is a wonderful thing, but leads to anachronisms 20 years later.
`-X` exists for anyone that doesn't want curl's divination
Backwards compatibility is a wonderful thing, but leads to anachronisms 20 years later.
`-X` exists for anyone that doesn't want curl's divination
curl is amazing. I thought I'd read somewhere that it's on Mars.
Yup! He has a badge for it on Github: https://github.com/bagder?tab=achievements
All the other github hosted software which is eligible for this achievement is available https://docs.github.com/en/account-and-profile/setting-up-an...
edit: oh no, log4j-2.11.0
edit: oh no, log4j-2.11.0
But for interactive CLI usage, I always find its command line options non intuitive - take this for example, I don’t think I can really remember the difference between -d and —data-url and —url-query if I don’t use it daily. I think that the price to pay to make it backward compatible.
For that reason, I would recommend httpie as a replacement, which provides a more intuitive interface.