Curl with Partial Files(daniel.haxx.se)
daniel.haxx.se
Curl with Partial Files
https://daniel.haxx.se/blog/2024/12/30/curl-with-partial-files/
20 comments
I'm sure this solves someone's problem, but reading https://everything.curl.dev/cmdline/variables.html makes me wonder whether curl really needs this added complexity
Folks using range requests will certainly welcome these as a great improvement. Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.
And, if you don’t need them, don’t use them! Have you seen how many options curl has? But never good ones to support range requests until now.
And, if you don’t need them, don’t use them! Have you seen how many options curl has? But never good ones to support range requests until now.
> Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.
This seems much more readable than the stuff introduced here, and I would still heavily prefer to do it this way.
This seems much more readable than the stuff introduced here, and I would still heavily prefer to do it this way.
I'm _pretty sure_ you're referring to `-C - -o $filename` and not this behavior. This one allows slurping ranges of local files and sending those in their entirety. I am open to maybe one day it does something crazypants with `-X PATCH --variable "fred@file[867-5309]"` or such but I doubt gravely that it does any such thing today
As shown here:
While digging into this, there's actually one for its modification time, too. Curl knows all:
As shown here:
$ python3 -m http.server 9090 &
# that doesn't matter, just return _something_ on :9090
$ ls -l .DS_Store
-rw-r--r--@ 1 mdaniel staff 18436 Dec 30 11:30 .DS_Store
$ curl -vo .DS_Store -C - http://127.0.0.1:9090/.DS_Store
** Resuming transfer from byte position 18436
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1:9090...
* Connected to 127.0.0.1 (127.0.0.1) port 9090
> GET /.DS_Store HTTP/1.1
> Host: 127.0.0.1:9090
> Range: bytes=18436-
> User-Agent: curl/8.7.1
> Accept: */*
While digging into this, there's actually one for its modification time, too. Curl knows all:
$ curl -v --time-cond .DS_Store -o .DS_Store -C - http://127.0.0.1:9090/.DS_Store
> If-Modified-Since: Mon, 30 Dec 2024 19:30:50 GMT
This is powerful stuff, but I can't help feeling like it's adding unnecessary bloat to an already-enormously complicated command. What is the advantage of:
curl --variable "pwd@secret;[0-31]" \
--expand-user daniel:{{pwd}} \
https://example.com/
curl --user daniel:`command-to-fetch-password` \
https://example.com/
The latter one shows your password in the process listing.
`mount /proc -o hidepid=1` recommended :)
First one will show how to get it :D
… which is not accessible to as many users as the process list is.
And can often be passed in bash via a filehandle, without a file.
Not all places you might run curl allow for shell tricks like process substitution or perhaps for subprocesses at all.
The main benefit seems to be context dependent quoting which is tricky to get right and in most cases (like in your example) omitted completely
What is the use-case for this feature? I get specifying byte ranges for parsing headers and whatnot, but I don't know a curl workflow that benefits from this.
The first one that comes to mind is something I haven’t done lately but used to do constantly, enough so that I wrote a custom fuse filesystem to combine isofuse with http handling, and that’s parsing an index from something with a range request then fetching only the part or parts I want. Think fetching only the trailing index from a zip archive and then fetching the specific file you want out of a 10tb zip rather than downloading the whole thing. Now, how often would I do this on the command line? Not sure, but I might not have bothered with fuse if I’d had the option.
Unless I'm gravely misunderstand this feature currently, it won't help that use case as the variable assignment only supports string literals, env cars, or existing files which wouldn't help your range request for the remote file
It would be a natural extension to treat the @ syntax as a uri (because, come on, it's curl!) if it contains a ":" but even that gets hairy because the next layer of indirection would be --variable-with-curlrc to allow setting the enormous number of parameters one would want when chasing the interior uri
It would be a natural extension to treat the @ syntax as a uri (because, come on, it's curl!) if it contains a ":" but even that gets hairy because the next layer of indirection would be --variable-with-curlrc to allow setting the enormous number of parameters one would want when chasing the interior uri
Pausing and resuming downloads, and fetching bytes in splits would turn curl into a download manager like aria2.
[deleted]
Nice! This will make my podcast blog to RSS generator a whole lot easier since I just need to grab some headers from the MP3 in addition to info the the page.
It's nice to see, but the extension on top of the cURL preexisting syntax is rough to look at.
Ever since stumbling upon Hurl a while ago, I've really enjoyed how easy to read the scripts are. Anyone that partially understands web/http is able to read and tell exactly what's going on in the low -> medium complexity scripts w/o even reading the docs.
Ever since stumbling upon Hurl a while ago, I've really enjoyed how easy to read the scripts are. Anyone that partially understands web/http is able to read and tell exactly what's going on in the low -> medium complexity scripts w/o even reading the docs.
This could be used to send a single file in a tar. Nice!