Wcurl: a curl wrapper to download files(samueloph.dev)
samueloph.dev
Wcurl: a curl wrapper to download files
https://samueloph.dev/blog/announcing-wcurl-a-curl-wrapper-to-download-files/
71 comments
It does sound like a request for “curl —-wget”, which I wholeheartedly support.
It's not a nice move to hijack someone else's program+popularity and include it as an option in yours...
That might be a nice improvement, but I believe people want a single command without any arguments for this use case.
I can't speak for other people. But for myself I'd rather have it as a --option than another command. I dislike tools that install a lot of different commands, it just gets harder to remember them all.
That argument doesn't make sense at first glance: how is lots of differen't commands harder to remember than lots of different --options?
one command can list all it's options with tool -h?
alias wget='curl --wget', then?
I use `curl -JLOC-`
It's easy to remember, JLO is jlo, C is continue, - for symmetry.
Stealing this, but had to lookup the parameters. It does work, but it should be `-C -` with a space. Also I like progress bars so my alias for now is:
* -J Use the server-specified Content-Disposition filename instead of extracting a filename from the URL
* -L Follow redirects
* -O Write output to a local file (download)
* -C - Resume transfers
Maybe a good mnemonic would be something along the lines of jlo liking hashtags?
curl -#JLOC -
* -J Use the server-specified Content-Disposition filename instead of extracting a filename from the URL
* -L Follow redirects
* -O Write output to a local file (download)
* -C - Resume transfers
Maybe a good mnemonic would be something along the lines of jlo liking hashtags?
>* -J Use the server-specified Content-Disposition filename instead of extracting a filename from the URL
Can't wrap my head around how -J maps to remote header name...still, this was super useful, thanks!
Can't wrap my head around how -J maps to remote header name...still, this was super useful, thanks!
My curl complains when -J/--remote-header-name and -C/--continue-at is combined.
$ curl -JLOC- "https://news.ycombinator.com/news.css?nZdOgS3Y18zj0ynCo50h"
curl: --continue-at and --remote-header-name cannot be combined
curl: try 'curl --help' or 'curl --manual' for more information[deleted]
is there a way to make curl do wget style progress output? i like curl but would love that kind of meter
"curl --progress-bar" does that
"wcurl --opts="--progress-bar" URL"
I was wondering if that should be the default for wcurl, but the bar only works for downloading single files, I was afraid of users thinking there's a bug whenever they downloaded more than one file if the bar wasn't there anymore.
"wcurl --opts="--progress-bar" URL"
I was wondering if that should be the default for wcurl, but the bar only works for downloading single files, I was afraid of users thinking there's a bug whenever they downloaded more than one file if the bar wasn't there anymore.
It isn't exactly what you are asking for but could be adapted to do that: https://github.com/dryark/curl_progress
alias wget='echo "using httpie as wget";http -d'
I appreciate it, but why not just:
alias wcurl="curl -# -O $1"
The alias is closer to:
Then, if you look at the script, it's basically that, with some minor cleaning up and nicer error handling.
alias wcurl = curl -# -O $1 --location --remote-name --remote-time --retry 10 --retry-max-time 10 --continue-at -"
Then, if you look at the script, it's basically that, with some minor cleaning up and nicer error handling.
FWIW, I think you can add some of those options to .curlrc if you wanted them to be added every time you invoke curl.
correct; .curlrc for the win!
and you can use `-q` to have it exclude the curlrc and use only args you pass to it. curl has such an amazing amount of power, but also means it has a lot of options to utilize a lot of things folks take for granted (exactly like retries, ect)
and you can use `-q` to have it exclude the curlrc and use only args you pass to it. curl has such an amazing amount of power, but also means it has a lot of options to utilize a lot of things folks take for granted (exactly like retries, ect)
Makes me think that probably a lot of unattended shell scripts out there should use -q in case someone has a .curlrc altering the behaviour of curl, and thus breaking the expectations of the script.
Maybe even should be a shellcheck rule.
Maybe even should be a shellcheck rule.
same can be said for wget with --no-config (and really any app that runs on any system) - if you're in the business of automating it, those config-features need to not be ignored (or down right overwritten to your scripts desire)
in docker-containers though, its more safe to assume no pre-loaded rc files are embedded (but 100% want to check your source-container for things like that) - but for running in some users workspace, the need to be careful is real.
most of the time though, those options should be safe; only really need to not have auto-retries are when trying to get super accurate date (like observing odd behavior, you dont want thing to simply "auto-work" when finetuning something, or triaging an upstream-server or dns issue)
i often write my scripts execing calls like `\curl -q ...` such that i get no user's alias of curl and ensures a consistant no-config amongst my system and others (although gnu curl vs mac curl (and other binaries of gnu vs mac vs busybox) are always the real fun part). (if the user has their own bash-function of curl then its on them for knowing their system will be special compared to others and results maybe inconsistent)
most of the time though, those options should be safe; only really need to not have auto-retries are when trying to get super accurate date (like observing odd behavior, you dont want thing to simply "auto-work" when finetuning something, or triaging an upstream-server or dns issue)
i often write my scripts execing calls like `\curl -q ...` such that i get no user's alias of curl and ensures a consistant no-config amongst my system and others (although gnu curl vs mac curl (and other binaries of gnu vs mac vs busybox) are always the real fun part). (if the user has their own bash-function of curl then its on them for knowing their system will be special compared to others and results maybe inconsistent)
Oh wow, I thought only function could take parameters !
It must be relatively recent because old those old answers do not mention it : https://stackoverflow.com/questions/7131670/make-a-bash-alia... https://stackoverflow.com/questions/34340575/zsh-alias-with-...
On the other hand I tested it in both bash and zsh, and it works in both
It must be relatively recent because old those old answers do not mention it : https://stackoverflow.com/questions/7131670/make-a-bash-alia... https://stackoverflow.com/questions/34340575/zsh-alias-with-...
On the other hand I tested it in both bash and zsh, and it works in both
Instead of -O you want to use
(I personally think this should have been the default from the beginning, -O should have set the behaviour for all following parameters until changed, but that is too late to change now.)
There is also --remote-header-name (-J) which takes the remote file name from the header instead of the URL, which is what wget does.
--remote-name-all
(I personally think this should have been the default from the beginning, -O should have set the behaviour for all following parameters until changed, but that is too late to change now.)
There is also --remote-header-name (-J) which takes the remote file name from the header instead of the URL, which is what wget does.
> There is also --remote-header-name (-J) which takes the remote file name from the header instead of the URL, which is what wget does.
I don't think that's the case, that behavior is opt-in as indicated in wget's manpage:
> This can currently result in extra round-trips to the server for a "HEAD" request, and is known to suffer from a few bugs, which is why it is not currently enabled by default.
I don't think that's the case, that behavior is opt-in as indicated in wget's manpage:
> This can currently result in extra round-trips to the server for a "HEAD" request, and is known to suffer from a few bugs, which is why it is not currently enabled by default.
Curl does not handle reconnections or use caching like wget, for example, by default. And many other things which includes actually quite many arguments. You can see the list from this project.
Why not just use wget if you need all that? Wget does yet more useful things.
MacOS comes with curl but no wget by default. Sure you can easily install it with homebrew. But it would be nice to have wget functionality by default.
Then you would need two tools instead of one
I've been trying to tell people my hammer is perfect for cutting their hair, they won't listen.
Until... I show them.
They become unresponsive, but their hair is removed.
Job done.
Until... I show them.
They become unresponsive, but their hair is removed.
Job done.
So? The tools are miniscule. Use the right tool for the right job. Shoehorning curl into the role of wget is silly.
BUT BUT BUT - neither wget nor curl let me read my email!
BRB, forking curl to add email client support.
https://programmingphilosophy.org/posts/zawinskis-law/
BRB, forking curl to add email client support.
https://programmingphilosophy.org/posts/zawinskis-law/
Good news then, because curl supports IMAP!
> Shoehorning curl into the role of wget is silly.
How is it silly if the identical functionality is there, but you just need to use more command-line arguments?
How is it silly if the identical functionality is there, but you just need to use more command-line arguments?
The functionality is not identical. Wget is much more high level. You should not need to be an HTTP and curl expert to download a file lol.
I felt weird reading the titles too, because maybe 2-3 years ago I have downloaded with `curl`, so why this?
And also, I am pretty sure `wget` can do it and better too.
As others pointed out, you can do that, you can also set them in .curlrc, or you can write a script if you want to allow for multiple URLs to be downloaded in parallel (not possible with an alias), or now you can just use wcurl :)
Note: wcurl sets a bit more flags than that, it also encodes the whitespaces in the URL and does parallel downloading of multiple URLs.
Note: wcurl sets a bit more flags than that, it also encodes the whitespaces in the URL and does parallel downloading of multiple URLs.
“or now you can just use wcurl :)”
Sadly I can’t since it is dependent upon the util-linux version of getopt which means it fails on bsd and macOS systems. Understandable since it is always available on the specific target the script was written for and it does make life easier.
Sadly I can’t since it is dependent upon the util-linux version of getopt which means it fails on bsd and macOS systems. Understandable since it is always available on the specific target the script was written for and it does make life easier.
argh, we are looking into fixing that now.
I knew getopt was linux-specific but I thoughts the only impact was that the long form argument (--opt) would not work. I turns out it doesn't run at all instead.
We should be able to fix this within the next few days, thank you!
I knew getopt was linux-specific but I thoughts the only impact was that the long form argument (--opt) would not work. I turns out it doesn't run at all instead.
We should be able to fix this within the next few days, thank you!
> We should be able to fix this within the next few days, thank you!
It's fixed now, should work in non-linux environments.
It's fixed now, should work in non-linux environments.
[deleted]
wcurl does a lot more than that, though: parallel downloads, retries, following redirects (and other things, that are all written in the blog post).
There's also some value in providing a ready-to-use wrapper for users that install the curl package. Maybe wcurl will also show up in other distributions after a while, especially since Daniel also likes the idea.
There's also some value in providing a ready-to-use wrapper for users that install the curl package. Maybe wcurl will also show up in other distributions after a while, especially since Daniel also likes the idea.
> parallel downloads, retries, following redirects
There's aria2
There's aria2
I felt the same but upon checking they do more than I expected. I like it.
You often want redirection too, so -L for --location.
Totally agree! That one has burned me a lot. I generally use 'curl -LO' now instead of wget.
[deleted]
I just don't get this, you can't remember two arguments so you invent a whole program.
Personally I never had any problems using curl to download files, nor remembering tar arguments.
Personally I never had any problems using curl to download files, nor remembering tar arguments.
I think that’s pretty ungenerous a take. Wrappers have a proven history of being really valuable contributions to tools (look at GCC and clang), I think making tools easier to use in a great goal.
but you will reach a point where remembering all tiny "tools" like this becomes more work than learning parameters of the actual tool
This happens so often. A developer could just read a man page or learn something deeply by example, but they don't. Instead they struggle for the rest of their lives in ignorance, even going so far as to adopt a hack to avoid having to learn a new thing. They will go far out their way to not learn a new thing, and over time, cause themselves untold misery.
I think there's a bit more nuance than just ignorance. In some ways, I think reinventing the wheel is almost inevitable when we require backwards compatibility but prefer sane defaults. The expectations around what exactly should happen when making an HTTP request have evolved over time, but we can't change the way that `curl` behaves by default with potentially breaking stuff all over the place. In the same way that I don't judge people who use MacOS because it "just works" instead of using Linux like me, I don't think it's fair to treat people who don't want to spend a lot of time learning how to ask a tool to do what another tool will do without needing to spend any extra effort as if they're somehow failing as competent developers.
Except there are about a million ways to do this that don't involve releasing another tool. Everyone I know (including myself) would make an alias, that's entirely what they're for. If you want to get really fancy, you could make it a function in your shell's rc file. Or if you're REALLY zesty, you could even write an entire shell script for it.
To a senior, tools like this make no sense, because they're unnecessary, and contribute to bloat and waste, both for systems, and for the time spent.
To a senior, tools like this make no sense, because they're unnecessary, and contribute to bloat and waste, both for systems, and for the time spent.
Many people cherish simplicity. They don't want to remember arcane spells of args. They want sane/comfy defaults. It has nothing to do with ignorance, they simply refuse to use something your way.
The point of having man pages is that you don't have to remember "arcane spells of args". For huge manpages or manpages in general there is certainly a learning curve on how to read and apply them effectively. But it's also an underrated way of learning about the system you use and deploy your applications on.
You don't listen/understand: man pages are for learning. Just imagine a user refusing to learn and not because of said user's ignorance. You probably didn't invent a new class of programs that do something absolutely new, you just wrote a program with poor design/defaults that require the user to learn how to use said program rather things be intuitive in general.
I can't even get developers to read error messages, let alone documentation. The number of times I've had senior (by title) individuals send me screenshots of stack traces asking for help is too damn high. This field is a joke.
Yes, it happens a lot to me as well. However it baffles me why I would ever want to download and install a tool when I could just read a man page and create an alias for curl and some args. It's a perfectly capable tool that comes with every conceivable distribution.
It is ignorance, because aliases and shell scripts have existed for eons, and this is an overkill solution for a non-issue. I have a bunch of aliases for common flag combinations; I haven't released any of them as entirely separate programs.
[deleted]
man pages vary in complexity/readability, and for occasional use, is probably way more painful
Sorry you're getting downvoted and probably flagged, but I agree with you. The fact that someone made an entirely separate thing for this is mind-boggling, when a simple shell alias solves this problem, and takes literally 60 seconds to set up.
But then again, this is the same industry that thinks shipping 600 MB of Node packages and an entire instance of Chromium is what it takes to use ffmpeg, so, I'm not surprised, just disappointed.
But then again, this is the same industry that thinks shipping 600 MB of Node packages and an entire instance of Chromium is what it takes to use ffmpeg, so, I'm not surprised, just disappointed.
Bah ? OP just added a thin-wrapper, to easy many people's lifes. Why write your own shell alias when it can be setup by default, for free ?