Ask HN: Why don't more software projects use BitTorrent for downloads?
92 comments
It's not really worth it for small downloads. I looked at one of my projects and it is an 880kb download.
If you are talking a gigabyte or more there is a big advantage in terms of (1) a completely reliable download, and (2) being able to split up the download into multiple streams. At home I have two ADSL lines and a load balancer and with bittorrent I get double-speed downloads.
If you are talking a gigabyte or more there is a big advantage in terms of (1) a completely reliable download, and (2) being able to split up the download into multiple streams. At home I have two ADSL lines and a load balancer and with bittorrent I get double-speed downloads.
Why don't large companies use it for updates? I have a lot of trouble downloading the 10+ gigabyte IOS downloads on DSL for some reason but torrenting works fine for larger files and stopping and resuming does not break torrenting unlike Apple's update process.
Some have. Blizzard used a variant of BitTorrent about 10 - 15 years ago to distribute the multi-gigabyte patches for their online games. Peak demand was very high with hundreds of thousands of users all on the same day.
It was built into the game installer / launcher. I think they've switched away from that since. I remember it causing some consternation at the time given some people pay by the byte for network service and felt it was unfair.
It was built into the game installer / launcher. I think they've switched away from that since. I remember it causing some consternation at the time given some people pay by the byte for network service and felt it was unfair.
iirc Windows does have a way that computers on a shared network can share the update with each other.
https://support.microsoft.com/en-us/windows/windows-update-d...
https://support.microsoft.com/en-us/windows/windows-update-d...
I don't know how they do it but when I download from Steam there are multiple streams and I get about 200% of the throughput of a single ADSL.
I don't know if this would be of interest to you, but https://lancache.net/ is really fun to setup. I assigned an old (unreliable) 500gb SSD to a Linux VM and run Lancache on it. A couple tweaks to Pi Hole and every system in your house will use with no configuration (and fallback to Steam if it's down).
If you only have a single gaming system it won't matter but if you have a few it's fun to watch the second one get 100MB/s Steam downloads in a house with 20MB/s internet.
If you only have a single gaming system it won't matter but if you have a few it's fun to watch the second one get 100MB/s Steam downloads in a house with 20MB/s internet.
Steam downloads are plain HTTP in separate file chunks. Easy to cache and load balance across connections.
They are also slow as dirt.
Taxes killed p2p...
https://youtu.be/5-sF7N1bfv0
https://youtu.be/5-sF7N1bfv0
Or more clearly stated, tax havens and offshoring revenue killed p2p.
Even without the tax havens, the jurisdictional issues seem killer -- if I pay for software in Kansas and download blocks from Texas, California, Puerto Rico and Canada, who all do I owe sales tax to?
Musescore is planning to use P2P sharing to download sample libraries (>1gb) and updates in version 4 (coming in the next few months).
Its redundancy. Just because the download is small, doesnt mean your webserver providing the downloads doesnt go down.
Sure, but if the webserver providing the downloads goes down, usually so does the website providing the torrent link.
Theres other ways if you can control the firewall ports, ie keep one open and then that could also be a website with links. It depends on how much extra thats built into an app.
Magnet link would still show up in the search engine cache.
Facebook reportedly was using Bittorrent to distribute their own application across their servers. I remember outreach presentations where they mentioned this over 10 years ago, and this article seems to have some corroborating details: https://arstechnica.com/information-technology/2012/04/exclu...
GitHub/GitLab releases makes it easy, free, and has virtually no overhead to distribute software to millions of users globally. Setting up torrents and making sure they stay seeded costs time and money, and torrents are less convenient for most users.
Those aren't necessarily mutually exclusive though. The Git hosts could also provide their releases by seeding them as an option.
We've had a few users complain that they had issues downloading large files from Github -- either S3 was blocked alltogether (China) or just very slow (Australia). That was a couple of years ago, don't know what the situation is like today.
I think offering BT as an alternative would be a good idea, especially for downloads > 100MB.
I think offering BT as an alternative would be a good idea, especially for downloads > 100MB.
Yep came to say the same, because a lot of software is already hosted in a lower friction way by GitHub and similar. So unless you have a need/want to host it yourself and maybe benefit from the optimizations then why bother?
You can add the web URL seed to the torrent, pretty cool feature.
Unfortunately web browsers don't support it, so it requires an external client to download.
There is WebTorrent, which uses WebRTC. instant.io uses it for file sharing, for example.
I got the impression that WebTorrent users can't download from BitTorrent users and vice versa, is that true?
Last I checked webtorrent users can download from webtorrent users. So they are isolated like that.
If I remember correctly, there is a server bridge that enables communication between webtorrent and normal bittorrent clients.
Why would you use BitTorrent to distribute files P2P if you're just going to place a server between the peers? Wouldn't it be better to just have the server distribute the files in that case?
It's used to bridge clients until more clients start supporting webtorrent directly. libtorrent (which for example deluge and qbittorrent use) already supports webtorrent (not sure which ones turn it on by default though).
The goal is basically to not have to use the bridge in the future.
Also one plus with a bridge compared to just downloading directly is that if a file becomes popular your server does not need to shoulder the full load since the webtorrent swarm can handle most of the load.
The goal is basically to not have to use the bridge in the future.
Also one plus with a bridge compared to just downloading directly is that if a file becomes popular your server does not need to shoulder the full load since the webtorrent swarm can handle most of the load.
Got a link to that?
>Unfortunately web browsers don't support it, so it requires an external client to download.
I guess I'm old, but it used to be common to - gasp! - install multiple applications. Now, the browser is basically everything.
I guess I'm old, but it used to be common to - gasp! - install multiple applications. Now, the browser is basically everything.
There's no reason to be condescending. Adding download friction for $0 of benefit for a developer makes no sense. I have had a BitTorrent client installed for more than 20 years, and I use it perhaps once a year these days (and only to download old video games that I can't get anywhere else).
It's been so long since I used a torrent that I don't know what program I'd use. I last used mutorrent but read they were putting crypto mining code in their client.
qBittorrent is cross-platform, open-source and malware-free. µTorrent has been not recommended for a while.
rtorrent or transmission are good CLI and GUI open source clients.
To be clear, I'm old too, vastly prefer applications rather than web browsers.
That and blocked on many firewalls, especially corporate ones.
Also, Opera used to support torrent downloads.
Also, Opera used to support torrent downloads.
The Brave browser has a BitTorrent client built in, though I've never used it. Not a major player in the browser market, though.
Brave support that native.
Try it.
Try it.
There's a ton of free HTTP mirroring available for open source; BitTorrent really has no advantage. Bandwidth is so cheap in general that BitTorrent is not needed.
Without Browser Support BT Requires additional client.
>which I imagine could incur substantial hosting costs over time.
And bandwidth is extremely cheap. ( Look at Cloudflare ) Lots of sponsorship and help for Open Source project or if you want to host it on Github for free. Bandwidth is also getting cheaper. May be there is a floor somewhere, but we dont seems to have that in sight.
>which I imagine could incur substantial hosting costs over time.
And bandwidth is extremely cheap. ( Look at Cloudflare ) Lots of sponsorship and help for Open Source project or if you want to host it on Github for free. Bandwidth is also getting cheaper. May be there is a floor somewhere, but we dont seems to have that in sight.
Many ISPs give customers much better download speeds than upload speeds. I remember personally having problems using bittorrent and seeding too much destroying the network for all using it.
I'm pretty sure OSS gets a free ride on mirrors, because of peering agreements: if traffic between networks is equal, nobody has a case that they're the stronger party on negotiations, so no money changes hands. If there's an asymmetry, theres' an argument that one side has more demand or need for peering and use that as leverage to demand payment. So smaller ISPs that want to balance out a lopsided consumer traffic, will set up mirrors to protect themselves from this.
Since OSS is free to redistribute, a loose federation of mirrors has set up to better match ISPs who want to send more bytes with projects that need to send a lot of bytes.
> Are there any big downsides to distributing torrents as opposed to traditional FTP/HTTP
Analytics is the big reason Mozilla shut down their community FTP mirrors, and bit torrent doesn't really solve that portion.
Since OSS is free to redistribute, a loose federation of mirrors has set up to better match ISPs who want to send more bytes with projects that need to send a lot of bytes.
> Are there any big downsides to distributing torrents as opposed to traditional FTP/HTTP
Analytics is the big reason Mozilla shut down their community FTP mirrors, and bit torrent doesn't really solve that portion.
The best downside that I've heard, at least for doing this automatically, is that not all users have cheap/available upload bandwidth. As long as you're able to opt in as a seeder, it's great.
For the least friction, you could make the download page use Webtorrent, which is a BitTorrent bridge using WebRTC datachannels.
When distributing over torrent, you need at least one seeder with a good connection. You obviously. So you still need to host a server with your package on it, and configure a torrent client to seed it.
I know a lot of HTTP/FTP storage space providers, or I can just put my binary on Github (Github Pages, or Release asset), or host it on a CDN like netlify. I think there is more solutions than real numbers between 0 and 1.
But I don't know a single "torrent seedbox" provider. Setting it up yourself is not complicated, but since it's DIY, this alone explains why it's not more wide-spread.
I know a lot of HTTP/FTP storage space providers, or I can just put my binary on Github (Github Pages, or Release asset), or host it on a CDN like netlify. I think there is more solutions than real numbers between 0 and 1.
But I don't know a single "torrent seedbox" provider. Setting it up yourself is not complicated, but since it's DIY, this alone explains why it's not more wide-spread.
There is a way that you can add a URL to a torrent file as a seed location, then you always have a backup seed for when there are no torrent seeds.
https://en.wikipedia.org/wiki/BitTorrent_%28protocol%29#Web_...
https://en.wikipedia.org/wiki/BitTorrent_%28protocol%29#Web_...
You don't need a seedbox. You just make the torrent file reference the normal HTTP download.
Can't you use S3 to seed a torrent?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec...
As mentioned, S3's torrent server is no longer a thing.
When it worked, it also was never that great. The WebSeed AWS ran was slow, the tracker was always problematic as well.
When it worked, it also was never that great. The WebSeed AWS ran was slow, the tracker was always problematic as well.
They seem to have deprecated support [1]
[1] https://news.ycombinator.com/item?id=27524549
[1] https://news.ycombinator.com/item?id=27524549
This is something I've wondered specifically for Musescore 4. They're releasing their Orchestral library as a separate 7.5gb download, but instead of using bit-torrent, they're making you download a download manager. Seems so bizarre.
A lot of Linux based OSes also have BitTorrent downloads.
Blizzard did this for a while for their WoW updates [0]. Not sure if they still do though.
[0] https://wowpedia.fandom.com/wiki/Blizzard_Downloader
[0] https://wowpedia.fandom.com/wiki/Blizzard_Downloader
A lot of ISPs nuke anything that looks like a torrent and send angry messages to their customers.
Is that actually true? Because I thought it was "content" owners hunting for torrents containing their stuff and then sending DMCA complaints (or local equivalent) to the ISPs of everyone in that particular swarm.
Are there really ISPs that just blanket block all torrents? I've never used one that does.
Are there really ISPs that just blanket block all torrents? I've never used one that does.
There's some local to me that don't necessarily block torrents, but will get upset.
Mostly just means I set up a VPN through Linode or whatever.
Mostly just means I set up a VPN through Linode or whatever.
And some corporate environments might be ok with non-business uses of Internet data, but uncomfortable talks will be had if they detect BitTorrent.
Bittorrent clients for windows and mac are generally malware infested pits of adware. Wherever i try to be virtuous and use one, i have to clean whatever i installed. I am sure i am not the only one.
Why not build your own install using the bittorrent libraries? The security model is more complicated, for one. If i was an architect and proposed that, i would have to understand the attack vectors and present why they are than the well known solution. And taking control of the software update channel is a massive risk.
Second, i have less control of the user experience. With http, i can pay for the right amount of bandwidth, or time on a cdn. If i implement bittorrent, i still have to have buy capacity, i just have a more complicated model for how much to buy.
Suggested updates can be spread over time - and need to be, for canary purposes. I think Android often pushes an app update over 4 or 5 days, by default? Steady state infra capacity, or even better, low priority which and be interrupted, is cheap.
Given the complexity and business risk (people can't download our software! Our binary got hijacked! Two code paths to test?) And the inexpensive nature of mirrors, and the competence of cdns, there are rarely causes where it would make sense.
Why not build your own install using the bittorrent libraries? The security model is more complicated, for one. If i was an architect and proposed that, i would have to understand the attack vectors and present why they are than the well known solution. And taking control of the software update channel is a massive risk.
Second, i have less control of the user experience. With http, i can pay for the right amount of bandwidth, or time on a cdn. If i implement bittorrent, i still have to have buy capacity, i just have a more complicated model for how much to buy.
Suggested updates can be spread over time - and need to be, for canary purposes. I think Android often pushes an app update over 4 or 5 days, by default? Steady state infra capacity, or even better, low priority which and be interrupted, is cheap.
Given the complexity and business risk (people can't download our software! Our binary got hijacked! Two code paths to test?) And the inexpensive nature of mirrors, and the competence of cdns, there are rarely causes where it would make sense.
Linux distros/archive.org and other legit users of BitTorrent tend to use HTTP seeds [1], so you can still calculate for bandwidth costs.
Commercial software like World of Warcraft used BitTorrent in their updater for years [2].
1: https://www.bittorrent.org/beps/bep_0019.html
2: https://wowpedia.fandom.com/wiki/Blizzard_Downloader
Commercial software like World of Warcraft used BitTorrent in their updater for years [2].
1: https://www.bittorrent.org/beps/bep_0019.html
2: https://wowpedia.fandom.com/wiki/Blizzard_Downloader
Deluge, qBittorrent, Transmission, oh my!
Blizzard has used BT for binary downloads for years. None of your fears came to pass.
Raspberry pi OS has both.
Regarding hosting, I am not exactly sure that it changes anything. You still have to seed the torrent, so the file needs to be hosted somewhere. What is reduced is the bandwidth, but you rarely pay for bandwidth.
I do like the torrent option, but it’s a little more work for the person who downloads.
Regarding hosting, I am not exactly sure that it changes anything. You still have to seed the torrent, so the file needs to be hosted somewhere. What is reduced is the bandwidth, but you rarely pay for bandwidth.
I do like the torrent option, but it’s a little more work for the person who downloads.
Because I dislike it so much as a user that I never got into it and never would have thought to propose it.
In Germany, before Fibre (and lol who has that), upload speeds for DSL and Cable have usually been only 10-20% of download speeds, and I know a lot of people who have stayed away from it for that reason alone. Yes, you can limit it, but it kinda kills some of the whole thing. Also the hassle with port forwarding etc.pp. I think I didn't have a client installed on any machine for the last 10 years.
In Germany, before Fibre (and lol who has that), upload speeds for DSL and Cable have usually been only 10-20% of download speeds, and I know a lot of people who have stayed away from it for that reason alone. Yes, you can limit it, but it kinda kills some of the whole thing. Also the hassle with port forwarding etc.pp. I think I didn't have a client installed on any machine for the last 10 years.
> upload speeds for DSL and Cable have usually been only 10-20% of download speeds
Why or how is that an issue? Anybody that contributes to the seeding, however small, matters. Because files are cut into a lot of pieces, there is no competition for the "faster upload speed".
> Also the hassle with port forwarding
That, I can understand, but it's fairly easy to open a port. And if your router supports it you can enable UPnP.
I am the exact opposite of you: as I user, I absolutely love it and it has to be one of my favorite piece of technology.
Why or how is that an issue? Anybody that contributes to the seeding, however small, matters. Because files are cut into a lot of pieces, there is no competition for the "faster upload speed".
> Also the hassle with port forwarding
That, I can understand, but it's fairly easy to open a port. And if your router supports it you can enable UPnP.
I am the exact opposite of you: as I user, I absolutely love it and it has to be one of my favorite piece of technology.
> Why or how is that an issue? Anybody that contributes to the seeding, however small, matters. Because files are cut into a lot of pieces, there is no competition for the "faster upload speed".
If it's a localized thing, for example a game client in your language, and everyone has shitty upload speed, the critical mass would need to be 10x as big so that not everyone has a shitty download rate. Maybe a few single cases soured me on the experience.
> port forwarding.
it's easy if you do 'port X to machine Y' but like in the above example, try 3 computers in the network who need the same 20GB download. it sucks. (esp if it's not /a file/ you can then locally distribute, but it's a game company's launcher)
If it's a localized thing, for example a game client in your language, and everyone has shitty upload speed, the critical mass would need to be 10x as big so that not everyone has a shitty download rate. Maybe a few single cases soured me on the experience.
> port forwarding.
it's easy if you do 'port X to machine Y' but like in the above example, try 3 computers in the network who need the same 20GB download. it sucks. (esp if it's not /a file/ you can then locally distribute, but it's a game company's launcher)
> critical mass would need to be 10x as big so that not everyone has a shitty download rate
Yes, but those who are downloading have - in the best case scenario - a better bandwidth and can therefore seed even more than before. From my experience, the scenario where nobody is seeding anymore - which wouldn't happen in this case - is much worse than the scenario where everybody has a slow upload speed. The more people download, the more they seed. Sure, there's a bottleneck at the start of the life of the torrent (essentially the creator of the torrent), but after it really evens out.
> it's easy if you do 'port X to machine Y' but like in the above example, try 3 computers in the network who need the same 20GB download
I'm sorry, I don't think I understood correctly. I'm going to answer as I understood: in that case, wouldn't you just attribute different ports to your local devices?
Yes, but those who are downloading have - in the best case scenario - a better bandwidth and can therefore seed even more than before. From my experience, the scenario where nobody is seeding anymore - which wouldn't happen in this case - is much worse than the scenario where everybody has a slow upload speed. The more people download, the more they seed. Sure, there's a bottleneck at the start of the life of the torrent (essentially the creator of the torrent), but after it really evens out.
> it's easy if you do 'port X to machine Y' but like in the above example, try 3 computers in the network who need the same 20GB download
I'm sorry, I don't think I understood correctly. I'm going to answer as I understood: in that case, wouldn't you just attribute different ports to your local devices?
A) If you don't use the cloud, bandwidth is so dirt cheap that nobody cares.
B) Some ISPs will throttle you heavily if you upload too much.
C) Some users still have to pay fees for upload traffic.
B) Some ISPs will throttle you heavily if you upload too much.
C) Some users still have to pay fees for upload traffic.
There are so many free CDNs and mirrors, especially for hosting open source.
Also most open source projects aren't particularly bandwidth hungry, why bother?
I suspect a big part of the reason is "more work for very little gain" since most users will just download over HTTP given the choice.
I'd love to see some stats from somewhere like Internet Archive that offers both, it'd be great to be proven wrong, but I suspect a number basically 0% after rounding of their data is downloaded via peer-to-peer BitTorrent traffic.
I'd love to see some stats from somewhere like Internet Archive that offers both, it'd be great to be proven wrong, but I suspect a number basically 0% after rounding of their data is downloaded via peer-to-peer BitTorrent traffic.
Twitter using BitTorrent on the Backend (2010):
>The project uses BitTorrent technology and is called Murder
https://www.bittorrent.com/blog/2010/02/09/twitter-using-bit...
>The project uses BitTorrent technology and is called Murder
https://www.bittorrent.com/blog/2010/02/09/twitter-using-bit...
A few companies I've worked at have used torrents to roll out new service binaries (multi-gb tarballs of Java jars) and periodic index deployments using torrents.
The biggest advantage is just diffusing a thundering heard without increasing latency considerably.
Shout-out to tTorrent.
The biggest advantage is just diffusing a thundering heard without increasing latency considerably.
Shout-out to tTorrent.
Maybe not for individual software projects, but I would like to see something like this for installing and updating packages via the OS's package manager. Given the fact that package repositories have mirror servers, it should definitely be possible.
Sounds like something that would have significant security impact, wouldn't it?
BT is resistant to content spoofing. So unless the users download arbitrary .torrent files from untrusted sources, it will be hard to inject malware this way. You'd need to find hash collisions for the contents of the torrent. It would have some privacy implications, as you end up announcing "I'm using this OS!" to your peers (well, it's a high probability that you are at least).
The same's true for ordinary package managers, but they've come with CVE's that allowed circumventing checksum validation [0]. Software mirror security doesn't seem to me (non-expert) a trivial or solved problem.
[0] https://news.ycombinator.com/item?id=8330386
[0] https://news.ycombinator.com/item?id=8330386
Blocked or otherwise impeded in some places: eg, unable to request peers from a tracker.
Spotify used to use BitTorrent, or something closely related.
Back in the early days, colleagues in my office would be falsely accused by IT of file sharing, when the only crime they were really committing was listening to Jack Johnson.
Back in the early days, colleagues in my office would be falsely accused by IT of file sharing, when the only crime they were really committing was listening to Jack Johnson.
If that was the very early days they were quite possibly sharing unlicensed tracks: https://torrentfreak.com/spotifys-beta-used-pirate-mp3-files...
Until everyone has synchronous internet it won’t need to be a thing. I’ve pointed out many times that between BitTorrent, tor, and bitcoin we’ve already done alleged web3, which is what we used to call the Dark Web.
Plus a whole number of other distributed projects over the years. Everything web3 is just hyped ideas that are already out there, in some cases for 10+ years.
That's a good point, you're right. I hadn't thought of it this way. What we want already exists, we just have to use it.
Windows 10 claims it distributes updates in a peer-to-peer manner. Or at least it used to. Does that still happen?
Delivery Optimisation? Yes, that's still a thing.
It doesn't have to be internet based, you can turn it down to just peers on your local network, or disable it entirely (if you have WSUS infrastructure and no need for it).
It doesn't have to be internet based, you can turn it down to just peers on your local network, or disable it entirely (if you have WSUS infrastructure and no need for it).
How is it better than HTTPS+CDNs both performance wise and implementation/maintainance complexity wise?
Valve hired Bram Cohen back in 2004, and AFAIK it is/was core to Steam's distribution system.
Why don't more software projects do this? Are there any big downsides to distributing torrents as opposed to traditional FTP/HTTP downloads?