Show HN: Exatorrent – Self-hostable Torrent client written in Go(github.com)
github.com
Show HN: Exatorrent – Self-hostable Torrent client written in Go
https://github.com/varbhat/exatorrent
111 comments
johnchristopher@localhost ~/Download
$ ./exatorrent-linux-amd64
Address => :5000
Directory => exadir
[INFO] 2021/08/25 22:48:04 Adding Admin user with username "adminuser" and password "adminpassword"
[INFO] 2021/08/25 22:48:05 Torrent Client Created
[INFO] 2021/08/25 22:48:05 Starting server on :5000
[INFO] 2021/08/25 22:48:05 Loaded 20 trackers from https://ngosang.github.io/trackerslist/trackers_best.txt
[INFO] 2021/08/25 22:48:05 Loaded 20 trackers in total , eliminating duplicates
[INFO] 2021/08/25 22:48:05 Added Loaded Trackers to Torrents
Thank you. I hope you like exatorrent
Hmmm, had a look, and the default listen address + default user / password are giving me shivers. @varbhat I'd advise you to change the default listen address from ":5000" to "127.0.0.1:5000" to only allow local connections. And maybe generate a password then for good measure, since you're printing it anyway :-)
I think he intends that it can be used as a seedbox. It can be a command line switch, though.
Tangential, I think adding support of http://webtorrent.io/ by bridging webtorrent peer with bittorent peer would greatly increase/revive the popularity of the protocol
exatorrent supports webtorrent peers already
That's great news! Is it designed to reusable as a library, for example to build a peertube native desktop client?
You can use
https://github.com/anacrolix/torrent
for those cases.
https://github.com/anacrolix/torrent
for those cases.
Didn't know. Savage! Good luck and wish you success!
Let's assume I want to run this on a Hetzner box. And let's assume all my activity is legit torrenting. Say, sharing linux distros. They are still not exactly going to be over the moon with my seedbox running on their servers, due to torrenting stigma.
You can replace 'Hetzner' with any VPS service here, really. What measures can one take to 'hide' torrenting activity?
You can replace 'Hetzner' with any VPS service here, really. What measures can one take to 'hide' torrenting activity?
I have run torrents on several VPS/Server provider (mostly OVH) and never had any issue.
Kimsufi onced nuked my server. Someone was running the usual blanket port scan and my IP was scanned. That was it. Weeks later someone replied to the ticket with an one liner “server hack suspected”. Then radio silence forever.
Good to know, thanks :)
I haven’t hidden my torrenting activity on my Hetzner box for years, and they haven’t kicked me off.
I assume their problem would be mostly with a huge amount of traffic, not torrent per se.
I assume their problem would be mostly with a huge amount of traffic, not torrent per se.
Yep, they don't care so long as you don't cause problems for them.
If only they didn't put 5.14 kernel into their rescue image this summer, but that's another story.
If only they didn't put 5.14 kernel into their rescue image this summer, but that's another story.
Why is kernel 5.14 an issue?
Best option is to force encryption on all bittorrent traffic. Most clients support full protocol encryption (PE or MSE) these days, so forcing it shouldn't be an issue. For libtorrent, you'd set in_enc_policy and out_enc_policy to pe_forced.
Whether a particular client exposes this or not depends. In the case of this project... looks like upstream github.com/anacrolix/torrent does support encrypted connections. Not clear how to force it to use _only_ encrypted connections though.
Whether a particular client exposes this or not depends. In the case of this project... looks like upstream github.com/anacrolix/torrent does support encrypted connections. Not clear how to force it to use _only_ encrypted connections though.
Well, one would find a provider who focuses on seedboxes. Usually these providers - at least the ones who last - put their boxes in a foreign data center and mix it with other traffic.
Better clients can periodically change between random ports and use encryption. I would expect such a client to require a pretty specific search to be discovered. What we can't really protect from is someone watching the global P2P traffic by posturing as another client, but that's a bit more involved than doing a wireshark survey.
[deleted]
I’ve been running private torrents on Hetzner for years and they’ve not kicked me out yet.
Only use private trackers :-)
Maybe something like https://github.com/haugene/docker-transmission-openvpn
Don’t see how this would help, only thing that would help is a tunnel out, like WireGuard to a VPN or less reputable provider.
You can tunnel traffic through NordVPN.
This is for rapidbay but should work for any service: https://github.com/hauxir/rapidbay/wiki/Setting-RapidBay-up-...
This is for rapidbay but should work for any service: https://github.com/hauxir/rapidbay/wiki/Setting-RapidBay-up-...
How much does this diminish performance?
Speed is probably 85-95% of normal speeds. Maybe less if the speed is gigabit. Depends on servers or VPN service.
Ok great, thanks :)
Maybe I shouldn't jinx it... but I run this container on hetzner, with a proprietary vpn service, and have never had any problems for a long long time now
Never had an issue, I encrypt everything fwiw.
What could be interesting is if you were able to add wireguard-go into your project.
It would be nice to have a single binary that can run traffic through a VPN and something I have not see done before.
It would be nice to have a single binary that can run traffic through a VPN and something I have not see done before.
Nice. I will consider.
You can route network of exatorrent through network/VPN .
Also, wireguard implemented in Linux kernel is faster than userspace implementation of wireguard-go . No?
You can route network of exatorrent through network/VPN .
Also, wireguard implemented in Linux kernel is faster than userspace implementation of wireguard-go . No?
> You can route network of exatorrent through network/VPN .
that always comes with the danger of not going through VPN if the connection somehow drops or you've restarted the machine its running on and the vpn started after the torrent client.
and the linux kernel implementation isnt necessarily available if its running in a docker container of some NAS or even on a windows machine.
it would also make it easier to use, allowing more people to utilize the tech.
that always comes with the danger of not going through VPN if the connection somehow drops or you've restarted the machine its running on and the vpn started after the torrent client.
and the linux kernel implementation isnt necessarily available if its running in a docker container of some NAS or even on a windows machine.
it would also make it easier to use, allowing more people to utilize the tech.
Thanks for suggesting me.
I will consider this.
I will consider this.
> that always comes with the danger of not going through VPN if the connection somehow drops or you've restarted the machine its running on and the vpn started after the torrent client.
My current setup uses namespaces to avoid this issue. Basically, one can create a wg interface in the default namespace and then move it to a new namespace and run applications there. The wg interface keeps routing the encrypted traffic through the interfaces in the default namespace, but as there is only the wg interface in the new namespace, there is no risk of leaking any traffic. It also doesn't need a network bridge and doesn't disturb the network configuration of the host.
I use lxc, but there are probably more lightweight options to just start an application in a different network namespace.
See: https://www.wireguard.com/netns/
My current setup uses namespaces to avoid this issue. Basically, one can create a wg interface in the default namespace and then move it to a new namespace and run applications there. The wg interface keeps routing the encrypted traffic through the interfaces in the default namespace, but as there is only the wg interface in the new namespace, there is no risk of leaking any traffic. It also doesn't need a network bridge and doesn't disturb the network configuration of the host.
I use lxc, but there are probably more lightweight options to just start an application in a different network namespace.
See: https://www.wireguard.com/netns/
What a great suggestion! I had debated rolling over to wg as my VPN provides it, however hadn't worked out how to manage the Binding for the torrent applications.
This relieves me having to manage restarts when the VPN connection issues for example a new IP. I currently just allow some scripts to refresh things when the uplink changes.
This relieves me having to manage restarts when the VPN connection issues for example a new IP. I currently just allow some scripts to refresh things when the uplink changes.
So for wireguard-nt and qbittorrent, qbitorrent has the ability to only send traffic on a particular interface (under advanced settings). I set this to be the WG interface and that works as a poor man's killswitch.
Userspace is slower but the idea is to have a single binary that can download torrents over wg. I have not seen anything like this before and it would make for a nice differentiator.
We use wireguard-go in our cli for connecting to vms over a private network. It is slower than wireguard in the kernel, but the convenience is hard to beat.
Here's an overview https://fly.io/blog/ssh-and-user-mode-ip-wireguard/
Here's an overview https://fly.io/blog/ssh-and-user-mode-ip-wireguard/
Have you tried https://github.com/KusakabeSi/wireguard-go-vpp ?
This is actually much more complicated than just using wireguard-go.
That currently just creates an interface, but that is just for raw packets. The kernel is the one that is still handling the IP connections.
To have a process do what you are implying would require an entire userspace TCP/IP stack.
These are pretty rare. Slip4netns is an example of one.
That currently just creates an interface, but that is just for raw packets. The kernel is the one that is still handling the IP connections.
To have a process do what you are implying would require an entire userspace TCP/IP stack.
These are pretty rare. Slip4netns is an example of one.
There's a user space network stack in gvisor, written in Go. And it's already included in wireguard-go.
See also https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ which has already done this for SSH through wireguard-go
See also https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ which has already done this for SSH through wireguard-go
That looks great, the only think that would make it even better for me would be a section to monitor RSS feeds and automatically download torrents from a set of rules into specific directories (preferably with the ability to unpack the download if the original files are compresses/splitted).
But that probably goes beyond the scope of this project.
Very cool nonetheless!
But that probably goes beyond the scope of this project.
Very cool nonetheless!
There are non-"self-hostable" torrent clients?
Yes, you can download torrents with Baidu Wangpan (https://pan.baidu.com/) which cannot be self-hosted.
AWS S3 used to have torrent client (seed-only), which could not be self-hosted.
AWS S3 used to have torrent client (seed-only), which could not be self-hosted.
Seed-only clients that can't be self-hosted, I can understand, because what you care about is not the client but the infrastructure that the client runs on (network, etc).
What's the use-case of Wangpan? I assume it's not "I don't want to torrent illegal things on my home network" because such a service could rat me out just as well.
What's the use-case of Wangpan? I assume it's not "I don't want to torrent illegal things on my home network" because such a service could rat me out just as well.
Baidu Wangpan is a dropbox-like service, that has torrent download as a small feature.
Imagine you want to stream a video file on your mobile, but:
1) The video file is only available via a torrent along with other files you don't want to watch right now, and
2) Your internet connection doesn't have great throughput, and
3) You will watch the file on a mobile device, so can compromise on resolution and compression.
The easiest thing would be to download the torrent on Baidu Wangpan, and then stream the parts you want to your mobile device.
Imagine you want to stream a video file on your mobile, but:
1) The video file is only available via a torrent along with other files you don't want to watch right now, and
2) Your internet connection doesn't have great throughput, and
3) You will watch the file on a mobile device, so can compromise on resolution and compression.
The easiest thing would be to download the torrent on Baidu Wangpan, and then stream the parts you want to your mobile device.
Amazing impressive work!
Was there a specific reason you built this or did you just want to create a showcase?
Was there a specific reason you built this or did you just want to create a showcase?
I built this as a project to learn things. I used to dislike myself for not being productive.
exatorrent helped me to learn new things ( go , databases , typescript , websocket , CI/CD ,etc.).
so,here it is.
exatorrent helped me to learn new things ( go , databases , typescript , websocket , CI/CD ,etc.).
so,here it is.
Really impressive! I have the same issue, its pretty motivating to see this. How long did it take you?
You have done an exceptional job. Congrats on the productivity!
Thank you. I hope you like it.
I used to torrent a lot, but gave up after my ISP adopted CGNAT and I lost the capability to forward ports locally. Do people adapted bittorrent to IPv6, since then, to circunvent this problem, or are we still stuck with IPv4 in this space?
I'm a newby in these topics. I'm pretty sure that bittorrent has worked in ipv6 for a long time now; actually, I find it unintuitive that it would care whether one or the other is used. Also, doesn't bittorrent work fine without port forwarding? I'm pretty sure that I used to run it that way for a long time when my ISP didn't give me a public IP.
It works, but the number of peers and speeds decrease a lot. At least in my experience. I haven't deep knowledge on how the protocol works.
Port forwarding will allow you to receive connections from clients that are themselves behind NATs, and would also allow you to participate in DHT which would allow you to discover clients faster.
Though in ideal cases (ones with high active seeders), that isn't going to affect performance that much. Torrent can also use UDP hole punching, but that can sometimes be finnicky behind CGNATs
Though in ideal cases (ones with high active seeders), that isn't going to affect performance that much. Torrent can also use UDP hole punching, but that can sometimes be finnicky behind CGNATs
Correct. At least one side must have an open port. So if that's you you can peer with everyone instead of only people with open ports.
People torrent just fine behind CGNAT though? I never understood how hole punching works but im pretty sure the protocol has something like that.
I've never had to do any port forwarding on my local router do torrent anything, I'm not sure what you were experiencing but it might have been transient.
Some VPN providers, like Mullvad, offer the option of port-forwarding.
I'm not sure if it would work for torrenting but I can't think of a reason why it wouldn't.
I'm not sure if it would work for torrenting but I can't think of a reason why it wouldn't.
> Some VPN providers, like Mullvad, offer the option of port-forwarding.
The thing is Mullvad offers dynamic Port-Forwarding only. They get to decide what port number they can grant for users and users cannot request for a specific port. There are VPNs that have static port-forwarding which is rare. There are less than 5 VPN providers that have static port-forwarding from what I remember and users can decide what port number they want (unless specific port number is taken prior). It would be a pain if someone have a range of applications that already using specific port prior and would have to change them to use the VPN's defined port number depending on how much applications they have to change.
The thing is Mullvad offers dynamic Port-Forwarding only. They get to decide what port number they can grant for users and users cannot request for a specific port. There are VPNs that have static port-forwarding which is rare. There are less than 5 VPN providers that have static port-forwarding from what I remember and users can decide what port number they want (unless specific port number is taken prior). It would be a pain if someone have a range of applications that already using specific port prior and would have to change them to use the VPN's defined port number depending on how much applications they have to change.
Not dynamic. Randomised. Once created, it doesn't change/
I have yet to see a torrent client that doesn't let you specify the public port it's listening on.
I have yet to see a torrent client that doesn't let you specify the public port it's listening on.
Impressive work. Did you implement the bittorrent protocol from scratch? which BEP specifically?
https://www.bittorrent.org/beps/bep_0000.html
https://www.bittorrent.org/beps/bep_0000.html
No , I used https://github.com/anacrolix/torrent to build exatorrent
Here is some advice I hope you find useful: I wouldn't hire you based on this because all I see is a shit ton of 3rd party libraries stitched together. The exact opposite of what I would like to see in a good engineering candidate.
So you may want to rethink your approach in terms of putting forth qualities that speak to your engineering prowess.
I also wouldn't personally use this software as it has so many 3rd party dependencies that one would find it hard to audit or make any robustness / security guarantees.
So you may want to rethink your approach in terms of putting forth qualities that speak to your engineering prowess.
I also wouldn't personally use this software as it has so many 3rd party dependencies that one would find it hard to audit or make any robustness / security guarantees.
Thank you .
Please Open Issue if you find any Bad or Buggy Code . I would correct it.
Also , exatorrent doesn't use much 3rd party libraries like you think
Please Open Issue if you find any Bad or Buggy Code . I would correct it.
Also , exatorrent doesn't use much 3rd party libraries like you think
Hey just a nitpick here but please don’t use spaces before punctuation.
For example: thank you. Also, exa…
For example: thank you. Also, exa…
And I cherish the fact that all software is build on shoulder of giant. I see "leverage" instead of shit ton of 3rd party libraries. But of course, maybe there are Linus Torvalds with a Putnam prize under their belt, what do I know.
This is not constructive feedback for a Show HN thread.
FFS. He's just looking for an internship.
He's demonstrating competence in writing go (and yes, selecting 3rd party libraries and stitching them together is often a skill you need in the Real World), using docker, understanding web apps with Svelte + typescript, etc. Scratching his own itch, literally the original hacker ethos.
Nice work, OP. Good luck in your search.
He's demonstrating competence in writing go (and yes, selecting 3rd party libraries and stitching them together is often a skill you need in the Real World), using docker, understanding web apps with Svelte + typescript, etc. Scratching his own itch, literally the original hacker ethos.
Nice work, OP. Good luck in your search.
You should add a NOTICE file to your repo, stating the fact that you have used portion of `anacrolix/torrent` OSS software:
https://github.com/anacrolix/torrent/blob/master/LICENSE
https://github.com/anacrolix/torrent/blob/master/LICENSE
Very cool project. I’d love to see more Go projects pop up because they’re absolutely great to learn Go. Have you had a chance to benchmark the maximal throughput? (I.e. 100MB/s on a 1Gbps max?)
I’ve been using Synapse (https://synapse-bt.org/synapse/, a torrent client written in Rust) recently and so far it was the only client fully saturating a 10Gbps NIC.
I’ve been using Synapse (https://synapse-bt.org/synapse/, a torrent client written in Rust) recently and so far it was the only client fully saturating a 10Gbps NIC.
I didn't benchmark the maximal throughput although I may in future .
Looks good. Congrats on this release! How is memory usage compared to e.g. Transmission?
Memory use is pretty low. It's around 5-6 MB initially and will increase as you add more torrents.
Also you can measure Memory Usage and other Stats in real time in Web Client of exatorrent(also there is API for it)
Also you can measure Memory Usage and other Stats in real time in Web Client of exatorrent(also there is API for it)
Please create Windows binary.
Is it possible to control it via web interface ?
Is it possible to control it via web interface ?
Yes, it is possible to control it via web interface.
I wanted to create Windows executable. I tried cross compiling using mingw and asked my friend to test it but it reported some error namely libstdc++ error and similar ones. So,i focussed first on releasing software first and look into this issue later.
I also wanted to release Binary for MacOS too ,so went to setup osxcross cross compiler but couldn't setup it properly too. I wanted to work on this after release.
So, Help is appreciated regarding this matter.
I wanted to create Windows executable. I tried cross compiling using mingw and asked my friend to test it but it reported some error namely libstdc++ error and similar ones. So,i focussed first on releasing software first and look into this issue later.
I also wanted to release Binary for MacOS too ,so went to setup osxcross cross compiler but couldn't setup it properly too. I wanted to work on this after release.
So, Help is appreciated regarding this matter.
[deleted]
Which legit activities are all you using torrent for?
Funnily enough, everything possible.
Everytime there is a torrent alternative to the direct download, I will choose torrent, always.
But, thanks to pirating stigma, torrenting hasn't proliferated anywhere near as much as it should.
I seed all linux distros that I use. Bandwidth is expensive when centralized. It pains me to think of dollars going toward paying for egress data rather than developer work for open source projects. Please download them via torrent instead of http, and then please seed!
Isn't most of the bandwidth spent on packages? Even immediately after the initial install you probably have gigabytes of updates + new packages.
I often wonder - who pays for pip packages that millions of server apps and CI/CD pipelines pull? There is some caching but still it’s gotta be enormous.
The official PyPI packkages are hosted on Fastly. I believe they donate the infrastructure costs through the Python Software Foundation, as the bandwidth is over 300TB a day, and would cost about 1.8 million dollars a month.
Source: https://dustingram.com/articles/2021/04/14/powering-the-pyth...
Source: https://dustingram.com/articles/2021/04/14/powering-the-pyth...
interesting article. where is the 300 TB number coming from?
> The “maximum” here is per-day, so this means that at the peak (which was the day I made these graphs) PyPI served nearly 900 terabytes over more than 2 billion requests per day.
> The “maximum” here is per-day, so this means that at the peak (which was the day I made these graphs) PyPI served nearly 900 terabytes over more than 2 billion requests per day.
This tweet from Dustin Ingram: https://twitter.com/di_codes/status/1235707819955032069
I think this number is dated but its what I stumbled across.
I think this number is dated but its what I stumbled across.
might be. either way the numbers are impressive although i am afraid this massive bandwidth is necessary because of the docker's and ci/cd's that don't use caches...
Yeah, it still blows my mind that nobody seems to have solved putting packages in torrents.
Apt-torrent - https://directory.fsf.org/wiki/Apt-torrent
I haven't used this a long time, but it used to be faster than mirrors where I was.
I haven't used this a long time, but it used to be faster than mirrors where I was.
Internet Archive collections are available as web seed torrents and download much faster than over HTTP because multiple servers are used concurrently.
Possible legit usage: https://academictorrents.com/
For seeding Big Buck Bunny of course. And Linux distros too. That's all :)
Downloading Have I Been Pwned's list of pwned passwords (https://haveibeenpwned.com/Passwords)
Linux distributions, downloads that are very popular, but where the servers can't keep up with it (i.e. small indie games people hosted the download links for themselves)
While I've yet to do it, I've heard of many people using it for peer to peer file transfer. I tend to transfer files between devices pretty often and have never found a perfect solution. I'd like to do the same soon enough once I figure out exactly how to do it.
Create a torrent for what you're trying to share, add a few open tracker announce urls[1] to it, then load it in you client. Then, anyone else who has a torrent client can download it with a magnet link[2].
One caveat is that either your client and router both have to support UPnP, or you have to manually open up a port for UDP traffic in your router and forward that to the machine you're running the client on.
I also use transmission-create to make torrents from the cli, but the UI of your client will have an option.
I use magnets to transfer all large files - although if there's anything confidential in the file, encrypt it first. The only thing keeping people from plucking your files out of the ether is a lack of interest.
-----
[1] Such as https://newtrackon.com/list
[2] Which are magnet:?xt=urn:btih:[insert torrent hash here]
Torrent hashes can be found in the UI of your client, or at the beginning of the actual .torrent file.
One caveat is that either your client and router both have to support UPnP, or you have to manually open up a port for UDP traffic in your router and forward that to the machine you're running the client on.
I also use transmission-create to make torrents from the cli, but the UI of your client will have an option.
I use magnets to transfer all large files - although if there's anything confidential in the file, encrypt it first. The only thing keeping people from plucking your files out of the ether is a lack of interest.
-----
[1] Such as https://newtrackon.com/list
[2] Which are magnet:?xt=urn:btih:[insert torrent hash here]
Torrent hashes can be found in the UI of your client, or at the beginning of the actual .torrent file.
I provide some large binaries and data dumps of my own projects as torrent as well.
PeerTube uses it, so as you watch videos you start sharing as a peer with others to reduce load on the host.
https://joinpeertube.org/
https://joinpeertube.org/
Usually downloading the contents of physical media that I have bought. It can be because I don't have anything to read it, or because I'm not at home.
Amusingly, Humble Bundle offers things like ebook purchases as torrent downloads.
Unironically, linux isos. Mostly debian and raspbian based images.
raspbian download
I have worked on this for past few months . exatorrent is BitTorrent Client written in Go. It can be run locally or be hosted in Remote Server with good resources to use that Server as SeedBox. It is Single Completely Statically Linked Binary with Zero External Dependencies .
exatorrent is simple yet feature-rich . It is lightweight and light on resources. It comes with Beautiful Responsive Web Client written in Svelte and Typescript , but thanks to documented WebSocket API of exatorrent , you can also write your own client or integrate with other services if you want to.
exatorrent can operate in Optional Multi-User Mode administrated by admin user/s but it can successfully be used by Single-User as well. Torrented Files are stored in local disk of device where it's run, which are then retrievable or streamable via HTTP.
I hope you like it . It's Open Source .
Also HN , I am looking for Internship . If you're interested , please email me