HackerTrans
TopNewTrendsCommentsPastAskShowJobs

sixtyfourbits

no profile record

Submissions

Pirate Sites Exploit 'Interplanetary File System' Gateways, Publishers Warn

torrentfreak.com
3 points·by sixtyfourbits·3 jaar geleden·0 comments

Dataset of 250K Books and 1.5M Scholarly Papers with Extracted Text Layers

old.reddit.com
3 points·by sixtyfourbits·3 jaar geleden·0 comments

The Dream Was Universal Access to Knowledge. The Result Was a Fiasco

nytimes.com
5 points·by sixtyfourbits·3 jaar geleden·2 comments

Publishers, Internet Archive agree to streamline digital book-lending case

reuters.com
7 points·by sixtyfourbits·3 jaar geleden·0 comments

Gradual removal of port forwarding from the IVPN service

ivpn.net
8 points·by sixtyfourbits·3 jaar geleden·1 comments

Standard Template Construct: Store and Search the Entirety of Human Knowledge

old.reddit.com
1 points·by sixtyfourbits·3 jaar geleden·0 comments

Telegram Copyright Takedowns Breed a Hydra of Z-Library Bots

torrentfreak.com
4 points·by sixtyfourbits·4 jaar geleden·0 comments

comments

sixtyfourbits
·2 jaar geleden·discuss
https://annas-archive.org/datasets/ia
sixtyfourbits
·3 jaar geleden·discuss
[flagged]
sixtyfourbits
·4 jaar geleden·discuss
IPFS provides the same privacy guarantees as bittorrent (i.e. none).

Use a VPN.
sixtyfourbits
·4 jaar geleden·discuss
https://gizmodo.com/amazon-secretly-removes-1984-from-the-ki...
sixtyfourbits
·4 jaar geleden·discuss
Fortunately someone had the foresight to make a backup:

https://news.ycombinator.com/item?id=32972923

Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.

Hopefully this will encourage more people to go back to the original libgen. I suspect Z-library's popularity was because of a better interface and larger collection, but I think lots of people didn't realize libgen offers all its books/papers for free without limits.
sixtyfourbits
·4 jaar geleden·discuss
It actually doesn't work so well, unfortunately.

Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this.

Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient.

Compare this to bittorrent:

- First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent.

- Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it.

On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state.

Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.