How we're using git-annex to make a backup of the Internet Archive(git-annex.branchable.com)
git-annex.branchable.com
How we're using git-annex to make a backup of the Internet Archive
https://git-annex.branchable.com/design/iabackup/
13 comments
See also: http://ascii.textfiles.com/archives/4609
Could http://ipfs.io/ help with this distributed archive effort?
Maybe? It looks like they're in contact; see Proposals at the bottom of http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK - it says a proposal is in progress.
Join up! This script will walk you through setup.
$ git clone https://github.com/ArchiveTeam/IA.BAK
$ cd IA.BAK
$ ./iabak
I still think that while backup solutions like this are technically very cool, the need for a working backup of the IA right now is so critical -- there is literally no backup, none, at all -- that it is just kind of insane that they're unwilling to consider straining their principles at all and doing something more prosaic in the interim like using tape even when people say "hey, we'll donate all the hardware needed". Nope, tape's too proprietary for them.
This is not the Internet Archive's own project, but rather the idea of Archive Team, an unaffiliated group. The Internet Archive presumably backs up their data and I think they have some older copies in the Netherlands and the new Library of Alexandria in Egypt.
http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
Not completely unaffiliated: The project is the brainchild of Jason Scott, who straddles both.
[deleted]
I'm an Archive Team member. If you'll donate the hardware, I've got a spot for it in my living room.
If you want to talk about low overhead redundancy schemes based on Reed Solomon coding, let me know, we used them to halve disk space requirements in QFS as opposed to HDFS (with no loss in reliability), and it could be stretched arbitrarily far to get at least some redundancy with an arbitrarily small overhead.
We opted not to do any fancy encoding of the data, even though it would make it easier to store (we would need fewer volunteers).
Instead we store individual files whole and unencoded on each volunteer's machine, allowing the volunteer to make use of them as they see fit; no one has to be able to contact other volunteers in order to get useful data.
Of course, since each volunteer's client is downloading files at random, you may not end up with the exact file you wanted (to pick an example at random, perhaps you wanted the fifth reel of the soundex index of the 1880 US Census for the state of Tennessee, but you wanted a collection of Lois Armstrong records from the 1920s instead). This turns out not to be a problem, however, as anyone can use git annex to manually pull down any file in the backup that they want. It gets counted as a copy by the backup system along with all the rest, and increases the redundancy of the backup.
Instead we store individual files whole and unencoded on each volunteer's machine, allowing the volunteer to make use of them as they see fit; no one has to be able to contact other volunteers in order to get useful data.
Of course, since each volunteer's client is downloading files at random, you may not end up with the exact file you wanted (to pick an example at random, perhaps you wanted the fifth reel of the soundex index of the 1880 US Census for the state of Tennessee, but you wanted a collection of Lois Armstrong records from the 1920s instead). This turns out not to be a problem, however, as anyone can use git annex to manually pull down any file in the backup that they want. It gets counted as a copy by the backup system along with all the rest, and increases the redundancy of the backup.
Assuming you can mostly saturate a gigabit port, please contact me. Email in profile via keybase.
Aye, there's the rub of course. I can saturate a gigabit port, but sadly my DSL cannot.
I appreciate the offer though; I'm sure if we really put our heads together we could make it work. Might be better for you to talk to IA though, rather than Archive Team. Pop in one Friday afternoon or send them an email.
Archive Team is a very loose confederation of folk who get angry when a website full of user-created content goes offline. We like to say that our primary virtues are rage, paranoia, and kleptomania, and backing up IA comes under the headings of paranoia and whimsy (an optional virtue) more than anything else.
IA is solid; they're not a startup to be acqui-hired, they're not Yahoo. We have time here for volunteers to step forward, each with a chunk of disk space and a desire to help out. We have time to invent something neat that can grow into a different way of doing things. In fact, we have time to do this in several different ways, all running parallel; I want to work on IPFS next.
I appreciate the offer though; I'm sure if we really put our heads together we could make it work. Might be better for you to talk to IA though, rather than Archive Team. Pop in one Friday afternoon or send them an email.
Archive Team is a very loose confederation of folk who get angry when a website full of user-created content goes offline. We like to say that our primary virtues are rage, paranoia, and kleptomania, and backing up IA comes under the headings of paranoia and whimsy (an optional virtue) more than anything else.
IA is solid; they're not a startup to be acqui-hired, they're not Yahoo. We have time here for volunteers to step forward, each with a chunk of disk space and a desire to help out. We have time to invent something neat that can grow into a different way of doing things. In fact, we have time to do this in several different ways, all running parallel; I want to work on IPFS next.