Ask HN: What is your personal photo/video storage and archival plan?
27 comments
I've personally learned the following lessons the hard way:
Lesson 1: "RAID isn't a backup." If all you do is copy your files to a NAS, they aren't safe. If you're NAS is taking snapshots onto a different physical spindle, that's good, but it's better to have your NAS backed up to a physically separate device, in case of cryptolocker attacks or hardware failures.
Lesson 2: if it's not automatic, it won't get done. Set up Resilio Sync or SyncThing or your NAS-specific backup software on your phone to pull photos and videos off and back then up automatically. Set up a cron job on your NAS to backup.
Lesson 3: consider an off-site backup. Backblaze has a great service, and so do other cloud services.
Lesson 4: don't use a digital asset manager that requires proprietary hardware or messes with your original files. Google photos, for example, is free, but garbles much of the metadata tags in your files, and downsamples your videos into a blurry mess (even if you use original quality). Failed photo startups are astoundingly prevalent. Use something that's been designed to continue even if the company doesn't.
Amusingly enough, I'm working on a photo startup (spoiler: it satisfies what I wanted from lesson 4!). It's currently in closed beta, if you're interested in trying it out and sharing feedback: https://blog.photostructure.com/introducing-photostructure/
Lesson 1: "RAID isn't a backup." If all you do is copy your files to a NAS, they aren't safe. If you're NAS is taking snapshots onto a different physical spindle, that's good, but it's better to have your NAS backed up to a physically separate device, in case of cryptolocker attacks or hardware failures.
Lesson 2: if it's not automatic, it won't get done. Set up Resilio Sync or SyncThing or your NAS-specific backup software on your phone to pull photos and videos off and back then up automatically. Set up a cron job on your NAS to backup.
Lesson 3: consider an off-site backup. Backblaze has a great service, and so do other cloud services.
Lesson 4: don't use a digital asset manager that requires proprietary hardware or messes with your original files. Google photos, for example, is free, but garbles much of the metadata tags in your files, and downsamples your videos into a blurry mess (even if you use original quality). Failed photo startups are astoundingly prevalent. Use something that's been designed to continue even if the company doesn't.
Amusingly enough, I'm working on a photo startup (spoiler: it satisfies what I wanted from lesson 4!). It's currently in closed beta, if you're interested in trying it out and sharing feedback: https://blog.photostructure.com/introducing-photostructure/
backblaze.com and wasabi.com are currently my favorites.
I’m no archival expert but here’s my lazy mans 3 location strategy.
I use iCloud photos by default, then open google photos app on my phone every once in awhile to have a duplicate cloud sync ($2mo). Then when I remember every month or so I open photos on my Mac and copy everything over to my synology nas which houses a master iPhoto library that is hundreds of GBs at this point. Using iPhoto is probably a mistake here since it’s fairly terrible, but not sure how else to reasonably manage all my photos. I’ve tried Picasa but came back.
I’ve been burned by apps like everpix cloud storage that suddenly shut down and gave me only a few days to download nearly a terabyte of now unsorted photos. Similar thing with Sony cloudstation. I wouldn’t trust any single cloud storage provider from screwing you at some point.
I use iCloud photos by default, then open google photos app on my phone every once in awhile to have a duplicate cloud sync ($2mo). Then when I remember every month or so I open photos on my Mac and copy everything over to my synology nas which houses a master iPhoto library that is hundreds of GBs at this point. Using iPhoto is probably a mistake here since it’s fairly terrible, but not sure how else to reasonably manage all my photos. I’ve tried Picasa but came back.
I’ve been burned by apps like everpix cloud storage that suddenly shut down and gave me only a few days to download nearly a terabyte of now unsorted photos. Similar thing with Sony cloudstation. I wouldn’t trust any single cloud storage provider from screwing you at some point.
SyncThing is pretty useful for copying files automatically between devices. It can be configured to say have every photo you upload automatically copied to another computer. It was a little tricky to try to make sure that when photos were deleted from the phone they weren't deleted from their backup/photo archive. I think I got that working. Just make sure you document your setup so that you can easily provision new devices when your setup changes. Might I also recommend getting some photos printed individually or in a book. The longevity of physical objects and accessibility of them is pretty hard to discount. It will also help your kid to be able to look at them as they grow older.
I use Android phone, dslr camera, & MacBook Pro.
I subscribe to iCloud. Also all photos from phone are synced to Google Photos in HQ format (free unlimited).
I use Dropbox to sync photos from phone to MBP.
Then I use Lightroom to move photos from Dropbox folder to my external SSD and free up space on Dropbox.
Also I use Lightroom 6 to import photos from DSLR and keep them on external SSD.
Then I open Apple Photos and import photos from SSD into Apple Photos. Since I have a small SSD in MBP, Apple Photos uses "Optimize Storage" setting. I can access these photos on iPad. I prefer iPad for photo and video editing.
Finally, I copy all content from my SSD to NAS backup.
It is a bit clunky. It would be really great if Apple Photos let you make backup on your local HD/NAS.
I subscribe to iCloud. Also all photos from phone are synced to Google Photos in HQ format (free unlimited).
I use Dropbox to sync photos from phone to MBP.
Then I use Lightroom to move photos from Dropbox folder to my external SSD and free up space on Dropbox.
Also I use Lightroom 6 to import photos from DSLR and keep them on external SSD.
Then I open Apple Photos and import photos from SSD into Apple Photos. Since I have a small SSD in MBP, Apple Photos uses "Optimize Storage" setting. I can access these photos on iPad. I prefer iPad for photo and video editing.
Finally, I copy all content from my SSD to NAS backup.
It is a bit clunky. It would be really great if Apple Photos let you make backup on your local HD/NAS.
FWIW, you could get rid of your DropBox steps if you used something like SyncThing or Resilio Sync on your phone and MBP.
Congrats on becoming a dad!
S P A C E maybe the solution you're looking for to store all your photos and videos.
- End-to-End Encrypted - your private key stays on your device(s) so only you can read your files.
- Blockchain-Backed - your files are immutable and indelible.
- Open Sourced - all source code is available from Github under Apache 2.0.
- Decentralized - you choose which providers store your files, and can setup your own computers as providers.
- Shareable - you can share a file with any other S P A C E user.
- Currently in Open Beta with an Android app and command line clients for Linux, Mac, and Windows. Your private key can be shared with your other devices so you can access your files from any of your (supported) devices.
There is still some manual work as you need to add the files to S P A C E, but I will look into supporting automatically adding files.
Aletheia Ware currently has two providers, one in San Francisco and the other in New York, but unfortunately neither offer offline tape backups. However, if someone else wants to setup this type of provider I would be happy to help.
The other comments had an interesting point;
- mceachen "Use something that's been designed to continue even if the company doesn't."
- devm0de "I wouldn’t trust any single cloud storage provider from screwing you at some point."
I designed S P A C E to continue even if my company, Aletheia Ware LLC, doesn't by open sourcing all the code, and enabling anyone to be their own providers. By having multiple parties able to provide these services independently you avoid the risk of having all your eggs in one basket - just like you shouldn't store all your data on one harddrive, you shouldn't store all your data with one cloud storage provider.
I would greatly appreciate any feedback you, or other HNers, have.
https://space.aletheiaware.com
https://github.com/AletheiaWareLLC
S P A C E maybe the solution you're looking for to store all your photos and videos.
- End-to-End Encrypted - your private key stays on your device(s) so only you can read your files.
- Blockchain-Backed - your files are immutable and indelible.
- Open Sourced - all source code is available from Github under Apache 2.0.
- Decentralized - you choose which providers store your files, and can setup your own computers as providers.
- Shareable - you can share a file with any other S P A C E user.
- Currently in Open Beta with an Android app and command line clients for Linux, Mac, and Windows. Your private key can be shared with your other devices so you can access your files from any of your (supported) devices.
There is still some manual work as you need to add the files to S P A C E, but I will look into supporting automatically adding files.
Aletheia Ware currently has two providers, one in San Francisco and the other in New York, but unfortunately neither offer offline tape backups. However, if someone else wants to setup this type of provider I would be happy to help.
The other comments had an interesting point;
- mceachen "Use something that's been designed to continue even if the company doesn't."
- devm0de "I wouldn’t trust any single cloud storage provider from screwing you at some point."
I designed S P A C E to continue even if my company, Aletheia Ware LLC, doesn't by open sourcing all the code, and enabling anyone to be their own providers. By having multiple parties able to provide these services independently you avoid the risk of having all your eggs in one basket - just like you shouldn't store all your data on one harddrive, you shouldn't store all your data with one cloud storage provider.
I would greatly appreciate any feedback you, or other HNers, have.
https://space.aletheiaware.com
https://github.com/AletheiaWareLLC
Has anyone found a solution for sorting and tagging images most efficiently? I have a one-year-old daughter and need to do it already.
Main goals:
- Tags like "2019", "3 weeks", "first step" etc
- I also take 3-4 pictures right one after another and would like to keep the best and delete the others
- Best of pictures.. choose a couple of pictures to summarize an event/vacations
I looked at different software but didn't find anything satisfying my needs. Maybe adding the tags in the metadata would be the best options so they are not software dependent?
Main goals:
- Tags like "2019", "3 weeks", "first step" etc
- I also take 3-4 pictures right one after another and would like to keep the best and delete the others
- Best of pictures.. choose a couple of pictures to summarize an event/vacations
I looked at different software but didn't find anything satisfying my needs. Maybe adding the tags in the metadata would be the best options so they are not software dependent?
Naming files by date solves the "2019" problem. It helps with the "three weeks" problem...just some math in the head. My advice is don't delete pictures. At the base level it's work and everyone is fallible. Photographically, the reason is that your opinion about better and worse photographs will evolve over time. You will learn from your mistakes. But you can't learn from the mistake of mistakenly deleting and image. Just file everything.
All things being equal, tags are beneficial. But time spent getting good at tagging comes at the expense of time spent getting better at making better pictures. In terms of tagging, give yourself years to get better at it. Five years from now, going back and sorting through pictures of your child won't be a horrible chore. It will be a joy.
I strongly recommend getting a printer and printing your best pictures. Printing will help you take better pictures. And even mediocre prints are better than anything on a screen.
All things being equal, tags are beneficial. But time spent getting good at tagging comes at the expense of time spent getting better at making better pictures. In terms of tagging, give yourself years to get better at it. Five years from now, going back and sorting through pictures of your child won't be a horrible chore. It will be a joy.
I strongly recommend getting a printer and printing your best pictures. Printing will help you take better pictures. And even mediocre prints are better than anything on a screen.
No commercial ties to them, but if you're in the US, Costco does prints that are cheap, and (as a former digital pre-press nerd, I can attest to) reasonably accurate color reproduction.
The nice 7-color inkjet I've got under my desk has been unused for years and years. Good riddance to fiddling with that ink-sucking, nozzle-coughing, fiddly beast.
The nice 7-color inkjet I've got under my desk has been unused for years and years. Good riddance to fiddling with that ink-sucking, nozzle-coughing, fiddly beast.
I tried the Costco route for exactly the reasons you mention. The technical qualities of the prints are better than what my inkjet produces. But in terms of what I am looking for, my inkjet produces prints I prefer because rapid feedback lets me iterate the image editing and my goals are orthogonal to prepress values.
These days, most photographic tools are oriented toward prepress because there is real money in B2B. I don't really care that much about accurate color reproduction between the real world and the print. I care about predictable correlation between what's on my screen and the print. I'm not aghast that tungsten lights produce warm images (thogh sometimes fluorescent lights are a problem).
Like you I forswore printers. Casual printing would often be a multi-hour or multi-day process because the printer demanded a new cartridge. And two days or two weeks later I would go to print and it would demand another.
About a year and a half ago, I broke down and bought a Canon Pro 100. After the gift card it was $99 and came with 50 13x19 sheets. I print frequently enough that I don't get clogged nozzles. I generally print 4x6. The paper is cheap. It uses less ink. The prints don't take much space. A few months ago, I switched to third party ink. It might not be as good as the OEM. Refilling is more fiddly than disposable cartridges, but it's way more under my control: no trip to the camera store or waiting for UPS. There's a guy on Youtube.
Costco printing made me see the value of printing. But in terms of what I do, it's like outsourcing to the lab back in the film days. The printer in my office is like having my own darkroom. But faster, cheaper, and easier. I can print iteratively at 10pm Sunday and into a Monday morning.
I'm thinking about adding a dye-sublimation printer: Canon Selphy. No fiddly bits, portable, and can run off battery. Somewhat like a Polaroid in terms of producing quick prints in the field. It costs out to around $0.30/4x6 print. Cheaper than a B&W 120 negative.
These days, most photographic tools are oriented toward prepress because there is real money in B2B. I don't really care that much about accurate color reproduction between the real world and the print. I care about predictable correlation between what's on my screen and the print. I'm not aghast that tungsten lights produce warm images (thogh sometimes fluorescent lights are a problem).
Like you I forswore printers. Casual printing would often be a multi-hour or multi-day process because the printer demanded a new cartridge. And two days or two weeks later I would go to print and it would demand another.
About a year and a half ago, I broke down and bought a Canon Pro 100. After the gift card it was $99 and came with 50 13x19 sheets. I print frequently enough that I don't get clogged nozzles. I generally print 4x6. The paper is cheap. It uses less ink. The prints don't take much space. A few months ago, I switched to third party ink. It might not be as good as the OEM. Refilling is more fiddly than disposable cartridges, but it's way more under my control: no trip to the camera store or waiting for UPS. There's a guy on Youtube.
Costco printing made me see the value of printing. But in terms of what I do, it's like outsourcing to the lab back in the film days. The printer in my office is like having my own darkroom. But faster, cheaper, and easier. I can print iteratively at 10pm Sunday and into a Monday morning.
I'm thinking about adding a dye-sublimation printer: Canon Selphy. No fiddly bits, portable, and can run off battery. Somewhat like a Polaroid in terms of producing quick prints in the field. It costs out to around $0.30/4x6 print. Cheaper than a B&W 120 negative.
When you add tags, make sure you do it to the file (or a sidecar). Don't store it in a proprietary library format, because failed photo software is so common now as to be cliche, and you'll lose all that work.
PhotoStructure[1] works like this: it reads from the keywords and tags added to either files directly or to sidecars.
[1] https://blog.photostructure.com/introducing-photostructure/
PhotoStructure[1] works like this: it reads from the keywords and tags added to either files directly or to sidecars.
[1] https://blog.photostructure.com/introducing-photostructure/
looks interesting.
"PhotoStructure tags all your photos and videos automatically with paths, like
When/2019/January, and
Camera/Apple/iPhone XS, and
Where/France/Paris/Musée d'Orsay."
Is this a standard way to save tags with a path? Or just the way your software does it?
"PhotoStructure tags all your photos and videos automatically with paths, like
When/2019/January, and
Camera/Apple/iPhone XS, and
Where/France/Paris/Musée d'Orsay."
Is this a standard way to save tags with a path? Or just the way your software does it?
Those hierarchical tags are actually automatically applied by PhotoStructure based on metadata your phone or camera stores. If you add keywords, like "first step" to a photo (by using the Windows file information dialog, for example), PhotoStructure would add "Keywords/first step" to that asset.
Most keywords I've seen in the wild are just a word or words, much like a hashtag, separated by comma or semicolon. There are hierarchical metadata fields, but they seem to be used only by professional media services, not consumer applications.
Most keywords I've seen in the wild are just a word or words, much like a hashtag, separated by comma or semicolon. There are hierarchical metadata fields, but they seem to be used only by professional media services, not consumer applications.
I loved using Lightroom for photos organization but lately I am just too lazy to organize photos.
With LR6, you can add tags during import.
Then you can use keyboard shortcuts like letter P to pick and X to reject photos. This makes culling and choosing best photos very fast.
There is also a VS mode where you can compare 2 photos side by side and pick one, then it compares it with next photo.
Then you can rate each photo by press number 1-5. I rarely did this.
With LR6, you can add tags during import.
Then you can use keyboard shortcuts like letter P to pick and X to reject photos. This makes culling and choosing best photos very fast.
There is also a VS mode where you can compare 2 photos side by side and pick one, then it compares it with next photo.
Then you can rate each photo by press number 1-5. I rarely did this.
Here is what I do:
Level 1 protection: Offline (only on when adding/updating) harddrive. USB Hard disk (non ssd) for example will do.
Level 2 protection: Always on Synology NAS at home with Raid 1 between 2 drives
Level 3 protection: Google Drive plan with 100Tb.
You work only in the level 2, always available in you network
You can automate between Level 1 and 2 if you connect the external hd to the nas and click the button to sync the photos.
You can automate between level 2 and 3 with synology software alone (in the package manager)
I hope this simplifies your life.
Level 1 protection: Offline (only on when adding/updating) harddrive. USB Hard disk (non ssd) for example will do.
Level 2 protection: Always on Synology NAS at home with Raid 1 between 2 drives
Level 3 protection: Google Drive plan with 100Tb.
You work only in the level 2, always available in you network
You can automate between Level 1 and 2 if you connect the external hd to the nas and click the button to sync the photos.
You can automate between level 2 and 3 with synology software alone (in the package manager)
I hope this simplifies your life.
>Google Drive plan with 100Tb.
Is there a grandfather plan or are you spending ~$1k / mo to store 100TB?
It looks like the highest listed plan is $300 for 30TB.
Is there a grandfather plan or are you spending ~$1k / mo to store 100TB?
It looks like the highest listed plan is $300 for 30TB.
Recently, I started treating digital recording media as write once. Basically SD cards are treated like film negatives. I shoot through to the end of an SD card, then throw it in a shoebox. That's my first level backup. It's as physical as a digital image can be. These days, a 64GB SD card is about the price of a roll of slide film. They're not free. But they're not precious enough to risk loss of data from reformatting.
The cards get copied to disks as I shoot of course. At the deep storage layer, same strategy. Write once and archive the media when it's full. That's where finished images go.
Write once is simpler. Not reformatting eliminates accidental reformats. It eliminates the decision of which images/videos to save (and which to delete). Unsalvageable images can become salvageable as my skill and knowledge improve. Too much work becomes a few clicks.
Caveat: I don't shoot much video. If I did, it would be more expensive. But not terribly so. And the price of physical storage media will keep going down.
Caveat {kinda, sorta}: There's incentive to continue to use relatively lower resolution cameras to produce smaller files to save storage space.
Caveat: I can't view all the original images on all devices. For me, that's not a big deal. The best once go on social media and/or get printed.
The cards get copied to disks as I shoot of course. At the deep storage layer, same strategy. Write once and archive the media when it's full. That's where finished images go.
Write once is simpler. Not reformatting eliminates accidental reformats. It eliminates the decision of which images/videos to save (and which to delete). Unsalvageable images can become salvageable as my skill and knowledge improve. Too much work becomes a few clicks.
Caveat: I don't shoot much video. If I did, it would be more expensive. But not terribly so. And the price of physical storage media will keep going down.
Caveat {kinda, sorta}: There's incentive to continue to use relatively lower resolution cameras to produce smaller files to save storage space.
Caveat: I can't view all the original images on all devices. For me, that's not a big deal. The best once go on social media and/or get printed.
[deleted]
Most, if not all, SD cards are not a viable archive medium.
Expect bitrot within 5 years. In testing PhotoStructure's image and video bitrot detector, I found all of my older (5+ year) cards were either unreadable or had file corruption.
Expect bitrot within 5 years. In testing PhotoStructure's image and video bitrot detector, I found all of my older (5+ year) cards were either unreadable or had file corruption.
I agree. But reformatting SD card seems a less useful backup strategy. Both in terms of accelerated bit rot and by adding human error into the mix. Basically, reformatting is worse than filling and storing. Filling and storing is less work. And the work it eliminates is prone to serious errors.
Right, I think he's not doubting your strategy of not reformatting, but more pointing out that the SD cards maybe shouldn't even be considered "first level backup" because they'll probably inevitably bitrot at some point.
Since the SD cards contain the originals, we're heading for trouble as soon as anyone applies formal semantics to 'backup'. We could call the SD cards "copies" but again, they contain the originals. I'm using 'backup' informally.
Concern about bitrotting originals is one reason to backup. Losing some images is a partial disaster. Losing all of them is total. Reformatting is exactly that. It's objectively worse than bitrot over time. It's objectively better than not "backing up" at all. It might be less contingent than an online solution with a credit card dependency over a period of five years.
Concern about bitrotting originals is one reason to backup. Losing some images is a partial disaster. Losing all of them is total. Reformatting is exactly that. It's objectively worse than bitrot over time. It's objectively better than not "backing up" at all. It might be less contingent than an online solution with a credit card dependency over a period of five years.
Even finding photos or sorting and tagging can be such a chore nowadays. It's also a privacy nightmare if you want to store photos online. God forbid you accidentally upload/archive some photos you did not mean to or realize were in your photos folder..
I think what I'm building might help you! It's called PhotoStructure[1], and it takes care of your privacy concerns by keeping your photos and videos on hardware you own. Your data stays on your computer, where it belongs.
[1] https://blog.photostructure.com/introducing-photostructure/
[1] https://blog.photostructure.com/introducing-photostructure/
Not exactly a backup, but https://lifeboxhq.com is trying to make digital time capsules for files and such that persist for a very long time.
Onsite Backup (Synology device) <---> PC (active copy) <---> Offsite Backup (Backblaze)
It's all automatic because if backups aren't automatic there is a lot of risk.
It's all automatic because if backups aren't automatic there is a lot of risk.
There is some manual work needed (not a lot) to make sure all are in sync. But boy I wish there was a fully automated way to do this that allowed sharing, lots of backups (even offline tape), and ability to collect and view videos and photos on all devices.
I’d love to hear how others are managing the long-term storage of photos and videos.