Ask HN: What is your approach for managing personal digital assets?
26 comments
Step 1. Get all of your shit off the cloud.
You don't own anything on the cloud. Yes, you can put stuff on there, but recognise this one fact. Out there, it doesn't belong to you. Nobody cares whether it remains safe or it disappears. especially don't have stuff on the cloud that doesn't exist anywhere else
To have it remain safe and under your control, it needs to be on your own server(s). And those servers need to be backed up daily.
Step 2. Have multiple backups.
Backups have been known to fail or die. If one backup goes, there should be at least one more.
Step 3. (And this might be sacrilege to some) Don't use encrypted drives.
If there is a problem with the encryption, the whole drive is irrevocably lost. If all files are stored individually, one or more files might get corrupted, but losing the entire drive would be very unlikely.
Step 4. Use hard copy. You know, that old stuff: paper.
Periodically, print out all your accounts, logins and passwords. Store that list in a readable documents package which will be read after your death, labelled such as 'Legal Documents'.
Step 5. Where have you copied all that stuff on your phone?
People too often forget to back up their phones. And it needs to be a backup that is not restricted to a particular company. So you've backed up your iPhone to Apple's Cloud. Are you certain that your next phone will also be an Apple? If it isn't can you replace all that stuff into your new Android phone?
You don't own anything on the cloud. Yes, you can put stuff on there, but recognise this one fact. Out there, it doesn't belong to you. Nobody cares whether it remains safe or it disappears. especially don't have stuff on the cloud that doesn't exist anywhere else
To have it remain safe and under your control, it needs to be on your own server(s). And those servers need to be backed up daily.
Step 2. Have multiple backups.
Backups have been known to fail or die. If one backup goes, there should be at least one more.
Step 3. (And this might be sacrilege to some) Don't use encrypted drives.
If there is a problem with the encryption, the whole drive is irrevocably lost. If all files are stored individually, one or more files might get corrupted, but losing the entire drive would be very unlikely.
Step 4. Use hard copy. You know, that old stuff: paper.
Periodically, print out all your accounts, logins and passwords. Store that list in a readable documents package which will be read after your death, labelled such as 'Legal Documents'.
Step 5. Where have you copied all that stuff on your phone?
People too often forget to back up their phones. And it needs to be a backup that is not restricted to a particular company. So you've backed up your iPhone to Apple's Cloud. Are you certain that your next phone will also be an Apple? If it isn't can you replace all that stuff into your new Android phone?
For around 15+ years I have maintained a super-curated deep taxonomy of files and folders, symlinking stuff, and so on. I've given up around 6 years ago switching to Emacs, actually almost all my files are org-mode managed, the current approach is:
- time-based notes, like
At first might sound complicated but it allow both an overview of anything, just looking at years binders notes, the maximum level of details going down to daily notes, and the maximum high level zoom via `lib` notes. Essentially I have a personal history, I can slowly change my notes having only to keep stuff under `lib` consistent, and having all things relevant to a certain timeframe immediately at hands.
That's to org-roam-node-find I can directly access any attached file just hitting a key and start typing something, it's a graph not a hierarchy, but when I'm on the desired results or around something I look for I can also "explore the surroundings" thanks to the time-based structure and org-mode links, org-roam backrefs and so on.
Anything is linked or attached in notes: my mails reside also on my iron indexed via notmuch, my files org-attached, some configs are created from notes (org-babel, tangled), something is attached and linked per file, some linked to the parent dir some files reside in dired etc. Basically I have a near-semantic, full-text searchable, metadata rich, tool to manage anything.
I've tried to do more with org-ql but honestly I'm not disciplined enough, I've tried the LLM way (khoj) and find very little useful results, so far direct access using org-mode heading as titles/searchable stuff with tags and optional org-roam aliases prove to be powerful enough. Sometimes when I fails to find something this way I going the counsel-rg way (full text ripgrep) and than I add a roam_alias matching the keywords I've tried before. I've also tried briefly recoll/counsel-recoll (xapian) but I've never have to use it except for testing.
I do miss phone/sms integration but having daily notes and org-capture at hand if something relevant happen I'll note in a daily note. If it became or I know it's relevant it will also be in some year-binder, if it's useful or became useful beyond an year it goes in lib, a kind of long-term archive. That's my current best compromise between "noise" and "order". Various automation do help noting anything, org-store-link/org-insert-link do help do link an email, so org-capture initiated from a mail, similarly yasnippet and hydra do help for recurrent transactions (all babel blocks get tangled to a tree where beancount import * anything), some helpers allow to link things to open with specific apps via elisp: links and so on.
It was a journey, I've hesitated much, I still have a gazillion of things to sort and automate properly, some broken/old automation etc BUT anything works smooth enough and I never lost a single bit of information nor I've failed to find something with nearly no effort so...
- time-based notes, like
~/org/yyyy/
notes/mm-dd.org
binders/{various,topic-based,notes}.org
work/{work,related,notes}.org
~/org/lib/{long,lasting,topic,based,notes}.org
~/data/ < org-attach tree
At first might sound complicated but it allow both an overview of anything, just looking at years binders notes, the maximum level of details going down to daily notes, and the maximum high level zoom via `lib` notes. Essentially I have a personal history, I can slowly change my notes having only to keep stuff under `lib` consistent, and having all things relevant to a certain timeframe immediately at hands.
That's to org-roam-node-find I can directly access any attached file just hitting a key and start typing something, it's a graph not a hierarchy, but when I'm on the desired results or around something I look for I can also "explore the surroundings" thanks to the time-based structure and org-mode links, org-roam backrefs and so on.
Anything is linked or attached in notes: my mails reside also on my iron indexed via notmuch, my files org-attached, some configs are created from notes (org-babel, tangled), something is attached and linked per file, some linked to the parent dir some files reside in dired etc. Basically I have a near-semantic, full-text searchable, metadata rich, tool to manage anything.
I've tried to do more with org-ql but honestly I'm not disciplined enough, I've tried the LLM way (khoj) and find very little useful results, so far direct access using org-mode heading as titles/searchable stuff with tags and optional org-roam aliases prove to be powerful enough. Sometimes when I fails to find something this way I going the counsel-rg way (full text ripgrep) and than I add a roam_alias matching the keywords I've tried before. I've also tried briefly recoll/counsel-recoll (xapian) but I've never have to use it except for testing.
I do miss phone/sms integration but having daily notes and org-capture at hand if something relevant happen I'll note in a daily note. If it became or I know it's relevant it will also be in some year-binder, if it's useful or became useful beyond an year it goes in lib, a kind of long-term archive. That's my current best compromise between "noise" and "order". Various automation do help noting anything, org-store-link/org-insert-link do help do link an email, so org-capture initiated from a mail, similarly yasnippet and hydra do help for recurrent transactions (all babel blocks get tangled to a tree where beancount import * anything), some helpers allow to link things to open with specific apps via elisp: links and so on.
It was a journey, I've hesitated much, I still have a gazillion of things to sort and automate properly, some broken/old automation etc BUT anything works smooth enough and I never lost a single bit of information nor I've failed to find something with nearly no effort so...
I religiously use Google contacts. It's the simplest way to keep people contacts up to date on Android.
I archive all important documents in specific folders by subject and date. This is backed up to back blaze with restic. https://restic.net/
I use https://ente.io for pictures. I convinced my wife to use it, and she agreed to auto share her photos so I don't nag her for copies. It had simple import from Facebook and Google.
I also keep extensive journals, which really helps to tie it all together. I can basically grep for hangouts, conversations, etc. https://jodavaho.io/tags/bullet-journal.html
I also separate work journal from personal, and have essentially a journal for each project. https://jodavaho.io/tags/bullet-journal.html again.
I religiously use Google calendar for all plans, you can easily search it for past events to get dates. See also journal.
I use task warrior, and use projects organized the same way as my project journal and file folders. https://taskwarrior.org/
So a project might be 'outrider/planner' and so the relevant docs are on ~/outrider/planner, tasks are in proj:outrider:planner and my notes are in docs dir with a common name like 'notebook.md' or logs/<date>.md
It's really that combo of using same method for folder structure, task project structure, notes, docs that keeps my life sane. Grep and FZF make it very easy to search.
I archive all important documents in specific folders by subject and date. This is backed up to back blaze with restic. https://restic.net/
I use https://ente.io for pictures. I convinced my wife to use it, and she agreed to auto share her photos so I don't nag her for copies. It had simple import from Facebook and Google.
I also keep extensive journals, which really helps to tie it all together. I can basically grep for hangouts, conversations, etc. https://jodavaho.io/tags/bullet-journal.html
I also separate work journal from personal, and have essentially a journal for each project. https://jodavaho.io/tags/bullet-journal.html again.
I religiously use Google calendar for all plans, you can easily search it for past events to get dates. See also journal.
I use task warrior, and use projects organized the same way as my project journal and file folders. https://taskwarrior.org/
So a project might be 'outrider/planner' and so the relevant docs are on ~/outrider/planner, tasks are in proj:outrider:planner and my notes are in docs dir with a common name like 'notebook.md' or logs/<date>.md
It's really that combo of using same method for folder structure, task project structure, notes, docs that keeps my life sane. Grep and FZF make it very easy to search.
Personally I also scatter files across cloud, laptop and smartphone, with no plan or structure; files end up where I last used them.
For now that works well for me though. Sorting by last modified and starring or adding items to favorites also helps. Password managers are also useful when it comes to storing sensitive data.
For now that works well for me though. Sorting by last modified and starring or adding items to favorites also helps. Password managers are also useful when it comes to storing sensitive data.
I've faced (and am still facing) this problem as well. I don't think there's a good single solution, especially for finding specific photos.
My current imperfect approach is to use Raycast / Alfred as a meta search tool to find things in various places. It doesn't unify things but it makes searching multiple places low friction.
Some specific tools, in case it helps.
- invoices, contracts, etc. go into paperless-ngx. Physical documents get scanned in.
- miscellaneous or unorganized documents into Nextcloud
- custom solution for searching text messages
- no solution whatsoever for images. I think semantic image search might do the job, although I haven't explored what tools exist for that.
Edit: formatting
My current imperfect approach is to use Raycast / Alfred as a meta search tool to find things in various places. It doesn't unify things but it makes searching multiple places low friction.
Some specific tools, in case it helps.
- invoices, contracts, etc. go into paperless-ngx. Physical documents get scanned in.
- miscellaneous or unorganized documents into Nextcloud
- custom solution for searching text messages
- no solution whatsoever for images. I think semantic image search might do the job, although I haven't explored what tools exist for that.
Edit: formatting
Tresorit is where all my stuff goes. Except photos from my phone, for that I use iCloud backup (not Photo sync - the encrypted full device backup)
My priorities for a solution for this was 1) Privacy 2) Security mostly in terms og availability (reliability) 3) works across the platforms the family uses, 4) do not want to spend any time managing the solution and 5) long term solution
I generally do not trust cloud services so Google Drive, Dropbox et al are not even in the discussion. Tresorit is an exception due to e2e encryption and company location (EU).
Tresorit is ridiculously expensive, and I pay for more iCloud storage. I have just accepted that if I want the priorities above (and esp. not having to managing it myself) I would have to pay for a quality product.
Proton (mail and drive) I would also trust for the same reasons I do Tresorit, and I think it’s less expensive, but it wasn’t around when I picked Tresorit
Edit: a big part of why Tresorit (and iCloud) solves the problem is actually not due to Tresorit, but being disciplined about sticking to only using one product. Sometimes that means not doing something I want if it’s not compatible with Tresorit, or accepting a more involved manual process. Like, MS Office documents I don’t use 365 and the Office storage (one drive), but the offline version of and a folder on disk synced to Tresorit.
My priorities for a solution for this was 1) Privacy 2) Security mostly in terms og availability (reliability) 3) works across the platforms the family uses, 4) do not want to spend any time managing the solution and 5) long term solution
I generally do not trust cloud services so Google Drive, Dropbox et al are not even in the discussion. Tresorit is an exception due to e2e encryption and company location (EU).
Tresorit is ridiculously expensive, and I pay for more iCloud storage. I have just accepted that if I want the priorities above (and esp. not having to managing it myself) I would have to pay for a quality product.
Proton (mail and drive) I would also trust for the same reasons I do Tresorit, and I think it’s less expensive, but it wasn’t around when I picked Tresorit
Edit: a big part of why Tresorit (and iCloud) solves the problem is actually not due to Tresorit, but being disciplined about sticking to only using one product. Sometimes that means not doing something I want if it’s not compatible with Tresorit, or accepting a more involved manual process. Like, MS Office documents I don’t use 365 and the Office storage (one drive), but the offline version of and a folder on disk synced to Tresorit.
I've moved to more and more plain-text or more common formats for others that will likely survive the test of time, such as JPEGs, PDFs.
So far, I'm also responsible for managing the digital assets of our family and the extended in-laws. Here is how I have organized so far, and there is a lot of room for improvement.
Every individual is a container (folder) and then is foldered further into the common ones -- medical/health, photos, documents, etc. So, if mother-in-law asked my wife to find the medical record for my father-in-law that we went to the clinic in AUG-2022, I go look at "/name/health/" and then is either in an "EventName" folder or a "Year/Date/Something". I can fine something within a few folders of separation for anything.
I tend to organize file, more aptly receipt, named something in the lines of "YYYY-MM-DD $xyz What was it.pdf". So, I know the date, the invoice/receipt value and what it might be without opening the file.
The photos are still in Apple Photos for convenience but is backed up every month to the highest resolution it was hot at in a JPEG and exported month as "/photos-backup/YYYY/photos-YYYY-MM" as part of my weekly, monthly, quarterly, and then yearly digital chores. As I do this regularly, they don't tend to pile up, even if I missed, say while traveling or I just got lazy and skip it.
The question I usually ask when using a tool/program is, “Can I walk out of this?”
Here is an example, when my daughter grows up enough and wants to own her digital assets, there is one folder with her and everything organized the way I've done, "Schools", "Health", "Creatives", "Photos", etc. She can then proceed the way she wants.
Of course, these are replicated across drives, cloud-drives, etc. My next fun project is to replicate my data across a few locations I frequent around the world -- a remote hilly hometown, the current home in Bangalore, another place I go frequently for work, etc.
(Typed in a bit of a haste. Will come back and edit.)
So far, I'm also responsible for managing the digital assets of our family and the extended in-laws. Here is how I have organized so far, and there is a lot of room for improvement.
Every individual is a container (folder) and then is foldered further into the common ones -- medical/health, photos, documents, etc. So, if mother-in-law asked my wife to find the medical record for my father-in-law that we went to the clinic in AUG-2022, I go look at "/name/health/" and then is either in an "EventName" folder or a "Year/Date/Something". I can fine something within a few folders of separation for anything.
I tend to organize file, more aptly receipt, named something in the lines of "YYYY-MM-DD $xyz What was it.pdf". So, I know the date, the invoice/receipt value and what it might be without opening the file.
The photos are still in Apple Photos for convenience but is backed up every month to the highest resolution it was hot at in a JPEG and exported month as "/photos-backup/YYYY/photos-YYYY-MM" as part of my weekly, monthly, quarterly, and then yearly digital chores. As I do this regularly, they don't tend to pile up, even if I missed, say while traveling or I just got lazy and skip it.
The question I usually ask when using a tool/program is, “Can I walk out of this?”
Here is an example, when my daughter grows up enough and wants to own her digital assets, there is one folder with her and everything organized the way I've done, "Schools", "Health", "Creatives", "Photos", etc. She can then proceed the way she wants.
Of course, these are replicated across drives, cloud-drives, etc. My next fun project is to replicate my data across a few locations I frequent around the world -- a remote hilly hometown, the current home in Bangalore, another place I go frequently for work, etc.
(Typed in a bit of a haste. Will come back and edit.)
Keeping backups of everything in a centralized directory/drive is useful since regardless of where the content may be normally (phone, different system) if it gets backed up to one place then it makes it searchable/indexable.
Edit: I should mention the kinds of backups I prefer to keep for structured data are in non-binary formats, which helps with searches. Otherwise just keeping organization sensible for future-me helps a great deal.
Edit: I should mention the kinds of backups I prefer to keep for structured data are in non-binary formats, which helps with searches. Otherwise just keeping organization sensible for future-me helps a great deal.
For me, I use a combination of things and I keep regular backups of everything.
Photos: Google Photo
Notes: Nextcloud +Joplin
Documents and invoices: Google Drive
I didn't move photos and documents to Nextcloud because the features for viewing photos and editing documents are not as good as Google.
Photos: Google Photo
Notes: Nextcloud +Joplin
Documents and invoices: Google Drive
I didn't move photos and documents to Nextcloud because the features for viewing photos and editing documents are not as good as Google.
For long-term preservation of important digital assets, I strongly recommend using archival-grade DVDs. While cloud services like Google Drive have demonstrated remarkable reliability in file storage over the years, surpassing the performance of most consumer-grade hard drives and USB drives, DVDs offer unparalleled reliability for local storage. This is the only approach to ensure your files remain accessible and secure over +50 years timeframe.
I keep "important" (as in I might need them for taxes or be asked for them again later) papers all in one folder organized by year and I'm pretty good at putting things there when I need to.
The rest is spread out across 2 google drive accounts and a Hetzner Nextcloud instance. I was really uncertain about the reliability of NC when I started using it but now it's been so reliable for quite some time and I'm thinking of moving more stuff there. Even though Hetzner offers automatic backups I think I still need another backup but haven't figured that part out yet.
Oh also I use syncthing to keep a few folders synced between my laptop, phone, and NC instance.
I will say though I've been long interested and searched for some ways that people actually organize their folders and such but haven't see anything too useful. I guess as long as you have good search it isn't as important.
The rest is spread out across 2 google drive accounts and a Hetzner Nextcloud instance. I was really uncertain about the reliability of NC when I started using it but now it's been so reliable for quite some time and I'm thinking of moving more stuff there. Even though Hetzner offers automatic backups I think I still need another backup but haven't figured that part out yet.
Oh also I use syncthing to keep a few folders synced between my laptop, phone, and NC instance.
I will say though I've been long interested and searched for some ways that people actually organize their folders and such but haven't see anything too useful. I guess as long as you have good search it isn't as important.
Photoprism for photos.
Plex for music, movies, and series.
Invoices I file physically.
For messaging I plan to move to Matrix such that my homeserver runs "bridges" for other platforms so that I can later on export and query messages. For emails I use notmuch.
For managing (storing) files I've concluded that it is impossible to find the ideal hierarchical structure for containing everything, so just go for reasonable structures that might have redundancy - for instance, accepting macos photos' way of organizing photos, adding in another directory for photos not managed by the photos app, plus photos that I include in my journal which is yet another set of directories.
For finding things, I've found it impossible to give sufficiently unique names to everything, and just go for names that are somewhat descriptive. If there is a way to annotate them with descriptions, I try to make the descriptions as complete as possible, am diligent with proper spelling and try to use canonical forms in my descriptions. I also try to settle on a standard set of words I use instead of their synonyms. This allows me to use standard searching mechanisms to find things without worrying about whether I am searching for the right word. I can with confidence to a system-wide search without worrying whether I'm searching using the right words: because I know which word from a set of synonyms to use, because I don't have to worry about finding misspelled variants, and because I know which order I will have put the words in
For finding things, I've found it impossible to give sufficiently unique names to everything, and just go for names that are somewhat descriptive. If there is a way to annotate them with descriptions, I try to make the descriptions as complete as possible, am diligent with proper spelling and try to use canonical forms in my descriptions. I also try to settle on a standard set of words I use instead of their synonyms. This allows me to use standard searching mechanisms to find things without worrying about whether I am searching for the right word. I can with confidence to a system-wide search without worrying whether I'm searching using the right words: because I know which word from a set of synonyms to use, because I don't have to worry about finding misspelled variants, and because I know which order I will have put the words in
This is what I use (primarily using self-hosted, open-source solutions where possible).
For documents like invoices, correspondence, etc, I scan them into paperless-ngx.[0] You can apparently set it up to monitor an email inbox and automatically store certain emails or attachments using rules, but I just take the old fashioned approach of adding scanning them in and saving them manually.
I backup all photos to a Nextcloud[1] instance. Periodically, I will go through all my recent photos and organise them. Travel-related photos go in trip-specific directory ("Travel -> [year] -> [trip name]"). Everything else just goes in a "Photos -> [year] -> [month]" directory. Unfortunately this won't help you find the photo of you wearing a cap unless you know where or when it was taken. I believe there are some AI-based solutions out there but I haven't tried them.
For contact details, I store these centrally on Nextcloud using its Contacts app[2] (which uses vCard and CardDAV). I can then access it using any standard contact book software (I use the CardBook addon[3] for Thunderbird and Fossify Contacts[4] on Android).
Similar for calendar - host it via Nextcloud (using CalDAV)[5] and access it from my preferred calendar software.
Joplin[6] for notes (again synced via Nextcloud).
The process is kind of half-automated, half-manual. I don't think there is a good solution to fully automate any of this stuff, particularly in the FOSS space. If you keep on top of organisation it's not that painful and doing things like organising photos and adding upcoming events to your calendar can be fun.
Nextcloud is really good at acting as a hub for your data and speaking to a variety of clients using traditional, open protocols like CalDAV and CardDAV.
0: https://docs.paperless-ngx.com/
1: https://nextcloud.com/
2: https://apps.nextcloud.com/apps/contacts
3: https://addons.thunderbird.net/en-GB/thunderbird/addon/cardb...
4: https://f-droid.org/packages/org.fossify.contacts/
5: https://apps.nextcloud.com/apps/calendar
6: https://joplinapp.org/
For documents like invoices, correspondence, etc, I scan them into paperless-ngx.[0] You can apparently set it up to monitor an email inbox and automatically store certain emails or attachments using rules, but I just take the old fashioned approach of adding scanning them in and saving them manually.
I backup all photos to a Nextcloud[1] instance. Periodically, I will go through all my recent photos and organise them. Travel-related photos go in trip-specific directory ("Travel -> [year] -> [trip name]"). Everything else just goes in a "Photos -> [year] -> [month]" directory. Unfortunately this won't help you find the photo of you wearing a cap unless you know where or when it was taken. I believe there are some AI-based solutions out there but I haven't tried them.
For contact details, I store these centrally on Nextcloud using its Contacts app[2] (which uses vCard and CardDAV). I can then access it using any standard contact book software (I use the CardBook addon[3] for Thunderbird and Fossify Contacts[4] on Android).
Similar for calendar - host it via Nextcloud (using CalDAV)[5] and access it from my preferred calendar software.
Joplin[6] for notes (again synced via Nextcloud).
The process is kind of half-automated, half-manual. I don't think there is a good solution to fully automate any of this stuff, particularly in the FOSS space. If you keep on top of organisation it's not that painful and doing things like organising photos and adding upcoming events to your calendar can be fun.
Nextcloud is really good at acting as a hub for your data and speaking to a variety of clients using traditional, open protocols like CalDAV and CardDAV.
0: https://docs.paperless-ngx.com/
1: https://nextcloud.com/
2: https://apps.nextcloud.com/apps/contacts
3: https://addons.thunderbird.net/en-GB/thunderbird/addon/cardb...
4: https://f-droid.org/packages/org.fossify.contacts/
5: https://apps.nextcloud.com/apps/calendar
6: https://joplinapp.org/
I try to maintain a structure for important stuff (documents, photos, decades of emails, etc.) that can be copied or backed up as needed.
If I find a need to create an order out of chaos (e.g., for thousands of the electronic books I have) I find it best to sketch up a top-level directory view of how I want it; then switching to this method and occasionally cleaning up one-off additions is not hard. My 2c.
If I find a need to create an order out of chaos (e.g., for thousands of the electronic books I have) I find it best to sketch up a top-level directory view of how I want it; then switching to this method and occasionally cleaning up one-off additions is not hard. My 2c.
I master all files on the home nas. No more files on a local laptop. Everything, including photos. Photos moved off of phone and stored on nas. Plex has access, and Home Assistant displays random family photos every couple of minutes.
Nas is replicated offsite via tunnel to another home. Occasional backup to a usb drive kept in a safe. No third party cloud.
Made the switch to Obsidian a couple of years ago for notes. A big step up in searchability. Combined with syncthing to replicate the vault on mobile phone.
Nas is replicated offsite via tunnel to another home. Occasional backup to a usb drive kept in a safe. No third party cloud.
Made the switch to Obsidian a couple of years ago for notes. A big step up in searchability. Combined with syncthing to replicate the vault on mobile phone.
My approach is a flat date-based file structure with a markdown-based Zettelkasten on top:
File are mostly in a flat structure. In ~/, I have directories like documents, photos, videos that directly contain all the files. For certain topic, I create separate directories (e.g., PhD, major projects). The filenames always start with an ISO date (yyyy-mm-dd). For documents, it could be like 2024-03-24_inv_google.pdf. Here, inv means it's an invoice and Google is the organization where the invoice comes from. After the date, there's always a three letter code that tells me the type of document. All physical documents are scanned and OCRed and put in the same structure.
On top of that I curate a Zettelkasten. It's just a directory with a set of markdown files (also a flat hierarchy). This Zettelkasten is layer of information on top of the remaining filesystem. The markdown files link to external resources (e.g., on the web) and also to internal files (e.g., (something)[../docuemnts/something.pdf]). This way I can browse my personal information like it's the web.
This file structure is my single source of truth. I try to export all other information as text files there (e.g., emails, contacts). I'm using restic to backup my entire home folder. I don't use any sophisticated tools, just a basic PDF reader, vim, Gimp, and other standard tools of the Linux ecosystem. I used to sync my home directory using unison but currently, I'm just using a single one.
File are mostly in a flat structure. In ~/, I have directories like documents, photos, videos that directly contain all the files. For certain topic, I create separate directories (e.g., PhD, major projects). The filenames always start with an ISO date (yyyy-mm-dd). For documents, it could be like 2024-03-24_inv_google.pdf. Here, inv means it's an invoice and Google is the organization where the invoice comes from. After the date, there's always a three letter code that tells me the type of document. All physical documents are scanned and OCRed and put in the same structure.
On top of that I curate a Zettelkasten. It's just a directory with a set of markdown files (also a flat hierarchy). This Zettelkasten is layer of information on top of the remaining filesystem. The markdown files link to external resources (e.g., on the web) and also to internal files (e.g., (something)[../docuemnts/something.pdf]). This way I can browse my personal information like it's the web.
This file structure is my single source of truth. I try to export all other information as text files there (e.g., emails, contacts). I'm using restic to backup my entire home folder. I don't use any sophisticated tools, just a basic PDF reader, vim, Gimp, and other standard tools of the Linux ecosystem. I used to sync my home directory using unison but currently, I'm just using a single one.
What do you use for scanning and OCRing? I haven't found a solution that _doesn't_ make me want to throw it all out and just retype the doc by hand.
I too am moving towards a flat hierarchy with dates after attempting Zettlekasten and PARA. I like your "Zettlekasten layer on top of the fs" idea. Def going to do something similar.
I too am moving towards a flat hierarchy with dates after attempting Zettlekasten and PARA. I like your "Zettlekasten layer on top of the fs" idea. Def going to do something similar.
I have some high end scanner (because I sometimes scan negatives from film photography). It has a built-in OCR that works pretty well. That's basically the only reason I need to boot MacOS because the Linux drivers are rather bland.
Not OP but for some stuff I only get on paper, so I need to scan I've simply have a rough "import" semi-automated workflow:
- scan to an image/as many as needed
- mogrify -deskew 90%
- for i in .png; do; convert $i ${i/ng/df}; rm $i; done
- pdftk cat output aname.pdf
- ocrmypdf --force-ocr -c -i (wrap tesseract and others)
I normally get a good enough scan. If it's not the case I might manually edit images than feed to the rest of the pipeline. Gimp G'Mic QT Repair Scanned Document filter works IME far better than unpaper and others, but it's not that automatic, need to be tuned case by case. However that's rare.
For notes, personally I go for time-based notes, similar to ZK but with the time-constraint, I've described the system in this page.
- scan to an image/as many as needed
- mogrify -deskew 90%
- for i in .png; do; convert $i ${i/ng/df}; rm $i; done
- pdftk cat output aname.pdf
- ocrmypdf --force-ocr -c -i (wrap tesseract and others)
I normally get a good enough scan. If it's not the case I might manually edit images than feed to the rest of the pipeline. Gimp G'Mic QT Repair Scanned Document filter works IME far better than unpaper and others, but it's not that automatic, need to be tuned case by case. However that's rare.
For notes, personally I go for time-based notes, similar to ZK but with the time-constraint, I've described the system in this page.
Not your drive, not your data.
For working storage, I use a local server with a Raid 1 array. The drive manufacture dates are offset by a year or more. I keep a spare drive on hand in case of failure.
Every 6 months or so I do a full backup to an external, portable Raid 1 array (again offset manufacture dates) that I keep at grandma's house.
For routine backup, I have a pair of microSD cards that I update every Friday. These are encrypted with Bitlocker in case they should ever be lost. These cards tend to fail over time, hence the 2 copies. I plug them in simultaneously (USB) into the server and run a batch file to update both. Any failure generally becomes obvious at this point. Bitlocker helps as a failure detection mechanism. Any corruption usually results in an unmountable drive.
One microSD goes into a fire proof safe. The other gets strapped to my wrist so I *always* carry it with me wherever I go (even the shower) so it is as safe as I am, if not safer.
The theory being that all I really need to worry about is my personal safety. But even if I should die in some horrible accident like a car or plane crash, authorities will expend a great deal of effort to recover my body --- and the attached microSD storage.
For this purpose, I have designed a 3D printed, discrete, secure locking microSD card holder that slips over a watch strap with the holder on the the inside against my wrist for protection. No one ever even notices it except me.
For working storage, I use a local server with a Raid 1 array. The drive manufacture dates are offset by a year or more. I keep a spare drive on hand in case of failure.
Every 6 months or so I do a full backup to an external, portable Raid 1 array (again offset manufacture dates) that I keep at grandma's house.
For routine backup, I have a pair of microSD cards that I update every Friday. These are encrypted with Bitlocker in case they should ever be lost. These cards tend to fail over time, hence the 2 copies. I plug them in simultaneously (USB) into the server and run a batch file to update both. Any failure generally becomes obvious at this point. Bitlocker helps as a failure detection mechanism. Any corruption usually results in an unmountable drive.
One microSD goes into a fire proof safe. The other gets strapped to my wrist so I *always* carry it with me wherever I go (even the shower) so it is as safe as I am, if not safer.
The theory being that all I really need to worry about is my personal safety. But even if I should die in some horrible accident like a car or plane crash, authorities will expend a great deal of effort to recover my body --- and the attached microSD storage.
For this purpose, I have designed a 3D printed, discrete, secure locking microSD card holder that slips over a watch strap with the holder on the the inside against my wrist for protection. No one ever even notices it except me.
Is the microsd card holder like this one? https://www.printables.com/model/691366-apple-watch-wristban...
Similar idea.
This looks somewhat heavier/bulkier than mine and I designed a latch/lever to grab and securely hold the card using the notch that is provided on one side. I'm pretty sure this is the intended purpose of the notch.
I'm also pretty sure that most ordinary microSD cards are reasonably waterproof. On the outside, they're solid molded plastic with contacts exposed. I've used different brands in the shower over long periods of time and gone swimming with them without any problem. I just wouldn't recommend powering them up wet --- might short the contacts.
This looks somewhat heavier/bulkier than mine and I designed a latch/lever to grab and securely hold the card using the notch that is provided on one side. I'm pretty sure this is the intended purpose of the notch.
I'm also pretty sure that most ordinary microSD cards are reasonably waterproof. On the outside, they're solid molded plastic with contacts exposed. I've used different brands in the shower over long periods of time and gone swimming with them without any problem. I just wouldn't recommend powering them up wet --- might short the contacts.
Has anyone done a (hopefully) systematic survey of the processes and software people use to store their stuff, sort of like a usesthis.com but just for storing assets, and how well that's worked over time? My guess is the successful strategies would look a lot like Brajeshwar's comment, a thoughtful plan that uses simple software and formats, some planning for the future, and, probably critically, regularly doing "digital chores".
There've been some efforts in the past to store everything and make it searchable, like the ancient Chandler project, and the possibly still alive Parkeep, none that have been more widely adopted than a strategy of put everything in Gmail, Dropbox, etc, and hope for the best, which is what I do, minus the regular diligence that people like Brajeshwar have.
Making and using anything more complex looks like it turns into a (very cool looking!) hobby in itself, like these:
https://thesephist.com/posts/monocle/
https://simonwillison.net/2020/Nov/14/personal-data-warehous...
https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
And yeah, the latter two also include storing and searching more than say email and photos, but maybe shows one's tendency to want to store and search everything.
There've been some efforts in the past to store everything and make it searchable, like the ancient Chandler project, and the possibly still alive Parkeep, none that have been more widely adopted than a strategy of put everything in Gmail, Dropbox, etc, and hope for the best, which is what I do, minus the regular diligence that people like Brajeshwar have.
Making and using anything more complex looks like it turns into a (very cool looking!) hobby in itself, like these:
https://thesephist.com/posts/monocle/
https://simonwillison.net/2020/Nov/14/personal-data-warehous...
https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
And yeah, the latter two also include storing and searching more than say email and photos, but maybe shows one's tendency to want to store and search everything.
Not much a survey, but in the past I've tried recoll (xapian wrapper, CLI and GUI, very simple to setup) only to realize that for my needs and the way I take notes (described in this page, another comment) I do not need full-text search except for emails (that are handled by notmuch, locally so indexed by xapian anyway).
I've tried for the web part archivy (pythonic web-app) and zotero, but they do not works like I want so I've dropped them and so far I've just pdf-render contents if I really want to save it beside a link/archive.ph and co.
I've tried for the web part archivy (pythonic web-app) and zotero, but they do not works like I want so I've dropped them and so far I've just pdf-render contents if I really want to save it beside a link/archive.ph and co.
I've long pondered on the idea of a unified, indexed database of all my personal data, which can also be stored on multiple removable drives (not your storage, not your data).
I've started work multiple times, but the scale of work required just scares me away.
Ideally, I'd like to see a timeline of everything that happened in my digital life - from photos, messages, health data, e-mails, even web sites I've visited.
For documents and files it should offer full text search and semantic search, etc.
The reason I want to have all my data in one index is not just for backup. As AI keeps evolving, I'll be able to use this data to train ever more capable AIs, which could eventually know pretty much everything about me and be able to sound like myself or even emulate me :).
Looking far into the future, this thing could eventually outlive me ...
I've started work multiple times, but the scale of work required just scares me away.
Ideally, I'd like to see a timeline of everything that happened in my digital life - from photos, messages, health data, e-mails, even web sites I've visited.
For documents and files it should offer full text search and semantic search, etc.
The reason I want to have all my data in one index is not just for backup. As AI keeps evolving, I'll be able to use this data to train ever more capable AIs, which could eventually know pretty much everything about me and be able to sound like myself or even emulate me :).
Looking far into the future, this thing could eventually outlive me ...
I have many files and digital content scattered everywhere. Some files and texts in my email accounts (personal and work). Some in Storage services such as Dropbox, GDrive, ... and some in messaging apps. There is also the photo gallery on my phone. These are photos, videos, text, pdfs, ... . The challenge comes in when I'm trying to find a specific file or info. I often find myself asking questions like:
- Where's the photo of me wearing a cap?
- Where are all my invoices from before 2023?
- Did my friend ever text me his phone number somewhere?
- ...
I'd really appreciate any advice or recommendations for tools or methods you've used that worked for you.