Ask HN: What do you use for a personal database?
45 comments
SQLite with the open source toolchain I've been building over the past five years:
https://datasette.io as the interface for running queries against (and visualizing) my data.
https://sqlite-utils.datasette.io/ as a set of tools for creating and modifying my databases (inserting JSON or CSV data, enabling full text search text)
https://dogsheep.github.io as a suite of tools for importing my personal data - see also this talk I gave about that project: https://simonwillison.net/2020/Nov/14/personal-data-warehous...
I'm also using GitHub issues and the new GitHub projects for a bunch of personal data - my notes and TODO items specifically - much of which I can then suck into my own SQLite databases using https://datasette.io/tools/github-to-sqlite
https://datasette.io as the interface for running queries against (and visualizing) my data.
https://sqlite-utils.datasette.io/ as a set of tools for creating and modifying my databases (inserting JSON or CSV data, enabling full text search text)
https://dogsheep.github.io as a suite of tools for importing my personal data - see also this talk I gave about that project: https://simonwillison.net/2020/Nov/14/personal-data-warehous...
I'm also using GitHub issues and the new GitHub projects for a bunch of personal data - my notes and TODO items specifically - much of which I can then suck into my own SQLite databases using https://datasette.io/tools/github-to-sqlite
Yup, I’m using mostly SQLite and tools I wrote using it these days, at least for personal data management.
I use this simple tool for most my notes now:
https://github.com/codazoda/nolific
For temporary notes I use my Ponder web app (it just uses the browsers DB) and is partially inspired by old AlphaSmart word processing machines:
https://github.com/codazoda/ponder
I use this simple tool for most my notes now:
https://github.com/codazoda/nolific
For temporary notes I use my Ponder web app (it just uses the browsers DB) and is partially inspired by old AlphaSmart word processing machines:
https://github.com/codazoda/ponder
I'm not actually sure what a personal database is. If I'm writing software, I love SQLite, but I don't have any software developed truly for personal use, although my automation server project does run a porch camera.
But even in software it's not common that I need a DB, since I do a lot of embedded. Most software I write uses text, with an eye to version control and SyncThing sync.
For personal information management I use Obsidian/Syncthing to take relatively unstructured notes, combined with Google assistant for reminders. Various bits of non critical data lives in assorted apps and clouds, like Garmin connect and Google location history.
On the very rare occasion I have any personal use for managing structured data, I use Google Docs or LibreOffice Calc, but that doesn't come up a lot.
More than what tools people use to manage data, I'm curious about the data itself. I always hear about programmers using Awk and sed, and all kinds of talk about processing data, but I'm not sure what people actually do with structured personal data.
But even in software it's not common that I need a DB, since I do a lot of embedded. Most software I write uses text, with an eye to version control and SyncThing sync.
For personal information management I use Obsidian/Syncthing to take relatively unstructured notes, combined with Google assistant for reminders. Various bits of non critical data lives in assorted apps and clouds, like Garmin connect and Google location history.
On the very rare occasion I have any personal use for managing structured data, I use Google Docs or LibreOffice Calc, but that doesn't come up a lot.
More than what tools people use to manage data, I'm curious about the data itself. I always hear about programmers using Awk and sed, and all kinds of talk about processing data, but I'm not sure what people actually do with structured personal data.
If you mean databases as in a "Personal Information System" (notes, tasks, catalogs, etc.), I use Grist and Tana.
Grist [1] is an "augmented" but still "classic" relational DB builder. It is (kind of) open-source, backed by SQLite, and you can self-host it. I like it because it is user-friendly while also being highly programmable (Python formulas, widget grid) and "data-friendly" (nice filters, Merge imports...)
Tana [2] is a less classic kind of DB. The main view is an "advanced" outliner (like Roam). But every line of text could also act as an "universal Form" and a "node" in a semantic graph DB. Wherever and whenever you type, you can turn your unstructured, freeform blocks of text into "as you go" structured data. This works well for Evergreen-Notes [3], Tasks, Contacts...
A bonus : for any kind of references (and not just citations), Zotero is very flexible. As local/opensource personal DBs go, it is a solid bookmark manager and pdfs-annotations store.
[1] https://www.getgrist.com/ [2] https://tana.inc/ [3] https://maggieappleton.com/evergreens
Grist [1] is an "augmented" but still "classic" relational DB builder. It is (kind of) open-source, backed by SQLite, and you can self-host it. I like it because it is user-friendly while also being highly programmable (Python formulas, widget grid) and "data-friendly" (nice filters, Merge imports...)
Tana [2] is a less classic kind of DB. The main view is an "advanced" outliner (like Roam). But every line of text could also act as an "universal Form" and a "node" in a semantic graph DB. Wherever and whenever you type, you can turn your unstructured, freeform blocks of text into "as you go" structured data. This works well for Evergreen-Notes [3], Tasks, Contacts...
A bonus : for any kind of references (and not just citations), Zotero is very flexible. As local/opensource personal DBs go, it is a solid bookmark manager and pdfs-annotations store.
[1] https://www.getgrist.com/ [2] https://tana.inc/ [3] https://maggieappleton.com/evergreens
Obsidian.md if I'm to take 'personal database' to mean how I store my random, occasionally interlinked information. So I suppose that's effectively the file system with some nice linking in markdown over the top of it.
Obsidian is by far the best solution and the one I've actually stuck to the most. Markdown, great plugins and all local files.
HackMD is great, and the fork of it called HedgeDoc has really taken it somewhere even greater.
not too sound dismissive, but one of the great things about obsidian is their ecosystem of plugins. I don't think this really rivals obsidian
Same. No other PKM ever clicked or worked for me.
My data - base (not a DB) is org-mode, org-roam managed, org-attached files. Actually org-roam use SQLite under the wood but I can say I do not use it directly...
In DB terms IF I have to write some personal software outside Emacs/org-mode I'll choose the simplest and lightest tools I can found, case by case, however so far I have encountered no personal cases...
In DB terms IF I have to write some personal software outside Emacs/org-mode I'll choose the simplest and lightest tools I can found, case by case, however so far I have encountered no personal cases...
Like others, not entirely sure what you mean here. May all-purpose "DB" or info store is OneNote, but I am making an effort to move as much as possible to TiddlyWiki this coming year.
TW is relatively future-proof, as it's just a single HTML file that encapsulates all of the content and app in a single file that can be moved anywhere or used in any browser, but it gives you powerful structuring, storage, and linking capabilities inside it. (I almost never use flat text files for reference any more except for the simplest things, or if a machine needs to read it...) It gives you the best of what your filesystem can do for you (and filesystems are DBs, especially these days) combined with browser/wiki capabilities, and you can even hack the JS if you're into that sort of thing. My perfect solution would munge together the capabilities of TiddlyWiki and Jupyter notebooks into one awesome live, flexible, portable, programmable metacontainer.
(I started on this with the first version of TW 20 years ago, combining test programs, data, and analysis/viz tools for sensor data collection into what I called "testacles" - test data receptacles capturing everything about a real-world test with IoT data, setup, docs, notes, code, data, etc. (I was building the world's first embedded IoT sensors at the time...))
BTW, thinking of adding BOOX's new Tab Ultra into my workflow - I've used pen on various Surface Pros for the last eight years (how OneNote became my go-to), but like the idea of not worrying about battery life. The Tab Ultra can run Android apps and a browser, so I should be able to use TW even if it's own pen note capture is too proprietary to be useful outside the device... we'll see.
TW is relatively future-proof, as it's just a single HTML file that encapsulates all of the content and app in a single file that can be moved anywhere or used in any browser, but it gives you powerful structuring, storage, and linking capabilities inside it. (I almost never use flat text files for reference any more except for the simplest things, or if a machine needs to read it...) It gives you the best of what your filesystem can do for you (and filesystems are DBs, especially these days) combined with browser/wiki capabilities, and you can even hack the JS if you're into that sort of thing. My perfect solution would munge together the capabilities of TiddlyWiki and Jupyter notebooks into one awesome live, flexible, portable, programmable metacontainer.
(I started on this with the first version of TW 20 years ago, combining test programs, data, and analysis/viz tools for sensor data collection into what I called "testacles" - test data receptacles capturing everything about a real-world test with IoT data, setup, docs, notes, code, data, etc. (I was building the world's first embedded IoT sensors at the time...))
BTW, thinking of adding BOOX's new Tab Ultra into my workflow - I've used pen on various Surface Pros for the last eight years (how OneNote became my go-to), but like the idea of not worrying about battery life. The Tab Ultra can run Android apps and a browser, so I should be able to use TW even if it's own pen note capture is too proprietary to be useful outside the device... we'll see.
A postgres database that is managed by a custom app that I wrote in Elixir / Phoenix / LiveView. Hosted for free on fly.io.
it stores:
- My TODO list (items -> entries -> checkboxes)
- My credit card transactions (scraped from alert emails with gmail api)
- A running log with miles/times/routes
i've implemented misc features to organize everything like content tagging, custom markdown rendering, etc.
it stores:
- My TODO list (items -> entries -> checkboxes)
- My credit card transactions (scraped from alert emails with gmail api)
- A running log with miles/times/routes
i've implemented misc features to organize everything like content tagging, custom markdown rendering, etc.
> I was curious what the members of HN use when they need a personal database.
Whichever seems appropriate for the given task I'm accomplishing.
I have some that are in PostgreSQL. Others in SQLite. A few that are just files organized in a filesystem directory hierarchy. There is really no "one size fits all" solution.
Whichever seems appropriate for the given task I'm accomplishing.
I have some that are in PostgreSQL. Others in SQLite. A few that are just files organized in a filesystem directory hierarchy. There is really no "one size fits all" solution.
“Personal database” as in database for personal software projects? Or something else like a “document”?
Outside of software dev I use a bit of Facebook to store social contacts, iOS contacts for phone numbers, my gmail inbox as the only source of email addresses… I use Google Keep to share a grocery shopping list.
I avoid like the plague anything that’s a file, script, database etc. I don’t take notes, don’t use web browser bookmarks or in any other way try to organize/persist/index data around me. I’m not 100% successful because I still self-host photos (hey Lightroom is SQLite!) but I’m getting there. Spotify is a godsend compared to organizing music files.
Outside of software dev I use a bit of Facebook to store social contacts, iOS contacts for phone numbers, my gmail inbox as the only source of email addresses… I use Google Keep to share a grocery shopping list.
I avoid like the plague anything that’s a file, script, database etc. I don’t take notes, don’t use web browser bookmarks or in any other way try to organize/persist/index data around me. I’m not 100% successful because I still self-host photos (hey Lightroom is SQLite!) but I’m getting there. Spotify is a godsend compared to organizing music files.
For my personal scripts that need databases, I mostly use SQLite. If I need to browse one of them I'll use TablePlus, but that's rare.
A couple of my older scripts still use JSON files on disk that are loaded into memory, which are actually perfectly fast for the purpose (fast enough that I see no perceptible difference between them and SQLite).
A couple of my older scripts still use JSON files on disk that are loaded into memory, which are actually perfectly fast for the purpose (fast enough that I see no perceptible difference between them and SQLite).
[deleted]
For most hacking projects I use arangodb. I kinda wish there was an OWL-supporting database that was straightforward to write apps in but I think there still isn’t. SQLite is a second in Python where file-oriented as opposed to client-server is acceptable. In Java I’d consider some other file-based db which is pure Java.
Try rdflib for Python. It can sort-of be like SQLite in ease of use. Anyway, I have used it for years for small RDF things.
I don't have need for a personal db, but I would likely choose MySQL since I know how to set up group replication, multiple writers, delayed replication, etc and none of it requires extra software or is particularly difficult--follow the instructions or RTM.
It also runs pretty well on smallish machine instances.
It also runs pretty well on smallish machine instances.
Another vote for SQLite until it needs to be accessed from multiple hosts, at which point I'd go to MySQL.
Why MySQL? I know it in and out, since I was a MySQL DBA for a few years. It doesn't surprise me anymore, and I can quickly get it to a point where it requires effectively no intervention on my side.
Why MySQL? I know it in and out, since I was a MySQL DBA for a few years. It doesn't surprise me anymore, and I can quickly get it to a point where it requires effectively no intervention on my side.
What is a personal database? What would I even store in it?
In any case, for files I use iCloud and for notes I use Obsidian.
In any case, for files I use iCloud and for notes I use Obsidian.
I like the question for how differently it gets interpreted;)
I personally would like to toy around with some graph database for knowledge management but haven’t found the time to get into it, yet.
Other than that, it’s dendron (and git).
I personally would like to toy around with some graph database for knowledge management but haven’t found the time to get into it, yet.
Other than that, it’s dendron (and git).
I use PostgreSQL in a container in my home lab. The backup is uploaded to GCP every morning 0200 with a 90 day retention policy.
PostgreSQL handles my finances, my database of quotes & collection of notes.
PostgreSQL handles my finances, my database of quotes & collection of notes.
Telegram Saved Messages, no joke.
I can send any file up to 2GB, any number of times. Can even set reminder messages. It's great because I don't have to go anywhere else to put things to remember.
I can send any file up to 2GB, any number of times. Can even set reminder messages. It's great because I don't have to go anywhere else to put things to remember.
Isn’t telegram likely infiltrated by the Russian state it is that FUD?
Total FUD. If anything, it was banned in Russia for years because Durov refused to share any data with the government.
for almost all my projects (from tracking personal expenses to building apps) I've been using Airtable. Yes I could deploy some db on fly.io or whatever, but Airtable comes w/ a great UX (e.g. can add stuff on my phone) and a great API, and fairly easy to "transition out of" if I need scale (which I've never needed).
It's slow as molasses to write and read though.
It's slow as molasses to write and read though.
What do you mean by "personal database?"
I used WikidPad to store my notes, etc. It used to be widely used, and cross platform... but that's broken lately.
Bleep. https://www.bleep.is
I use SQLite for smaller things (Think desktop applications) and MySQL for anything else
For personal projects I use Firestore. Performant and a very generous free tier.
Same here. I use Firebase/Firestore for MVPs and projects where I don't expect the data to be ultra relational in principle.
Firestore is bad at some things (e.g. try running a "where" filter query for 'date > X', but "orderBy" some field that isn't 'date').
Also obviously Firestore is very bad for any kind of text searching, especially searching for fuzzy strings.
Also Firestore gets expensive if you're expecting to do a ton of reads/writes in a day.
But for 80% ~ 90% of projects I swear by Firestore.
Firestore is bad at some things (e.g. try running a "where" filter query for 'date > X', but "orderBy" some field that isn't 'date').
Also obviously Firestore is very bad for any kind of text searching, especially searching for fuzzy strings.
Also Firestore gets expensive if you're expecting to do a ton of reads/writes in a day.
But for 80% ~ 90% of projects I swear by Firestore.
I try to use SQLite3 for everything I can, and Postgres if I need more.
Not sure why anyone would need a "personal database."
Or what would one store in it. Does it mean your own personal business? Like a to-do list? Use a spreadsheet.
Git repo with md files.
Sqlite + Django work nice too.
Evernote
Libreoffice Base
The file system.
Notion
If anyone would like to say what they use, and why they chose it, I would appreciate hearing about it.
Thanks in advance.