HackerTrans
トップ新着トレンドコメント過去質問紹介求人

joistef

no profile record

コメント

joistef
·6 か月前·議論
For Drive and Photos, Relica can do this. Full disclosure: I develop the software.

It uses restic under the hood (so you get deduplication, encryption, immutable snapshots) and rclone for cloud connectivity. Backs up to your NAS or multiple destinations, runs on a schedule, GUI instead of cron jobs and scripts. $5/month for the software, storage is yours.

Gmail is harder. It's not a filesystem so restic/rclone-based tools don't handle it. Your best bet there is probably Google Takeout on a schedule (annoying but works), or something like gmvault if you're comfortable with Python scripts.

The lockout fear is real. One option: backup Google Drive → your NAS → a second cloud provider that isn't Google (Or Relica Cloud for managed redundancy). Relica can chain those so you upload once and it replicates. Check relicabackup.com
joistef
·6 か月前·議論
A few things that might help:

The Delta Lake suggestion is tempting but I don't think it actually solves your core problem - you still need to rewrite rows to populate a new column with non-default values. Their deletion vectors help with row-level updates, not column additions.

The separate table approach is probably your best bet, but I'd avoid row_number joins. If you can do a one-time rewrite to add a content hash or deterministic row ID, your annotation table becomes (row_id, score) and the join is trivial. Yes, it's a one-time cost, but it pays off every time you add a new annotation column.

If a rewrite is truly off the table, the overlay pattern works: keep annotations in separate parquet files keyed by (file_path, row_group_index, row_offset_within_group). DuckDB handles this reasonably well. The ergonomics aren't great but it's honest about the tradeoff you're making.

The parquet metadata idea is clever but metadata is size-limited and not designed for per-row data - you'd be fighting the format.

What query engine are you using? That might change the calculus.