Show HN: Csvdb – Git-friendly CSV directories that convert to SQLite or DuckDB(github.com)
github.com
Show HN: Csvdb – Git-friendly CSV directories that convert to SQLite or DuckDB
https://github.com/jeff-gorelick/csvdb
2 comments
I admire what you are doing. It obviously serves you well and you hope it helps others.
I'm familiar with how to import CSV data with SQLite and I could probably learn how to do it with DuckDB with ease. Your tool introduces a new and rather shallow abstraction layer with commands that work for both databases.
The world needs tools that provide a great deal of utility, are trustworthy, and keep getting more reliable and capable. I'm not sure that yours will qualify.
I'm familiar with how to import CSV data with SQLite and I could probably learn how to do it with DuckDB with ease. Your tool introduces a new and rather shallow abstraction layer with commands that work for both databases.
The world needs tools that provide a great deal of utility, are trustworthy, and keep getting more reliable and capable. I'm not sure that yours will qualify.
csvdb converts between a directory of CSV files and SQLite/DuckDB databases. The CSV side is the source of truth you commit to git. The database side is what you query.
The key design decision is deterministic output. Rows are sorted by primary key, so identical data always produces identical CSV files. This means git diffs show only actual data changes, not row reordering noise.
A few details that took some thought:
- NULL handling: CSV has no native NULL. By default csvdb uses \N (the PostgreSQL convention) to distinguish NULL from empty string. Roundtrips are lossless.
- Format-independent checksums: csvdb checksum produces the same SHA-256 hash whether the input is SQLite, DuckDB, or a csvdb directory. Useful for verifying conversions.
- Schema preservation: indexes and views survive the roundtrip, not just tables.
Written in Rust. Beta quality -- the file format may still change.
https://github.com/jeff-gorelick/csvdb