Show HN: Sqlite3-dump - a fast SQLite to CSV and parquet(github.com)
github.com
Show HN: Sqlite3-dump - a fast SQLite to CSV and parquet
https://github.com/i64/sqlite3-dump
a fast zero-copy sqlite implementation only for dumping the contents of a sqlite3 file
4 comments
Thanks for sharing!
If converting SQLite to CSV/parquet is a one-off task for you, you can also import SQLite files (among the other 50+ formats for tabular data) and export to multiple supported formats (CSV and parquet included) using Datagrok (https://datagrok.ai/). Just drag-and-drop the sqlite file, then use the "export" button on top. Everything happens inside the browser (so there is a limit on the max size of the dataset, probably around 1GB).
Disclaimer - I'm one of the developers of Datagrok :)
If converting SQLite to CSV/parquet is a one-off task for you, you can also import SQLite files (among the other 50+ formats for tabular data) and export to multiple supported formats (CSV and parquet included) using Datagrok (https://datagrok.ai/). Just drag-and-drop the sqlite file, then use the "export" button on top. Everything happens inside the browser (so there is a limit on the max size of the dataset, probably around 1GB).
Disclaimer - I'm one of the developers of Datagrok :)
$ duckdb f.db -c 'COPY table1 TO table1.csv;COPY table1 TO table1.parquet;'
on my machine that i did the basic run, the one in the link is way more faster.
``` $ time ./duckdb_cli-linux-amd64 ./basic_batched.db -c "COPY user TO 'user.csv'" 100% (00:00:20.55 elapsed)
real 0m24.162s user 0m22.505s sys 0m1.988s ```
``` $ time ./duckdb_cli-linux-amd64 ./basic_batched.db -c "COPY user TO 'user.parquet'" 100% (00:00:17.11 elapsed)
real 0m20.970s user 0m19.347s sys 0m1.841s ```
``` $ time cargo run --bin parquet --release -- basic_batched.db user -o out.parquet Finished `release` profile [optimized] target(s) in 0.11s Running `target/release/parquet basic_batched.db user -o out.parquet` Database opened in 14.828µs
SQLite to Parquet Exporter ========================== Database: basic_batched.db Page size: 4096 bytes Text encoding: Utf8 Output: out.parquet Batch size: 10000
Exporting table: user Output file: out.parquet
real 0m6.052s user 0m10.455s sys 0m0.537s ```
``` $ time cargo run --bin csv --release -- basic_batched.db -t user -o out.csv Finished `release` profile [optimized] target(s) in 0.03s Running `target/release/csv basic_batched.db -t user -o out.csv`
real 0m6.453s user 0m5.252s sys 0m1.196s ```
``` $ time ./duckdb_cli-linux-amd64 ./basic_batched.db -c "COPY user TO 'user.csv'" 100% (00:00:20.55 elapsed)
real 0m24.162s user 0m22.505s sys 0m1.988s ```
``` $ time ./duckdb_cli-linux-amd64 ./basic_batched.db -c "COPY user TO 'user.parquet'" 100% (00:00:17.11 elapsed)
real 0m20.970s user 0m19.347s sys 0m1.841s ```
``` $ time cargo run --bin parquet --release -- basic_batched.db user -o out.parquet Finished `release` profile [optimized] target(s) in 0.11s Running `target/release/parquet basic_batched.db user -o out.parquet` Database opened in 14.828µs
SQLite to Parquet Exporter ========================== Database: basic_batched.db Page size: 4096 bytes Text encoding: Utf8 Output: out.parquet Batch size: 10000
Exporting table: user Output file: out.parquet
user: 100000000 rows (310.01 MB) - 5.85s (17095636 rows/sec)
real 0m6.052s user 0m10.455s sys 0m0.537s ```
``` $ time cargo run --bin csv --release -- basic_batched.db -t user -o out.csv Finished `release` profile [optimized] target(s) in 0.03s Running `target/release/csv basic_batched.db -t user -o out.csv`
real 0m6.453s user 0m5.252s sys 0m1.196s ```
Nice! Thank you