JasonPunyon

no profile record

comments

Thanks for taking it for a spin! I'm working on why this is slow now.

You may remember a carbon copy of this event from a year ago. https://meta.stackexchange.com/questions/389922/june-2023-da...

Discussion from then https://news.ycombinator.com/item?id=36257523

JasonPunyon

2년 전discuss

If anyone wants their data back in a way they can use it, it's right here https://seqlite.puny.engineering

And I'd be remiss if I didn't point out that their trade dress is MIT licensed. https://stackoverflow.design

Have fun.

JasonPunyon

2년 전discuss

Thanks so much, that’s really kind.

JasonPunyon

2년 전discuss

Good catch! Fix going up now.

JasonPunyon

2년 전discuss

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination

On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

JasonPunyon

2년 전discuss

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

JasonPunyon

Submissions

Why is the total number of questions on Stack Overflow shrinking lately?

Stack Overflow has stopped publishing data dumps to the Internet Archive

Farey Numbers and Linked Lists

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

SEqlite: Minimal Stack Exchange Data Dump in SQLite Format

comments