JasonPunyon

no profile record

Submissions

comments

anno scorsodiscuss

Thanks for taking it for a spin! I'm working on why this is slow now.

You may remember a carbon copy of this event from a year ago. https://meta.stackexchange.com/questions/389922/june-2023-da...

Discussion from then https://news.ycombinator.com/item?id=36257523

JasonPunyon

2 anni fadiscuss

If anyone wants their data back in a way they can use it, it's right here https://seqlite.puny.engineering

And I'd be remiss if I didn't point out that their trade dress is MIT licensed. https://stackoverflow.design

Have fun.

JasonPunyon

2 anni fadiscuss

Thanks so much, that’s really kind.

JasonPunyon

2 anni fadiscuss

Good catch! Fix going up now.

JasonPunyon

2 anni fadiscuss

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination

On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

JasonPunyon

2 anni fadiscuss

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

JasonPunyon

Submissions

Why is the total number of questions on Stack Overflow shrinking lately?

Stack Overflow has stopped publishing data dumps to the Internet Archive

Farey Numbers and Linked Lists

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

SEqlite: Minimal Stack Exchange Data Dump in SQLite Format

comments