JasonPunyon

в прошлом годуdiscuss

Thanks for taking it for a spin! I'm working on why this is slow now.

2 года назадdiscuss

You may remember a carbon copy of this event from a year ago. https://meta.stackexchange.com/questions/389922/june-2023-da...

Discussion from then https://news.ycombinator.com/item?id=36257523

JasonPunyon

2 года назадdiscuss

If anyone wants their data back in a way they can use it, it's right here https://seqlite.puny.engineering

And I'd be remiss if I didn't point out that their trade dress is MIT licensed. https://stackoverflow.design

Have fun.

JasonPunyon

2 года назадdiscuss

Thanks so much, that’s really kind.

JasonPunyon

2 года назадdiscuss

Good catch! Fix going up now.

JasonPunyon

2 года назадdiscuss

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination

On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

JasonPunyon

2 года назадdiscuss

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

JasonPunyon

3 года назадdiscuss

I mean fine, write an article that overweights recent events to weave a tale of the decline of Stack Exchange...but don't leave out the moderator strike, the data dump fiasco, or the CEO that simply doesn't get it. Jon Ericson's writing on the issue is much more informative. https://jlericson.com/

JasonPunyon

3 года назадdiscuss

https://stackoverflow.blog/2023/05/10/a-message-from-prashan...

JasonPunyon

Submissions

Why is the total number of questions on Stack Overflow shrinking lately?

Stack Overflow has stopped publishing data dumps to the Internet Archive

Farey Numbers and Linked Lists

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

SEqlite: Minimal Stack Exchange Data Dump in SQLite Format

June 2023 Data Dump is missing

comments