JasonPunyon

Thanks for taking it for a spin! I'm working on why this is slow now.

You may remember a carbon copy of this event from a year ago. https://meta.stackexchange.com/questions/389922/june-2023-da...

Discussion from then https://news.ycombinator.com/item?id=36257523

JasonPunyon

2 年前議論

If anyone wants their data back in a way they can use it, it's right here https://seqlite.puny.engineering

And I'd be remiss if I didn't point out that their trade dress is MIT licensed. https://stackoverflow.design

Have fun.

JasonPunyon

2 年前議論

Thanks so much, that’s really kind.

JasonPunyon

2 年前議論

Good catch! Fix going up now.

JasonPunyon

2 年前議論

Nice post!

Yep, my process is similar. It goes...

  - decompress (users|posts)  
  - split into batches of 10,000  
  - xsltproc the batch into sql statements  
  - pipe the batches of statements into sqlite in parallel using flocks for coordination

On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.

JasonPunyon

2 年前議論

This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.

JasonPunyon

3 年前議論

I mean fine, write an article that overweights recent events to weave a tale of the decline of Stack Exchange...but don't leave out the moderator strike, the data dump fiasco, or the CEO that simply doesn't get it. Jon Ericson's writing on the issue is much more informative. https://jlericson.com/

JasonPunyon

3 年前議論

https://stackoverflow.blog/2023/05/10/a-message-from-prashan...

JasonPunyon

投稿

Why is the total number of questions on Stack Overflow shrinking lately?

Stack Overflow has stopped publishing data dumps to the Internet Archive

Farey Numbers and Linked Lists

SEqlite – Minimal Stack Exchange Data Dump in SQLite Format

SEqlite: Minimal Stack Exchange Data Dump in SQLite Format

June 2023 Data Dump is missing

コメント