HackerTrans
TopNewTrendsCommentsPastAskShowJobs

peterdstallion

no profile record

Submissions

DuckDB can be 5x faster than Spark at 500M record files

blog.dataexpert.io
4 points·by peterdstallion·vor 10 Monaten·1 comments

A New Wave: From Big Data to Small Data

fabi.ai
2 points·by peterdstallion·vor 10 Monaten·0 comments

comments

peterdstallion
·vor 10 Monaten·discuss
I am a paying subscriber to Gemini, Claude and OpenAI.

I don't know if it's me, but over the last few weeks I've got to the conclusion ChatGPT is very strongly leading the race. Every answer it gives me is better - it's more concise and more informative.

I look forward to testing this further, but out of the few runs I just did after reading about this - it isn't looking much better
peterdstallion
·vor 10 Monaten·discuss
This test was done on a small dev laptop with 16GB of RAM, scanning a 500M row (record) 23GB Parquet file. DuckDB proved to be 5x faster.

A bit of an obvious one - small data tech is faster at small data. It serves more of a lower bound reminder of what "small data" is nowadays.

The article rightly starts with:

> Processing power on laptops has increased dramatically over the last twenty years. This allows single laptops to accomplish what we needed multi-node Spark clusters to do ten years ago.
peterdstallion
·vor 10 Monaten·discuss
The technological progress is astonishing. But so is the societal degradation that came with it.
peterdstallion
·vor 10 Monaten·discuss
It's my go-to keyboard. I can't recommend it enough. My only regret is I didn't buy it earlier.

It's definitely way more ergonomic - I believe that's incredibly important if you're going to be spending decades in front of a screen. At the very least don't make it easy to be hunched over.

Last but not least - coming from a non-mechanical keyboard before - the switch is super satisfying.
peterdstallion
·vor 2 Jahren·discuss
Nope, they seem to have their own catalog library that translates it: https://github.com/awslabs/s3-tables-catalog