OK. I've read it a few times and still don't understand. Where is the distributed part? You store data in a single transaction into postgres. What/who is notifying the message queue?
I'm wondering. Just wondering? Will they ever support multiple storage engines like MariaDB? Having a storage engine that support OLTP or OLAP or append-only would be cool. I totally understand if they don't want to do that.
I just came out of an initial interview for a rather senior enterprise position. It was bad. It was bad in a sense that they really didn't know what to do. The senior manager (EVP level person) asked me about LLM for code generation. I told them about my aws kiro-cli experience. They literally asked me to sit down and show them how they can do it. I'm pretty sure they want me to back for another round.
This whole thing reminds me of when I was in school, showing old timers who to use MS Office and VBA.
We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.
When you say compatibility issues, you mean they are more problematic or less?
It’s pretty common to read Parquet into Arrow for transport.
I'm confused by this. Are you referring to Arrow Flight RPC? Or are you saying distributed analytic engine use arrow to transport parquet between queries?
I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.
You'll be amazed on what the new breed of engineers are using Redis for. I personally saw an entire backend database using Redis with RDB+AOF on. If you redis-cli into the server, you can't understand anything because you need to know the schema to make sense of it all.
I lost all my college projects due to source forge bullshit decades ago. And old pictures from digital/film cameras. Right now, I run both google/apple photos so I have 2 backups of my pics and videos.
Honestly, after 20 some years in technology, I don't think it's possible to back up everything unless you are willing to pay and constantly work at it.
I've seen this type of advice a few times now. Now I'm not a database expert by any stretch of imagination, but I have yet to see UUID as primary key in any of the systems I've touched.
Are there valid reasons to use UUID (assuming correctly) for primary key? I know systems have incorrectly expose primary key to the public, but assuming that's not the concern. Why use UUID over big-int?
Yes, we have to in order to use a lot of the features. The core issue for us is really the way Django assumes code represents database state. In normal webapp where the application has full control of the database, that's a good idea. But our databases are overloaded for simple transactions, analytics, users managements, jobs and AI. Business uses the databases in various ways such as Power BI, Aquastudio, etc.. Django app is actually a tiny part of the database. As you can imagine, we duck tape the heck out of the databases, and Django goes bonkers when things aren't matching.