Why is modern data architecture so confusing? And what made sense for me(exasol.com)
exasol.com
Why is modern data architecture so confusing? And what made sense for me
https://www.exasol.com/hub/data-warehouse/architecture/
10 comments
yup, it's an ad in disguise.
Exasol accelerates your queries by up to 6969x btw in case you missed it
Real medium and large companies are so much messier. Almost guaranteed to have different iterations of each architecture and multiple competing architectures all running in parallel, with divided siloed and opposing ownership and perverse incentives and all the rest. Show me the spaghetti dataflow chart of an org and I will reverse-engineer the history of power struggles, resume-engineering and fads and failures that created it :)
Hilarious how true this can be, at some point I worked at a place that had three different competing setups for data workflows, with completely different stacks in all the possible ways: different programming languages, data stores, pipeline orchestrators, etc.
An absolute mess of technologies that no single person could make sense, backfilling when something went wrong could need 5-10 people to coordinate.
The running joke was that the data engineering department was trying to compete with the frontend devs on how fast they could throw a whole architecture out for a new fad.
An absolute mess of technologies that no single person could make sense, backfilling when something went wrong could need 5-10 people to coordinate.
The running joke was that the data engineering department was trying to compete with the frontend devs on how fast they could throw a whole architecture out for a new fad.
I’m a data engineering student who recently decided to shift from a non-tech role into tech, and honestly, it’s been a bit overwhelming at times. This guide I found really helped me bridge the gap between all the “bookish” theory I’m studying and how things actually work in the real world.
For example, earlier this semester I was learning about the classic three-tier architecture (moving data from source systems → staging area → warehouse). Sounds neat in theory, but when you actually start looking into modern setups with data lakes, real-time streaming, and hybrid cloud environments, it gets messy real quick.
I’ve tried YouTube and random online courses before, but the problem is they’re often either too shallow or too scattered. Having a sort of one-stop resource that explains concepts while aligning with what I’m studying and what I see at work makes it so much easier to connect the dots.
Sharing here in case it helps someone else who’s just starting their data journey and wants to understand data architecture in a simpler, practical way.
I’ve tried YouTube and random online courses before, but the problem is they’re often either too shallow or too scattered. Having a sort of one-stop resource that explains concepts while aligning with what I’m studying and what I see at work makes it so much easier to connect the dots.
Sharing here in case it helps someone else who’s just starting their data journey and wants to understand data architecture in a simpler, practical way.
If you put ETL and ELT in the same layer you have missed the essence of data platform architecture schools in the last few years. DW is ETL. Data lake is ELT. Then you mix and match (e.g. lakehouse etc.) The distinction between transformation post or ante ingestion is the major thing to drill into. The next one to master is streaming versus batch and after those you start hitting interesting problems like orchestration, snapshots and consistency layers. Not too complex a domain, but it requires some practical requirements to have to find these things out.
It's an ad / a SEO blog thing to drive people into the maws of whatever it is they're selling.
I don't feel intellectuelly stimulated reading this.
I don't feel intellectuelly stimulated reading this.
The article lost me after reading the first paragraphs. It just seems too academic.
I have heard exasol is a very performant database but using closed software can be a risk, I would rather deploy open source software.
I have heard exasol is a very performant database but using closed software can be a risk, I would rather deploy open source software.
There’s nothing academic about this, it’s an ad.
As an academic, that hurts. Academic good; ad bad.
As an academic, that hurts. Academic good; ad bad.
Now here's the same user's first comment, posted a few weeks ago:
[begins]
That’s a fair point—DuckDB’s lightweight design and intuitive UX are big reasons it’s gained traction, especially for analytics on the desktop or in embedded scenarios. But when it comes to “primetime” in the sense of enterprise-grade analytics—think massive concurrency, complex workloads, and scaling across distributed environments— Exasol I see as one of the solution.
DuckDB is fantastic for local analytics and prototyping, but when your needs move into enterprise territory—where performance, reliability, and manageability at scale become critical.
[ends]
Doesn't read quite so much like "overwhelmed previously-non-technical engineering student who'd be relieved to find some explanation of how things work in the real world", does it?
And, astonishingly, that comment was on ... a post from the Exasol blog, just like this one. Which had a number of positive comments from new accounts (another user even remarked on it).
Add to that the very LLMish feel of said user's comments (they made three on the previous Exasol post, all responding to others. Their openings: "Absolutely!", "That's a fair point—", and "Totally agree—") and the fact that one of the more transparently-astroturfing other comments also looks like it was written by an LLM, and the fact that the three HN posts this user has interacted with are (1) this one which they posted, (2) a previous instance of posting the same article, and (3) the aforementioned previous Exasol blog post ... and something definitely feels fishy to me.