Dismantling ELT: The Case for Graphs, Not Silos(jack-vanlightly.com)
jack-vanlightly.com
Dismantling ELT: The Case for Graphs, Not Silos
https://jack-vanlightly.com/blog/2024/11/26/dismantling-elt-the-case-for-graphs-not-silos
15 comments
> provide “data APIs” to data teams
I feel like I'm reading part of Data Mesh again.
I feel like I'm reading part of Data Mesh again.
Hah. That is a good reference. Never seen a concept boosted so much.
As a 20 year veteran the biggest opportunity in new data space is potential for LLM unstructured data parsing and intelligent api integrations to break silos
This article assumes the business has a logical progression for its plans in question and then makes logical and somewhat rational decisions that lead to a operational estate vs a data estate when in realty it’s about budgets and pet projects and c suite whims and frank got fired after he brought in the last data engineering team who ended up building a data lake with a dashboard that one VP uses and know one else understands
Something gets built =>based on a limited dev budget it leads to something else getting built on top of it maybe if a consultant sells it well to an exec - ultimately it all becomes a complex spaghetti house of cards… or “data mesh”
This article assumes the business has a logical progression for its plans in question and then makes logical and somewhat rational decisions that lead to a operational estate vs a data estate when in realty it’s about budgets and pet projects and c suite whims and frank got fired after he brought in the last data engineering team who ended up building a data lake with a dashboard that one VP uses and know one else understands
Something gets built =>based on a limited dev budget it leads to something else getting built on top of it maybe if a consultant sells it well to an exec - ultimately it all becomes a complex spaghetti house of cards… or “data mesh”
I missed the part where there was an actionable takeaway about how I as a practitioner am supposed to differently.
I hate to say it, but good documentation is the key here. Visualizing data as interconnected nodes breaks down silos, aligns teams and makes it easier to build reusable, loosely-coupled systems.
It's a great theoretical point.
> The tools, the practices, all built around the premise that software and data teams don’t work closely together.
In particular, the various teams, even the same team with itself, end up being separated along a timeline.
Information Technology ends up being the embarrassment we all face that our data are rarely, if ever, showing up dressed for work.
> The tools, the practices, all built around the premise that software and data teams don’t work closely together.
In particular, the various teams, even the same team with itself, end up being separated along a timeline.
Information Technology ends up being the embarrassment we all face that our data are rarely, if ever, showing up dressed for work.
Conway’s law works both ways: if you want a system to be designed in a certain way, you create an organization who is building it in that way.
So, if you want a data mesh, build a mesh org. Not the easiest job.
So, if you want a data mesh, build a mesh org. Not the easiest job.
Since the article uses lots of big words, phrases and memes (when a few simple words would suffice), and assumes you know it already, Conway's law is simply that your technical architecture will reflect your social/organizational architecture.
"Big words, phrases and memes"? You got this distressed about an article using industry standard terminology for its target audience and assuming that you'd read (or would read) the post it linked to on Conway's law?
I flat out disagree with the premise of that article. I think it is written from the perspective of an engineer that is primarily in the operational domain and only see the value of analytics domain as a supplier of data to the operational side.
The the truth is there are many many uses of data that isn’t transactional/operational and frankly have no need to be real time at all. So all the building and maintaining of graph network to deliver near real time data is just an unnecessarily over-engineered and unnecessary and costly complexity.
If your operational use cases are indeed near real time, please find some optimal way to do that. But don’t you dare assume many other data use cases need that
The the truth is there are many many uses of data that isn’t transactional/operational and frankly have no need to be real time at all. So all the building and maintaining of graph network to deliver near real time data is just an unnecessarily over-engineered and unnecessary and costly complexity.
If your operational use cases are indeed near real time, please find some optimal way to do that. But don’t you dare assume many other data use cases need that
Yeah, 100 people can literally implement what he is saying in 100 different way.
reminds me nifi plus kafka
Honestly the whole landscape seems broken and unproductive at this point.
Countless vendors, platforms, cloud environments, industry/technical jargon - all with different pricing models, SLAs, tooling, etc etc.
Getting anything usable is a challenge and most orgs spin in a never ending cycle of data integration/normalization work that produces little business value.
My advice to teams now is simplify, reduce, streamline - get to the kernel of what you think you need and protect it all costs. Most of the shiny new objects being pitched as silver bullets are just ways for other people to make money off your margin.