This is a strange feeling for me. Something new and exciting from AWS besides another mass layoff? What’s wrong with those people? It’s almost like they care or something. Definitely sus. If you’re like me, you’ve spent decades under the big thumb of that not-so-benevolent AWS Cloud.
Our lives and work are consumed by that company; every twitch they make sends ripple effects down the long terminal halls of bits and bytes, eventually breaking some 5-year-old boto3 call in Python running on a decrepit EC2 instance via cron.
It's a race to the bottom, as per usual, inside the inner tech sanctum where legions of software grifters burn tokens in dark terminals, sending up a sweet-smelling aroma to the Sardaukar licking up the costly sacrifice of the faithful.
I'm going to tell your mom. I’m going to do it. You can’t stop me. And you, Grandma, for that matter. I’m calling them right now, like now. You’re an addict; you have to stop. Someone has to stop you. Might as well be me.
It started out harmless, didn’t it?
But now things have changed, you’re hooked on The Vibes. They got their claws in you. Look at yourself. You should be ashamed. Sitting high on your Macbook Tower all these years, God’s programming gift to humankind, those LLMs came along, you were enamored, Gastown and all, those tasty token morsels.
I don’t know what it is about Apache Arrow, that GOAT of data engineering, that snuck in like a weasel through the backdoor, we all woke up one day and found out Arrow is the Atalas of the data world, holding up the systems we depend on and take for granted. One name you might, or might not have heard rattling around is “Apache Arrow Flight” or “Arrow Flight.”
I hadn’t thought about it much lately, but depending on your point of view, Kafka is either at the height of its rise or on a slow downward spiral. Maybe both? We do live in the age of abstraction; businesses at large seem to be less willing to pay hordes of Platform Engineers to babysit complex architecture.
If we had a dollar for every time someone came along to become the Apache Parquet killer, we would all be living on the side of a mountain tending to our alpacas. A boy can dream, can’t he?
Yeah, so … I’ve heard rumbling and mumblings about, here and there. But I had yet to try it out for myself. I trust nothing I can’t put my hands on. Something about being raised in the cornfields of the Midwest, always be skeptical of anything that seems like Black Magic.
It’s hard to find the bright, shining stars amid the doom and gloom the tech world seems to be floundering in. When the going gets tough, I like to remind myself that there are lots of new and exciting tools released in the last few years, most of which, when combined, have not been part of the great LLM training material, leaving some fun left to explore.
Apache Arrow entered the data scene quietly; for years, it languished in obscurity, unheard of and uncared for by the data community. Back in the olden days of 2022, which feels like another world, I was happily using and writing about Arrow as a data processing tool. A lot has changed since then, and Arrow has catapulted its way into everyday data engineering conversations.
Ok, Spark isn’t dead. Before you leave, I’m sorry for lying to you. Sorta. Kinda. Not really.
Undoubtedly Apache Spark has reached its zenith, shot like a rocket out of the Databricks barrel into the sky. The world is shifting though, even if ever so imperceptible.
First, open source Spark has been taken over, corralled and tammed by the corporate workings and money trains. It’s the fault of no one really, it’s the world we live in. Very few open source projects truly stand the test of time, uncorrupted by outside forces.
ypically, when the Harvard Business Review publishes something, especially techy, people tend to pay attention. Well, that is, unless it goes against the ultra-psyops-capitalism that drives most of the known world, in the form of extracting every useful drop of blood and life from the glassy-eyed masses that are too exhausted or addicted to the doom-scroll to look up for a minute.
For years, I've wondered at the intricate designs of MAKE files, venvs, poetry, binaries, uvs, yml, and many other such spells that engineers through the ages have deemed necessary to provide a consistent runtime environment. Project to project, company to company, it all changes.
I’ve been wanting to hit on this little topic for a few months, but spring has sprung, and the woods and river are irresistible in their calling to me. Also, if I’m going to be poor and homeless, living in a van down by the river because AI has taken my crappy job, then why should I even write anything about my destroyer?
Let’s keep moving and talk about the guiding principles of architectural design for the data platform. These principles have technical implications, but many are logical and theoretical and should inform our decisions when choosing tooling, frameworks, and architectural approaches.
Some of these principles overlap with later sections on the building blocks of a data platform, but that is simply because of their particular importance.
Building data platforms that actually work and provide value has to be grounded in the day-to-day realities of the messy data systems you will encounter in the real world.
Neither life nor the systems we will encounter exists in a vacuum. Every single data platform from the Fortune 500 company to the twelve-person startup is going to have a certain amount of …
Well, I had my moment, the tipping or boiling point, I guess you would call it. I’ve had enough, and, in a moment of fury, I ripped Polars from its Lambda throne and supplanted it with DuckDB. Been a long time coming.
When you first consider a data platform as a whole, the architecture and infrastructure serve as the foundation upon which everything else is built. For our purposes, we can use the two words, architecture and infrastructure, interchangeably, not precisely, but close enough.
Look, you don’t have to, but if you want to, here we go. Also, you should probably go find one of those elusive AI Engineers that are now showing up on LinkedIn full of wisdom and foresight. Maybe they know what’s going on. Either way, I’ve built a few multi—agent POCs, some on the Databricks platform and others more bespoke.
I’ve poked and prodded enough “AI stuffy stuff” to at least make myself dangerous. Put me in, coach.
Sometimes I lean back in my chair, before work starts, and dream of the time long ago, before AI ruined everything. Back in the olden days when simplicity was > than complexity. It seems those days are long gone.
People all of a sudden worship at the altar of Claudius Maximus Spit Codeius. Overnight, for the most part, all the classic dos and don’ts of software development best practices went out the window. The faster you can spit the code, and the more of it you spit … the better.