“When you discuss languages it almost becomes like a
religious argument. But really a language is just a tool.
It’s like arguing, well, which one is better: a hammer or a
screwdriver? You tell me what you want to do with it and
I’ll tell you which one is better.”
At this point a pipeline built on top of Stitch / Fivetran /
dbt is far more reliable than one built on top of custom-
built Airflow tasks.
If you hire a data engineer who just wants to muck around in
the backend and hates working with less-technical folks,
you’re going to have a bad time.
1) Machine Learning Productionization
2) Being a source of data expertise (consulting) with other
developers (working on services or the main product) in
the organziation
Issues around the performance of Python and programs
written in it have far wider consequences than startup
time. During all the time any Python program is running,
its host machine is consuming power that typically depends
on pumping CO2 into the atmosphere. If most of that power is
wasted, the effects go far beyond extra money to buy
it, or to operate extra servers, or users who wait a
little longer. The carbon footprint of a Python program
that runs throughout a data center, or many data centers,
adds up.