I think it's really about an interplay of mental models and domains. Hers's my one personal datapoint. I did my undergrad in math and discovered functional programming back then. Got real excited about it, read the Learn You a Haskell book and then sort of nothing happened. But I waited on that "mind expanding" stuff to pay off.
Then I got my first job, where they really only cared that I was good in python. Cranked out a bunch of python services, and got good at cloud stuff along the way. Decided FP and that kind of thing was "school stuff."
A few years later I found myself working at another shop doing backend Go/Kotlin. We were increasingly writing a lot of streaming stuff on Kafka, and could tell we weren't really "doing it right." Just gut feel that something was not as good as it could be. So we brought in a senior eng with a bunch of Kafka experience, who just sort of coincidentally had been working in Scala during that experience. We onboarded him onto Kotlin, but the code pretty quickly picked up a Scala style and flavor.
After about a year I couldn't shake the feeling of how perfectly this kind of functional thinking mapped to stream processing services. Especially when you're working on a framework like KStreams or Flink, being able to describe your transformations with these pure non-nullable functions was just perfect for the domain. When you squint a lot of the Option/Either monad stuff really helps you think about your programs as a linear flow of data, which is exactly what stream processing is.
When you look at a lot of the Java api's in the space of big data, they look functional. AFAIK thats not because a bunch of Java devs decided to use all the cutting edge Java functional features. Its because the stuff people were building in Scala just fit the domain so much better. So the Java ecosystem took those ideas and everyone is for the better.
At no point did I develop a superpower. Now I'm back to writing Python stuff in a new domain, and those insights aren't so valuable day to day. I try to make my code a bit more linear and functional, but can increasingly tell its not the practical choice. But if somebody needed me to write a high throughput reliable stream processing thing, I would immediately jump back to those patterns.
"Simulate querying industry-scale ML datasets locally with real SQL statements without having to access, manage or pay for industry-scale infrastructure."
Matt and Sam here from SyntheticDB. We'll be keeping an eye on this thread so if you have any questions about the project just ask. Any and all feedback is extremely welcome.
Then I got my first job, where they really only cared that I was good in python. Cranked out a bunch of python services, and got good at cloud stuff along the way. Decided FP and that kind of thing was "school stuff."
A few years later I found myself working at another shop doing backend Go/Kotlin. We were increasingly writing a lot of streaming stuff on Kafka, and could tell we weren't really "doing it right." Just gut feel that something was not as good as it could be. So we brought in a senior eng with a bunch of Kafka experience, who just sort of coincidentally had been working in Scala during that experience. We onboarded him onto Kotlin, but the code pretty quickly picked up a Scala style and flavor.
After about a year I couldn't shake the feeling of how perfectly this kind of functional thinking mapped to stream processing services. Especially when you're working on a framework like KStreams or Flink, being able to describe your transformations with these pure non-nullable functions was just perfect for the domain. When you squint a lot of the Option/Either monad stuff really helps you think about your programs as a linear flow of data, which is exactly what stream processing is.
When you look at a lot of the Java api's in the space of big data, they look functional. AFAIK thats not because a bunch of Java devs decided to use all the cutting edge Java functional features. Its because the stuff people were building in Scala just fit the domain so much better. So the Java ecosystem took those ideas and everyone is for the better.
At no point did I develop a superpower. Now I'm back to writing Python stuff in a new domain, and those insights aren't so valuable day to day. I try to make my code a bit more linear and functional, but can increasingly tell its not the practical choice. But if somebody needed me to write a high throughput reliable stream processing thing, I would immediately jump back to those patterns.