Ask HN: Development/Engineering Epiphanies from your experience?
Would love to hear inputs on less technical epiphanies you've had during development of a new thing, especially at the human interfaces (dev teammates, customer, management), but machine interfaces could also be interesting (something you learned about your computer/workbench/mill).
21 comments
Choice of language does matter.
For instance if you commit in 2022 to java/scala - you subscribe to heavy 20yo over-engineering legacy, whether you want it or not. No matter how fancy and eloquent your frameworks are, or how smart is your founding team — sooner or later this will catch your project.
The benefits may still outweigh the drawbacks for your particular case (though i never seen such cases except some external infra constraints like dependency on critical jvm service etc.) - but it is an important factor to consider that so many overlook.
For instance if you commit in 2022 to java/scala - you subscribe to heavy 20yo over-engineering legacy, whether you want it or not. No matter how fancy and eloquent your frameworks are, or how smart is your founding team — sooner or later this will catch your project.
The benefits may still outweigh the drawbacks for your particular case (though i never seen such cases except some external infra constraints like dependency on critical jvm service etc.) - but it is an important factor to consider that so many overlook.
>you subscribe to heavy 20yo over-engineering legacy
Maybe the key word here is "over-engineering" but given the age of Python, Ruby, Javascript (IE: other languages in popular use, all of which are of a similar age) I'm curious how you see Java as being different in this regard. Seems like you've done some thinking on this point so I'm genuinely curious to hear your perspective.
Maybe the key word here is "over-engineering" but given the age of Python, Ruby, Javascript (IE: other languages in popular use, all of which are of a similar age) I'm curious how you see Java as being different in this regard. Seems like you've done some thinking on this point so I'm genuinely curious to hear your perspective.
Ruby Python and js has their own downsides and benefits, I just took java as an example to illustrate the point — language is not just a syntax, vm and standard lib, it’s always bundled with some traditions, mentality, patterns that one should take into consideration.
> “if you commit in 2022 to java/scala - you subscribe to heavy 20yo over-engineering legacy, whether you want it or not”
Fully committed to Java on services side and Scala for data transforms and Spark jobs.
The Java services have 500k-1MM TPS.
For a dev onboarding, setting up personal dev environment on Mac or their VM and creating personal cloud stack takes less than 10 min.
some of the services are up to 4-5 years old. couple spun up end of q3 this year. I think those are getting more like 100k TPS.
Devs are pretty happy, we have decent roadmap for next year, and operational support is totally reasonable. Zero pages all month (knock on wood).
We have high test coverage, including unit and integ tests, canaries, and full ci/cd. A couple code paths use AWS X-ray but with very low sample rate to avoid blowing up costs.
All the services are Java. The Scala I think is even simpler and more straightforward. Spark and a few other things orchestrated using Airflow. The Airflow scripts are in Python though.
Let me know more about your prediction for our over engineering. People are happy (close 1:1 feedback from people I mentor, their managers, and managers manager), and we’re making money [knock on wood].
Why is this not good enough and how can it be better?
Btw we have dedicated software engineers in test. Their tooling is also using Java in quite a few places, and their partnership with core dev teams is very strong. Haven’t gotten negative feedback from them or their leadership either.
Fully committed to Java on services side and Scala for data transforms and Spark jobs.
The Java services have 500k-1MM TPS.
For a dev onboarding, setting up personal dev environment on Mac or their VM and creating personal cloud stack takes less than 10 min.
some of the services are up to 4-5 years old. couple spun up end of q3 this year. I think those are getting more like 100k TPS.
Devs are pretty happy, we have decent roadmap for next year, and operational support is totally reasonable. Zero pages all month (knock on wood).
We have high test coverage, including unit and integ tests, canaries, and full ci/cd. A couple code paths use AWS X-ray but with very low sample rate to avoid blowing up costs.
All the services are Java. The Scala I think is even simpler and more straightforward. Spark and a few other things orchestrated using Airflow. The Airflow scripts are in Python though.
Let me know more about your prediction for our over engineering. People are happy (close 1:1 feedback from people I mentor, their managers, and managers manager), and we’re making money [knock on wood].
Why is this not good enough and how can it be better?
Btw we have dedicated software engineers in test. Their tooling is also using Java in quite a few places, and their partnership with core dev teams is very strong. Haven’t gotten negative feedback from them or their leadership either.
I can just congratulate you on a great job done.
You may as well have said that Amazon is living on JVM mostly and look how rich and happy they are as company.
It would be a much stronger argument :)
Sometimes things happen not thanks to, but inspite of.
You may as well have said that Amazon is living on JVM mostly and look how rich and happy they are as company.
It would be a much stronger argument :)
Sometimes things happen not thanks to, but inspite of.
I’m assuming this is sarcasm.
I am not working at Amazon. If anything, you can view my post history and see my disfavor of big tech.
If you have something more substantial to say, please do it, but a sarcastic/snark remark isn’t jt
I am not working at Amazon. If anything, you can view my post history and see my disfavor of big tech.
If you have something more substantial to say, please do it, but a sarcastic/snark remark isn’t jt
It is not sarcastic (tiny bit of sarcasm is pointed towards Amazon if any, because they are actually using JVM a lot).
I'm just saying that your arguments doesn't refute my point, nor supports it. Your success with java doesn't mean that java world is not full of legacy and suffering for many other teams.
I'm just saying that your arguments doesn't refute my point, nor supports it. Your success with java doesn't mean that java world is not full of legacy and suffering for many other teams.
> “because they are actually using JVM a lot”
It has nothing to do with my company, and so interesting your rebuttal to my comment was snark and about an entirely different company.
Here’s your original comment.
> “if you commit in 2022 to java/scala - you subscribe to heavy 20yo over-engineering legacy, whether you want it or not”
And now you said:
> “Your success with java doesn't mean that java world is not full of legacy and suffering for many other teams.”
You painted a broad brush stroke, and I shared personal evidence of a successful business running on JVM.
Now you’re painting another brush, indirectly saying this case is exception or somehow doesn’t matter.
If you have something meaningful to say, or any particular experience on what my organization should watch out for (some blind spot), call it out.
It feels like you’re not here to teach or inform, this is more about you being right. Everyone can have an opinion. They’re worthless unless you have something actually constructive to add.
It has nothing to do with my company, and so interesting your rebuttal to my comment was snark and about an entirely different company.
Here’s your original comment.
> “if you commit in 2022 to java/scala - you subscribe to heavy 20yo over-engineering legacy, whether you want it or not”
And now you said:
> “Your success with java doesn't mean that java world is not full of legacy and suffering for many other teams.”
You painted a broad brush stroke, and I shared personal evidence of a successful business running on JVM.
Now you’re painting another brush, indirectly saying this case is exception or somehow doesn’t matter.
If you have something meaningful to say, or any particular experience on what my organization should watch out for (some blind spot), call it out.
It feels like you’re not here to teach or inform, this is more about you being right. Everyone can have an opinion. They’re worthless unless you have something actually constructive to add.
And also people who love java would never complain, they tend to call what others call “overengineering” a “scalable clean architecture”.
And more “clean” it is- more happy they are.
I’ve been there myself.
And more “clean” it is- more happy they are.
I’ve been there myself.
> they tend to call what others call “overengineering” a “scalable clean architecture”.
I’m not sure what this means or what you’re referring to. It seems the premise of your argument is that Java is bad and over engineering, and anyone who disagrees is fundamentally misinformed, bias, and wrong.
With your argument, it’s actually impossible to have any kind of debate or constructive discussion because the premise is the other biased and stupid. You went from attacking the tech to actually attacking the people.
I’m not sure what this means or what you’re referring to. It seems the premise of your argument is that Java is bad and over engineering, and anyone who disagrees is fundamentally misinformed, bias, and wrong.
With your argument, it’s actually impossible to have any kind of debate or constructive discussion because the premise is the other biased and stupid. You went from attacking the tech to actually attacking the people.
I did a lot of job hopping and can tell you all projects are a mess. So for me it's all about the money. And having a hobby project where I can have fun.
Companies are wasting truckloads of money on maintenance nightmares.
Business value trumps tech issues/nice solutions/priorities/etc every time. Or put an other way: the guy paying for the party decides what goes and doesnt go, regardless of how technically correct you are (or how correct your technical argument is). To put it bluntly, most of the time the message will be: “F your tech debt and long term considerations, just deliver business value in the short run.”.. just delivered in nicer wording.
HOWEVER here is the epiphany. All of the above talk comes from the “business side” and they only speak business language. You speak the tech language which the business people do not understand (and thus you can get the last word on tech things). If you learn to speak the business language then you will be in the unique situation of speaking both tech and business, and you can use that to frame and control talks with the business people. That will enable you to steer the “all mighty - value now” people into decisions that incorporate tech concerns without them explicitly knowing this, enabling you to deliver value in a way that also allows you to provide your team with an environment where they can do great tech things. That will keep your team happy and delivering which will keep the business happy with your team.
The bonus epiphany is: This is not even a bad thing for the business or misleading, since baking due and proper tech cosiderstions into business prioritization and decisions will benefit the business in the long run. They’ll just never realize it.
HOWEVER here is the epiphany. All of the above talk comes from the “business side” and they only speak business language. You speak the tech language which the business people do not understand (and thus you can get the last word on tech things). If you learn to speak the business language then you will be in the unique situation of speaking both tech and business, and you can use that to frame and control talks with the business people. That will enable you to steer the “all mighty - value now” people into decisions that incorporate tech concerns without them explicitly knowing this, enabling you to deliver value in a way that also allows you to provide your team with an environment where they can do great tech things. That will keep your team happy and delivering which will keep the business happy with your team.
The bonus epiphany is: This is not even a bad thing for the business or misleading, since baking due and proper tech cosiderstions into business prioritization and decisions will benefit the business in the long run. They’ll just never realize it.
Deep learning is clever matrix multiplications with some clever loss functions.
When I first started learning theoretical deep learning I was afraid of it since it seemed so difficult. Then I realized that it's just doing matrix multiplication with some clever tricks to introduce inductive bias, and clever loss functions that teach you how to optimize the parameters of your matrices. And a lot of data.
When I first started learning theoretical deep learning I was afraid of it since it seemed so difficult. Then I realized that it's just doing matrix multiplication with some clever tricks to introduce inductive bias, and clever loss functions that teach you how to optimize the parameters of your matrices. And a lot of data.
Naming things is probably the most important part of programming. If you’re struggling to give a function a good name it probably means that it’s the wrong level of abstraction. For example if you find yourself naming a function something like “submitAndParseResult” that’s probably an indication that it should be broken up into two functions.
I used to give a code refactoring exercise as a part of my interview process and I remember one candidate spent most of the exercise just naming the variables and breaking sections of the code into separate functions that didn’t actually improve the performance but made it much more readable. I hired him on the spot.
I used to give a code refactoring exercise as a part of my interview process and I remember one candidate spent most of the exercise just naming the variables and breaking sections of the code into separate functions that didn’t actually improve the performance but made it much more readable. I hired him on the spot.
>I used to give a code refactoring exercise as a part of my interview process
I really like this - smart way to get a dialog between yourself and the candidate over something tangible. Great way to see if you can work together, can communicate effectively together, plus gets in their head about how they will approach work.
I really like this - smart way to get a dialog between yourself and the candidate over something tangible. Great way to see if you can work together, can communicate effectively together, plus gets in their head about how they will approach work.
Thanks! I like refactoring exercises because they tend to select for real world experience over classes/studying. And when I think about the most difficult technical problems I’ve worked on it’s never been about selecting the right algorithm or data structure, it’s always involved making legacy code adapt to new requirements when the previous developer is no longer around and the requirements weren’t properly documented.
I'd agree with this, and it's one of the beefs I have with how we tend to do interviews. When I think too about more involved problems I've worked on, there have been times I've had to consider several different data types or algorithms, and potentially build a prototype and refine it. This "instant gratification" we demand of interviewees to churn out some algorithm or data type in an interview isn't really helpful relative to seeing how they actually think.
All of programming is moving data, the better you understand WHAT you're moving, the easier and cleaner the code.
If you implement a fixed size binary tree as an array, breadth first search is simply iterating over its elements.
Never underestimate the ability of end-users to do the wrong thing.
Abstraction doesn't mean you don't have to learn the underlying technology.
Most performance problems come from not understanding the underlying technology due to abstraction.
Being a software developer doesn't mean you understand how the thing works.
Setting expectations is the #1 job of the account team.
Abstraction doesn't mean you don't have to learn the underlying technology.
Most performance problems come from not understanding the underlying technology due to abstraction.
Being a software developer doesn't mean you understand how the thing works.
Setting expectations is the #1 job of the account team.