The Prime Video microservices to monolith story(adrianco.medium.com)
adrianco.medium.com
The Prime Video microservices to monolith story
https://adrianco.medium.com/so-many-bad-takes-what-is-there-to-learn-from-the-prime-video-microservices-to-monolith-story-4bd0970423d4
405 comments
Were those startups that went out of business did so because of their stacks/architecture, or are you confusing correlation with causation? And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.
> Were those startups that went out of business did so because of their stacks/architecture, or are you confusing correlation with causation?
As far as I am aware, and I can tell, it wasn't because of the stacks / architecture, but lack of product market fit. The fact that the response to lack of PMF was to double down on product features and tech (instead of a business pivot), in my opinion, was what made the investor money go poof and the startup die down. I've seen this happen' in too many places (as an employee or consultant) to be a coincidence.
> And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.
Different stacks for different use cases. If I can't get money with a shitty PHP product offering, it's probably best to figure out why instead of attributing it to my tech stack.
As far as I am aware, and I can tell, it wasn't because of the stacks / architecture, but lack of product market fit. The fact that the response to lack of PMF was to double down on product features and tech (instead of a business pivot), in my opinion, was what made the investor money go poof and the startup die down. I've seen this happen' in too many places (as an employee or consultant) to be a coincidence.
> And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.
Different stacks for different use cases. If I can't get money with a shitty PHP product offering, it's probably best to figure out why instead of attributing it to my tech stack.
I have totally seen that too. For me one of the biggest dangers in any tech project, startups especially is caring about outputs more than outcomes. "Look at all the things we're building! Look at how hard we're working!" But those outputs don't matter if they aren't creating more value for the customer.
> And, there is a good reason people shy away from PHP
I agree, I also strongly dislike Angular and MySQL in addition to PHP. But I also think a much better indicator for success is discipline. Angular, PHP and MySQL, to their credit, aren’t crazy Turing tarpits, you can write good, successful software with them.
I’ve never seen a dysfunctional project where any of the inherent flaws of those technologies even registered on the same scale as the sheer malpractice of everyone working on the product. And this is also the problem with trendy and flashy stacks, developers not learning the stacks properly and using them as intended. Probably due to commercial pressure to push prototypes to production instead of rebuilding properly after learning the stack. I think that should be the takeaway from the original comment, that CTOs need to either budget for building it twice, or stick to a stack everyone knows inside and out.
I agree, I also strongly dislike Angular and MySQL in addition to PHP. But I also think a much better indicator for success is discipline. Angular, PHP and MySQL, to their credit, aren’t crazy Turing tarpits, you can write good, successful software with them.
I’ve never seen a dysfunctional project where any of the inherent flaws of those technologies even registered on the same scale as the sheer malpractice of everyone working on the product. And this is also the problem with trendy and flashy stacks, developers not learning the stacks properly and using them as intended. Probably due to commercial pressure to push prototypes to production instead of rebuilding properly after learning the stack. I think that should be the takeaway from the original comment, that CTOs need to either budget for building it twice, or stick to a stack everyone knows inside and out.
It's called survivorship bias (1).
The only people pointing to PHP as the right choice that kept them from ruin are those that used it successfully. Therefore, they were building something that fit well with PHP's benefits and didn't expose too many of it's weaknesses. But no one knows for sure exactly what about that project made it a good candidate for a PHP monolith, because we don't really have any science around software engineering, just hearsay and fashion.
Having rescued several 10+ person year PHP projects from ruin, none of them were on the edge because of the language itself. However, the people building those systems made an incredible mess in a short time. I think each language attracts a certain type of developer and comes with it's own community baggage. The PHP community has some highly questionable best practices. But this is true of all programming language specific communities. They tend to be insular and have a lot of "winning the last war" advice for working around the shortcomings of the language's previous major version that have been fixed in the current major version.
(1) https://en.wikipedia.org/wiki/Survivorship_bias
The only people pointing to PHP as the right choice that kept them from ruin are those that used it successfully. Therefore, they were building something that fit well with PHP's benefits and didn't expose too many of it's weaknesses. But no one knows for sure exactly what about that project made it a good candidate for a PHP monolith, because we don't really have any science around software engineering, just hearsay and fashion.
Having rescued several 10+ person year PHP projects from ruin, none of them were on the edge because of the language itself. However, the people building those systems made an incredible mess in a short time. I think each language attracts a certain type of developer and comes with it's own community baggage. The PHP community has some highly questionable best practices. But this is true of all programming language specific communities. They tend to be insular and have a lot of "winning the last war" advice for working around the shortcomings of the language's previous major version that have been fixed in the current major version.
(1) https://en.wikipedia.org/wiki/Survivorship_bias
> Having rescued several 10+ person year PHP projects from ruin
How did you rescue them? What are your views on Laravel?
How did you rescue them? What are your views on Laravel?
The teams were generally adding more bugs per release then they were fixing. Many bugs were caused by duplicated code: "I fixed the bug, but only in 1/18th of the places it manifests."
The projects were using Zend framework 2. We didn't change really anything about the text stack at all.
The solutions were generally three pronged. This isn't to bash anyone, but every failing project I've seen is a skill, talent, or discipline issue, not a technical one. So recusing projects tends to be about training, mentorship, and guidance. Sure, you fix things along the way, but that's more to prove to the devs that you know what you're talking about, so they start to fix things themselves.
First, start a root cause analysis on bugs before fixing them. This often meant that instead of another if statement in the UI because of bad state, there was a larger fix needed to prevent the bad state entirely. These bigger changes were scary, but they quickly turned these projects from spending all their time in emergencies into only a few hours a week.
Second tool was to deduplicate and unit test code as much as possible. No more copy-pasting whole files every time there was a slightly different use-case required.
Third, to stop treating the database like a PHP ORM connection and start treating it like a distinct tool. Things like turning serialized PHP arrays from a single column into separate tables allowed us to start using SQL to query data instead of pulling all records into memory to filter and sort. This rapidly improved performance, which meant fewer support issues.
All boring, bog standard stuff. One project we started a second service in .NET to be the location of new features because half the team didn't want to use PHP anymore. The idea was they could slowly migrate endpoints to the .NET project. But that didn't survive. They got bored with a slow and safe conversion and convinced upper management to approve a total rewrite in .NET with a whole different team building out a microservice mess. The rewrite has been ongoing for 18 months now without a release. I heard they finished their first feature a few months ago, but still aren't live. Meanwhile the PHP project just keeps chugging along. Just goes to show, it's rarely the technology that causes issues, it's the team.
I don't really know anything about Laravel. I'm not really a PHP guy, I'm a project rescuer. Just happens to be most projects I get called into are PHP, C#, and Visual Basic. I've heard really good things about Laravel, but even ZF2 can be made safe, secure, performant, and reliable. It just takes discipline and the right team. That being said, I would consider PHP to be a language that is hard to hire for. There's just so many developers that flood the inbox who don't even know what a function is.
I will say, when I took a contract to build a greenfield web dashboard at the start of the year, I chose .NET 6 and C#. The platform and ecosystem is fine. I've shipped Java, Python, JavaScript, Ruby, Perl, OCaml, Haskell, Clojure, F#, and VB.NET. I picked C# for this project because the company already has a couple dozen C# developers who are pretty talented. I figured if I shipped them something they were comfortable with maybe they'd hire me again down the road.
The projects were using Zend framework 2. We didn't change really anything about the text stack at all.
The solutions were generally three pronged. This isn't to bash anyone, but every failing project I've seen is a skill, talent, or discipline issue, not a technical one. So recusing projects tends to be about training, mentorship, and guidance. Sure, you fix things along the way, but that's more to prove to the devs that you know what you're talking about, so they start to fix things themselves.
First, start a root cause analysis on bugs before fixing them. This often meant that instead of another if statement in the UI because of bad state, there was a larger fix needed to prevent the bad state entirely. These bigger changes were scary, but they quickly turned these projects from spending all their time in emergencies into only a few hours a week.
Second tool was to deduplicate and unit test code as much as possible. No more copy-pasting whole files every time there was a slightly different use-case required.
Third, to stop treating the database like a PHP ORM connection and start treating it like a distinct tool. Things like turning serialized PHP arrays from a single column into separate tables allowed us to start using SQL to query data instead of pulling all records into memory to filter and sort. This rapidly improved performance, which meant fewer support issues.
All boring, bog standard stuff. One project we started a second service in .NET to be the location of new features because half the team didn't want to use PHP anymore. The idea was they could slowly migrate endpoints to the .NET project. But that didn't survive. They got bored with a slow and safe conversion and convinced upper management to approve a total rewrite in .NET with a whole different team building out a microservice mess. The rewrite has been ongoing for 18 months now without a release. I heard they finished their first feature a few months ago, but still aren't live. Meanwhile the PHP project just keeps chugging along. Just goes to show, it's rarely the technology that causes issues, it's the team.
I don't really know anything about Laravel. I'm not really a PHP guy, I'm a project rescuer. Just happens to be most projects I get called into are PHP, C#, and Visual Basic. I've heard really good things about Laravel, but even ZF2 can be made safe, secure, performant, and reliable. It just takes discipline and the right team. That being said, I would consider PHP to be a language that is hard to hire for. There's just so many developers that flood the inbox who don't even know what a function is.
I will say, when I took a contract to build a greenfield web dashboard at the start of the year, I chose .NET 6 and C#. The platform and ecosystem is fine. I've shipped Java, Python, JavaScript, Ruby, Perl, OCaml, Haskell, Clojure, F#, and VB.NET. I picked C# for this project because the company already has a couple dozen C# developers who are pretty talented. I figured if I shipped them something they were comfortable with maybe they'd hire me again down the road.
Very common for older PHP projects to store PHP serialized strings in the database. Horrible performance and of course gets easily corrupted of not handling concurrent updates properly.
Rewrites rarely succeeds, it is a typical fallacy for a team to believe that their problems lie in the language they are using and not in themselves. Seen it too many times.
Rewrites rarely succeeds, it is a typical fallacy for a team to believe that their problems lie in the language they are using and not in themselves. Seen it too many times.
The point wasn't that you should use PHP, it was to not jump on stack trends. There is a name for your kind of fallacy.
> there is a good reason people shy away from PHP
Please share it. And don't rely on articles that are more than a decade and 3 or 4 versions of PHP old.
Please share it. And don't rely on articles that are more than a decade and 3 or 4 versions of PHP old.
IMHO this very discussion factors in whether you'll want a PHP stack or not.
If you're looking for pragmatic devs that don't care much about the trends, the community or peers' general opinion about a technology, going with PHP will get you this type in spades. Especially if you're looking for experienced engineers.
If you want devs that care about thoughtful designs, elegant solutions and want to enjoy the community, it will be a lot more difficult (not impossible, but it's an uphill battle)
And this has little to do with PHP's viability per se, there's just so much baggage that most devs will self select in one side or another. A language's base design philosophy matters, and it's always better to chose a stack that matches your own philosophy.
If you're looking for pragmatic devs that don't care much about the trends, the community or peers' general opinion about a technology, going with PHP will get you this type in spades. Especially if you're looking for experienced engineers.
If you want devs that care about thoughtful designs, elegant solutions and want to enjoy the community, it will be a lot more difficult (not impossible, but it's an uphill battle)
And this has little to do with PHP's viability per se, there's just so much baggage that most devs will self select in one side or another. A language's base design philosophy matters, and it's always better to chose a stack that matches your own philosophy.
Afaik you cannot discern null and undefined. So, not there and "null"? Either that or I'm going to have to have a good talk with my colleague
Go have your talk [1]. But that's a pretty weak argument anyway given that:
1. Other languages also do not have this distinction (Python for example AFAIK),
2. It's relatively useless. I get the value of it in JavaScript which has a lot of asynchronous code and frontend where the behavior may indeed be very different between null and undefined, but in a server-side application I don't think the nuance is necessary at runtime, and potential programming mistakes involving that can be caught by a linter.
[1] https://stackoverflow.com/questions/6905491/how-to-tell-whet...
1. Other languages also do not have this distinction (Python for example AFAIK),
2. It's relatively useless. I get the value of it in JavaScript which has a lot of asynchronous code and frontend where the behavior may indeed be very different between null and undefined, but in a server-side application I don't think the nuance is necessary at runtime, and potential programming mistakes involving that can be caught by a linter.
[1] https://stackoverflow.com/questions/6905491/how-to-tell-whet...
https://stackoverflow.com/questions/6905491/how-to-tell-whet...
At the language level PHP has about the same problems as JS.
Thankfully it looks uglier so people get an early warning before using it for everything.
At the language level PHP has about the same problems as JS.
Thankfully it looks uglier so people get an early warning before using it for everything.
> Afaik you cannot discern null and undefined.
This is a discussion for us tech folks. When it comes to business. Customers don't give a rat's ass about this as long as the product provides them with value.
If you're flooded with customers and the tech stack is a pain in the ass, perhaps it would be a good time to start considering this, but until you have this problem, the technical aspects are just wasting investor money and / or engineering time.
This is a discussion for us tech folks. When it comes to business. Customers don't give a rat's ass about this as long as the product provides them with value.
If you're flooded with customers and the tech stack is a pain in the ass, perhaps it would be a good time to start considering this, but until you have this problem, the technical aspects are just wasting investor money and / or engineering time.
When I (an average developer) accidentally check for null but not undefined in the code and it causes a bug, the customer might care a lot.
There is always a balance to strike between development and delivery. When creating a system with high potential failure costs, it's important to select a technology stack that minimizes common errors for developers.
For most companies, issues like `undefined | null` bugs can be resolved with just a few code adjustments, resulting in only minor inconvenience for customers. As long as problems are addressed and the product provides value, customers will continue to support it.
There are CEOs and CTOs that feel differently. I wish them luck.
For most companies, issues like `undefined | null` bugs can be resolved with just a few code adjustments, resulting in only minor inconvenience for customers. As long as problems are addressed and the product provides value, customers will continue to support it.
There are CEOs and CTOs that feel differently. I wish them luck.
> Afaik you cannot discern null and undefined. So, not there and "null"?
So?
Don't similar footguns appear in Python and JS? Why single out PHP for footguns when the other languages are no better?
While I'm not a PHP programmer, I have used most programming languages, and PHP doesn't seem any worse than the competitors.
So?
Don't similar footguns appear in Python and JS? Why single out PHP for footguns when the other languages are no better?
While I'm not a PHP programmer, I have used most programming languages, and PHP doesn't seem any worse than the competitors.
How about the fact that most of us are experienced and can attest that PHP is the worst language we've ever used in our careers?
I don't think so, i've used PHP at work for more than 10 years till 7.4 point mostly Symfony, it has it's own quirks but i still feel PHP really pragmatic. I don't have any strong feelings against it.
Nowadays im on django/python because im in a python shop and I like python too.
Nowadays im on django/python because im in a python shop and I like python too.
Yeah, I feel like PHP is a massive hiring risk. I personally wouldn't want to touch any PHP code unless I'm paid a massive premium for it.
(I mostly write Python professionally, and use Python, Swift and Rust in personal projects in case anyone is wondering.)
(I mostly write Python professionally, and use Python, Swift and Rust in personal projects in case anyone is wondering.)
I can think of far worse fates
Some people write whole-ass applications in bash, the psychopaths
Some people write whole-ass applications in bash, the psychopaths
> Some people write whole-ass applications in bash, the psychopaths
*giggles nervously, then changes the subject.
*giggles nervously, then changes the subject.
I disagree; it's not the language that's bad, it's the community. Too much tech debt that was never removed, too many people that profess to be experts on the internet and give bad examples, and too many people full of themselves building and rebuilding their own frameworks. That was the issue 10, 15 years ago anyway, things have improved but it means people have to pick established frameworks like Laravel.
You can't really separate a language from its community, though.
You're getting paid to deliver business logic, which means most work is going to be gluing together libraries. This inherently means relying on community-written code, and related stuff like documentation and StackOverflow answers.
If the community is bad, the language isn't worth using.
You're getting paid to deliver business logic, which means most work is going to be gluing together libraries. This inherently means relying on community-written code, and related stuff like documentation and StackOverflow answers.
If the community is bad, the language isn't worth using.
Genuine question, when was the last time you wrote PHP?
5 years ago and I wish I hadn't.
It was fine in the late 90s.
Very adequate for the hosting landscape.
I have not touched it since that time
Spoken like someone who's never used Matlab :)
> indexes start at 1
- no one
-- Matlab
- no one
-- Matlab
Pascal had indexes starting at 1 too.
Starting indexes at 0 makes sense when you want to enable the developer to manipulate memory (for example, C)
Indexes starting at 1 matches with a more natural ordinal sequencing.
Starting indexes at 0 makes sense when you want to enable the developer to manipulate memory (for example, C)
Indexes starting at 1 matches with a more natural ordinal sequencing.
The most evil was Visual Basic where you could configure the default starting index (OPTION BASE 1), leading to weird bugs when multiple people worked on the same codebase.
I don’t understand. Either your saying adding or removing elements to an array doesn’t manipulate memory, or you’re saying adding or removing elements in an array does manipulate memory, but the developer doesn’t care due to the GC.
If it’s the second option, why has (almost) every other language chosen with a GC to start arrays with 0?
If it’s the second option, why has (almost) every other language chosen with a GC to start arrays with 0?
> indexes starting at...
"I don't understand the word 'index'".
- a functional programmer
"I don't understand the word 'index'".
- a functional programmer
Personally I think the indicator of success is not necessarily X stack or Y stack, it's what stack are you most comfortable with? I think one of the problems with those startups you mentioned that used flashy stacks is they are not only fighting a battle to build the product, they are also fighting a battle to learn their way around the new stack they just adopted, and learning all the quirks with it.
I don't think one stack is necessarily better than the other, it's all about how well do the developers know the tech stack in question.
I don't think one stack is necessarily better than the other, it's all about how well do the developers know the tech stack in question.
http://boringtechnology.club/ remains relevant, and I try to apply it wherever I go with... varying results.
One issue though is that it seems more difficult to find competent developers for boring stacks, because they're always looking for the next challenge, the next thing they consider interesting or what looks good on their profile (vs what is best for their employer).
One issue though is that it seems more difficult to find competent developers for boring stacks, because they're always looking for the next challenge, the next thing they consider interesting or what looks good on their profile (vs what is best for their employer).
> it seems more difficult to find competent developers for boring stacks, because they're always looking for the next challenge, the next thing they consider interesting or what looks good on their profile (vs what is best for their employer).
The people who job hop every 18 months and have flipped from ML to web3 to LLM's in the past 4 years are not the people you want to hire. If someone has 5-6 years experience writing Java code, they'll be fine if your tech stack is C#, and vice versa. There might be a bit of a learning curve pulling people from python or rails onto typed languages (or vice versa), but in my experience that's something that can be overcome easily enough. If you're willing to give someone a month to ramp up with the expectation they'll stay longer than 2 years, that seems to be a nice tradeoff to make.
The people who job hop every 18 months and have flipped from ML to web3 to LLM's in the past 4 years are not the people you want to hire. If someone has 5-6 years experience writing Java code, they'll be fine if your tech stack is C#, and vice versa. There might be a bit of a learning curve pulling people from python or rails onto typed languages (or vice versa), but in my experience that's something that can be overcome easily enough. If you're willing to give someone a month to ramp up with the expectation they'll stay longer than 2 years, that seems to be a nice tradeoff to make.
I'd love to work on a "boring" stack and just ship features and build stuff but... companies are like a herd, and they all ask for the same microservices' technologies when interviewing, no matter what they are building.
So if I stay with a boring stack, I'd lose competitive advantage and face the potential of being "un-hirable" in the not-so-distant future.
So if I stay with a boring stack, I'd lose competitive advantage and face the potential of being "un-hirable" in the not-so-distant future.
Well ... that depends a lot on where you want to be hired. There is tons of companies which are not at all that interesting like the latest startup but deliver awesome stuff all the time. IMHO, there is not a correlation between developer technology and delivering awesome stuff. There is a correlation between management style and mental capability to delivering awesome stuff.
I mean obviously we all have to move ourselves from "Fortran boring" to "Java boring" every decade or two, but being un-hirable because of "Java boring" is not a thing.
I mean obviously we all have to move ourselves from "Fortran boring" to "Java boring" every decade or two, but being un-hirable because of "Java boring" is not a thing.
Senior DevSecOps Engineer responsible for converged, cross-cloud cyber-resilience platform security
vs
unix greybeard who writes bash scripts to grep secrets in git repo
They're the same, but the word salad ver is more understandable to VC and HR types
vs
unix greybeard who writes bash scripts to grep secrets in git repo
They're the same, but the word salad ver is more understandable to VC and HR types
Maybe the word "boring" scare some people away, but it is a good contrast to exciting new technology anyway. "Mature" would probably the more appropriate term. Many times the exciting stuff is the problem domain itself. But developers tend to be more excited about new technologies than about customer needs.
Maybe you just aren't paying enough money for a developer to actually stay put and be happy with their role, so the only ones that apply are those who merely use the role as a playground to polish up their resume at your expense?
I agree with you that choosing a boring stack and focusing on down to earth, understandable and pragmatic architectural principles offers good probabilities to get a maintainable product.
I would like to interject, though, that stack and architecture do not weigh equally on these probabilities. My observation has been that tech organizations fail when they loose their grip on pragmatism, regardless the stack.
That doesn't mean that you should force the latest, unproven, JS framework on 200 engineers but if you have the culture for it one can easily succeed with react+JVM, elm+Haskell, etc.
One interesting thing also is access to devs as each tech stack will attract different groups of the Dev society which will in turn affect your success chances.
I would like to interject, though, that stack and architecture do not weigh equally on these probabilities. My observation has been that tech organizations fail when they loose their grip on pragmatism, regardless the stack.
That doesn't mean that you should force the latest, unproven, JS framework on 200 engineers but if you have the culture for it one can easily succeed with react+JVM, elm+Haskell, etc.
One interesting thing also is access to devs as each tech stack will attract different groups of the Dev society which will in turn affect your success chances.
There is a middle ground to this. Sooner or later you have to upgrade. There might be a security upgrade or the old system doesn't play nice with other systems. So you are forced to upgrade to something more non-boring, even if you are using the same tech. It also get harded to hire developers. I doubt there is many devs that want to work on Angular 2 these days.
The middle ground is that once the tech becomes the actual problem, it might be time to investigate possibilities, until then, customers don't care, business people doesn't care and the bottom line doesn't care. There are not that many companies in the world where the tech stack is their competitive advantage.
> I doubt there is many devs that want to work on Angular 2 these days.
From my experience, a product that makes money and is able to pay people at market rates, will find said people. Not everyone is an EMACS wielding, Linux wizard that refuses to touch tech out of idealistic concerns.
The problem, in my experience, is more often lack of PMF than tech stack (I've seen customers give insane leeway to actually valuable products).
> I doubt there is many devs that want to work on Angular 2 these days.
From my experience, a product that makes money and is able to pay people at market rates, will find said people. Not everyone is an EMACS wielding, Linux wizard that refuses to touch tech out of idealistic concerns.
The problem, in my experience, is more often lack of PMF than tech stack (I've seen customers give insane leeway to actually valuable products).
I think it's because it's fun to work at a company that is tech-first (as opposed to just using tech as a tool for growing business) but it leads to doing things in complicated and risky ways "because we're a tech company"
I guess it also depends on the person. I get off on providing customer value by addressing business needs. I scratch the other itch late at night with unspeakable horrors that never see the light of day.
That sounds more like survivorship bias, startups typically fail over things such as product market fit, running out of cash, not how well the stack operates.
One way to run out of cash is to hire 50 expensive devs to build a microservices hairball with Kafkas and all the latest toys, as opposed to having 5 devs just knock it out, faster and better, with a single Python app and a Postgres database.
We used to build real-scale systems with a stack of this kind when we had fewer features and less CPU power.
We used to build real-scale systems with a stack of this kind when we had fewer features and less CPU power.
If you have a predictable workload (i.e. we ingest 100,000 videos a month with n size) you should be looking at it from that perspective -- how much compute you are going to need, and when. Serverless works well for predictably unpredictable, infrequent workloads that don't need to have perfect output ("good enough").
The big mistake I see people make is trying to be overly clever and predict the future workload needs. It never works out like that. Design for your current workload now, if your architecture can handle 10x of that great! Each time your scale from 10x workload you will likely need to redesign the system anyway, so you shouldn't pay that tax unless you absolutely have to.
There are a lot of limitations of serverless, the big one I experienced was inability to control the host environment and limitations on the slice of memory/CPU you get, such that you must take that into consideration when designing your atomic work units. Also paying the distributed computing tax is real and occurs at development and runtime -- things like job tracking and monitoring are important when you have 10,000 processes running -- you start to get into the territory of things that basically never happen on a single machine or small cluster, but become problems with thousands of disparate machines / processes.
The big mistake I see people make is trying to be overly clever and predict the future workload needs. It never works out like that. Design for your current workload now, if your architecture can handle 10x of that great! Each time your scale from 10x workload you will likely need to redesign the system anyway, so you shouldn't pay that tax unless you absolutely have to.
There are a lot of limitations of serverless, the big one I experienced was inability to control the host environment and limitations on the slice of memory/CPU you get, such that you must take that into consideration when designing your atomic work units. Also paying the distributed computing tax is real and occurs at development and runtime -- things like job tracking and monitoring are important when you have 10,000 processes running -- you start to get into the territory of things that basically never happen on a single machine or small cluster, but become problems with thousands of disparate machines / processes.
>Design for your current workload
Please be my friend.
The bulk of my job these days is sitting in design reviews trying to convince people that just making up scenarios where you need to scale to 100x isn't actually engineering. It's exhausting self-indulgence. Nothing is easier than inventing scenarios where you'll need to "scale". It's borderline impossible to get software folks to just live in the world that exists in front of their eyes.
"Scale" should go back to its original meaning: change relative to some starting point. Slap a trend line on what you know. Do some estimation for what you don't. Design accordingly.
Please be my friend.
The bulk of my job these days is sitting in design reviews trying to convince people that just making up scenarios where you need to scale to 100x isn't actually engineering. It's exhausting self-indulgence. Nothing is easier than inventing scenarios where you'll need to "scale". It's borderline impossible to get software folks to just live in the world that exists in front of their eyes.
"Scale" should go back to its original meaning: change relative to some starting point. Slap a trend line on what you know. Do some estimation for what you don't. Design accordingly.
Believe me, our startup offering a coffee marketplace for fans of TANO*C HARDCORE music who want to contribute to Azeri poverty elimination efforts is going to need to scale from hundreds of users to hundreds of millions in a matter of months and you need to design ahead for that.
This has been a problem in a few places I've worked at that decided to build "microservices" (read: simple apps that moved the complexity to a higher architectural level, i.e. by having every service talk to every other service over a REST API or event bus), not because it solved a problem they had, but because it MIGHT solve a problem they'd LIKE to have. Cargo cult, in a nutshell.
My current employer is going all-in on serverless because it solved a problem they had with performance. The problem wasn't solved by serverless, but by moving away from an old fashioned and unscalable solution. The real problem or bottleneck - a centralized SAP database - has not been solved yet. They would have achieved the same results if they rebuilt their API in a generic Java monolith.
Here's my prediction: when the crack team of consultants that powered through building a serverless architecture leaves (because they will, one because they're consultants and two because they get bored when the problem has been solved), they won't be able to find competent developers to take over and the whole thing will be rebuilt again in something they CAN find developers for. I mean it's just NodeJS, but the architecture is distributed and much harder to manage than in a simple monolithic app.
My current employer is going all-in on serverless because it solved a problem they had with performance. The problem wasn't solved by serverless, but by moving away from an old fashioned and unscalable solution. The real problem or bottleneck - a centralized SAP database - has not been solved yet. They would have achieved the same results if they rebuilt their API in a generic Java monolith.
Here's my prediction: when the crack team of consultants that powered through building a serverless architecture leaves (because they will, one because they're consultants and two because they get bored when the problem has been solved), they won't be able to find competent developers to take over and the whole thing will be rebuilt again in something they CAN find developers for. I mean it's just NodeJS, but the architecture is distributed and much harder to manage than in a simple monolithic app.
> just making up scenarios where you need to scale to 100x isn't actually engineering.
Even for "peak" Amazon's concern seemed limited to about ~5x daily TPS maximums unless one had extraordinary evidence.
The counter-balance to limiting resource balooning to 5x scale is introducing Erlang-B modelling. Depending on how many 9s you require, you may need way more service availability than expected.
The 100x calculations are probably doubly wrong (both too large and too small), providing negative value false confidence.
Even for "peak" Amazon's concern seemed limited to about ~5x daily TPS maximums unless one had extraordinary evidence.
The counter-balance to limiting resource balooning to 5x scale is introducing Erlang-B modelling. Depending on how many 9s you require, you may need way more service availability than expected.
The 100x calculations are probably doubly wrong (both too large and too small), providing negative value false confidence.
With few exceptions, the motivation for scale that isn't driven by a real need in the real world is because someone wants to experiment with distributed systems and is looking for an excuse. I think it comes from a good place -- wanting to learn something new, or add a new skill to your toolbox.
However, what happens is most of the time the product or company will never even reach that scale so that effort will be wasted. If you're not able to close the loop and actually support a running system at scale, with a real workload (not simulated or ab stats) it actually does the opposite: you don't learn what really works and what doesn't. It's easy to convince yourself you have built a system that can handle such scale based off back of the envelope calculations, synthetic testing, or cargo-culting.
Most of what actually happens with scale in the real world has to do with old & boring things like appropriate choice of data structures, cache design, load balancing, locality, etc. and has nothing to do with distributed computing, serverless, etc. which are just tools that have specific characteristics that might make them good choices or not.
However, what happens is most of the time the product or company will never even reach that scale so that effort will be wasted. If you're not able to close the loop and actually support a running system at scale, with a real workload (not simulated or ab stats) it actually does the opposite: you don't learn what really works and what doesn't. It's easy to convince yourself you have built a system that can handle such scale based off back of the envelope calculations, synthetic testing, or cargo-culting.
Most of what actually happens with scale in the real world has to do with old & boring things like appropriate choice of data structures, cache design, load balancing, locality, etc. and has nothing to do with distributed computing, serverless, etc. which are just tools that have specific characteristics that might make them good choices or not.
Yeah unless you are designing the replacement for a system that has already reached it's scalability limits you shouldn't be worrying too much other than not doing very silly things architecturally.
When you are designing replacements though you need to have an idea of what your scalability runway is before the next re-design will be possible. Sometimes that is just 2x, often times it's 10x, occasionally it's 100x but it's all situational.
When you are designing replacements though you need to have an idea of what your scalability runway is before the next re-design will be possible. Sometimes that is just 2x, often times it's 10x, occasionally it's 100x but it's all situational.
> The big mistake I see people make is trying to be overly clever and predict the future workload needs. It never works out like that. Design for your current workload now, if your architecture can handle 10x of that great! Each time your scale from 10x workload you will likely need to redesign the system anyway, so you shouldn't pay that tax unless you absolutely have to.
I’d also add that the problem you hit first is almost certainly going to be something you aren’t expecting. I’ve seen so many times where people spent time on the cool sexy problems and then years dealing with the scaling problems in their algorithms or other services instead.
I’d also add that the problem you hit first is almost certainly going to be something you aren’t expecting. I’ve seen so many times where people spent time on the cool sexy problems and then years dealing with the scaling problems in their algorithms or other services instead.
“For every order of magnitude change everything breaks.” An oversimplified rule of thumb - but I think it applies here.
Obviously, the article is microservice apologia, but...
> They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS...
No, it was no longer a microservice; it became a plain service, as in SOA. It was no longer micro. That's the whole point.
They could have saved time and money had they just made the thing a plain service to begin with. This thing was not sufficiently complicated to warrant the complexity of starting with a bunch of microservices.
The article says many hot takes have missed the point, but I think what we're seeing here is an example of the two sides talking past one another since the author hasn't appreciated the opposition's arguments at all.
> They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS...
No, it was no longer a microservice; it became a plain service, as in SOA. It was no longer micro. That's the whole point.
They could have saved time and money had they just made the thing a plain service to begin with. This thing was not sufficiently complicated to warrant the complexity of starting with a bunch of microservices.
The article says many hot takes have missed the point, but I think what we're seeing here is an example of the two sides talking past one another since the author hasn't appreciated the opposition's arguments at all.
Yes, and that’s not the only example of microservice apologia:
> They state in the blog that this was quick to build, which is the point. When you are exploring how to construct something, building a prototype in a few days or weeks is a good approach.
First, it’s a huge stretch to say its simpler to use microservices. Anything distributed has to deal with consistency, dropped messages, serialization, propagation latency etc. If you choose to ignore those aspects, that doesn’t make it simpler, it just leaves a wrapped gift of complexity to your future coworkers who will have to maintain it.
Secondly, this wasn’t the case of building something exploratory for future unpredictable workloads. All the requirements were already available. This tells you an important story: Amazon engineers were not able to estimate upfront how much “serverless” resources were needed, and how this “microservice mesh” would perform. This isn’t surprising, because the more complex a system is, the harder it is to predict how it’ll work under some workload. And it doesn’t help that the microservice preachers have been actively discouraging developers to think about infrastructure.
I don’t have a horse in this race. I often hold off with judgment until I have seen the defending side speak. In this case, the defense only strengthens the cause for concern.
> They state in the blog that this was quick to build, which is the point. When you are exploring how to construct something, building a prototype in a few days or weeks is a good approach.
First, it’s a huge stretch to say its simpler to use microservices. Anything distributed has to deal with consistency, dropped messages, serialization, propagation latency etc. If you choose to ignore those aspects, that doesn’t make it simpler, it just leaves a wrapped gift of complexity to your future coworkers who will have to maintain it.
Secondly, this wasn’t the case of building something exploratory for future unpredictable workloads. All the requirements were already available. This tells you an important story: Amazon engineers were not able to estimate upfront how much “serverless” resources were needed, and how this “microservice mesh” would perform. This isn’t surprising, because the more complex a system is, the harder it is to predict how it’ll work under some workload. And it doesn’t help that the microservice preachers have been actively discouraging developers to think about infrastructure.
I don’t have a horse in this race. I often hold off with judgment until I have seen the defending side speak. In this case, the defense only strengthens the cause for concern.
What's the difference between a microservice and an SOA service?
One thing to note is that the transition from smaller to larger services tends to be straight forward, while cutting into smaller can be tricky. Thus IMO there is some merit to keeping them small in the beginning until you can analyse them under production workloads.
That being said, in this case here I think some very simple performance / cost modelling would have shown the issues with serverless already in the beginning. I do find serverless architecture useful, but not for such a case with heavy base load. Furthermore, data locality is also an important aspect to consider in anything with strong latency or throughput requirements - serverless or not.
That being said, in this case here I think some very simple performance / cost modelling would have shown the issues with serverless already in the beginning. I do find serverless architecture useful, but not for such a case with heavy base load. Furthermore, data locality is also an important aspect to consider in anything with strong latency or throughput requirements - serverless or not.
It's the other way around - start with monolith (because it's easier to change things, it's just single pr addressing all places at once) and then, possibly few years later, whatever has crystalized and has clear boundaries with little to no changes coming in or changes contained within this boundary - can be potentially extracted.
Just look how over-represented RoR is/was as bootstrap tech in known, successful companies.
Microservices is not yes/no - it's a slider. You may find sweet spot at ie. 50/50 split like ie. github does, have more or less microservices while keeping core in monolith or services under single versioned monorepo.
Microservices are good for satellite services like system integration, pre/post-processing, gateways etc.
As a rule of thumb whatever can fit single (tech lead + team)'s "head" - can be monolithic (single monolith or set of services under single versioned monorepo managed by that team). Their job is to provide stable apis/uis - otherwise it can be seen as black box by other teams.
This is natural way things evolve (teams are created around naturally bounded concepts) and the suggestion is - don't break it by creating mismatches, keep it in harmony. If something creates measurable problems, slowly form team around it with new tech lead from existing team and let it grow on it's own - it'll extract itself to separate subsystem by itself. It doesn't have to be done overnight.
It's astounding how many people use "scale" as chupacabra to scare everybody in the meeting room without clearly defining what they mean by "scale". To make decision around changing architecture to scale you need to precisely define what it means, have metrics, have benchmarks showing current limits, proof it's a problem now or near future and focus just on those actual issues, if they even exist.
Just look how over-represented RoR is/was as bootstrap tech in known, successful companies.
Microservices is not yes/no - it's a slider. You may find sweet spot at ie. 50/50 split like ie. github does, have more or less microservices while keeping core in monolith or services under single versioned monorepo.
Microservices are good for satellite services like system integration, pre/post-processing, gateways etc.
As a rule of thumb whatever can fit single (tech lead + team)'s "head" - can be monolithic (single monolith or set of services under single versioned monorepo managed by that team). Their job is to provide stable apis/uis - otherwise it can be seen as black box by other teams.
This is natural way things evolve (teams are created around naturally bounded concepts) and the suggestion is - don't break it by creating mismatches, keep it in harmony. If something creates measurable problems, slowly form team around it with new tech lead from existing team and let it grow on it's own - it'll extract itself to separate subsystem by itself. It doesn't have to be done overnight.
It's astounding how many people use "scale" as chupacabra to scare everybody in the meeting room without clearly defining what they mean by "scale". To make decision around changing architecture to scale you need to precisely define what it means, have metrics, have benchmarks showing current limits, proof it's a problem now or near future and focus just on those actual issues, if they even exist.
> (because it's easier to change things, it's just single pr addressing all places at once) and then, possibly few years later, whatever has crystalized and has clear boundaries with little to no changes coming in or changes contained within this boundary - can be potentially extracted.
From my experience you get the best of both worlds by having a mono-repo but try to keep your services small-ish. E.g. for a reporting framework we have separate services for source data extraction, one for view transformations and one for exports. We can always recombine that (and we do actually integrate it locally in a single process for dev/debugging purposes), but it does enforce some good practices in keeping the boundaries clean IMO.
Note that I wasn't arguing at all to just go blindly all-in on Microservices, I was just saying there is some merit and YMMV.
From my experience you get the best of both worlds by having a mono-repo but try to keep your services small-ish. E.g. for a reporting framework we have separate services for source data extraction, one for view transformations and one for exports. We can always recombine that (and we do actually integrate it locally in a single process for dev/debugging purposes), but it does enforce some good practices in keeping the boundaries clean IMO.
Note that I wasn't arguing at all to just go blindly all-in on Microservices, I was just saying there is some merit and YMMV.
Yes, we do it as well - we have local monolith workspace package that combines all services and simulators in single node process. We don't use it on any environments, just for local development and ci. It works very well (very fast, very little resources and with simulators for external services you can work even on a plane without internet if you want and system behaves very much like real one in production).
And yes, non-microservices doesn't imply monolith. We have many services. But they're not microservices because they don't have their own data store and distinct versioning/deployment - they're part of monorepo.
And yes, non-microservices doesn't imply monolith. We have many services. But they're not microservices because they don't have their own data store and distinct versioning/deployment - they're part of monorepo.
I see a lot of about architecture, service design and complexity. One of the key missing piece is around Amazon culture. Promo-driven design is a real deal at Amazon. I am sure this design got someone promoted to principal engineer or higher. If you cannot build complexity into the design, how can you prove your caliber and get promoted.
Definitely a real phenomenon. Obscure overengineering also serves another purpose: if you want to build a mini-empire, people leave you alone because they don’t have time to fully grok your design, so they can’t provide meaningful feedback. In these companies the burden-of-proof falls on the critic of a design, not the designer. Simply saying “this looks too complex” is treated as “oh, they couldn’t understand this sophisticated piece of engineering”. This is a cultural issue. If your peers don’t grok it quickly, you should return to the drawing board, or at least provide evidence for the extraordinary circumstances that warrants all that complexity.
The problem with statement like "this looks too complex" is that it lacks substance and is used as a stick to beat people with, and often has alternative meanings.
To combat this, you need to justify the complaint.
In the case of lambda, I think being tied into an ecosystem of things that you can't run locally is a net loss, but Amazon has done a terrific job at making a bunch of people not super comfortable with compute, see just running a share-nothing process as the way to bridge the gap, like PHP never existed.
Same thing happens with IT orgs that end up owning cloud. You get azure, regardless of whether you want such a thing.
To combat this, you need to justify the complaint.
In the case of lambda, I think being tied into an ecosystem of things that you can't run locally is a net loss, but Amazon has done a terrific job at making a bunch of people not super comfortable with compute, see just running a share-nothing process as the way to bridge the gap, like PHP never existed.
Same thing happens with IT orgs that end up owning cloud. You get azure, regardless of whether you want such a thing.
> The problem with statement like "this looks too complex" is that it lacks substance and is used as a stick to beat people with, and often has alternative meanings.
I’ve never seen anyone say this at $faang-gig. Ive certainly never been it used as a stick. While it’s true that any process can be abused, I’m taking about default burden of proof.
> To combat this, you need to justify the complaint.
Yeah, of course you can’t just say a magic word and expect others to fall in line. I’m claiming there should be justification for proposed increases in complexity. There is an asymmetry in where adding complexity is so trivial it is often accidental, whereas reducing complexity is labor intensive and so difficult that it’s often not cost-effective to even attempt it. Because of this asymmetry, complexity accumulates uncontrollable over time, unless you actively reduce it (expensive) or prevent it before it is implemented (cheaper). Otherwise your software engineers turns into software alchemists, imo.
I’ve never seen anyone say this at $faang-gig. Ive certainly never been it used as a stick. While it’s true that any process can be abused, I’m taking about default burden of proof.
> To combat this, you need to justify the complaint.
Yeah, of course you can’t just say a magic word and expect others to fall in line. I’m claiming there should be justification for proposed increases in complexity. There is an asymmetry in where adding complexity is so trivial it is often accidental, whereas reducing complexity is labor intensive and so difficult that it’s often not cost-effective to even attempt it. Because of this asymmetry, complexity accumulates uncontrollable over time, unless you actively reduce it (expensive) or prevent it before it is implemented (cheaper). Otherwise your software engineers turns into software alchemists, imo.
Well, Amazon running applications on AWS Lambda is more like them having a general purpose cluster which all their services borrow when they need to run some boring unpredictable tasks than it being some bizarre remote service thing. I mean, it's theirs, it's technically local. Local enough.
The article is more about bundling together lambda functions into horizontally scalable containers. Paraphrasing the author:
> I think the prime video team's presentation should have been called "Moving from lambda to container microservices"
There is nothing particularly bad about this move. Prototype something in a FaaS, move it to a container based microservices when you know which features you want and how they perform.
Not doing much FaaS prototyping myself but I agree with the author that FaaS beats containers in time to production (even if the difference might be negligible in some cases)
> I think the prime video team's presentation should have been called "Moving from lambda to container microservices"
There is nothing particularly bad about this move. Prototype something in a FaaS, move it to a container based microservices when you know which features you want and how they perform.
Not doing much FaaS prototyping myself but I agree with the author that FaaS beats containers in time to production (even if the difference might be negligible in some cases)
Exactly. The article is stating that there is nothing wrong with the move Prime Video did, and that it was a "serverless to containers" refactoring that somehow the Internet misrepresented as "microservices to monolith" and kicked up viral "even Amazon admits that microservices is overengineering!" meme discussions. Which was obvious to anyone who had read the original post that was not the case, but most people were commenting on what they thought the (incorrect) title meant.
I am disappointed that the top comment and all of the discussion is once again not related to the article at hand in the slightest and just focused on the easy "complex is bad, guys!" dunk.
I am disappointed that the top comment and all of the discussion is once again not related to the article at hand in the slightest and just focused on the easy "complex is bad, guys!" dunk.
> they had some overly chatty calls between AWS lambda functions and S3. They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS, and which is invoked via a lambda function. This is only one of many microservices that make up the Prime Video application. The problem is that they called this refactoring a microservice to monolith transition, when it’s clearly a microservice refactoring step
If I understood the post at all, I disagree.
One can spend hundreds of HN comments discussing technology stacks, monoliths, etc, and this is important: it affects maintainability, developer hours, and money spent on orchestration. For some applications, almost the entire workload is orchestration, and this discussion makes sense.
But for this workload, actual work is being done, and it can be quantified and priced. So let’s get to the basics: an uncompressed 1080p video stream is something like 3 Gbps. It costs a certain amount of CPU to decompress. It costs a certain amount of CPU to analyze.
These have price tags. You can pay for that CPU at EC2 rates or on-prem rates or Lambda rates or whatever. You can calculate this! And you pay for that bandwidth. You can pay for 3Gbps per stream using S3: you pay S3 and you pay EC2 (or whatever) because that uses 1/10th of the practical maximum EC2 <-> S3 bandwidth per instance. Or you pay for the fraction of main memory bandwidth used if you decode and analyze on the same EC2 instance (or Lambda function or whatever). Or you pay for the fraction of cache bandwidth used if you decide partial frames and analyze without ever sending to memory.
And you’ll find that S3 is not “chatty”. It’s catastrophic. In fact, any use of microservices here is likely catastrophic if not carefully bound to the same machine.
If I understood the post at all, I disagree.
One can spend hundreds of HN comments discussing technology stacks, monoliths, etc, and this is important: it affects maintainability, developer hours, and money spent on orchestration. For some applications, almost the entire workload is orchestration, and this discussion makes sense.
But for this workload, actual work is being done, and it can be quantified and priced. So let’s get to the basics: an uncompressed 1080p video stream is something like 3 Gbps. It costs a certain amount of CPU to decompress. It costs a certain amount of CPU to analyze.
These have price tags. You can pay for that CPU at EC2 rates or on-prem rates or Lambda rates or whatever. You can calculate this! And you pay for that bandwidth. You can pay for 3Gbps per stream using S3: you pay S3 and you pay EC2 (or whatever) because that uses 1/10th of the practical maximum EC2 <-> S3 bandwidth per instance. Or you pay for the fraction of main memory bandwidth used if you decode and analyze on the same EC2 instance (or Lambda function or whatever). Or you pay for the fraction of cache bandwidth used if you decide partial frames and analyze without ever sending to memory.
And you’ll find that S3 is not “chatty”. It’s catastrophic. In fact, any use of microservices here is likely catastrophic if not carefully bound to the same machine.
This is called mechanical sympathy and most architects and developers do not have it.
Computer architectures encompass 13 orders of magnitude of performance! That's roughly the difference between something like a trivial function call processing data in L1 cache to a remote call out to something in another region.
People often make relatively "small" mistakes of 3 or 4 orders of magnitude, which is still crazy if you think about it, but that's considered to be a minor sin relative to making the architecture diagram look pretty.
Computer architectures encompass 13 orders of magnitude of performance! That's roughly the difference between something like a trivial function call processing data in L1 cache to a remote call out to something in another region.
People often make relatively "small" mistakes of 3 or 4 orders of magnitude, which is still crazy if you think about it, but that's considered to be a minor sin relative to making the architecture diagram look pretty.
So it didn't make sense to separate some functionality. So what? No one who advocates microservices would say you can cut a service along any arbitrary line and expect good results.
Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system.
So it's humorous to see an Amazon team that is doing things correct and holistically in contrast to the sales and marketing bullshit that AWS and Cloud evangelists spew.
Us nerds have a great way of missing the point and bikeshedding on the technicals (which is also great sometimes!)
So it's humorous to see an Amazon team that is doing things correct and holistically in contrast to the sales and marketing bullshit that AWS and Cloud evangelists spew.
Us nerds have a great way of missing the point and bikeshedding on the technicals (which is also great sometimes!)
It's amazing how cloud companies managed to convince so many people in software engineering that software engineering wasn't a core competence in producing software.
I think the difference was simple vs easy. They made a few of the things very easy.
Yeah cause engineers aren’t ones typically doing the buying
Same reason MSFT SWEs use Mac over Windows -- they've seen too much.
> Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system.
As someone who knows a little bit about what AWS recommends to customers when first hand, I can absolutely guarantee you that AWS does not recommend serverless to everyone.
The vast majority of projects that consultants do by revenue at any cloud consulting company (including AWS’s own internal company) is regular old VMs.
As someone who knows a little bit about what AWS recommends to customers when first hand, I can absolutely guarantee you that AWS does not recommend serverless to everyone.
The vast majority of projects that consultants do by revenue at any cloud consulting company (including AWS’s own internal company) is regular old VMs.
Thank you.
It seems that most of the takes against this article are coming from people that have opinions about AWS based on nothing.
Sure, AWS is selling Lambda/Serverless services. Sure, they are making articles/videos on microservices architecture. But they are also selling EC2 instances. They are also making articles/videos on monolithic architecture.
I have never come across any content from AWS telling that this service/architecture/mindset is the best and every company should use/implement it.
Anyone who went through the Well-Architected Framework would be completely aware of that.
But I guess it's easier to mock some folks over the Internet than taking time to grab some knowledge and reflect.
It seems that most of the takes against this article are coming from people that have opinions about AWS based on nothing.
Sure, AWS is selling Lambda/Serverless services. Sure, they are making articles/videos on microservices architecture. But they are also selling EC2 instances. They are also making articles/videos on monolithic architecture.
I have never come across any content from AWS telling that this service/architecture/mindset is the best and every company should use/implement it.
Anyone who went through the Well-Architected Framework would be completely aware of that.
But I guess it's easier to mock some folks over the Internet than taking time to grab some knowledge and reflect.
I am so baffled by this whole discussion. Isn’t serverless (aka event-driven distributed systems made up of small components, with no reserved capacity) just one of many options? I write code for 8-bit controllers that I’d never run on Lambda (even though I’m a Lambda/Step Functions freak). I need 90s server software that can only run in a VM.
It’s a complicated world and your favorite tool/model just isn’t going to cut it everywhere. Move on maybe?
It’s a complicated world and your favorite tool/model just isn’t going to cut it everywhere. Move on maybe?
I still don't really get it. There are so many simple, easy to build, cheap to run three tier frameworks out there so why bother with all the hassle of porting at all? If your system gets super busy then you can repurpose the code with higher performance and well tuned DBs later. I spent 15 years doing that for some of the busiest systems in the world, but those started on a single cpu.
At my client, we go Serverless first in a bastardized state, NestJS. The benefit is quick development at a low cost. We have a series of calculations. Each read from a common S3 and write to it. This allows the teams to do their work with independent deployments.
Each one costs about $5 a month. That’s rather hard to get in the AWS EC2 world. They are also easier to manage. We don’t have to manage keys or certain. There is no redeploy after 90s that comes with EC2.
However these are low access, slowish apps. Maybe a 1,000 calls a day. They can take a minute to do their thing. Seems like a nice match for lambdas.
Benefit of NestJS is if we ever needed to move out of lambda we have a tradition all Express app.
Each one costs about $5 a month. That’s rather hard to get in the AWS EC2 world. They are also easier to manage. We don’t have to manage keys or certain. There is no redeploy after 90s that comes with EC2.
However these are low access, slowish apps. Maybe a 1,000 calls a day. They can take a minute to do their thing. Seems like a nice match for lambdas.
Benefit of NestJS is if we ever needed to move out of lambda we have a tradition all Express app.
Useful insight, but whether it's 5$ or 50$ a month, compared to dev costs that's a rounding error pretty much everywhere.
That's why we went with NestJS. It's like Spring but for TypeScript. We can run everything locally. Boot up the NestJS service at localhost:8080 with LocalStack running on the side for access to S3 and DynamoDB compatible, working services.
If you write your NestJS application with nice boundaries (SOA or DDD) you can split it up as it scales. It's the same approach a lot of .NET guys take, write it modularly and scale out just the bits that need the throughput.
The point is: managers and keywords
Yeah, I don't really understand why people are making a big deal about this. Two lambda functions passing data to step functions? Not exactly the best serverless architecture. And if you think about the problem of checking video streams, there is really no reason to scale different parts of it independently because you just need one process to watch the video and check for errors. Plus they still deployed the new solution on ECS, so it isn't really a monolith application that runs on a fixed number of servers. It's a real nothing story - the Prime Video team was just doing some engineering maintenance.
Honestly the bigger story is that it was mischaracterized from the beginning, but I would imagine that is more a result of things coming together from multiple different systems and teams more than the way someone designed this stuff.
Honestly the bigger story is that it was mischaracterized from the beginning, but I would imagine that is more a result of things coming together from multiple different systems and teams more than the way someone designed this stuff.
What's funny to me, is that I focused on something completely different; how independent these teams at Amazon are. The move from pretty different infrastructure services, all at the same company. I can't think of a single place I've worked at in 20 years where these infra teams wouldn't be either: a.) mandated, i.e., "you will stick with lambda/kubernetes/whatever", or b.) review and be involved in the architectural decisionmaking of the video service approach, i.e., have to politick your way around any deviation.
I guess the shift from "mono to micro" just isn't very interesting to me. You can usually change your definition of either concept to fit your architecture. This just seemed like the team did the math and revised their approach. Good for them!
I guess the shift from "mono to micro" just isn't very interesting to me. You can usually change your definition of either concept to fit your architecture. This just seemed like the team did the math and revised their approach. Good for them!
This is in part where the "It's always Day 1" mentality manifests at Amazon.
There's only one programming language that is forbidden (PHP - for wrong reasons), and only one cloud vendor that is mandated (obvious). But beside that, teams are able to use literally whatever technology they want. There are some technologies that impose themselves because of the ecosystem, but you can always do your own sauce if needed.
I've been in two teams that had basically 90% different tech stacks, but it's never a problem, and it doesn't really ever come up in design reviews (unless there's a real reason, not for personal preferences).
There's only one programming language that is forbidden (PHP - for wrong reasons), and only one cloud vendor that is mandated (obvious). But beside that, teams are able to use literally whatever technology they want. There are some technologies that impose themselves because of the ecosystem, but you can always do your own sauce if needed.
I've been in two teams that had basically 90% different tech stacks, but it's never a problem, and it doesn't really ever come up in design reviews (unless there's a real reason, not for personal preferences).
Curious though, what is the reason for banning PHP? It must be pretty specific if they singled out just PHP.
I don't know the full history because it's from way before I joined, but it is honestly a terrible take (by a guy who's now Distinguished Engineer at Apple, no less!) explaining that it is impossible to write secure applications in PHP.
Basically, they used "the historical number of vulnerabilities identified in applications developed using a technology" as metric to determine how insecure a technology is.
For the argument here, they looked at the CVE list, where at the time (in 2012, with the list existing since 1999) 40% of all software vulnerabilities recorded were in PHP applications. This led to the conclusion that PHP is insecure by nature.
Of course, he didn't mention that at the time, PHP was also used by 80% of all websites, because that would have made his argument worthless.
That wiki page explaining that is still up. It's so baffling to me when the argument violates so many of Amazon's leadership principles.
Basically, they used "the historical number of vulnerabilities identified in applications developed using a technology" as metric to determine how insecure a technology is.
For the argument here, they looked at the CVE list, where at the time (in 2012, with the list existing since 1999) 40% of all software vulnerabilities recorded were in PHP applications. This led to the conclusion that PHP is insecure by nature.
Of course, he didn't mention that at the time, PHP was also used by 80% of all websites, because that would have made his argument worthless.
That wiki page explaining that is still up. It's so baffling to me when the argument violates so many of Amazon's leadership principles.
Yeah that is a pretty poor reason, thanks for the nugget though that is pretty interesting.
I could think of a bunch of legitimate reasons to want to ban it but they would also hit a bunch of other languages as a result - hence why I was curious how it could be so specific.
It wouldn't surprise me if LOC for LOC PHP written today is much more secure than JS because of the same dynamic, just JS is the one that is now on 80% of new code written by beginners.
I could think of a bunch of legitimate reasons to want to ban it but they would also hit a bunch of other languages as a result - hence why I was curious how it could be so specific.
It wouldn't surprise me if LOC for LOC PHP written today is much more secure than JS because of the same dynamic, just JS is the one that is now on 80% of new code written by beginners.
Even worse than that - Amazon spent dozens of engineer years migrating their wiki from PHP MediaWiki to Java XWiki, pretty much for that reason only, and for dubious customer benefit. There was a very epic post mortem at the end of it.
I didn't mention that in my original response, but the guy that wrote this document putting the nail in the coffin for PHP at Amazon said that ultimately it was too expensive to even TRY to assess PHP's safety. The amount he quoted was only 250,000$.
Looking back it's so obvious that it was an opinionated crusade from one guy, I simply don't understand how one of the most influential programming languages ever got dismissed by a single dude with fallacious arguments in less than a thousand words.
Looking back it's so obvious that it was an opinionated crusade from one guy, I simply don't understand how one of the most influential programming languages ever got dismissed by a single dude with fallacious arguments in less than a thousand words.
Amazon is actually better thought of as a collection of independent startups with their own software architectures, roadmaps and P&Ls. There’s commonly used infrastructure but very little is really mandated at the company level.
Exactly. I'd say the enforcement is on org/PE (Principal Engineer) level, cascading down to Senior engineers.
Once you're designing something beyond a small app, you'd have to pass PE review anyway.
Once you're designing something beyond a small app, you'd have to pass PE review anyway.
Isn't that just a massive maintenance mess?
Am I alone in feeling there is nothing left to be said on this topic? Correct application design, and balancing that against the constraints of getting a product out the door, the requirements of a particular application, and the resources on hand (skill-sets, developers, infrastructure) does not in my opinion boil down to "microservices vs monolith".
Both strategies have tradeoffs, both have produced successes and failures, and choice of approach seems (obviously) too specific to a given context and application for general advice to apply.
Both strategies have tradeoffs, both have produced successes and failures, and choice of approach seems (obviously) too specific to a given context and application for general advice to apply.
Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftman just knows its tools better. There is no magic bullet here.
> Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftsman just knows its tools better. There is no magic bullet here.
Rational take, but I see the debate similar to Roman vs Arabic numerals.
Keeping a tally? Roman. Need to use operators? Arabic. Sometimes you can keep a tally in Arabic (not ideal), and sometimes you can do basic operations on Roman numerals (not ideal).
However, when you want to start using variables, only one tool enables this easily.
I can't architect the types of redundant and properly separated interoperable systems with a monolith that microservices otherwise enable.
So the desire to move forward isn't the need to find a magic bullet, but the next evolution of an existing ability that unlocks new capabilities...
(I don't think calculus would have been discovered using Roman numerals)
Rational take, but I see the debate similar to Roman vs Arabic numerals.
Keeping a tally? Roman. Need to use operators? Arabic. Sometimes you can keep a tally in Arabic (not ideal), and sometimes you can do basic operations on Roman numerals (not ideal).
However, when you want to start using variables, only one tool enables this easily.
I can't architect the types of redundant and properly separated interoperable systems with a monolith that microservices otherwise enable.
So the desire to move forward isn't the need to find a magic bullet, but the next evolution of an existing ability that unlocks new capabilities...
(I don't think calculus would have been discovered using Roman numerals)
No, but you can start a business with a singlesystem, keep things well decoupled (hard / software engineer is hard).. Then you gradually run some parts of the systems in different docker containers, then at some point you completely decoupled systems and have different teams handle different service boundaries. You could still have the core system be a big bigger, and sometimes referred to as monolith. But monolith vs microservices is a polarises the discussion, its a gradual scale. Its the engineering, and keeping things separated that matters.
You're not alone. What doesn't get said enough about this topic is "it doesn't matter". The architecture and design, for the most part, doesn't matter. Because no matter what is chosen, there will be pitfalls. Failing to account for those pitfalls will lead to problems, and changing the design or architecture to avoid the pitfalls will just introduce new pitfalls.
IMHO, the right path is to first become educated about each design and its pitfalls, so that when the time comes, you can quickly pick a design, and move swiftly onto dealing with the pitfalls, because that's where the actual problems lie.
IMHO, the right path is to first become educated about each design and its pitfalls, so that when the time comes, you can quickly pick a design, and move swiftly onto dealing with the pitfalls, because that's where the actual problems lie.
Among the maxims on Lord Naoshige's wall there was this one:
"Matters of great concern should be treated lightly."
Master Ittei commented,
"Matters of small concern should be treated seriously."Of course there is nothing technical to be said. There is plenty of personal anecdote and associated feelings to be shared however, which is what homo sapiens primarily enjoy :)
You get double the thrill if you think you are engaging in a technical conversation all the while arguing anecdotal experiences, beliefs and biases.
You get double the thrill if you think you are engaging in a technical conversation all the while arguing anecdotal experiences, beliefs and biases.
What an excellent observation! I will quote this often. Thank you! I’d even generalize it a bit further.
“You get double the thrill if you think you are engaging in logical reasoning, all the while arguing anecdotal experiences, beliefs and biases.”
“You get double the thrill if you think you are engaging in logical reasoning, all the while arguing anecdotal experiences, beliefs and biases.”
I think there are an increasing number of people that believe that microservice is not in the same risk category as a normal application design. A design we really need to stop calling "monolith" as to PMs, management, etc. it sounds ancient.
So the claim that "both strategies have tradeoffs" is not true for a lot of people. Microservies have bigger tradeoffs and consequences than a normal application design.
So the claim that "both strategies have tradeoffs" is not true for a lot of people. Microservies have bigger tradeoffs and consequences than a normal application design.
[deleted]
I do wonder if anyone actually read the Amazon post before creating their hot takes.
The crux of the post is simple and obvious to anyone who's ever actually developed "low-level" (used loosely) video libraries. Microservices are not suited for video because exchange of video between microservices is substantially more costly than bits of JSON.
The crux of the post is simple and obvious to anyone who's ever actually developed "low-level" (used loosely) video libraries. Microservices are not suited for video because exchange of video between microservices is substantially more costly than bits of JSON.
I don’t quite understand the point in all this. Design tradeoffs are still a thing; did we all just forget?
“Abortions for some... miniature American flags for others!”
“Abortions for some... miniature American flags for others!”
Software architecture is (incorrectly) seen as NP-hard by practitioners, so they avoid it, and instead focus on implementing the current "best practices." However, "best practices" can't be best if they're situational, so we arrive at very local maxima, such as microservices for all.
Many comments seem to miss the point that nowadays native AWS cloud applications are meant to be built by combining various AWS services using Lambda as the glue between them. It's not about serverless vs non-serverless: it's about using cloud services for the purpose they are each good at. In many cases you might actually have more IaC code than Lambda handler code, especially if you can leverage services like AppSync and Step Functions. The application is no longer a set of microservices; it's a set of CDK / CloudFormation stacks and the resources that they define, some of which might be microservices.
> I do think microservices were over sold as the answer to everything, and I think this may have arisen from vendors who wanted to sell Kubernetes with a simple marketing message that enterprises needed to modernize by using Kubernetes to do cloud native microservices for everything
Yes, they absolutely were oversold, Kubernetes marketing was part of it. There was also a lot of resume-driven development, and simply "look, new shiny object!". A lot of young engineers entered the workforce over the last couple of decades. That often comes along with "forget this old stuff, what's the new stuff?".
I use KISS as much as possible with every project I'm involved in desinging.
The microservices push got the point where a dev shops with 10 people had it all broken into microservices and running Kubernetes to manage low-traffic front ends, an API and a database. I would see some of these systems and the only thing I could think was "... but why?".
Do microservices have a place? Yes. Have I seen them used where they shouldn't be? Yes. Have I seen more poorly implemented microservices architecture than good ones. Yes.
Over-engineering the tech stack, or selecting one part of it vs. another based on which is trendier, only leads to more frusturation, difficulty reasoning about it, added development time ... all of which means it will cost more.
Yes, they absolutely were oversold, Kubernetes marketing was part of it. There was also a lot of resume-driven development, and simply "look, new shiny object!". A lot of young engineers entered the workforce over the last couple of decades. That often comes along with "forget this old stuff, what's the new stuff?".
I use KISS as much as possible with every project I'm involved in desinging.
The microservices push got the point where a dev shops with 10 people had it all broken into microservices and running Kubernetes to manage low-traffic front ends, an API and a database. I would see some of these systems and the only thing I could think was "... but why?".
Do microservices have a place? Yes. Have I seen them used where they shouldn't be? Yes. Have I seen more poorly implemented microservices architecture than good ones. Yes.
Over-engineering the tech stack, or selecting one part of it vs. another based on which is trendier, only leads to more frusturation, difficulty reasoning about it, added development time ... all of which means it will cost more.
That reminds me of a somewhat funny story. I was listening to a presentation about a startup with about 10 people as you mentioned doing hospital digitalization (gist of it seemed to be sending patient data from one doctor to another). They explained that they wanted to choose a “robust” architecture so they chose microservices (kubernetes and all). I thought that was a bit odd for something hosted at a hospital locally especially because they had like 7 or so services (and maybe even two databases but not sure on that one).
Well, we later asked them how much data they were even handling and (after a bit of side-stepping) they said it was around 200mb excluding images.
That was my “but why?” moment.
Apparently you need 7 services to have a web frontend for 200mb of data with most of it probably never accessed and hospitals can suddenly get three new floors overnight.
Seeing all the acolytes of serverless get triggered by this has been enjoyable to watch
I'm surprised Amazon published what they did. AWS make a lot of money selling hyped up buzz word tech to incompetent companies.
Prime Video, Lambda, EC2, everything in discussions still revolves around Amazon.
Serverless isn't "cheap". It's rightsized. If you aren't serving 24/7, it makes sense to move to an on-demand model. However if you compare the unit of compute pricing, Lambda is comparable to other services, and likely more expensive than buying compute wholesale. You can just buy it in smaller portions.
Cant' we just ask the "Application Deployer" to upload an EAR, published by the "Application Assembler", and run it on any of the spec-compatible servers?
These problems have been solved decades ago.. 2.5 to be exact
These problems have been solved decades ago.. 2.5 to be exact
The original, unedited title is "So many bad takes — What is there to learn from the Prime Video microservices to monolith story"
I think Agile workflow has a part to play in this. It embraces complexity and moving fast to build out buggy features, where every new bug becomes a new user story. It feels productive but it just creates more work than you need.
Nothing is ever truly a mistake, it's much more about understanding what you are trading off between real hardware, cloud servers, something like fly or heroku or vercel and purer serverless functions on AWS or whatever.
Seeing as how a lot of players are going in on serverless (vercel, netlify, deno, cloudflare) the cynic in me wonders if this is a calculated move on part of Amazon to bring people back to using ec2 .. etc
for a new project that I had to architect, we had a section that was web facing... and it was previously prototyped with React. When i spoke to the team none of them really had React experience but most had AngularJS experience. A lot of the benefits we saw was intermediate to expert knowledge in our stack and it was having an open conversation with the development team + organization to understand it instead of proposing something that's way out in left field!
Dup. Already discussed on HN less than a day ago [0].
[0] https://news.ycombinator.com/item?id=35847988
[0] https://news.ycombinator.com/item?id=35847988
No, it's a different link. The one in this topic is addressing the link you posted.
The points in the Prime Video article still stand. People overuse microservices and serverless and stuff gets expensive quickly.
It’s interesting that there are folks who think there must be only one architecture to be the best in all cases.
It reeks of inexperience and being highly opinionated followers of popular opinion instead of playing with both personally.
So few projects actually scale to benefit from microservices where its a benefit.
One’s interpretation and benefit for microservices is fine. It’s just not guaranteed to be factual in many used cases.
First, the line of where scaling benefits from microservices is also moving higher each year due to the same underlying reason:
Whether Monolith or Microservices, the increased performance of Linux networking, availability of inexpensive ram and cpu along with much more optimized databases like Postgres or mysql from the past 10-15 years ago seems to have been missed by many people who seem to have set an interpretation in place and never looked back.
When horsepower gets faster and cheaper each year the architectural best practices of microservices adoption was likely relative to its year of adoption.
What to do?
Keep your stack unapologetically simple. Don’t handcuff your development speed when the complexity arrives to rearchitect things anyways. Because the first version will have the wrong assumptions about what customers want.
How?
Consider monorepos to let you start as monolith for velocity and move to microservices as needed
Monolithic apps can have a service based architecture internally (monorepos can be pretty handy), which in turn lets startups ship much more regularly than they ought think for monoliths.
You can design folders in a monorepo at the start to fuel a monolith and keep a path to go to micro services, where and if needed.
To know this enough experience (both good and bad) is needed with both microservices and monoliths.
It reeks of inexperience and being highly opinionated followers of popular opinion instead of playing with both personally.
So few projects actually scale to benefit from microservices where its a benefit.
One’s interpretation and benefit for microservices is fine. It’s just not guaranteed to be factual in many used cases.
First, the line of where scaling benefits from microservices is also moving higher each year due to the same underlying reason:
Whether Monolith or Microservices, the increased performance of Linux networking, availability of inexpensive ram and cpu along with much more optimized databases like Postgres or mysql from the past 10-15 years ago seems to have been missed by many people who seem to have set an interpretation in place and never looked back.
When horsepower gets faster and cheaper each year the architectural best practices of microservices adoption was likely relative to its year of adoption.
What to do?
Keep your stack unapologetically simple. Don’t handcuff your development speed when the complexity arrives to rearchitect things anyways. Because the first version will have the wrong assumptions about what customers want.
How?
Consider monorepos to let you start as monolith for velocity and move to microservices as needed
Monolithic apps can have a service based architecture internally (monorepos can be pretty handy), which in turn lets startups ship much more regularly than they ought think for monoliths.
You can design folders in a monorepo at the start to fuel a monolith and keep a path to go to micro services, where and if needed.
To know this enough experience (both good and bad) is needed with both microservices and monoliths.
> It’s interesting that there are folks who think there must be only one architecture to be the best in all cases.
> To know this enough experience (both good and bad) is needed with both microservices and monoliths.
Over the years I have worked with a huge number of people who are looking for Silver Bullets and Best Practices, generally for one (or both) of these reasons:
- To be blameless if the architecture doesn't work out (building with microservices is a Best Practice, it's not my fault it got very expensive)
- To cut down on the number of deep-thinking decisions they have to make
And to be fair on the second point, I don't necessarily disagree with the premise; if these kinds of decisions allow you to get going quicker, it's not necessarily the wrong choice at the moment. What everyone needs to keep in mind, though, is that if you don't do deep architectural thinking and planning at the start of a project (or even if you do sometimes...), you'll have to revisit it later. Often the same forces and pressures that led to taking shortcuts at the early architectural stages of a project will continue to be present down the road, and it'll be more work to fix it later than to do it right the first time.
It's a tough balance. If there's a high risk that the project won't be alive in 6 months it probably doesn't matter. If it's a prototype that's very likely to get into and stay in production, then it really does matter.
> To know this enough experience (both good and bad) is needed with both microservices and monoliths.
Over the years I have worked with a huge number of people who are looking for Silver Bullets and Best Practices, generally for one (or both) of these reasons:
- To be blameless if the architecture doesn't work out (building with microservices is a Best Practice, it's not my fault it got very expensive)
- To cut down on the number of deep-thinking decisions they have to make
And to be fair on the second point, I don't necessarily disagree with the premise; if these kinds of decisions allow you to get going quicker, it's not necessarily the wrong choice at the moment. What everyone needs to keep in mind, though, is that if you don't do deep architectural thinking and planning at the start of a project (or even if you do sometimes...), you'll have to revisit it later. Often the same forces and pressures that led to taking shortcuts at the early architectural stages of a project will continue to be present down the road, and it'll be more work to fix it later than to do it right the first time.
It's a tough balance. If there's a high risk that the project won't be alive in 6 months it probably doesn't matter. If it's a prototype that's very likely to get into and stay in production, then it really does matter.
Totally agree on the tough balance. At best it's a trade off of different forms of technical, process, and business debt.
Doing deep enough architectural thinking and planning at the start to focus on maintaining flexibility (which can be achieved in many ways) without premature optimization or engineering can go a long way. Basically, don't work yourself into a corner or concrete too quickly by trying to do too much, or too little is an approach I try to keep in mind. Technical and architectural debt comes with interest as well.
One way to reduce the architectural risk is to make the first version a throwaway. It can burn through assumptions. Similar to design thinking, going broad and converging twice can help get a better sense of vector,
Doing deep enough architectural thinking and planning at the start to focus on maintaining flexibility (which can be achieved in many ways) without premature optimization or engineering can go a long way. Basically, don't work yourself into a corner or concrete too quickly by trying to do too much, or too little is an approach I try to keep in mind. Technical and architectural debt comes with interest as well.
One way to reduce the architectural risk is to make the first version a throwaway. It can burn through assumptions. Similar to design thinking, going broad and converging twice can help get a better sense of vector,
There are fundamental reasons why microservices have become a bit less popular though. If you see a good, elegant microservice architecture, it looks very appealing: independently scalable parts, greater redundancy, etc.
The problem comes when you try to start a new project and begin to realize that except in simple cases, it's exceptionally hard to know ahead of time how you would best be served to split an application up into services. Once there's a network boundary between two pieces of code, there's now an API that needs to be managed carefully: deployments of the two pieces of code may not be in lockstep, rolling one back may be necessary, etc. If you got the split wrong, the refactoring to fix that could be extremely hard.
By comparison, splitting logical services out of a monolith isn't always a cakewalk, but it's usually more straightforward, especially since as you mentioned, it's totally possible to just share the same codebase between services. It could even be the same binary invoked differently.
My feeling is that it's easier to split a monolith than to join microservices that were developed separately together; they might have different codebases and possibly even be written in different programming languages. Therefore, it's probably better to start with a monolith even if you're pretty damn sure you will need some services split off later to scale.
Serverless on the other hand I am more skeptical of. I like certain products that provide generic-ish abstractions as a service: Cloud Run for example seems like a good idea, at least on paper, and Amazon Aurora seems relatively reasonable, too. But I think that they are often too expensive and do not provide enough value to justify it, especially since a lot of serverless tools don't provide terribly generic APIs or services, leading to substantial lock-in.
Both microservices and serverless APIs were sold as near-panaceas to the woes of traditional monoliths, but it's a lot more complicated than that.
edit: When I typed this response, the message I was replying to was different/much shorter.
The problem comes when you try to start a new project and begin to realize that except in simple cases, it's exceptionally hard to know ahead of time how you would best be served to split an application up into services. Once there's a network boundary between two pieces of code, there's now an API that needs to be managed carefully: deployments of the two pieces of code may not be in lockstep, rolling one back may be necessary, etc. If you got the split wrong, the refactoring to fix that could be extremely hard.
By comparison, splitting logical services out of a monolith isn't always a cakewalk, but it's usually more straightforward, especially since as you mentioned, it's totally possible to just share the same codebase between services. It could even be the same binary invoked differently.
My feeling is that it's easier to split a monolith than to join microservices that were developed separately together; they might have different codebases and possibly even be written in different programming languages. Therefore, it's probably better to start with a monolith even if you're pretty damn sure you will need some services split off later to scale.
Serverless on the other hand I am more skeptical of. I like certain products that provide generic-ish abstractions as a service: Cloud Run for example seems like a good idea, at least on paper, and Amazon Aurora seems relatively reasonable, too. But I think that they are often too expensive and do not provide enough value to justify it, especially since a lot of serverless tools don't provide terribly generic APIs or services, leading to substantial lock-in.
Both microservices and serverless APIs were sold as near-panaceas to the woes of traditional monoliths, but it's a lot more complicated than that.
edit: When I typed this response, the message I was replying to was different/much shorter.
Apologies, I hit submit accidentally sooner than intended.
Splitting a monolith, in a monorepo, can be conspired to break into microservices a bit easier than going the other way. There's a lot more tooling becoming available.
The committment to cloud, serverless, devops has occured when the complexity of administering servers was high. Banking on progress, the administering servers has become exponentially simpler. The recent project from the basecamp guys (mrsk?) is an example of this.
You could almost roll your own "serverless" with enough postgres exposing itself as a queue and/api. Once you outgrow it, so be it. Tools like Hasura are quite interesting.
Splitting a monolith, in a monorepo, can be conspired to break into microservices a bit easier than going the other way. There's a lot more tooling becoming available.
The committment to cloud, serverless, devops has occured when the complexity of administering servers was high. Banking on progress, the administering servers has become exponentially simpler. The recent project from the basecamp guys (mrsk?) is an example of this.
You could almost roll your own "serverless" with enough postgres exposing itself as a queue and/api. Once you outgrow it, so be it. Tools like Hasura are quite interesting.
Even monorepos are not perfect for evolving to microservices. It’s incredibly hard to refactor generally. We still lack good monorepo tooling. Tests take huge amounts of time to run. Multiple languages becomes painful especially ones with slow compile speed like Rust.
Someone needs to make a programming language that encourages you to write code that can be easily split and recombined. Something that can do remoting well.
Someone needs to make a programming language that encourages you to write code that can be easily split and recombined. Something that can do remoting well.
I do wish there was more out there about the different ways to design a monorepo and the pros and cons of each.
It took me a while to compile a structure I thought was good to start with but also (hopefully) flexible.
It took me a while to compile a structure I thought was good to start with but also (hopefully) flexible.
> Monolithic apps can have a service based architecture internally
Not only that, but if you absolutely must (for example different scaling requirements), forking out a service from a monolith and deploying it as microservice is relatively straightforward. Provided source code interdependencies have been maintained cleanly before, which can be done with support from tools.
Not only that, but if you absolutely must (for example different scaling requirements), forking out a service from a monolith and deploying it as microservice is relatively straightforward. Provided source code interdependencies have been maintained cleanly before, which can be done with support from tools.
I definitely agree. More than once I've taken a monolith and sliced out a small piece of it to run as a separate service. That's been a significantly less painful task than trying to aggregate multiple microservices into a less expensive monolith, especially when multiple microservices use the same underlying packages but different conflicting versions or different global configuration.
The interesting thing about doing this is how things kind of end back at n-tier architecture, where one can have abstractions between each layers of the cake to more easily swap in/out technologies as well.
Yeah, definitely. In my experience, the trick is that without doing analysis up front, you don't know exactly what the n-tiers are going to look like. Jumping right to microservices right off the bat is putting a lot of extra overhead at the start of the project (defining HTTP or gRPC or whatever APIs, working out the boilerplate for each project, etc.) without knowing whether the boundaries are appropriate or whether a given component is likely to end up being a hotspot that will need independent scaling.
When you start with a monolith that isn't just a ball of spaghetti (e.g. clear module boundaries), you don't have to put as much work into the whole big picture and it's easier to evolve the internal architecture of one larger service than N smaller services. And if you do discover that one piece of it is appropriate to split off (hotspot, reasonably transactional instead of interactive calling, etc), then you just replace the guts of that module with whatever RPC mechanism you're using and the rest of the application doesn't really need to care about it.
When you start with a monolith that isn't just a ball of spaghetti (e.g. clear module boundaries), you don't have to put as much work into the whole big picture and it's easier to evolve the internal architecture of one larger service than N smaller services. And if you do discover that one piece of it is appropriate to split off (hotspot, reasonably transactional instead of interactive calling, etc), then you just replace the guts of that module with whatever RPC mechanism you're using and the rest of the application doesn't really need to care about it.
Analysis is important, if we don't know what the problem is, or how to solve it, finding the problem is the first undertaking, and the architecture for it often expands greatly to me.
I often end up viewing the start of apps without a problem to solve might benefit from being a dynamically generated interface set by from a combination of feature flags from subscription to a plan.
That degree of flexibility can seem extreme, but until the patterns from the chaos don't emerge, it can be pretty tough.
As a total aside - Is there/why isn't there a discourse forum for architecture folks to chat about things like this?
This has been one of the most enjoyable threads I've been a part of in HN in recent memory.
I often end up viewing the start of apps without a problem to solve might benefit from being a dynamically generated interface set by from a combination of feature flags from subscription to a plan.
That degree of flexibility can seem extreme, but until the patterns from the chaos don't emerge, it can be pretty tough.
As a total aside - Is there/why isn't there a discourse forum for architecture folks to chat about things like this?
This has been one of the most enjoyable threads I've been a part of in HN in recent memory.
> As a total aside - Is there/why isn't there a discourse forum for architecture folks to chat about things like this?
Occasionally I see good discussions on Twitter. I just tried to find a couple of names of people and sadly I can't seem to remember any of them. Sorry.
Otherwise... the discourse generally seems to be "this is The One True Way of doing things and if you're not you should feel bad". And that's not helpful :)
Occasionally I see good discussions on Twitter. I just tried to find a couple of names of people and sadly I can't seem to remember any of them. Sorry.
Otherwise... the discourse generally seems to be "this is The One True Way of doing things and if you're not you should feel bad". And that's not helpful :)
How do you work around DB transactions? For example, in many Spring apps the transaction is common across all DB calls for an HTTP request. Would you put @RequiresNew? Many data sources don’t support this for a singular DB.
This has been posted like 6 times already man.
Serverless first is a mistake. It should be "serverless where it's distinct advantages are actually useful" which is way less common than people think.
Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong.
All the points noted in the slides etc just come from poor engineering leadership. i.e decisions/choices, etc. Constaints do speed up development but they can be enforced by leadership, not by putting yourself in an endless pit of technical debt.
The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making.
Otherwise assuming you have the mandate necessary to get stuff built the right way just stick with the Boring Tech stack of JVM + PG + k8s. It solves all problems regardless of size, there are minimal decisions to be made (really just Spring or Quarkus atm) and you won't spend time dicking around with half-baked serverless integration crap. Instead you can just use battle tested high performance libraries for literally everything.
Don't get sucked into this serverless nonsense. It's always been bad and it probably always will be. When it's out of fashion Boring Tech will still be here.
Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong.
All the points noted in the slides etc just come from poor engineering leadership. i.e decisions/choices, etc. Constaints do speed up development but they can be enforced by leadership, not by putting yourself in an endless pit of technical debt.
The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making.
Otherwise assuming you have the mandate necessary to get stuff built the right way just stick with the Boring Tech stack of JVM + PG + k8s. It solves all problems regardless of size, there are minimal decisions to be made (really just Spring or Quarkus atm) and you won't spend time dicking around with half-baked serverless integration crap. Instead you can just use battle tested high performance libraries for literally everything.
Don't get sucked into this serverless nonsense. It's always been bad and it probably always will be. When it's out of fashion Boring Tech will still be here.
Really strong language without any backing arguments whatsoever. You can’t just pick some random tech and decide this is boring tech so it’s good and so the other is bad and has always been. This type of comment is so incredibly meaningless and infuriating.
Please explain why you are saying what you’re saying so at least we can learn from it.
Please explain why you are saying what you’re saying so at least we can learn from it.
I do think specifying JVM/PG/k8s in that comment was a mistake. Boring tech will be different for different people and companies. Boring tech doesn't just mean it's time-tested and battle-hardened, it means you have deep knowledge across the company because you've used it for so long. I'm in a Python shop, so Python is my boring default tech stack choice. If I introduced C++ into our tech stack, it would be not-boring tech for us, despite C++ not being a cutting edge language.
k8s is funny to call "boring tech" because it's notoriously complex. But if your company has set up dozens of k8s clusters and has a bunch of k8s experts on staff, it can be considered "boring tech." It's all relative.
k8s is funny to call "boring tech" because it's notoriously complex. But if your company has set up dozens of k8s clusters and has a bunch of k8s experts on staff, it can be considered "boring tech." It's all relative.
I think the big thing with "boring" vs "exciting" tech is people pick exciting tech for reasons other than it being a good technical fit (lets board the hype train!!!). Bad results then happen.
Boring tech is usually more stable and more mapped out. Everyone knows where its good and where its not. Other people have already had whatever problems you have.
I'll always bet on the boring tech over the hyped exciting tech.
Boring tech is usually more stable and more mapped out. Everyone knows where its good and where its not. Other people have already had whatever problems you have.
I'll always bet on the boring tech over the hyped exciting tech.
Serverless actual pros:
"infinite" (meaning wallet level) scalability.
Integrated deployment tooling usually.
Integrated ingress usually.
Those last 2 really don't matter though as neither problem is that difficult if you know what you are doing. They both would have been a huge selling point in a world where k8s doesn't exist. Alas it does and thus mostly moot.
The first however is a big deal as managing very bursty loads on k8s can actually be a bit of a challenge. This is doubly true when you are using some sort of queue and don't have good queue scaling metrics for whatever reason. Serverless lets you cop out of managing that by using infinite concurrency.
Ok so now cons of serverless:
Turns all your code into spaghetti, this is probably the biggest one.
Cold boot performance is highly varible and generally worsens with added function complexity but this depends on tech choice.
Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
Lock-in is awful, migration story is awful.
Amount of Infrastructure as Code overhead, ala Terraform, CloudFormation, whatever else floats your boat to support serverless is insane. The sheer number of objects that need to defined, kept in sync etc is just bonkers. Not to mention if you have multiple services setup like this it's all boilerplate you either need to abstract or carry around.
Hides serious programming deficiencies like memory leaks because context is thrown away after servicing request.
Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
All of the above leads to increasing pressure to make poor architectural decisions to work around limitations of the runtime.
I could probably go on but that is a reasonable set for you.
EDIT: Actually I forgot the worst con. The cost. The $/CPU/RAM cost of Serverless is beyond bonkers. When I was pricing up Cloud Functions vs Spot instances on GKE I think I came to the conclusion Cloud Functions are about 300x as expensive. If your code can run on serverless it can probably run on Spot instances which you can just add to your GKE deployment using another node pool and only schedule things that can handle nodes going away within 60s there and instantly reap 300x cost savings. This is because Serverless is sold at a massive premium to CPU list prices and Spot is sold at like ~30-40% of list prices depending on machine type (for those curious when I last checked T2D was the best for price/performance for highly single thread performance dependent workloads, ala Node.js)
"infinite" (meaning wallet level) scalability.
Integrated deployment tooling usually.
Integrated ingress usually.
Those last 2 really don't matter though as neither problem is that difficult if you know what you are doing. They both would have been a huge selling point in a world where k8s doesn't exist. Alas it does and thus mostly moot.
The first however is a big deal as managing very bursty loads on k8s can actually be a bit of a challenge. This is doubly true when you are using some sort of queue and don't have good queue scaling metrics for whatever reason. Serverless lets you cop out of managing that by using infinite concurrency.
Ok so now cons of serverless:
Turns all your code into spaghetti, this is probably the biggest one.
Cold boot performance is highly varible and generally worsens with added function complexity but this depends on tech choice.
Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
Lock-in is awful, migration story is awful.
Amount of Infrastructure as Code overhead, ala Terraform, CloudFormation, whatever else floats your boat to support serverless is insane. The sheer number of objects that need to defined, kept in sync etc is just bonkers. Not to mention if you have multiple services setup like this it's all boilerplate you either need to abstract or carry around.
Hides serious programming deficiencies like memory leaks because context is thrown away after servicing request.
Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
All of the above leads to increasing pressure to make poor architectural decisions to work around limitations of the runtime.
I could probably go on but that is a reasonable set for you.
EDIT: Actually I forgot the worst con. The cost. The $/CPU/RAM cost of Serverless is beyond bonkers. When I was pricing up Cloud Functions vs Spot instances on GKE I think I came to the conclusion Cloud Functions are about 300x as expensive. If your code can run on serverless it can probably run on Spot instances which you can just add to your GKE deployment using another node pool and only schedule things that can handle nodes going away within 60s there and instantly reap 300x cost savings. This is because Serverless is sold at a massive premium to CPU list prices and Spot is sold at like ~30-40% of list prices depending on machine type (for those curious when I last checked T2D was the best for price/performance for highly single thread performance dependent workloads, ala Node.js)
> cons of serverless:
> Turns all your code into spagetti, this is probably the biggest one.
Overall agree with you, but I don't think spaghetti is the right term here - even if the typo is corrected :)
Spaghetti code is more of a term for monolithic mess where you have a hard time following all the code logic (cyclomatic complexity) and dependencies.
Understanding & tracking dependencies is actually easier with serveless, it's pretty clear what dependency that one tiny micro service has.
But where it becomes worse is regarding the code logic mess, you've now turned it in a distributed system problem where, in addition to having a even hard time with cyclomatic complexity, you also have to think about transient failures/retries at each service boundaries.
That and the other problems you describe (performance hit etc)
> Turns all your code into spagetti, this is probably the biggest one.
Overall agree with you, but I don't think spaghetti is the right term here - even if the typo is corrected :)
Spaghetti code is more of a term for monolithic mess where you have a hard time following all the code logic (cyclomatic complexity) and dependencies.
Understanding & tracking dependencies is actually easier with serveless, it's pretty clear what dependency that one tiny micro service has.
But where it becomes worse is regarding the code logic mess, you've now turned it in a distributed system problem where, in addition to having a even hard time with cyclomatic complexity, you also have to think about transient failures/retries at each service boundaries.
That and the other problems you describe (performance hit etc)
> Spaghetti code is more of a term for monolithic mess where you have a hard time following all the code logic (cyclomatic complexity) and dependencies.
I started using distributed monolith.
I started using distributed monolith.
I saw this described recently as the "Lambda Pinball" architecture.
* https://www.thoughtworks.com/radar/techniques?blipid=2019110...
* https://www.thoughtworks.com/radar/techniques?blipid=2019110...
When even Thoughtworks thinks you are doing it wrong you know you are in trouble.
How about spaghetti architecture.
I like that one.
> Spaghetti code is more of a term for monolithic mess where you have a hard time following all the code logic (cyclomatic complexity) and dependencies.
The only thing worse than a monolithic mess, is the same thing but distributed
The only thing worse than a monolithic mess, is the same thing but distributed
Yeah I completely agree with all of that.
> Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
Or just set the Lambda concurrency to a fixed value proportional to your MySQL DB's capacity. It's an easy parameter.
> Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
This is just nonsense. What is "unpredictable" about the runtime? What is difficult about enabling X-Ray?
> Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
"All-serverless" is just as stupid an idea as "no-serverless."
Aside from lock-in, your "cons" are just the cons of anyone trying to build large-scale systems without knowing what they are doing. There is absolutely no way anyone who read the documentation would run into your cons of serverless.
https://docs.aws.amazon.com/wellarchitected/latest/serverles...
Or just set the Lambda concurrency to a fixed value proportional to your MySQL DB's capacity. It's an easy parameter.
> Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
This is just nonsense. What is "unpredictable" about the runtime? What is difficult about enabling X-Ray?
> Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
"All-serverless" is just as stupid an idea as "no-serverless."
Aside from lock-in, your "cons" are just the cons of anyone trying to build large-scale systems without knowing what they are doing. There is absolutely no way anyone who read the documentation would run into your cons of serverless.
https://docs.aws.amazon.com/wellarchitected/latest/serverles...
> Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-...
https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-...
> Turns all your code into spaghetti, this is probably the biggest one.
This is a nonsensical claim. Nothing about serverless requires 'spaghetti' code. Vice versa, nothing aboud non-serverless deployments suddenly ensures well written code.
> Cold boot performance is highly varible and generally worsens with added function complexity but this depends on tech choice.
For many use-cases this isn't a problem. If it is a problem, then sure, don't use serverless.
> Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
Sure, but:
A) You get a lot of scalability before that becomes an issue. B) Setting up a proxy isn't exactly hard. (I get one for free since we use Hasura).
> Lock-in is awful, migration story is awful.
This is nonsense.
Our serverless code runs in docker containers, running perfectly normal ASP.net code. This code could be deployed to a VM, appengine, or k8s in less than a day if needed.
> Amount of Infrastructure as Code overhead, ala Terraform, CloudFormation, whatever else floats your boat to support serverless is insane. The sheer number of objects that need to defined, kept in sync etc is just bonkers. Not to mention if you have multiple services setup like this it's all boilerplate you either need to abstract or carry around.
One of the advantages of our use of serverless is that we don't yet need any of that complexity.
> Hides serious programming deficiencies like memory leaks because context is thrown away after servicing request.
What you descried is literally not a serious deficiencies. If you code leaks memory, but never runs long enough for that to have meaningful impact, it's not a problem.
> Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
The serverless solutions we use have observability built in for free... and datadog automatically can pull that in if you wnat a starting point for more.
> Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
If you have something that must be long running, serveless isn't a great pick for that one thing (at least, witht he solutions I'm aware of).
That said, there are some elegant ways to handle that using serverless (e.g. by breaking it into smaller tasks with shorter durations)... and going non-serverless isn't a silver bullet, I've seen terrible architectures for long-running code in all kinds of deployments.
> All of the above leads to increasing pressure to make poor architectural decisions to work around limitations of the runtime.
Completely ignoring that serverless can encourage better architectural decisions.
> EDIT: Actually I forgot the worst con. The cost. The $/CPU/RAM cost of Serverless is beyond bonkers. When I was pricing up Cloud Functions vs Spot instances on GKE I think I came to the conclusion Cloud Functions are about 300x as expensive.
Curious.
Let's have a 1s task on Cloud Run with 1 CPU and 1GB of memory:
CPU: $0.00002400/s Memory: $0.00000250/s Request: $0.40 / million
Total = $0.0000269 for 1 request lasting 1s.
Current spot pricing for C3 instances is:
$0.003086 / vCPU hour $0.000413 / GB hour
Or for 1s that would be: 0.00000097/s
Which is 27x cheaper - so you're already off by an order of a magnitude.
It gets worse - that's 1GB of RAM, however the VM needs RAM for it's own OS etc, so maybe you need to provision 2GB of memory, a bit more CPU etc.
The minimum billing time for compute instances according to docs is 60s... so 60x cost if you only need to do this hourly or less.
You're entirely responsible for all time spent, including startup, shut down, idle etc. This just adds more and more cost.
The expected availability model is completely different - serverless should be available in milliseconds, seconds at the worst. Spot instances have no such expectations.
It's a fundamentally different model - a spot VM can't loadbalance, serve code or anything without devops. So now we're spending likely $XXX/hour on devops to try to save fractions of pennies an hour.
When this is all said and done, I don't think you really understand the cost or use-cases of serverless well - and dramatically underestimate the costs of the solutions you present.
This is a nonsensical claim. Nothing about serverless requires 'spaghetti' code. Vice versa, nothing aboud non-serverless deployments suddenly ensures well written code.
> Cold boot performance is highly varible and generally worsens with added function complexity but this depends on tech choice.
For many use-cases this isn't a problem. If it is a problem, then sure, don't use serverless.
> Infinite concurrency is a bit of a curse. Want to use that MySQL DB that contains all your business critical data? Better use some MySQL Serverless proxy nonsense because unbounded connections isn't fun.
Sure, but:
A) You get a lot of scalability before that becomes an issue. B) Setting up a proxy isn't exactly hard. (I get one for free since we use Hasura).
> Lock-in is awful, migration story is awful.
This is nonsense.
Our serverless code runs in docker containers, running perfectly normal ASP.net code. This code could be deployed to a VM, appengine, or k8s in less than a day if needed.
> Amount of Infrastructure as Code overhead, ala Terraform, CloudFormation, whatever else floats your boat to support serverless is insane. The sheer number of objects that need to defined, kept in sync etc is just bonkers. Not to mention if you have multiple services setup like this it's all boilerplate you either need to abstract or carry around.
One of the advantages of our use of serverless is that we don't yet need any of that complexity.
> Hides serious programming deficiencies like memory leaks because context is thrown away after servicing request.
What you descried is literally not a serious deficiencies. If you code leaks memory, but never runs long enough for that to have meaningful impact, it's not a problem.
> Unpredictable runtime makes observability more complicated, probably forcing you into poor decisions as a result, i.e prom push gateway.
The serverless solutions we use have observability built in for free... and datadog automatically can pull that in if you wnat a starting point for more.
> Runtime/execution time limits mean people trying to do all-serverless end up doing horrible things to accomplish long-running tasks.
If you have something that must be long running, serveless isn't a great pick for that one thing (at least, witht he solutions I'm aware of).
That said, there are some elegant ways to handle that using serverless (e.g. by breaking it into smaller tasks with shorter durations)... and going non-serverless isn't a silver bullet, I've seen terrible architectures for long-running code in all kinds of deployments.
> All of the above leads to increasing pressure to make poor architectural decisions to work around limitations of the runtime.
Completely ignoring that serverless can encourage better architectural decisions.
> EDIT: Actually I forgot the worst con. The cost. The $/CPU/RAM cost of Serverless is beyond bonkers. When I was pricing up Cloud Functions vs Spot instances on GKE I think I came to the conclusion Cloud Functions are about 300x as expensive.
Curious.
Let's have a 1s task on Cloud Run with 1 CPU and 1GB of memory:
CPU: $0.00002400/s Memory: $0.00000250/s Request: $0.40 / million
Total = $0.0000269 for 1 request lasting 1s.
Current spot pricing for C3 instances is:
$0.003086 / vCPU hour $0.000413 / GB hour
Or for 1s that would be: 0.00000097/s
Which is 27x cheaper - so you're already off by an order of a magnitude.
It gets worse - that's 1GB of RAM, however the VM needs RAM for it's own OS etc, so maybe you need to provision 2GB of memory, a bit more CPU etc.
The minimum billing time for compute instances according to docs is 60s... so 60x cost if you only need to do this hourly or less.
You're entirely responsible for all time spent, including startup, shut down, idle etc. This just adds more and more cost.
The expected availability model is completely different - serverless should be available in milliseconds, seconds at the worst. Spot instances have no such expectations.
It's a fundamentally different model - a spot VM can't loadbalance, serve code or anything without devops. So now we're spending likely $XXX/hour on devops to try to save fractions of pennies an hour.
When this is all said and done, I don't think you really understand the cost or use-cases of serverless well - and dramatically underestimate the costs of the solutions you present.
As someone who went with Lambda for my current project, I don't agree with this take at all. In fact, it's as boring of an implementation as you can imagine - a single "fat function" developed in .NET serving the entire web application. Can be easily transplanted onto any other hosting paradigm, but Lambda is incredibly cheap (free at my current usage) and incredibly scalable should I hit some inflection point. Learning K8s would have been quite an overkill.
We are thinking the exact same thing (w/ Az Functions). Keeping it simple as possible means you can forklift the whole thing in one afternoon to a different provider. It also means you could potentially even go back to an on-prem stack or whatever.
We are looking at taking our .NET6 monolith and "inverting" it so the entry point is our logical HTTP resources instead of the various DI monstrosities that emerge from main.
The reasoning for us is that we don't want ownership over all of the horrible things we could get wrong. If a <TLS1.2 connection some how gets established and something naughty happens because we forgot to update some startup.cs boilerplate, it's currently our pain today. Like legally. If we are just eating pre-approved/filtered/etc HTTP requests from inside our hermetically-sealed function environment, we get to cast a lot more blame than we otherwise would be able to.
I really don't get why you'd go to serverless and then shit it up with a lot of extra complexity. The whole point in my view is to inherit the compliance packages of your hyperscalar by way of directly using their native primitives in as standard a way as possible. This means you get to 'sail' through audits and engage up-market clients that are typically out of reach for everyone else.
We are looking at taking our .NET6 monolith and "inverting" it so the entry point is our logical HTTP resources instead of the various DI monstrosities that emerge from main.
The reasoning for us is that we don't want ownership over all of the horrible things we could get wrong. If a <TLS1.2 connection some how gets established and something naughty happens because we forgot to update some startup.cs boilerplate, it's currently our pain today. Like legally. If we are just eating pre-approved/filtered/etc HTTP requests from inside our hermetically-sealed function environment, we get to cast a lot more blame than we otherwise would be able to.
I really don't get why you'd go to serverless and then shit it up with a lot of extra complexity. The whole point in my view is to inherit the compliance packages of your hyperscalar by way of directly using their native primitives in as standard a way as possible. This means you get to 'sail' through audits and engage up-market clients that are typically out of reach for everyone else.
If you have a monolithic entrypoint, is it really serverless? You're simply using the Lambda infra as a run-on-demand server, something Heroku has been offering for 10+ years. Why would you pay extra when you can deploy to a simple VM or managed platform costing 100x less?
What you described would be more akin to using ECS Fargate to run the workload. What they described sounds like a single .NET lambda behind API Gateway, so the infrastructure of ingesting requests (and handling TLS termination, DDoS mitigation, WAF, etc.) all happens on the AWS side. All you have to do is handle the routing inside of the lambda using regular ASP.NET Core. It would also scale up infinitely and down to zero with no work on their end. This isn’t similar to running something on Heroku on an allocated server with fixed overhead at all.
Indeed it would be more like Fargate in terms of container runtime vs lambda runtime, but the line between those is pretty blurry. Fly.io for example does containers but they spawn faster than some cloud function platforms. It's not really relevant for the discussion.
> Heroku on an allocated server with fixed overhead at all
I don't think you're familiar with Heroku then? I'm not talking about dedicated or managed servers. That's exactly what their 'dynos' platform has been doing since ~2008, though less granular: https://www.heroku.com/dynos/scaling
"All you have to do is handle the routing inside of the lambda using regular ASP.NET Core" also applies to both fly.io and Heroku. That's what I'm getting at. If you don't have a collection of independent cloud functions to execute, and you are spinning up a little monolith for every request, the platform seems like overkill.
> Heroku on an allocated server with fixed overhead at all
I don't think you're familiar with Heroku then? I'm not talking about dedicated or managed servers. That's exactly what their 'dynos' platform has been doing since ~2008, though less granular: https://www.heroku.com/dynos/scaling
"All you have to do is handle the routing inside of the lambda using regular ASP.NET Core" also applies to both fly.io and Heroku. That's what I'm getting at. If you don't have a collection of independent cloud functions to execute, and you are spinning up a little monolith for every request, the platform seems like overkill.
What makes it serverless is the ability to spin up instances on demand (including down to zero when not in use), while paying only when the resources are in use. Given Lambda's generous free tier, I'm literally paying nothing for it. That would not be the case for any other platform.
Because it doesn’t actually cost less? You have to have a LOT of invocations to match the cost of the entry level 6 dollar digital ocean box. And with that you have to manage that box vs serverless you don’t have to spend engineering time (which costs an order of magnitude more) on maintenance, a big deal for small shops without dedicated/capable ops teams.
[deleted]
k8s is not as hard as its advertised
I implemented self-hosted k8s in my current company and had to learn it from the ground. I'm not a professional, but I learned a thing or two.
My opinion is as follows:
Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux.
Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard.
Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before.
Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy.
Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works.
Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well.
Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.
My opinion is as follows:
Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux.
Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard.
Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before.
Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy.
Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works.
Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well.
Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.
Agree on storage being the hard part. You either need a robust SAN to store persistent volumes or you need to store persistent volumes on the local storage of kubelets which is a major pain. Once a pod binds a local pvc to a kubelet that pod must now run on that kubelet for the rest of eternity. If you want to get rid of a kubelet with a local pvc you either need the application be durable to data loss (which many are not) or you need to implement some sort of stateful pod draining logic.
When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.
When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.
One of my recent conclusions is that it's not so much that Kubernetes on its own is so very complicated; it's that if someone tries to glue their own container platform together from Kubernetes plus another dozen or so cloud-native projects and integrate the whole thing, it can get wickedly complex very quickly.
Right — Kubernetes was evolved in the context of being a Google Cloud service, and despite now having other deployments, still is mostly architected around the assumption that your Kubernetes cluster is surrounded by a featureful IaaS platform providing things like managed load balancers, VM auto-scaling pools, iSCSI volumes on a dedicated SAN network, non-throughput-bottlenecked object storage (and a container image repo on top of that!), a centralized logs and metrics sink, etc. A k8s cluster expects these like a computer expects a bootable storage device.
If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects.
(And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects.
(And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
There are other container platforms like Red Hat OpenShift as well (disclaimer: I work there) either operated for you or on-prem (doesn't need to be on top of OpenStack and typically isn't these days). I'm biased but Kubernetes complexity tends to be conflated with the complexity of a DIY container platform which can be significant, especially given that Kubernetes itself has generally made the decision to keep its scope relatively narrow.
Your comment highlights that you really don't understand what "serverless" is. We run a monolith running on App Engine "NodeJS Flexible". All of our code is basically a single Node monolith, and under the covers many of GCP's newer serverless options (like Cloud Run and Cloud Functions V2) run on the same infrastructure -it's basically just containers, and they even offer versions that let you simply deploy any Docker containerized app.
But compared to the maintenance burden I've seen with other similarly sized startups that use k8s, we just spend way, way less time on infrastructure management.
On a related note, this kind of language, "The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making." which is basically "if you don't agree with me you're an idiot", says a lot more about the speaker than the speakers usually realize, and it's a huge, giant red flag.
But compared to the maintenance burden I've seen with other similarly sized startups that use k8s, we just spend way, way less time on infrastructure management.
On a related note, this kind of language, "The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making." which is basically "if you don't agree with me you're an idiot", says a lot more about the speaker than the speakers usually realize, and it's a huge, giant red flag.
Yeah I am familiar with those (Cloud Run, Fargate et al) and they are much less egregious than Lambda and Cloud Functions. As for if they qualify as Serverless though I don't know, it seems poorly defined.
How about I clarify that my stance mostly relates to Functions as a Service so it's clear that while I wouldn't choose Cloud Run over GKE that is hugely different to FaaS.
How about I clarify that my stance mostly relates to Functions as a Service so it's clear that while I wouldn't choose Cloud Run over GKE that is hugely different to FaaS.
I think Cloud Run, Fargate and Azure Container Apps all fall more into the category of "platform-as-a-service" rather than "function-as-a-service", in the same way that EKS or GKE could be categorised as a PAAS.
EDIT: I also think they're _excellent_ to use for development. The migration from ECS + Fargate to EKS + EC2 is not a difficult migration from the application side, and can be done relatively piecemeal, in my experience.
EDIT: I also think they're _excellent_ to use for development. The migration from ECS + Fargate to EKS + EC2 is not a difficult migration from the application side, and can be done relatively piecemeal, in my experience.
JVM+PG+k8s is absolutely not "boring tech" to me.
LAMP, this is boring tech. You can switch Apache for Nginx, and MySQL for another SQL database, but that's the idea. People still develop in PHP today, and performance-wise, assuming you follow standard good development practices, and good servers, it can take a lot. If it was good enough to get Facebook started, it is most likely good enough for you, and the situation has much improved since the early days of Facebook, a lot of it thanks to Facebook itself, and also better hardware. PHP security is not as terrible as it once was too.
At work we have a couple of Spring projects, and I don't think I have hated a tech stack more than that. If you to illustrate the "action at a distance" anti-pattern, look no further. I'm sure most Spring web apps could be implement in PHP for a fraction of the size, both in terms of application code and memory. I don't do k8s, so I have no real opinion except that it looks like overkill in many cases. As for Postgres, nothing against it, it is a very good SQL database, though my opinion is that you should limit yourself to basic SQL as much as you can, so that you are not dependent on a particular database.
LAMP, this is boring tech. You can switch Apache for Nginx, and MySQL for another SQL database, but that's the idea. People still develop in PHP today, and performance-wise, assuming you follow standard good development practices, and good servers, it can take a lot. If it was good enough to get Facebook started, it is most likely good enough for you, and the situation has much improved since the early days of Facebook, a lot of it thanks to Facebook itself, and also better hardware. PHP security is not as terrible as it once was too.
At work we have a couple of Spring projects, and I don't think I have hated a tech stack more than that. If you to illustrate the "action at a distance" anti-pattern, look no further. I'm sure most Spring web apps could be implement in PHP for a fraction of the size, both in terms of application code and memory. I don't do k8s, so I have no real opinion except that it looks like overkill in many cases. As for Postgres, nothing against it, it is a very good SQL database, though my opinion is that you should limit yourself to basic SQL as much as you can, so that you are not dependent on a particular database.
I don't get how a Spring app rewritten in PHP, assuming a framework is used, reduces code significantly given that PHP5+ is a dynamically-typed Java clone with the same mind-numbing boilerplate.
> All the points noted in the slides etc just come from poor engineering leadership. i.e decisions/choices, etc.
Hindsight is 20/20. It's easy to make these claims after the fact. In the moment it's far more complex and the decisions made at the time take in a lot of factors. Oversimplifying things and placing blame on a tiny thing doesn't push the discussion and learnings forward.
Lambda came out in late 2014 and Step Functions came out in late 2016. It's entirely likely that this team was an early adopter of the tech. Internally a lot of these AWS technologies have a roadmap that sound like they will solve a lot of your issues but those roadmaps get tossed fairly regularly as priorities shift at the product level and AWS-wide level.
We won't be able to get the info on what went on here (likely the team that improved things doesn't even know due to turn over). But watering it down to be "poor engineering leadership" is off the mark and way too simplistic.
Hindsight is 20/20. It's easy to make these claims after the fact. In the moment it's far more complex and the decisions made at the time take in a lot of factors. Oversimplifying things and placing blame on a tiny thing doesn't push the discussion and learnings forward.
Lambda came out in late 2014 and Step Functions came out in late 2016. It's entirely likely that this team was an early adopter of the tech. Internally a lot of these AWS technologies have a roadmap that sound like they will solve a lot of your issues but those roadmaps get tossed fairly regularly as priorities shift at the product level and AWS-wide level.
We won't be able to get the info on what went on here (likely the team that improved things doesn't even know due to turn over). But watering it down to be "poor engineering leadership" is off the mark and way too simplistic.
I wasn't talking about the Prime Video case, I was talking about the slides in Adrians blog post which compare Serverless with traditional development.
IMO if you are facing those problems it has to be poor leadership, there is no excuse for why you shouldn't be able to make effective decisions if you know what you are doing.
IMO if you are facing those problems it has to be poor leadership, there is no excuse for why you shouldn't be able to make effective decisions if you know what you are doing.
I don't care for serverless, but your suggestion that the only right way to build software is by using the JVM, and the only right way to use the JVM is to use Spring or Quarkus is... puzzling to say the least.
I might as well say that you should run everything on PHP. I guess that's boring too?
I might as well say that you should run everything on PHP. I guess that's boring too?
You can sub JVM for something else but you should probably stick with PG and k8s.
What I am saying is the highest quality tools are still the best and will remain that way. As other things come and go they absorb the best ideas from the new stuff without losing what makes them good.
Sometimes the new stuff survives (Node/TS/Go/Rust) sometimes it doesn't (Clojure, Scala, arguably Ruby), some we are waiting to find out.
But betting on JVM and .NET has been winning strategy from a technical perspective for a long time now.
Also worth mentioning no, PHP probably doesn't work here because it's not "general purpose" enough IMO. JVM can handle almost any workload shape, the main one it doesn't do well in is memory constrained environments (though it's possible if you really want to). PHP, Node/TS, Python, Ruby all suffer from being single threaded so can't take advantage of properly large machines without a ton of extra complexity and you pay a high memory overhead for most of those strategies.
Go is OK, I hate writing it but it can do all the things so if that floats your boat go for it. Rust is also perfectly fine, I just find it is actually pretty slow to write because changing requirements often mean changing structure which can be harder to manage in Rust vs Java/Kotlin/C#.
Anyways, wasn't meant to be a thing about languages. Just pick a good runtime or AoT lang of choice, write everything in it, stick to conventional frameworks, choose PostgreSQL, run the lot on k8s. That is the core takeaway.
What I am saying is the highest quality tools are still the best and will remain that way. As other things come and go they absorb the best ideas from the new stuff without losing what makes them good.
Sometimes the new stuff survives (Node/TS/Go/Rust) sometimes it doesn't (Clojure, Scala, arguably Ruby), some we are waiting to find out.
But betting on JVM and .NET has been winning strategy from a technical perspective for a long time now.
Also worth mentioning no, PHP probably doesn't work here because it's not "general purpose" enough IMO. JVM can handle almost any workload shape, the main one it doesn't do well in is memory constrained environments (though it's possible if you really want to). PHP, Node/TS, Python, Ruby all suffer from being single threaded so can't take advantage of properly large machines without a ton of extra complexity and you pay a high memory overhead for most of those strategies.
Go is OK, I hate writing it but it can do all the things so if that floats your boat go for it. Rust is also perfectly fine, I just find it is actually pretty slow to write because changing requirements often mean changing structure which can be harder to manage in Rust vs Java/Kotlin/C#.
Anyways, wasn't meant to be a thing about languages. Just pick a good runtime or AoT lang of choice, write everything in it, stick to conventional frameworks, choose PostgreSQL, run the lot on k8s. That is the core takeaway.
I mean, I currently work on the JVM, I just have enough experience with different languages (including Ruby, which I don't think is quite dead yet) and tech stacks to know that they all suck in their own way. The JVM, and specifically Java, and even more specifically Spring, have a lot of annoying warts, it's just that other languages have different warts. I agree that you shouldn't try to rewrite everything in an obscure, untested technology, but beyond that, there's a number of options with enough tradeoffs. But it seems we're largely in agreement there.
I don't necessarily agree with the "keep everything in one language / stack" sentiment. There's certainly companies that overdo it, having 20 different languages at the same time. But there's also valid reasons for keeping certain tech stacks separate (e.g. having to do ML / data analysis stuff in Python). In the worst case, the "use only a single language" mindset leads to disasters like GWT, which is something that my current company unfortunately still uses because some Java developers (that don't work for the company any more) thought that JavaScript was beneath them.
I also wouldn't choose PHP, for the record. I would like to trash it here, but I've never actually been forced to use it, so I can't even comment on it.
I'm not so sure about k8s. I think you can get away with much simpler deployment models, too. At some point the overhead of having to manage everything manually (or a need for zero-downtime deployments) might make the added complexity of k8s more bearable, but it's certainly added complexity and not necessary everywhere and all the time.
I 100% agree about postgres, it's so good and feature-rich that you should probably really only ever use another database if you have very specific needs (except maybe also adding redis as a cache, because redis is also very good).
I don't necessarily agree with the "keep everything in one language / stack" sentiment. There's certainly companies that overdo it, having 20 different languages at the same time. But there's also valid reasons for keeping certain tech stacks separate (e.g. having to do ML / data analysis stuff in Python). In the worst case, the "use only a single language" mindset leads to disasters like GWT, which is something that my current company unfortunately still uses because some Java developers (that don't work for the company any more) thought that JavaScript was beneath them.
I also wouldn't choose PHP, for the record. I would like to trash it here, but I've never actually been forced to use it, so I can't even comment on it.
I'm not so sure about k8s. I think you can get away with much simpler deployment models, too. At some point the overhead of having to manage everything manually (or a need for zero-downtime deployments) might make the added complexity of k8s more bearable, but it's certainly added complexity and not necessary everywhere and all the time.
I 100% agree about postgres, it's so good and feature-rich that you should probably really only ever use another database if you have very specific needs (except maybe also adding redis as a cache, because redis is also very good).
Quarkus also has a lot of annoying warts, as does just plain ol' Java.
Lord, if I never see another Builder pattern again in my life, I may die a happy man.
Lord, if I never see another Builder pattern again in my life, I may die a happy man.
I haven't yet used Quarkus, but in my opinion the solution to Spring's problems with startup times isn't to run everything in GraalVM, it's to declare dependencies statically instead of creating a mess of annotations that have to be interpreted at runtime. Kotlin IMHO provides ways of doing that rather elegantly.
Agreed. Builder.build() wtf!!?
I don't know about absorbing the best ideas. Java, even after the introduction of raw string literals, still can't handle a regex without escaping metacharacters. Perl had that one solved in the 90s.
> I might as well say that you should run everything on PHP. I guess that's boring too?
PHP can't run everything though. If we go by the traditional model it's per request and fails at things like websockets. It's also less performant than a JVM solution, so definitely can't serve every use case.
PHP can't run everything though. If we go by the traditional model it's per request and fails at things like websockets. It's also less performant than a JVM solution, so definitely can't serve every use case.
[deleted]
Serverless first is a great way to start building your business. As long as you don’t have strict latency requirements. You focus on your business, while the infrastructure requirements are handled for you.
The point of engineering is to deliver the best solution and the right cost. Cost needs to be evaluated on multiple dimensions - actual dollars, time required to build, time required to maintain.
Nothing stops you from using Postgres with Lambda. AWS has done a bunch of work to make this possible.
I would recommend, creating the correct layer of separation in your code, so that when the times comes to migrate from Lambda or other serverless functions to Fargate, cloudrun, or even containers run on VMs managed by K8S, you have to make minimal changes to get up and running.
The point of engineering is to deliver the best solution and the right cost. Cost needs to be evaluated on multiple dimensions - actual dollars, time required to build, time required to maintain.
Nothing stops you from using Postgres with Lambda. AWS has done a bunch of work to make this possible.
I would recommend, creating the correct layer of separation in your code, so that when the times comes to migrate from Lambda or other serverless functions to Fargate, cloudrun, or even containers run on VMs managed by K8S, you have to make minimal changes to get up and running.
I’m building something with Spring Boot, Kotlin, Postgres and containers, with a small team. It’s far from boring, it’s the most fun I’ve had in ages.
Spring is fun until you need to do anything mildly complex and then you tear your hair out trying to find the right documentation. You have to navigate myriad of sub projects. They really really need to sort out the SEO. It’s atrocious.
The only way to learn Spring is to read the latest manuals of every subproject you think you might need, cover to cover.
Even then you might not figure out what awkward collection of beans you need to override the magic Spring does.
The only way to learn Spring is to read the latest manuals of every subproject you think you might need, cover to cover.
Even then you might not figure out what awkward collection of beans you need to override the magic Spring does.
I found Spring to have a little bit of an understandability cliff. Coming from Guice though once I understood the container and had a feel for the general patterns it was easy.
I think if you try to use it without understanding the magic you can probably get pretty far but you will always feel lost when things don't work just like the example. Understanding how all the runtime wiring works at the DI/IoC layer generally makes it pretty easy to work out what is going on though.
I think if you try to use it without understanding the magic you can probably get pretty far but you will always feel lost when things don't work just like the example. Understanding how all the runtime wiring works at the DI/IoC layer generally makes it pretty easy to work out what is going on though.
Yeah I very much prefer Kotlin now. It's Boring Tech but at the same time pleasant because of it's high expressivity and tooling.
Also I find it way easier to convince people coming from a dynamic language background to try it out vs Java which they find less approachable. Which is ironic as Kotlin is by far the more complicated language vs Java.
Also I find it way easier to convince people coming from a dynamic language background to try it out vs Java which they find less approachable. Which is ironic as Kotlin is by far the more complicated language vs Java.
> Also I find it way easier to convince people coming from a dynamic language background to try it out vs Java which they find less approachable.
It's just a sad fact of Kotlin is not Java. There are lots of people out there that just hate on Java as a hype train.
It's just a sad fact of Kotlin is not Java. There are lots of people out there that just hate on Java as a hype train.
Yeah it's sad. Java is really really good these days.
The amount of complexity created in the desperate attempt to avoid learning k8s APIs has at this point, dramatically out complexified k8s itself.
I see the anti-K8S sentiment so much on hackernews. Sure, there’s a time and a place for K8S, but I see a lot of point-blank objection to it for often weak reasons.
It's bloated
Often this translate to "It has features I PERSONALLY don't need or understand"
Often this translate to "It has features I PERSONALLY don't need or understand"
I generally find people that don't like k8s don't understand it or more commonly even the reasons why it exists.
If your job hasn't been to write Cloudformation/Terraform/Pulumi/Chef/Puppet/Salt/Ansible/Capistrano/Packer for a living, you might not appreciate it.
As someone that has been around the block I think k8s is the best thing since sliced bread.
If your job hasn't been to write Cloudformation/Terraform/Pulumi/Chef/Puppet/Salt/Ansible/Capistrano/Packer for a living, you might not appreciate it.
As someone that has been around the block I think k8s is the best thing since sliced bread.
The quadratic function that represents the hand waving away of k8s complexities causes a stack overflow exception.
[deleted]
Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome.
The labelling of JVM + K8 as boring, to posture why it's automatically superior to "serverless integration crap" is exactly the sort of thing poor engineering leadership does to avoid having the sort of difficult conversations necessary. There's no metrics involved, no studies, no prototypes, no financial assessments, none of that it's purely based on emotion.
You can envision the same exact thing happening (but to the preference of serverless) on the Prime Video team.
The labelling of JVM + K8 as boring, to posture why it's automatically superior to "serverless integration crap" is exactly the sort of thing poor engineering leadership does to avoid having the sort of difficult conversations necessary. There's no metrics involved, no studies, no prototypes, no financial assessments, none of that it's purely based on emotion.
You can envision the same exact thing happening (but to the preference of serverless) on the Prime Video team.
It's not emotion. I make money from porting people off of poor technical choices onto solid ones. It comes from experience working with all these various systems.
I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff.
It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.
I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff.
It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.
I remember beanstalk. We used it for years at twitch. We got a shit ton of mileage out of it. It's easy to talk shit about the problems of the old tech, and beanstalk has a lot of issues. But it also let us build a ton of stuff much easier than the alternatives.
This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.
This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.
> I make money from porting people off of poor technical choices onto solid ones
So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.
So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.
My opinion from doing both these ports and greenfield is this:
None of the tools that say they help you move faster matter after the first 3 months.
It's true they take less code and pieces to get started with. No k8s manifests, no Dockerfiles (usually) but if you don't hate your future self you will still use IaC to create the API Gateways/function contexts/DynamoDB/whatever, so you will still be dicking around with at least one of Terraform/Pulumi/similar.
The thing is the setup cost of a traditional lang + pg + k8s setup amortised over 3 months is essentially nothing and yet you have a setup that will last you effectively forever.
If your company wasn't going to die because of being a few days slower in that initial 3 month period you have came out ahead. You have a forever architecture and you avoided a whole bunch of costs and technical debt in the process.
Maybe the calculus is different if you don't have the same depth of infrastructure experience I do but for me it's obvious this is a faster and better way of delivering a prototype that gives it proper room to grow into production software.
None of the tools that say they help you move faster matter after the first 3 months.
It's true they take less code and pieces to get started with. No k8s manifests, no Dockerfiles (usually) but if you don't hate your future self you will still use IaC to create the API Gateways/function contexts/DynamoDB/whatever, so you will still be dicking around with at least one of Terraform/Pulumi/similar.
The thing is the setup cost of a traditional lang + pg + k8s setup amortised over 3 months is essentially nothing and yet you have a setup that will last you effectively forever.
If your company wasn't going to die because of being a few days slower in that initial 3 month period you have came out ahead. You have a forever architecture and you avoided a whole bunch of costs and technical debt in the process.
Maybe the calculus is different if you don't have the same depth of infrastructure experience I do but for me it's obvious this is a faster and better way of delivering a prototype that gives it proper room to grow into production software.
The IAC piece is the same across both. Except with k8s you’re managing infrastructure across cloud, k8s, potentially frameworks like helm, etc.
And on a large team there’s very little likelihood that everyone understands the different configuration layers. There’s very little likelihood that people understand the scope that changes to configuration might touch either. But when you work with CDK, it’s one tool to understand.
Faster and better way to grow into production ignores team level problems and team level constraints. At 10, 20, 100 engineers this is fine. But when you have dozens of teams you’re almost certainly losing efficiency by being prescriptive.
I find that this type of pedantry really surrounds the k8s ecosystem. I think k8s is an enterprise scale tool that has uses in enterprise applications. But the idea of suggesting it to everyone is lunacy
And on a large team there’s very little likelihood that everyone understands the different configuration layers. There’s very little likelihood that people understand the scope that changes to configuration might touch either. But when you work with CDK, it’s one tool to understand.
Faster and better way to grow into production ignores team level problems and team level constraints. At 10, 20, 100 engineers this is fine. But when you have dozens of teams you’re almost certainly losing efficiency by being prescriptive.
I find that this type of pedantry really surrounds the k8s ecosystem. I think k8s is an enterprise scale tool that has uses in enterprise applications. But the idea of suggesting it to everyone is lunacy
I am more of a generalist than you are but I have worked with a Kubernetes setup and I've also worked with going all-in on Lambda and other AWS services managed by CDK and I don't share your intuition. Besides the initial setup, the former also has other ongoing maintenance things that are harder to deal with. It does give you more flexibility if whatever you're doing isn't a good fit for those tools, though (though obviously nothing stops you from provisioning EC2 instances or whatever through other tools).
I have a hard time imagining anyone doing anything non trivial with CDK and walking away happy.
Why? Do you have something specific to say about why you dislike it? I think it’s pretty good.
Oddly I'm not able to reply directly to re-thc, so I hope you don't mind me commenting here that - I suspect that they're not deploying with a CDK-defined Code Pipeline (starter article[0], in-depth example[1]), which makes cross account/region deployments so ludicrously smooth and simple that it often has newbies refuse to believe that it can have worked _that_ easily.
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
[1] https://pipelines.devops.aws.dev/
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
[1] https://pipelines.devops.aws.dev/
If the reply link doesn't appear you can still reply by clicking on the timestamp. I don't know why it works that way.
Odd - it's showing up now. I'll remember for next time. Thanks!
It’s just… half baked. It feels like the CDK team constantly has to work circles around the lacking functionality in CFN. Want to upload a docker image to ECR, use this hacky lambda function. Want to create an object in an S3 bucket? Use this other hacky lambda function.
Then a lot of the resources depend on each other in funky ways, but they’re not always implicitly defined, so whether everything will be created in the right order is anyone’s guess and you often have to define the dependencies manually anyway.
Then when it fails to bring up/modify a stack you get the helpful message that resource x failed to create, but nearly never why.
Like, CDK is marginally better than plain CF, but something like Terraform or Pulumi is so much better that it’s hard to imagine anyone would want to use anything else.
Also, like someone else said, the stacks are directly bound to specific accounts/regions. When I bring domething up that is my QA environment, I’d enjoy it if I didn’t have to split every stack in twain (cloudfront certificates that can only be in us-east-1 or something).
Oh, and when you finally work through all these issues and work on your app for a few months, the dreaded message will come up… “Maximum of 500 resources in stack reached”.
Now enjoy splitting it all up.
Then a lot of the resources depend on each other in funky ways, but they’re not always implicitly defined, so whether everything will be created in the right order is anyone’s guess and you often have to define the dependencies manually anyway.
Then when it fails to bring up/modify a stack you get the helpful message that resource x failed to create, but nearly never why.
Like, CDK is marginally better than plain CF, but something like Terraform or Pulumi is so much better that it’s hard to imagine anyone would want to use anything else.
Also, like someone else said, the stacks are directly bound to specific accounts/regions. When I bring domething up that is my QA environment, I’d enjoy it if I didn’t have to split every stack in twain (cloudfront certificates that can only be in us-east-1 or something).
Oh, and when you finally work through all these issues and work on your app for a few months, the dreaded message will come up… “Maximum of 500 resources in stack reached”.
Now enjoy splitting it all up.
> It feels like the CDK team constantly has to work circles around the lacking functionality in CFN. Want to upload a docker image to ECR, use this hacky lambda function. Want to create an object in an S3 bucket? Use this other hacky lambda function.
It feels like you have a different boundary of what constitutes "infrastructure" than the CDK team do. I'd call both of those examples content _within_ an infrastructure component. Would you expect CDK to initialize an SQS Queue with a message on it? Or a DynamoDB table with data in it? I guess you could make that argument, but to me _populating_ your infrastructure is a very different task from _making_ it - it's an application-level task, not a infrastructure-level one.
Most of the rest of your comments are pretty subjective - I can't say I agree with most of them or have experienced them, but I also don't disbelieve you that they've happened to you.
Not sure what you mean by having an issue with "stacks [being] directly bound to specific accounts/regions", though. Why is that an issue? Why would you _want_ a stack that spans multiple account/regions? I just successfully created and validated a Certificate in us-west-2, so I'm not sure what your parenthetical means.
It feels like you have a different boundary of what constitutes "infrastructure" than the CDK team do. I'd call both of those examples content _within_ an infrastructure component. Would you expect CDK to initialize an SQS Queue with a message on it? Or a DynamoDB table with data in it? I guess you could make that argument, but to me _populating_ your infrastructure is a very different task from _making_ it - it's an application-level task, not a infrastructure-level one.
Most of the rest of your comments are pretty subjective - I can't say I agree with most of them or have experienced them, but I also don't disbelieve you that they've happened to you.
Not sure what you mean by having an issue with "stacks [being] directly bound to specific accounts/regions", though. Why is that an issue? Why would you _want_ a stack that spans multiple account/regions? I just successfully created and validated a Certificate in us-west-2, so I'm not sure what your parenthetical means.
Cross region/account usage is terrible. It's also 1/2 baked. Some things have higher level constructs but some don't. Lots of things have to be "escaped" to get it to work.
If you're just on a single region/account then yes it's better than alternatives.
If you're just on a single region/account then yes it's better than alternatives.
I suspect that you're not deploying with a CDK-defined Code Pipeline (starter article[0], in-depth example[1]), which makes cross account/region deployments so ludicrously smooth and simple that it often has newbies refuse to believe that it can have worked _that_ easily.
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
[1] https://pipelines.devops.aws.dev/
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
[1] https://pipelines.devops.aws.dev/
"So it was successful enough for long enough " Which doesn't necessarily make it a good choice at any point. It just says it wasn't enough of a bad technical choice to make the product sink.
OK, sure, it's not ironclad proof that it was the right decision. But it's certainly not proof it was the wrong decision when it was made.
"It works and makes money" is at the root of so much shoddy workmanship in software I think you're making a different point than what you're trying to.
The argument of TFA is that starting with serverless and evolving away from it bit by bit as it is revealed to be necessary is a sound strategy most of the time. Whatever you may think of that argument, “I have been hired to optimize projects built using serverless and get them off of it” is not a rebuttal.
Thanks for rehashing how this thread got started for me?
What is this different point you think I’m making then?
>this is a great way to prototype ideas or get a first version of the product out there
As long as it works and makes money it excuses shoddiness. Publishing a draft to fish for revenue is the reason we have so much poor software running out there. After it's in production, you will not be able to make a business case to re-make it right.
As long as it works and makes money it excuses shoddiness. Publishing a draft to fish for revenue is the reason we have so much poor software running out there. After it's in production, you will not be able to make a business case to re-make it right.
Apparently you will since the other guy in this thread has made this a specialization.
This has nothing to do with software. Same applies for things like buildings where the developers get away with so many defects even after people pay huge $$$ for it.
If you consider building your service that should take 3000 man hours to build in 300k instead, overspending by a factor if a 100, and then still succeeding, then yeah, you could consider that a success.
I’d consider it a massive, massive waste.
I’d consider it a massive, massive waste.
I have no idea what you’re arguing with this one.
You can build exactly the same application even if you throw different amounts of money at it.
Large corpo spends 2M on a CRUD app and is happy with the result (e.g. consider it a success).
Another organisation (with more skilled people) builds exactly the same thing for just 50K, and also consider it a success.
The fact that it’s a success doesn’t mean that it wasn’t silly.
Sorry kind of lost track of the original thread here too.
Large corpo spends 2M on a CRUD app and is happy with the result (e.g. consider it a success).
Another organisation (with more skilled people) builds exactly the same thing for just 50K, and also consider it a success.
The fact that it’s a success doesn’t mean that it wasn’t silly.
Sorry kind of lost track of the original thread here too.
Bit of survivorship bias there! No-one is going to hire you to port off of serverless, if they're happy with it.
Ergo you only see cases where it isn't working well.
Ergo you only see cases where it isn't working well.
Yeah that is fair. There is probably serverless chugging away fine in lots of places.
Doesn't make it better than alternatives but still a valid point.
Doesn't make it better than alternatives but still a valid point.
I use serverless successfully - handing tens of millions of annual revenue.
We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.
We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.
I too have a small team that built a greenfield application that generates about $100M annually for a large hotel chain. Serverless allowed us to get to market fast and everyone in the team is a feature developer — 0 infra folks. I think that’s the biggest value we have been able to derive from this compared all the other internal App teams who struggle with release cadences, experimentation and TTV.
But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule.
But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure.
To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.
But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule.
But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure.
To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.
Seconded, just got done with one of these. Literally 1200 separate lambdas, step functions, hundreds of kinesis streams, and sqs topics plus a smattering of ecs fargate tasks for when invariably something takes longer than the max lambda time. And oh god all the stupid supporting infrastructure to build layers.
There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs.
The only downside has been not being able to meme about how we're web scale anymore.
There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs.
The only downside has been not being able to meme about how we're web scale anymore.
Not being able to make jokes about how fast it is to serve requests from /dev/zero must be absolutely killing your team.
However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example.
Curious if you ended up also saving money in the process?
However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example.
Curious if you ended up also saving money in the process?
One potential reason to advocate for serverless architecture is it is relatively stateless and event-driven and therefore modular and easy to reason about.
Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?
Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?
State can be part of it. For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.
Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path.
However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached.
Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically.
Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.
Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path.
However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached.
Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically.
Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.
> Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path
By off the beaten path you mean a few lines of yaml to configure a FIFO SQS queue and have it trigger a Lambda?
By off the beaten path you mean a few lines of yaml to configure a FIFO SQS queue and have it trigger a Lambda?
I notice the GP hasn't responded to this. If Lambdas couldn't be a target for SNS or SQS, I wouldn't bother with them either. But that has been a core architectural pattern since 2018.
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simp...
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simp...
(quite impressed by the sheer number of comments you have made in this thread, btw)
> For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.
Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
> It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
> For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.
Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
> It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
I don't necessarily agree with everything OP is saying, and I certainly don't speak for them, but I completely understand what they mean by a big ball of microservice spaghetti. The spaghetti is distributed across a massive literal network of dependencies instead of being distributed off a giant graph of object dependencies, or goto statements or something. The spaghetti is real and just as messy in the microservice world as the monolith world. I should know, we inherited a massive one.
But this shouldn't be too much of a revelation since there is no silver bullet in taming software architecture. Architecting software is very subjective, and more often than not we make the wrong decisions and have to untangle the mistakes further down the road.
I don't necessarily agree with everything OP is saying, and I certainly don't speak for them, but I completely understand what they mean by a big ball of microservice spaghetti. The spaghetti is distributed across a massive literal network of dependencies instead of being distributed off a giant graph of object dependencies, or goto statements or something. The spaghetti is real and just as messy in the microservice world as the monolith world. I should know, we inherited a massive one.
But this shouldn't be too much of a revelation since there is no silver bullet in taming software architecture. Architecting software is very subjective, and more often than not we make the wrong decisions and have to untangle the mistakes further down the road.
> I completely understand what they mean by a big ball of microservice spaghetti
Sure, I understand that too, but that's not what was said. Here it is again:
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
Sure, I understand that too, but that's not what was said. Here it is again:
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
> Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.
I evaluated it at the time. I don't have the context loaded to remember exactly why it didn't work for my use-case but I'm going to look again because CFs platform moves pretty fast and perhaps it could be simpler than what we are currently doing.
> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
Elsewhere it's been coined Lambda Pinball, perhaps you find that a more pleasing description?
> Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
Yes, both of which are generally necessary in distributed systems. i.e not only is it a hard problem in and of itself it requires solving sub-problems which are also some of the hardest problems.
I think you may have made my points for me.
I evaluated it at the time. I don't have the context loaded to remember exactly why it didn't work for my use-case but I'm going to look again because CFs platform moves pretty fast and perhaps it could be simpler than what we are currently doing.
> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
Elsewhere it's been coined Lambda Pinball, perhaps you find that a more pleasing description?
> Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
Yes, both of which are generally necessary in distributed systems. i.e not only is it a hard problem in and of itself it requires solving sub-problems which are also some of the hardest problems.
I think you may have made my points for me.
I think in general you’re right on the money and have been throughout your comments here. Lambda pinball is a great term as well. I agree that serverless incentivizes a “nanoservice” approach that flies in the face of patterns like classes/factories or DDD, since everything is atomic you lose the concept of a service or domain boundary and everything can get muddled as hell.
Statelessnes is really a prerequisite for proper horizontal scaling (with small exceptions), so best practices for e.g. spring boot web applications will lead to a stateless architecture as well. Same goes for event-driven -> amqp, mqtt (and jms for java) plug easily into most modern web-frameworks, you can be as event-driven as you want to (which in my experience is an overhead to plain http / rest which needs to be justified)
> Statelessnes is really a prerequisite for proper horizontal scaling
I could understand making this claim w.r.t. Idempotence. Claiming statelessnes as a requirement strikes me as non-trivial enough to require some evidence or a citation thereto.
I could understand making this claim w.r.t. Idempotence. Claiming statelessnes as a requirement strikes me as non-trivial enough to require some evidence or a citation thereto.
Modular, maybe (though sometimes it's just introducing a network hop in between dependencies). Event-driven being easy to reason about? No way, never.
Yah, I was confused by that too. Out of:
1) event-driven
2) procedural
3) functional
4) inheritance
I would say event-driven is tied for last as the hardest to reason about.
1) event-driven
2) procedural
3) functional
4) inheritance
I would say event-driven is tied for last as the hardest to reason about.
I'd say it comes down to your choice of events. I also find these systems very easy to reason about.
Instead of your IDE very easily telling you on click the callers or callees of things you'd rather have events where you have no idea who might be processing or sending a particular event in your system of hundreds of microservices?
Yeah, totally easy!
Yeah, totally easy!
I think you're conflating problems here. IDLs allow interface definitions for interactions that cross compute and administrative boundaries, and event-driven architectures do not require compute distribution (Erlang/beam).
Different people have different strengths and anxieties. The suggestion that, with some regularity, the lack of (local) ambiguity of event message arrival may be of higher leverage relative to complexity than increasing the surface of an invokable interface does not seem beyond the pale.
Ultimately relinquishing a total temporal ordering is going to be painful regardless of whether it's local or distributed.
Different people have different strengths and anxieties. The suggestion that, with some regularity, the lack of (local) ambiguity of event message arrival may be of higher leverage relative to complexity than increasing the surface of an invokable interface does not seem beyond the pale.
Ultimately relinquishing a total temporal ordering is going to be painful regardless of whether it's local or distributed.
That's fair enough if you use something that actually does this and also keeps track of users of your event. Not saying it is unsolvable.
What I've seen though is that in many cases it is unsolved for and it's very hard to reason about "if I remove this event because it's 5 versions too old already and I don't want to keep this backwards compatibility layer, will something "at the other end of our system I never even interact with" puke?
Granted some orgs are so big that it makes sense to solve for these large landscapes of services with backwards compatibility ad absurdum in order to allow anything to get done (been at that bank, done that). But reason about the entire system it does not make easier.
At many smaller orgs these services were also not built because of organizational issues that needed to be solved but because "everyone does this now" and there's no proper registry of properly defined interfaces and trying to find out if a change you were going to make is going to take you forever because you do need to version something or if you can just change things as needed is way harder than it's supposed to be at that size org.
What I've seen though is that in many cases it is unsolved for and it's very hard to reason about "if I remove this event because it's 5 versions too old already and I don't want to keep this backwards compatibility layer, will something "at the other end of our system I never even interact with" puke?
Granted some orgs are so big that it makes sense to solve for these large landscapes of services with backwards compatibility ad absurdum in order to allow anything to get done (been at that bank, done that). But reason about the entire system it does not make easier.
At many smaller orgs these services were also not built because of organizational issues that needed to be solved but because "everyone does this now" and there's no proper registry of properly defined interfaces and trying to find out if a change you were going to make is going to take you forever because you do need to version something or if you can just change things as needed is way harder than it's supposed to be at that size org.
It's freeing you to reason about only one step rather than the entire system at all times.
Fair enough that it's freeing overall if your org is so large that it is faster to always be 100% backwards compatible for aeons.
I repeat: most orgs that go all in on this are actually so small that being so compatible is more of a drag than understanding the part of the system that is attached to the interface you are trying to change ;)
I repeat: most orgs that go all in on this are actually so small that being so compatible is more of a drag than understanding the part of the system that is attached to the interface you are trying to change ;)
>stateless and event-driven and therefore modular
Non-sequitur
Non-sequitur
As a one man shop, I made a decision years ago to try Beanstalk for a particular app backend / API. It's the only "serverless" deployment I've done. It still works fine for that purpose, and it's less trouble than manually deploying to multiple servers. Forced updates are an occasional headache but at least they're all in one place. The code does a lot of serverside computation for a lot of client connections, so that part needs to scale, and since there's relatively little database stuff it just connects to a single RDS instance.
I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.
I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.
Yeah Beanstalk was pretty simple and decent for it's time.
I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end.
These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.
For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.
I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end.
These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.
For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.
> You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.
How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.
How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.
I have done one massive AWS + kops > GKE migration and several smaller migrations between AWS/GCP/Azure.
My experience leads me to believe the main problems stem from adopting IaaS/PaaS specific stuff outside of k8s. If you aren't doing that they are very portable and even if you are if you are using standard stuff like managed PostgreSQL/MySQL then it's still easy.
Where that breaks down is very vendor specific stuff, ala Spanner, queues/logs like SQS/PubSub etc. If you are using those and your architecture is tied to specific semantics of them you are in for a rough time.
I personally try to avoid any and all of those. Not for vendor lock-in reasons but because I greatly dislike not being able to run everything locally in exactly the same configuration it will be deployed in.
So I run PostgreSQL in k8s using Patroni, Kafka or Pulsar for queue/log, etc and that is about it usually. I generally don't need/use other things. Maybe memcached if there is a good need for it over say Caffiene (in-memory cache).
This is awesome because it means it's super super easy to have staging/UAT environment or even preview environments on PRs and I can run integration tests against the whole stack E2E without spinning up/manipulating cloud provider resources.
My experience leads me to believe the main problems stem from adopting IaaS/PaaS specific stuff outside of k8s. If you aren't doing that they are very portable and even if you are if you are using standard stuff like managed PostgreSQL/MySQL then it's still easy.
Where that breaks down is very vendor specific stuff, ala Spanner, queues/logs like SQS/PubSub etc. If you are using those and your architecture is tied to specific semantics of them you are in for a rough time.
I personally try to avoid any and all of those. Not for vendor lock-in reasons but because I greatly dislike not being able to run everything locally in exactly the same configuration it will be deployed in.
So I run PostgreSQL in k8s using Patroni, Kafka or Pulsar for queue/log, etc and that is about it usually. I generally don't need/use other things. Maybe memcached if there is a good need for it over say Caffiene (in-memory cache).
This is awesome because it means it's super super easy to have staging/UAT environment or even preview environments on PRs and I can run integration tests against the whole stack E2E without spinning up/manipulating cloud provider resources.
One nice thing about Beanstalk was that you didn't (don't) have to deal with containers on the toolchain at all. You basically zip the directory structure you'd normally serve and it just gets rolled out to all the instances. What I figured out early (the hard way) was that it wasn't worth the time trying to screw around with leader instances that installed and ran cron jobs or anything like that. I just have an little separate ec2 instance that acts as a heartbeat for cleaning up the DB and buckets, rather than trying to get Beanstalk to do that consistently. I think it's extremely well suited for one layer, the API; and counterintuitively I find it easier not to containerize that, because it gives me more flexibility to switch DBs or storage engines without touching the API, and without having to rework a container and redeploy the API itself.
[deleted]
Have you tried Cloudflare Workers? I work at Cloudflare but I’m genuinely curious what your feedback is on it as it does work quite differently from Lambda/Cloudflare functions.
Yes actually, currently using in production. I actually advocated for the move to Cloudflare and we ended up adopting workers because Cloudflare unfortunately has broken stale-while-revalidate support. So I built an implementation in Workers that abuses the Cache API to store our own TTLs and also our revalidating state. It works pretty well as we don't end up with too many isolates per POP but it's not perfect.
I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation.
I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge.
We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for.
Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.
I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation.
I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge.
We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for.
Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.
Interesting. Since I TL KV currently I’m particularly interested in the ways it’s not working for you. We’re in the process of rolling out major changes here that might help you here so hit me up on the KV Discord or my username at cloudflare.com. I actually do have a meeting with the Cache team on Monday about something related and I’ll ask about it if you could message me what specifically is broke / what the semantics you want are (I’m not intimately familiar with all the HTTP semantics).
Shot you an email.
I must admit I didn't go super deep on KV so I will come by discord at some point. I can share what we are doing and maybe it turns out KV is better suited.
I must admit I didn't go super deep on KV so I will come by discord at some point. I can share what we are doing and maybe it turns out KV is better suited.
Grrrt autocomplete. Lambda/CloudFunctions.
> I make money from porting people off of poor technical choices
I make money by delivering features my customers care about.
We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.
I make money by delivering features my customers care about.
We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.
I actually think Beanstalk/Cloud Run/friends are fine. I should have specified that Functions as a Service is the main mess I think people should be avoiding.
If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.
If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.
Has GAE come and gone? Surely it’s still there, and capable of running the JVM? K8s seems like overkill for small teams which don’t already have experience
GAE is actually still pretty good tbh. However it has it's limitations and even though there are things like Cloud Run et al I still find GKE is the best option.
> I make money from porting people off of poor technical choices onto solid ones.
Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?
Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?
Amen, brother. Love you.
> Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome.
For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership.
Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better.
Why the hell hire super smart people when you have to tell them how to do things?
For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership.
Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better.
Why the hell hire super smart people when you have to tell them how to do things?
Those smart engineers should be encouraged to devote their brainpower to other aspects of the business and technology stack.
Far too often teams use an opportunity to build something new as a science project to use all the latest wiz bang cloud features when the boring stack would work just fine.
For those that truly want to innovate on that aspect of the stack, they should be part of a central infrastructure team, whose job it is to build scalable solutions for the group.
Far too often teams use an opportunity to build something new as a science project to use all the latest wiz bang cloud features when the boring stack would work just fine.
For those that truly want to innovate on that aspect of the stack, they should be part of a central infrastructure team, whose job it is to build scalable solutions for the group.
You can hire anyone for postgres and k8s.
Not so much the fancy new hotness that will be weirdware in three years.
Not so much the fancy new hotness that will be weirdware in three years.
Replying to your deleted comment on another thread:
IMHO if possible, grow your business without investors and retain control. Jason Cohen has an amazing talk on bootstrapping, with some very solid advice from his own experiences: https://m.youtube.com/watch?v=otbnC2zE2rw
Look at co-working spaces near you? Our business worked in an incubator for a bit, and the problems we saw with investors in other startups were eye-watering. We retained 100% ownership, and that was the right decision for us. Learn from the mistakes of others at different stages just by being around them.
Flick through my previous comments on VC for external links that explain my points better: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
I was a founder, and I don’t know shit about VC except through reading, so be appropriately skeptical about my thoughts and only take what is of value for your circumstances.
Here’s a post that turned up this morning on Angel investors: https://open.substack.com/pub/colingardiner/p/what-to-look-f...
Beware: the VC model and the whole industry can at times be parasitic, self-serving, and definitely has vibes of Ponzi or pyramid schemes, and the worst excesses are hidden from view if you are not in the industry.
Good fortune.
IMHO if possible, grow your business without investors and retain control. Jason Cohen has an amazing talk on bootstrapping, with some very solid advice from his own experiences: https://m.youtube.com/watch?v=otbnC2zE2rw
Look at co-working spaces near you? Our business worked in an incubator for a bit, and the problems we saw with investors in other startups were eye-watering. We retained 100% ownership, and that was the right decision for us. Learn from the mistakes of others at different stages just by being around them.
Flick through my previous comments on VC for external links that explain my points better: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
I was a founder, and I don’t know shit about VC except through reading, so be appropriately skeptical about my thoughts and only take what is of value for your circumstances.
Here’s a post that turned up this morning on Angel investors: https://open.substack.com/pub/colingardiner/p/what-to-look-f...
Beware: the VC model and the whole industry can at times be parasitic, self-serving, and definitely has vibes of Ponzi or pyramid schemes, and the worst excesses are hidden from view if you are not in the industry.
Good fortune.
> Why the hell hire super smart people when you have to tell them how to do things?
A very hard question. One side is that every smart engineer will do what's best for their use case.
The other is now you might have 100 smart engineers with 100 smart different solutions which can cause repeated work, integration issues, missed common solutions, etc.
Both approaches have lots of pros cons and tradeoffs
A very hard question. One side is that every smart engineer will do what's best for their use case.
The other is now you might have 100 smart engineers with 100 smart different solutions which can cause repeated work, integration issues, missed common solutions, etc.
Both approaches have lots of pros cons and tradeoffs
Generally the best approach there is to elevate a bunch of your smartest engineers into an architecture team that a) set good standards but also b) appreciates the nuance of problems sufficiently to know when to deviate.
A small team can probably only handle one tech stack, an org with 10 teams can probably handle 10 stacks if they needed to but can get efficiency by standardising on some number less than 10.
However making those calls shouldn't be coming down from the CTO, it should be made among the architects that actually design and build the systems.
A small team can probably only handle one tech stack, an org with 10 teams can probably handle 10 stacks if they needed to but can get efficiency by standardising on some number less than 10.
However making those calls shouldn't be coming down from the CTO, it should be made among the architects that actually design and build the systems.
Where I'd consider a (good) CTO should be somehow responsible for the high-level architectural decisions. Accumulate enough expertise and know-how (via "architects" or rather very senior engineers) and then discuss the pros/cons and at the end weigh each argument and decide what may work best.
The smart have to design for the average of devs and not the smartest.
The smart design for easily creating beginners in thenfuture to work with what exists.
Too often smart devs have to be clever with coding, where clever architecture can often be more flexible.
The smart design for easily creating beginners in thenfuture to work with what exists.
Too often smart devs have to be clever with coding, where clever architecture can often be more flexible.
I agree with him that serverless should not be the default and there should be a clear justification for needing those features.
He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.
He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.
Exactly, not everything needs scale.
For a low traffic, low use system, lambda + dynamo might just be the cheapest and easiest to implement and maintain.
And there are generally more of those type of systems.
For a low traffic, low use system, lambda + dynamo might just be the cheapest and easiest to implement and maintain.
And there are generally more of those type of systems.
Depends? Is your load consistent and latency sensitive? Then having a small ECS instance may be a lot better.
Yes, to call Serverless hype train and then talk of K8s as the "boring solid choice" is pretty funny.
Fully agree with this - it completely ignores the context and domain for a one size fits all solution, which is precisely what is wrong with engineering today - lack of holism.
The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong."
I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.
This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your switching/egress costs
The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong."
I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.
This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your switching/egress costs
> I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.
I agree with OP. If you hand me a binary and instructions on how to run it, I can have it running on either AWS or DO in a container, connected to a managed database with snapshots enabled in about 30 minutes. If the toolchain is easy to set up, I can have CI/CD with 0-downtime deployments with auto-rollback in another 30 minutes, and you won't need to touch either of those until you're hitting thousands of concurrent users for a typical web app.
> you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)
this is FUD of the n'th degree - if you suspect AWS are spying on your managed RDS data, I don't know what to tell you.
I agree with OP. If you hand me a binary and instructions on how to run it, I can have it running on either AWS or DO in a container, connected to a managed database with snapshots enabled in about 30 minutes. If the toolchain is easy to set up, I can have CI/CD with 0-downtime deployments with auto-rollback in another 30 minutes, and you won't need to touch either of those until you're hitting thousands of concurrent users for a typical web app.
> you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)
this is FUD of the n'th degree - if you suspect AWS are spying on your managed RDS data, I don't know what to tell you.
Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate
And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.
And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.
> Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate
With 30 minutes work, I've set up a production quality solution that I won't have to touch for months/years, and has almost no vendor lock-in. Containers are portable, you can run them on AWS, DO, GCP, or install docker on a bare metal instance, it's up to you. Until you _do_ decide to migrate, it's pretty much 0 upkeep (I checked one of the work projects I stood up and we haven't touched the infrastructure side of it for 9 months).
> And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.
That's not what you said at first, you said "you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)". You're telling AWS that you're running containers and a database.
With 30 minutes work, I've set up a production quality solution that I won't have to touch for months/years, and has almost no vendor lock-in. Containers are portable, you can run them on AWS, DO, GCP, or install docker on a bare metal instance, it's up to you. Until you _do_ decide to migrate, it's pretty much 0 upkeep (I checked one of the work projects I stood up and we haven't touched the infrastructure side of it for 9 months).
> And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.
That's not what you said at first, you said "you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)". You're telling AWS that you're running containers and a database.
Yes, and I’m sure after you set things up with “no vendor lock in”, you came back years later when the company grew and organized a large migration where you had to deal with budgets, transferring data, regression testing, going through architecture review boards, the PMO, permissions, compliance, training, etc.
I’m also sure after many years no vendor specific corner cases snuck in that you didn’t realize were there
I’m also sure after many years no vendor specific corner cases snuck in that you didn’t realize were there
> poor engineering leadership.
Or fantastic engineering leadership.
OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails.
K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.
Or fantastic engineering leadership.
OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails.
K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.
Having done the dreaded AWS + kops -> GKE migration I must agree, it was hard but nowhere near as hard as it could have been.
> simple
> k8s
Pick one
> k8s
Pick one
Eh. There are different types of complexity.
k8s has a nice contained local complexity that I prefer.
It's not simple internally but it's simple to use if you know how it works.
So really it's not "k8s isn't simple" it's more "nothing is simple" so pick your preferred type of complexity.
k8s has a nice contained local complexity that I prefer.
It's not simple internally but it's simple to use if you know how it works.
So really it's not "k8s isn't simple" it's more "nothing is simple" so pick your preferred type of complexity.
> It's not simple internally but it's simple to use if you know how it works.
Isn’t that literally true about anything?
Isn’t that literally true about anything?
> it's purely based on emotion
Maybe charitably assume it’s based on years of experience doing this exact thing?
It’s certainly my experience.
Maybe charitably assume it’s based on years of experience doing this exact thing?
It’s certainly my experience.
Stop with the confident silver bullets. There are effective teams using every stack out there, including serverless. The professional chooses the right tool for the job and team. This is an instance of one change between options that happened to blow in the direction you like, but there are good options at all points of the continuum.
I've used most languages in on-prem and distributed settings, from app to web app to micro services to serverless to single DB's of all kinds to distributed everything, in all clouds, and all have their place and time and team. Sometimes I'd arrive with a perfectly confident view and then learn that actually there is another truth. Sometimes a 50-page stored procedure is perfect. Microservices is awesome in the right context, monolith in the right context, just stop with the simple answers because you have a particular lens that works for you.
What a single lens does really well is that it removes options and that makes you faster at how you solve most problems. Good for you, simplicity is excellent. But don't diss professionals and their leaders who have made choices and then adjust them based on the data.
I've used most languages in on-prem and distributed settings, from app to web app to micro services to serverless to single DB's of all kinds to distributed everything, in all clouds, and all have their place and time and team. Sometimes I'd arrive with a perfectly confident view and then learn that actually there is another truth. Sometimes a 50-page stored procedure is perfect. Microservices is awesome in the right context, monolith in the right context, just stop with the simple answers because you have a particular lens that works for you.
What a single lens does really well is that it removes options and that makes you faster at how you solve most problems. Good for you, simplicity is excellent. But don't diss professionals and their leaders who have made choices and then adjust them based on the data.
There is a big difference between learning the nuances of engineering as you go and re-adjusting your viewpoint vs bandwagoning on a new tool because it's cool even though you can't tie any of it's advantages back to your problem statement.
That isn't making a choice and adjusting based on the data, that is just shoddy engineering.
It should have been obvious to anyone at a cursory glance that not only did serverless not offer anything to this application it would be actively detrimental. Simple napkin math of data rate of a 1080p video stream should have made that immediately clear.
Just because you can use something to get a task done doesn't make it a good choice. I can still write pretty decent Perl. It gets the job done and for some protocols that haven't changed in a long time like LDAP the existing CPAN libraries are still some of the best. But that isn't the right choice is it?
I don't think it's defensible to argue that we should build thing sub-standardly just to give people more choice. That is like arguing that real engineers should have more freedom in choice in how they design bridges. No one would agree with that.
We get away with it because our bridges are much cheaper to replace and noone dies if they break or have to be rewritten because they rotted too much to run anymore.
Right now as it stands arguments for serverless aren't technical in nature. They are a best an appeal to "the traditional way is slow" or some notion of "infinite scalability", the first of which isn't true at all and the latter doesn't matter for anyone that is arguing in favor of serverless.
Also even if it was "faster" (I don't agree but lets postulate) that probably wouldn't make it a good decision either unless it was proportionally faster vs it's costs (not just dollar costs, architecture, complexity of managed objects, etc). Which I highly doubt would be the case.
As you can see from a technical POV it's just very very hard to rationalise a position for serverless > Boring Tech. For me it would take too much mental gymnastics.
I was perhaps a little strong but my points are solid.
If it was good it would have proven advantages by now. K8s adoption curve vs serverless adoption curve tells the real story of quality IMO, especially given how much bigger of a move it was to k8s from what preceeded it.
That isn't making a choice and adjusting based on the data, that is just shoddy engineering.
It should have been obvious to anyone at a cursory glance that not only did serverless not offer anything to this application it would be actively detrimental. Simple napkin math of data rate of a 1080p video stream should have made that immediately clear.
Just because you can use something to get a task done doesn't make it a good choice. I can still write pretty decent Perl. It gets the job done and for some protocols that haven't changed in a long time like LDAP the existing CPAN libraries are still some of the best. But that isn't the right choice is it?
I don't think it's defensible to argue that we should build thing sub-standardly just to give people more choice. That is like arguing that real engineers should have more freedom in choice in how they design bridges. No one would agree with that.
We get away with it because our bridges are much cheaper to replace and noone dies if they break or have to be rewritten because they rotted too much to run anymore.
Right now as it stands arguments for serverless aren't technical in nature. They are a best an appeal to "the traditional way is slow" or some notion of "infinite scalability", the first of which isn't true at all and the latter doesn't matter for anyone that is arguing in favor of serverless.
Also even if it was "faster" (I don't agree but lets postulate) that probably wouldn't make it a good decision either unless it was proportionally faster vs it's costs (not just dollar costs, architecture, complexity of managed objects, etc). Which I highly doubt would be the case.
As you can see from a technical POV it's just very very hard to rationalise a position for serverless > Boring Tech. For me it would take too much mental gymnastics.
I was perhaps a little strong but my points are solid.
If it was good it would have proven advantages by now. K8s adoption curve vs serverless adoption curve tells the real story of quality IMO, especially given how much bigger of a move it was to k8s from what preceeded it.
> I don't think it's defensible to argue that we should build thing sub-standardly just to give people more choice
That's like a village of straw men. Stop going around telling everyone they're fucking idiots. We'd probably agree 90% of the time, dealing down the intensity just makes for a better discussion.
They "already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale (our target was to monitor thousands of concurrent streams and grow that number over time)"
Good principle #1, reuse. They launched faster than starting from scratch. They changed direction and modified some components, and then published, against what marketing would probably have wanted. Real problems are when you paint yourself into a corner and can't fix it. They fixed it.
Amazon, keep telling us what lessons you've learned because it helps the whole industry.
(Edited to remove some of my own incendiary annoyance.)
That's like a village of straw men. Stop going around telling everyone they're fucking idiots. We'd probably agree 90% of the time, dealing down the intensity just makes for a better discussion.
They "already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale (our target was to monitor thousands of concurrent streams and grow that number over time)"
Good principle #1, reuse. They launched faster than starting from scratch. They changed direction and modified some components, and then published, against what marketing would probably have wanted. Real problems are when you paint yourself into a corner and can't fix it. They fixed it.
Amazon, keep telling us what lessons you've learned because it helps the whole industry.
(Edited to remove some of my own incendiary annoyance.)
> Serverless first is a mistake. It should be "serverless where it's distinct advantages are actually useful" which is way less common than people think.
Wouldn't that approach take away the remaining advantages of Serverless while still leaving you with the cost?
My understanding was that Serverless is supposed to free you from the burden of maintaining a complete software stack, including configuring the db, server, application runtime etc and just focus on the business logic - a bit like one of the old "sharehost"/LAMPP setups from a few decades ago.
If that is true, then you'd be well-advised to either go all-in and structure your entire application as lambda functions, or don't use functions at all and go with a "traditional" kubernetes deployment.
As both strategies come with considerable complexity cost, combining them would give you the cost of both. I think you'd want to avoid that if possible.
Wouldn't that approach take away the remaining advantages of Serverless while still leaving you with the cost?
My understanding was that Serverless is supposed to free you from the burden of maintaining a complete software stack, including configuring the db, server, application runtime etc and just focus on the business logic - a bit like one of the old "sharehost"/LAMPP setups from a few decades ago.
If that is true, then you'd be well-advised to either go all-in and structure your entire application as lambda functions, or don't use functions at all and go with a "traditional" kubernetes deployment.
As both strategies come with considerable complexity cost, combining them would give you the cost of both. I think you'd want to avoid that if possible.
Somewhat. I think if I was to ever use serverless for something it would be in concert with a traditional setup so in a way yes, you pay 2x the deployment complexity because now you are orchestrating 2 environments.
However if you have a usecase that is absolutely a perfect fit for serverless i.e incredibly bursty, unpredicable, very short runtime. To the point it would be to the detriment of other services hosted on the same infrastructure i.e would need say an isolated cluster anyway then yeah, serverless could be a cost worth paying.
However if you have a usecase that is absolutely a perfect fit for serverless i.e incredibly bursty, unpredicable, very short runtime. To the point it would be to the detriment of other services hosted on the same infrastructure i.e would need say an isolated cluster anyway then yeah, serverless could be a cost worth paying.
Main issue of lambda and similar offerings is that you're pretty much bound to the cloud.
Kubernetes is much more open standard. It's much easier to move to other cloud or self-host.
It's more about insurance and keeping other options.
Managed Kubernetes, managed databases are a thing and you should use them if you can. Just don't depend on cloud-specific stuff too much.
Kubernetes is much more open standard. It's much easier to move to other cloud or self-host.
It's more about insurance and keeping other options.
Managed Kubernetes, managed databases are a thing and you should use them if you can. Just don't depend on cloud-specific stuff too much.
Doesn't JVM here result in some amount of "dicking around" with respect to which JVM language to use? Or do you mean that it doesn't matter, as long as it's all JVM? Or do you mean to just pick java?
One thing I've seen in my experience generally buying this "standardize on boring tech" take, is that it's really common these days for a company to be doing some amount of "serious" data science work, and most people seem to really want to do that in python, and it also seems to benefit pretty significantly from some kind of OLAP database system. So adding those things onto your "boring" stack becomes JVM+psql+k8s+python+snowflake/databricks/duckdb(?).
But I agree with your real point about serverless.
One thing I've seen in my experience generally buying this "standardize on boring tech" take, is that it's really common these days for a company to be doing some amount of "serious" data science work, and most people seem to really want to do that in python, and it also seems to benefit pretty significantly from some kind of OLAP database system. So adding those things onto your "boring" stack becomes JVM+psql+k8s+python+snowflake/databricks/duckdb(?).
But I agree with your real point about serverless.
For backend, JVM implies Java. Although Kotlin is safe choice as well, it's not as popular so I wouldn't suggest to use it, unless you have good reasons to (like team of Android developers who decided to build backend).
I actually use Kotlin. :)
I am happy with either Kotlin or Java though.
Reasons for Kotlin is mostly nullability typing though @NotNull gets you pretty far in Java these days with the tooling configured correctly.
I like the syntax a bit more and extension functions are a nice sugar vs util classes with static functions.
The main reason however is I have a lot more luck convincing Typescript developers to try Kotlin than Java which is relevant when most of my colleagues have primarily done Typescript.
Either or though IMO, both are great choices for backend development and they interoperate really well.
I am happy with either Kotlin or Java though.
Reasons for Kotlin is mostly nullability typing though @NotNull gets you pretty far in Java these days with the tooling configured correctly.
I like the syntax a bit more and extension functions are a nice sugar vs util classes with static functions.
The main reason however is I have a lot more luck convincing Typescript developers to try Kotlin than Java which is relevant when most of my colleagues have primarily done Typescript.
Either or though IMO, both are great choices for backend development and they interoperate really well.
> For backend, JVM implies Java.
... no it doesn't.
... no it doesn't.
Generally speaking your data-science stuff wouldn't fit into the serverless bucket anyway though.
Also generally Python is just used for terminal processing in the Big Data world, all the pieces in between are JVM anyway.
Also generally Python is just used for terminal processing in the Big Data world, all the pieces in between are JVM anyway.
Yeah, I mean if you're already a JVM shop then you could probably run Spark yourself (which can be painful) and use SparkSQL to dump to Python (maybe on databricks, though I'm not a massive fan) and do your fancy ML stuff there.
Alternatively, you could just use a read-only PG replica and Python here, which would suffice for most DS needs (unless your core product is DS related or you have huuuuuggggggggeeee data (which you probably don't)).
Alternatively, you could just use a read-only PG replica and Python here, which would suffice for most DS needs (unless your core product is DS related or you have huuuuuggggggggeeee data (which you probably don't)).
Yeah that is fair too. Most people have tiny data. Then there is places I have worked... where 10TB qualified as small data that you "just query with Presto" and where more exotic things were needed for actually interacting with the "big data".
All relative in the end.
All relative in the end.
Yeah, snowflake is super nice for analytics, but PG will get you a long way with less complexity (given that you're already using it for OLTP).
This doesn't mesh with my experience of the current breakdown in usage between python and the jvm for these use cases.
+1 that k8s really isn't that much more challenging configuration syntax than most 'serverless' solutions (which is a bit different than FaaS as in the article). The difficulty with k8s is (w/ a managed solution) generally the cost of running the control plane though most IaaS will 'give' you a zonal one for free to encourage more spend because inevitably you don't just need one, but ideally one per environment per app/product maybe even per engineer if you want eng to be able to stand up their own 'full' environments of beefier apps w/ lots of components and dependencies.
Serverless first is probably not a mistake in general. But we may have a different definition? For me Serverless means no need to administer a server (in the sense of "an operating system").
So baremetal, vps, = serverful
And lambdas, app engines, firebase, ... = serverless
There is a space where you are deploying a monolith app, and letting the cloud provider do the scaling, server management, environment management. But at any time you could run that same app happily on a server.
I think starting off there is not a mistake.
I think k8s is an advanced move. You can start off with the cloud built-in container management solution, then move to k8s later if you really need it.
So baremetal, vps, = serverful
And lambdas, app engines, firebase, ... = serverless
There is a space where you are deploying a monolith app, and letting the cloud provider do the scaling, server management, environment management. But at any time you could run that same app happily on a server.
I think starting off there is not a mistake.
I think k8s is an advanced move. You can start off with the cloud built-in container management solution, then move to k8s later if you really need it.
yeah, I think I should have said something along the lines "Functions as a Service first is a mistake" but whatever, I didn't think people would focus so much on what they consider serverless is.
IMO managed container runtimes are PaaS, are both FaaS and PaaS serverless? Who knows, we don't seem to have hard definitions but in my mind PaaS predates serverless becoming a fashionable word so I associate with FaaS and FaaS-like things only.
IMO managed container runtimes are PaaS, are both FaaS and PaaS serverless? Who knows, we don't seem to have hard definitions but in my mind PaaS predates serverless becoming a fashionable word so I associate with FaaS and FaaS-like things only.
Safest to say the exact thing (or list of things) I reckon!
Technically "serverless" should mean "no server" which means "no process fielding network requests" i.e. "unable to accept inbound network requests" - so some kind of batch processing system, or scheduled lambdas :-)
Technically "serverless" should mean "no server" which means "no process fielding network requests" i.e. "unable to accept inbound network requests" - so some kind of batch processing system, or scheduled lambdas :-)
I can't help but laugh at this when I think of how easy it is to imagine the exact same tenor in a post dismissing Kubernetes as newfangled, overcomplicated garbage.
Yeah there probably were people that wrote those. Not me though, I had been through the whole wringer before k8s so I knew what it was when I saw it.
I was working on a competitor to k8s at the time though so my knowledge of the subject matter definitely influenced my opinion of it.
I thought it was unstable though, largely due to dependency on etcd which was also unstable at the time. Thankfully that got sorted out though.
I was working on a competitor to k8s at the time though so my knowledge of the subject matter definitely influenced my opinion of it.
I thought it was unstable though, largely due to dependency on etcd which was also unstable at the time. Thankfully that got sorted out though.
I disagree very much. I don't think any blanket statement like that makes sense.
I recently started building a service and need to keep costs down. Serverless is perfect for it. My service has low to moderate traffic, costs pennies to run on AWS Lambda and MongoDB Atlas. If I had gone the boring route of JVM + PG + k8s, putting aside the time it would take to defamiliarize myself with anything on the JVM, the cost to run this service would have been in the hundreds of dollars a month vs the pennies. Interestingly the most expensive part of my current setup is the NAT Router on my VPC. With JVM + PG + k8s it would have been PG or K8S depending on where I ran PG.
I do agree that there is a misconception with containers taking longer than server less. I don't think either takes longer considering the current tooling available.
Seems like you got burned on Serverless at some point, I'm sorry that happened, but for many people and teams it is a productivity multiplier and can be a big cost cutting solution.
I recently started building a service and need to keep costs down. Serverless is perfect for it. My service has low to moderate traffic, costs pennies to run on AWS Lambda and MongoDB Atlas. If I had gone the boring route of JVM + PG + k8s, putting aside the time it would take to defamiliarize myself with anything on the JVM, the cost to run this service would have been in the hundreds of dollars a month vs the pennies. Interestingly the most expensive part of my current setup is the NAT Router on my VPC. With JVM + PG + k8s it would have been PG or K8S depending on where I ran PG.
I do agree that there is a misconception with containers taking longer than server less. I don't think either takes longer considering the current tooling available.
Seems like you got burned on Serverless at some point, I'm sorry that happened, but for many people and teams it is a productivity multiplier and can be a big cost cutting solution.
> the cost to run this service would have been in the hundreds of dollars a month vs the pennies
And maybe this again proves the initial comment's point.
> I do agree that there is a misconception with containers taking longer than server less. I don't think either takes longer considering the current tooling available.
And (the misconception) that it costs more.
> AWS Lambda and MongoDB Atlas
If you take a managed database with a free tier and compare it there are similarly managed databases (SQL) based with a free tier. Not really fair to say PG will cost more.
This also comes down to the world is not just AWS. k8s is cheap in a lot of places and definitely not hundreds a month.
And maybe this again proves the initial comment's point.
> I do agree that there is a misconception with containers taking longer than server less. I don't think either takes longer considering the current tooling available.
And (the misconception) that it costs more.
> AWS Lambda and MongoDB Atlas
If you take a managed database with a free tier and compare it there are similarly managed databases (SQL) based with a free tier. Not really fair to say PG will cost more.
This also comes down to the world is not just AWS. k8s is cheap in a lot of places and definitely not hundreds a month.
I’m going to have to disagree with this.
There are different shades of serverless, and you don’t start at an R620 plugged into a colo rack. You grow into it OR start with a large enough problem to demand it.
You start with GitHub pages or Cloudflare pages where all you have is CSS, HTML, and JS served from the equivalent of a global CDN that you didn’t have to build for essentially free.
For a mom and pop shop, how many times are they going to have to contact you over the next 20 years for a piece of software doing its job running on GitHub pages? Practically zero, unless it’s a feature request. Can’t say the same for a Linux box plugged into the wall under your desk or an R620 at the colo downtown.
Then you graduate to sprinkling a little bit of persistent state in. When you need a little bit of persistent state for a small mom and pop, deciding to vet colo vendors and signing that shop up to be responsible for sysadmining an R620 with a Linux kernel is irresponsible. You find a serverless vendor for your DB layer (like the Firebase of yore) or a set of CRUD handlers written on infra that fully manages the language runtime, operating system, kernel, and infra for your customer (like Cloudflare workers).
How many times is the customer going to need to contact you over the next 20 years for your Cloudflare worker? Probably only a handful. Odds are, significantly less than an R620 plugged into a server rack at the colo downtown.
If you outgrow this, you probably are starting to look at VMs or colo.
Serverless is real. It’s not lambda. It’s pervasive. It’s provides substantial value in ongoing maintenance and engineering costs if you pick the right vendor. And a majority of the market demands it as a starting point.
An engineer is expensive and most customers aren’t in a position to practice the craft themselves. The best service we can provide is to minimize their dependence on our field.
There are different shades of serverless, and you don’t start at an R620 plugged into a colo rack. You grow into it OR start with a large enough problem to demand it.
You start with GitHub pages or Cloudflare pages where all you have is CSS, HTML, and JS served from the equivalent of a global CDN that you didn’t have to build for essentially free.
For a mom and pop shop, how many times are they going to have to contact you over the next 20 years for a piece of software doing its job running on GitHub pages? Practically zero, unless it’s a feature request. Can’t say the same for a Linux box plugged into the wall under your desk or an R620 at the colo downtown.
Then you graduate to sprinkling a little bit of persistent state in. When you need a little bit of persistent state for a small mom and pop, deciding to vet colo vendors and signing that shop up to be responsible for sysadmining an R620 with a Linux kernel is irresponsible. You find a serverless vendor for your DB layer (like the Firebase of yore) or a set of CRUD handlers written on infra that fully manages the language runtime, operating system, kernel, and infra for your customer (like Cloudflare workers).
How many times is the customer going to need to contact you over the next 20 years for your Cloudflare worker? Probably only a handful. Odds are, significantly less than an R620 plugged into a server rack at the colo downtown.
If you outgrow this, you probably are starting to look at VMs or colo.
Serverless is real. It’s not lambda. It’s pervasive. It’s provides substantial value in ongoing maintenance and engineering costs if you pick the right vendor. And a majority of the market demands it as a starting point.
An engineer is expensive and most customers aren’t in a position to practice the craft themselves. The best service we can provide is to minimize their dependence on our field.
> How many times is the customer going to need to contact you over the next 20 years for your Cloudflare worker? Probably only a handful. Odds are, significantly less than an R620 plugged into a server rack at the colo downtown.
This has nothing to do with "serverless". Heroku did the same thing well before serverless was available.
Keeping a simple app running on heroku over the past 15 years has required about 3 hours admin total since I set it up (moving from an old stack to a newer one).
This has nothing to do with "serverless". Heroku did the same thing well before serverless was available.
Keeping a simple app running on heroku over the past 15 years has required about 3 hours admin total since I set it up (moving from an old stack to a newer one).
Agreed. Serverless is a spectrum. Heroku for sure sits on that spectrum.
At it core serverless is about delegation of responsibility and how much responsibility you can push across the vendor boundary.
At it core serverless is about delegation of responsibility and how much responsibility you can push across the vendor boundary.
Wow what an emotional and biased take.
Architecture isn’t like Lord of the Rings where one ring rules them all.
What if the dev team doesn’t know Java? Are you suggesting they be forced to learn it?
What if the company has 20 applications with complex interactions? Are you suggesting they all be baked into k8s and then have someone manage containers full-time? What if no one in the company understands containers?
I’ve been at this for 40 years and will tell you there is no common architecture environment.
There can be any mix of business logic, services, data storage, containers, events, queues, redundancy, reports, analytics, external systems, 3rd party black boxes, skill sets and levels, budget, tech stack, and leadership.
Trying to pigeonhole any architecture into that mix is the very definition of amateur.
Architecture isn’t like Lord of the Rings where one ring rules them all.
What if the dev team doesn’t know Java? Are you suggesting they be forced to learn it?
What if the company has 20 applications with complex interactions? Are you suggesting they all be baked into k8s and then have someone manage containers full-time? What if no one in the company understands containers?
I’ve been at this for 40 years and will tell you there is no common architecture environment.
There can be any mix of business logic, services, data storage, containers, events, queues, redundancy, reports, analytics, external systems, 3rd party black boxes, skill sets and levels, budget, tech stack, and leadership.
Trying to pigeonhole any architecture into that mix is the very definition of amateur.
> Trying to pigeonhole any architecture into that mix is the very definition of amateur.
I’ve also been at this 40 years and what you ended with is worded unnecessarily crass, in that folks of course learn perspective over time.
I’ve also been at this 40 years and what you ended with is worded unnecessarily crass, in that folks of course learn perspective over time.
I think I was matching the same tone as the previous comment.
“I don’t think you need unnecessary complexity. You should use Kubernetes”
There is a certain lack of self awareness.
What exactly do you think that serverless is besides spinning up VMs on demand where you don’t have to maintain the orchestration?
There is a certain lack of self awareness.
What exactly do you think that serverless is besides spinning up VMs on demand where you don’t have to maintain the orchestration?
I personally find k8s to be much simpler to use.
Both are complex internally, k8s does a better job of preventing that complexity from leaking IMO.
You need to know roughly the same amount of things to use both.
On k8s to deploy something simple you will need:
1. A deployment to actually run your app.
2. A service describe how to talk to your app.
3. An ingress to route requests to the service.
On Lambda w/API Gateway:
1. Lambda Function
2. IAM Role
3. API Gateway objects (Deployment, RestApi, Stage)
More if you expand out the policies and API gateway objects. These become -really- burdensome if you are using IaC like Pulumi or Terraform.
In practice using k8s these days is essentially managed also, ale GKE/EKS/AKS so the real differences are down to:
1. API, the FaaS etc ones are proprietary, k8s is standardized.
2. Runtime environment and execution limits, varies by FaaS, k8s has none.
3. Access to durable storage, FaaS only object/HTTP based storage, you can attach EBS-like to your k8s containers.
4. Architecture incentives. FaaS highly incentivises microservices, k8s isn't really opinionated. FaaS is more like cgi-bin on steroids, k8s is daemons on a box but with fancy management.
5. Scaling. FaaS scale "infinitely", k8s scales depending on settings of cluster autoscaler.
So generally you get as good or better from k8s, the API complexity is about the same except now it's standard no lock-in, no enforced runtime limits, you have durable containers so you can attach a debugger, etc.
Hopefully this shows a bit why I think k8s is at least as simple as serverless if not more simple.
Both are complex internally, k8s does a better job of preventing that complexity from leaking IMO.
You need to know roughly the same amount of things to use both.
On k8s to deploy something simple you will need:
1. A deployment to actually run your app.
2. A service describe how to talk to your app.
3. An ingress to route requests to the service.
On Lambda w/API Gateway:
1. Lambda Function
2. IAM Role
3. API Gateway objects (Deployment, RestApi, Stage)
More if you expand out the policies and API gateway objects. These become -really- burdensome if you are using IaC like Pulumi or Terraform.
In practice using k8s these days is essentially managed also, ale GKE/EKS/AKS so the real differences are down to:
1. API, the FaaS etc ones are proprietary, k8s is standardized.
2. Runtime environment and execution limits, varies by FaaS, k8s has none.
3. Access to durable storage, FaaS only object/HTTP based storage, you can attach EBS-like to your k8s containers.
4. Architecture incentives. FaaS highly incentivises microservices, k8s isn't really opinionated. FaaS is more like cgi-bin on steroids, k8s is daemons on a box but with fancy management.
5. Scaling. FaaS scale "infinitely", k8s scales depending on settings of cluster autoscaler.
So generally you get as good or better from k8s, the API complexity is about the same except now it's standard no lock-in, no enforced runtime limits, you have durable containers so you can attach a debugger, etc.
Hopefully this shows a bit why I think k8s is at least as simple as serverless if not more simple.
Anyone who says that they aren’t “locked-in” hasn’t done a large migration and dealt with both the technical and organizational complexity of any large migration.
I’ve been involved with organizations that just literally wanted to do a “lift and shift” of 100s of VMs and databases. They were definitely didn’t have any “vendor lock in”. Guess how long it took? Guess how much regression testing it took and planning?
I’ve been involved with organizations that just literally wanted to do a “lift and shift” of 100s of VMs and databases. They were definitely didn’t have any “vendor lock in”. Guess how long it took? Guess how much regression testing it took and planning?
Months probably. Took the team I was on ~6mo to complete a very large AWS -> GKE migration.
If we weren't using k8s it probably would have been impossible though. We were lucky we used fairly minimal managed services, mostly RDS and a tiny bit of SQS and Kinesis which were ported to Kafka as part of the move.
If we weren't using k8s it probably would have been impossible though. We were lucky we used fairly minimal managed services, mostly RDS and a tiny bit of SQS and Kinesis which were ported to Kafka as part of the move.
When I talked to people at Amazon it sounded like there was a huge push to dogfeed Lambda. The guy was almost worshipping it in the interview and appalled that I wasn't interested in it. :)
IMO serverless is incredibly useful when latency isn't important. Backend stuff, metrics, analytics, APIs, etc should be serverless. It allows you to scale incredibly easily at basically 0 cost.
For everything else, you have to test and examine the use cases and performance requirements. In general if there's a UI that's doing work it shouldn't be serverless. Work = latency = bad.
For everything else, you have to test and examine the use cases and performance requirements. In general if there's a UI that's doing work it shouldn't be serverless. Work = latency = bad.
How are you going to throw away the competitive advantage of serverless first if you’re a startup? You’re just asking to lose to whichever of your competition that goes serverless first.
However, if you’re a massive company like Amazon working on a less-than-cutting edge project, you certainly have the resources to avoid serverless up front to avoid rewrite costs down the line.
However, if you’re a massive company like Amazon working on a less-than-cutting edge project, you certainly have the resources to avoid serverless up front to avoid rewrite costs down the line.
LOL.
Competitive advantage comes from understanding user needs, great UX, and a smooth running sales machine. No-one cares that you built with technology X, whatever that is. There is no silver bullet.
Competitive advantage comes from understanding user needs, great UX, and a smooth running sales machine. No-one cares that you built with technology X, whatever that is. There is no silver bullet.
While I agree with the conclusion, competitive advantage is by definition very relative. It might be what you say, if you're producing some sort of saas offering. But it might be something completely different as well.
[deleted]
I'm confused by your language since you can do "serverless" or "FaaS" or "cloudfunctions" on k8s.
> Boring Tech stack of JVM + PG + k8s.
We live in a new time where k8s is "boring" and grouped in as such.
We live in a new time where k8s is "boring" and grouped in as such.
For infra folk that lived through the "before time" it certainly is and damn is it refreshing that it's become boring at most places. :)
Is jvm pg and k8s boring enough?
K8s is overkill. Jvm is heavy. Pg is fine but relational dbs generally suck.
K8s is overkill. Jvm is heavy. Pg is fine but relational dbs generally suck.
What is better than relational dbs (and for what purposes)?
I've been learning k8s fundamentals recently after using borg for a long time and then awhile kind of just poking at k8s trying to use it without learning it. Honestly I think it seems like about as simple as a solution to a broad set of real common problems could plausibly be.
I've been learning k8s fundamentals recently after using borg for a long time and then awhile kind of just poking at k8s trying to use it without learning it. Honestly I think it seems like about as simple as a solution to a broad set of real common problems could plausibly be.
Relational dbs are based on relational model. It’s a giant black box of magic. And it’s magic is based on a bunch of constraints to make the relational algebra work. It’s really quite flawed for modern web apps.
It ... really isn't a black box of magic? It's a transparent box of - as your comment points out - implementing a particular mathematical model.
You're gonna need to give me some more on why it's "quite flawed for modern web apps". I may even agree with you, but it's tough to know from what you've written so far.
You're gonna need to give me some more on why it's "quite flawed for modern web apps". I may even agree with you, but it's tough to know from what you've written so far.
> isn't a black box of magic
Referring mostly to the query planner - you have no control over it, and you have to dive deep to figure out why it's making certain plans, and how it collects statistics to make these calls. Plus the planner essentially gives up at 6+ joins.
> quite flawed for modern web apps
Try listening to updates on a query.
Referring mostly to the query planner - you have no control over it, and you have to dive deep to figure out why it's making certain plans, and how it collects statistics to make these calls. Plus the planner essentially gives up at 6+ joins.
> quite flawed for modern web apps
Try listening to updates on a query.
elaborate on how relational databases suck?
They're not web scale.
https://youtu.be/b2F-DItXtZs
https://youtu.be/b2F-DItXtZs
Try streaming updates to a query without polling…which is essentially all web apps.
It’s fundamentally the wrong approach and people don’t realize. The relational model is good for analytics.
Think about how often you need to know the internals or you have to debug perf by trying to get a black box to do what you want.
Devs should be a layer deeper.
It’s fundamentally the wrong approach and people don’t realize. The relational model is good for analytics.
Think about how often you need to know the internals or you have to debug perf by trying to get a black box to do what you want.
Devs should be a layer deeper.
Very interesting that they were able to reduce costs by 90%, though also agree with the opinions mentioned here.
What if Amazon take bad choices in "something" and my company is tied up with its services? I know that Prime is always "Amazon" and that's okay. But for a startup (but not only) imho it's better starts with something more "standard" than something as serverless that is obviously an abstraction above other tecnologies.
The nth posting of this story.
https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=fal...
https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=fal...
I'm starting to be convinced that the EU should regulate acceptable software design patterns. This will prevent us all from heaving to implement techbro cargo cult pseudo-innovations like "Serverless for everything" /s
EU should stop regulating what is acceptable and let society naturally come to conclusions. Limiting software designs seems like an area the government shouldn't be involved with.
Regulate cloud providers if necessary not abstract patterns
Regulate cloud providers if necessary not abstract patterns
microservices? to sling video files over the internet? tell me you're over-engineering without telling me you're over-engineering
The whole industry is stupid at this point. I don’t think anyone knows what they are doing. Everyone just wants to play with the new shiny thing. Half the engineers probably have ADHD which leads to this behavior of seeking novelty.
Everyone is obsessed with their shitty stack they used at their last company, and this shit will keep going on and on.
The only things that matters is how easy is something to debug and to change. Think about how often you have heard that as any kind of priority. So many shitty decisions would be avoided if people started thinking about that.
Everyone is obsessed with their shitty stack they used at their last company, and this shit will keep going on and on.
The only things that matters is how easy is something to debug and to change. Think about how often you have heard that as any kind of priority. So many shitty decisions would be avoided if people started thinking about that.
Alan Kay once pointed out something along the line that when excitement exceeds knowledge you get fashion. For 10 years I’ve been in and around languages that were declared dead by the HN crowd because there wasn’t enough new hype around it. It’s not a perfectly inverse correlation, but honestly at this point when I see HN is excited about something new it raises red flags for me.
I see this more on twitter to be honest. HN, of late, has been a great source of info on the ‘boring’ stacks e.g Rails.
Twitter is where you get the JavaScript ‘influencers’ who appeal to a core demographic of engineers with FOMO who signal their ‘in status’ by waxing lyrical about the latest framework
Twitter is where you get the JavaScript ‘influencers’ who appeal to a core demographic of engineers with FOMO who signal their ‘in status’ by waxing lyrical about the latest framework
Not to pat myself on the back but fashion was the exact analogy I came up while describing the situation. When everyone begins focusing on tech as an end in itself as opposed to it serving a bigger purpose we end up here.
And add to that the zero cost of capital of 2010s, it lead to engineers being given almost free reign which is generally a bad idea. “we are a technology company” would attract VC money, as if tech itself was a product of every darn startup.
And add to that the zero cost of capital of 2010s, it lead to engineers being given almost free reign which is generally a bad idea. “we are a technology company” would attract VC money, as if tech itself was a product of every darn startup.
>And add to that the zero cost of capital of 2010s
I suspect the timing of ChatGPT release just as interest rates started going up is a ploy to layer more obscurity into tech choices to drive the hype harder ad keep the gravy train going even as capex becomes more costly.
I suspect the timing of ChatGPT release just as interest rates started going up is a ploy to layer more obscurity into tech choices to drive the hype harder ad keep the gravy train going even as capex becomes more costly.
It's easy to sit back and say that, but I can almost guarantee that if you actually follow this up with a set of opinions about what stack / architecture / approach is easiest to debug and change, you will find yourself back in the argument with the rest of us stupid people.
I believe there's a lot more variables than "how easy is something to debug and change". The issue is that how easy something is to debug and change is very contextual to the point in the lifetime of an application, the industry the application will live in, and the changing requirements as the application grows in complexity.
Even the simplest interpretation of your statement leads to considerable argument. Often the ease of change and the ease of debugging are at odds with one another, where a highly dynamic approach might make change a breeze, but make debugging more difficult.
I believe there's a lot more variables than "how easy is something to debug and change". The issue is that how easy something is to debug and change is very contextual to the point in the lifetime of an application, the industry the application will live in, and the changing requirements as the application grows in complexity.
Even the simplest interpretation of your statement leads to considerable argument. Often the ease of change and the ease of debugging are at odds with one another, where a highly dynamic approach might make change a breeze, but make debugging more difficult.
This very much. I find that often times we are looking at a narrow subset of the factors that lead to certain choices or technologies. Yes, there is a lot of fluff sometimes but there’s also the cycle of wanting to improve on existing pain points.
Things like serverless appeal to businesses due to the much smaller logistical implications it presents to a business. Some businesses enjoy that (at least as promised), It’s all taken care of and the team you need to manage it is likely to be smaller. It’s pay as you go for what would otherwise imply more human beings and contracts on your plate.
I imagine at some point those baked in costs and redundancies, the luxury of having huge scale on demand in a general sense, will be much higher than something thoughtfully designed when a company has understood their needs well enough after operating at scale.
For what it’s worth I prefer bare metal clusters :)
Things like serverless appeal to businesses due to the much smaller logistical implications it presents to a business. Some businesses enjoy that (at least as promised), It’s all taken care of and the team you need to manage it is likely to be smaller. It’s pay as you go for what would otherwise imply more human beings and contracts on your plate.
I imagine at some point those baked in costs and redundancies, the luxury of having huge scale on demand in a general sense, will be much higher than something thoughtfully designed when a company has understood their needs well enough after operating at scale.
For what it’s worth I prefer bare metal clusters :)
Serverless in the way we do it today is the best example of something that is impossible to debug.
Every choice people have to make in tech today comes with this crazy tradeoff of where you get some benefit like scalability, but then it becomes impossible to debug or change.
What's crazy about modern tech is that we've created a world where its so difficult to change things that its analogous to building pipes under asphalt roads. You have to spend all night with a construction crew digging up the road just to change a small thing, and then you have to put it all back again. Except that we work with digital stuff (any physical stuff is completely abtracted) where we can theoretically change anything instantly. The fact that we use shipping container analogies is ironic, if you've ever seen how long shipping loading/unloading and transport actually takes. We need to get back to the computer as the bicycle of the mind...at the moment it is not.
I think we will look back on this era of tech as we do to the era of the secretary on the typewriter...and laugh at the inefficiency.
Every choice people have to make in tech today comes with this crazy tradeoff of where you get some benefit like scalability, but then it becomes impossible to debug or change.
What's crazy about modern tech is that we've created a world where its so difficult to change things that its analogous to building pipes under asphalt roads. You have to spend all night with a construction crew digging up the road just to change a small thing, and then you have to put it all back again. Except that we work with digital stuff (any physical stuff is completely abtracted) where we can theoretically change anything instantly. The fact that we use shipping container analogies is ironic, if you've ever seen how long shipping loading/unloading and transport actually takes. We need to get back to the computer as the bicycle of the mind...at the moment it is not.
I think we will look back on this era of tech as we do to the era of the secretary on the typewriter...and laugh at the inefficiency.
Agreed on the trade offs, not sure if to the same degree though. It seems to me that it’s critical to know if you’re gonna end up going “against the grain” on a platform that makes as many decisions as a serverless one does.
I agree that "easy to debug" is subjective. But it's insanity to act like that doesn't mean that some architectures are objectively easier to debug than others. How can splitting something up into different services and introducing network calls make it easier? You're not reducing complexity at all. It makes debugging harder.
It would indeed be insanity to argue that, which is why I didn’t. The argument for splitting into micro services is generally that particular teams would own particular services and therefore make debugging/change easier for them, at the cost of increased overall complexity of the overall system. Anyone arguing it reduces complexity is being very hopeful.
I personally think an org should generally cling to a so called monolith as long as they possibly can. But it’s very hard to be prescriptive without knowing particulars of the problems an org is trying to solve.
I personally think an org should generally cling to a so called monolith as long as they possibly can. But it’s very hard to be prescriptive without knowing particulars of the problems an org is trying to solve.
> set of opinions about stack
The stack doesn't exist today. It could have if we didn't jump so quickly to new technology and all these different prog languages.
I've just never heard someone pitch new tech with: "easy to debug and change". It's always: look at this contrived example with some new esoteric feature.
The stack doesn't exist today. It could have if we didn't jump so quickly to new technology and all these different prog languages.
I've just never heard someone pitch new tech with: "easy to debug and change". It's always: look at this contrived example with some new esoteric feature.
I get your cynicism because I've made the same complaint. But I wouldn't universalize it, because I think it's more of a marketing and context issue than an issue fundamental to the whole ecosystem.
Startups often prioritize time-to-market for what is effectively a prototype. Startups get a lot of hype. Therefore, frameworks that focus on time to market over refactoring / debugging get a lot of hype. Sometimes those frameworks actually internally put a lot of effort into refactoring / debugging, but when it comes to marketing themselves, they don't.
But some frameworks are marketed more towards mature or large organizations, and they certainly do emphasize refactoring / debugging. The JVM and .NET runtimes are big examples here. Their debugging and operational monitoring capabilities are several classes above most of the competition, and their primary and tool-vendor ecosystems continually push further debugging capabilities as features.
Startups often prioritize time-to-market for what is effectively a prototype. Startups get a lot of hype. Therefore, frameworks that focus on time to market over refactoring / debugging get a lot of hype. Sometimes those frameworks actually internally put a lot of effort into refactoring / debugging, but when it comes to marketing themselves, they don't.
But some frameworks are marketed more towards mature or large organizations, and they certainly do emphasize refactoring / debugging. The JVM and .NET runtimes are big examples here. Their debugging and operational monitoring capabilities are several classes above most of the competition, and their primary and tool-vendor ecosystems continually push further debugging capabilities as features.
> JVM and .NET runtime
Maybe I'm too caught up in startup-land, as I haven't touched these tools in a long time. The debugging tools and IDE support on these platforms have always been great.
The Visual Studio install time, the huge frameworks, slow start time, bad package managers, were such killers for productivity.
Maybe I'm too caught up in startup-land, as I haven't touched these tools in a long time. The debugging tools and IDE support on these platforms have always been great.
The Visual Studio install time, the huge frameworks, slow start time, bad package managers, were such killers for productivity.
Half the engineers are made to believe they are God's gift to humanity and like so totally clever. But we all spend our time building banal and ultimately dumb shit nobody wants. But people construct their entire identity around tech anyway.
It is no surprise the place is littered with ivory towers.
It is no surprise the place is littered with ivory towers.
There are a lot of financial incentives pushing engineers to seek novelty, even if they might not necessarily want to. Promotions and raises are heavily tied to projects introducing or inventing shiny new tech. Heck, even VC funding for starting a business is tied to this. One might also view this as the human condition: a bias to solve problems by adding rather than subtracting.
> even VC funding for starting a business is tied to this
This is the main problem.
Excluding VC, businesses exist to make profit. The purpose of IT and engineering is to enable that business or make it operate more efficiently - engineering and technology should never be a goal in and of itself.
Given this situation, the most efficient technology (that enables faster delivery of business software) would win. "Shiny" stuff would never win in terms of mindshare unless it's actually good and enables faster delivery than the status quo.
Now add VCs to the mix. Suddenly you have an infinite supply of free money that distorts the usual market dynamics, allowing businesses to keep operating with no regard to profitability. The downwards pressure of the market pushing for businesses to operate more profitably is no longer there. As an engineer, you no longer have the pressure of building technology for the purposes of making the business more efficient - you lost that valuable feedback loop.
Now run this for 10 years. Entire careers will be built on this - why even try to build a profitable business when you can be a "serial startup CEO" and merely use the right amount of shiny to solicit endless VC funding to continue the adventure? Imagine a well-meaning, budding engineer joining such a startup and learning such a corrupted idea of "engineering" with none of the usual objectives, pressures and feedback loops of engineering.
In the end, you end up with "senior" engineers (with over-inflated titles) whose entire careers are built on technology that wouldn't actually pass muster in an actual business under the normal market pressures. But they don't even know anything else or why that is wrong (because they've started their career during this charade and never had any taste of what real engineering is), so they keep pushing for these (now-inadequate) solutions, deluded that they are doing the right thing.
This is the main problem.
Excluding VC, businesses exist to make profit. The purpose of IT and engineering is to enable that business or make it operate more efficiently - engineering and technology should never be a goal in and of itself.
Given this situation, the most efficient technology (that enables faster delivery of business software) would win. "Shiny" stuff would never win in terms of mindshare unless it's actually good and enables faster delivery than the status quo.
Now add VCs to the mix. Suddenly you have an infinite supply of free money that distorts the usual market dynamics, allowing businesses to keep operating with no regard to profitability. The downwards pressure of the market pushing for businesses to operate more profitably is no longer there. As an engineer, you no longer have the pressure of building technology for the purposes of making the business more efficient - you lost that valuable feedback loop.
Now run this for 10 years. Entire careers will be built on this - why even try to build a profitable business when you can be a "serial startup CEO" and merely use the right amount of shiny to solicit endless VC funding to continue the adventure? Imagine a well-meaning, budding engineer joining such a startup and learning such a corrupted idea of "engineering" with none of the usual objectives, pressures and feedback loops of engineering.
In the end, you end up with "senior" engineers (with over-inflated titles) whose entire careers are built on technology that wouldn't actually pass muster in an actual business under the normal market pressures. But they don't even know anything else or why that is wrong (because they've started their career during this charade and never had any taste of what real engineering is), so they keep pushing for these (now-inadequate) solutions, deluded that they are doing the right thing.
I don't have ADHD but I find your comment quite offensive.
It also seems that people without ADHD don't like innovation? That either came out wrong, or ...boh?
It also seems that people without ADHD don't like innovation? That either came out wrong, or ...boh?
I do have ADHD and op is spot on.
The software industry is filled to the brim with midwits who think their idea is the only right idea even though its nothing special and they’re ignoring actual problems
As op said - when’s the last time you heard someone talk about writing code that’s easy to debug? Never. Yet 90%+ of companies will make you pass arbitrary algorithms tests
The field is dominated by A types who confuse their neuroticism and ego for talent
The software industry is filled to the brim with midwits who think their idea is the only right idea even though its nothing special and they’re ignoring actual problems
As op said - when’s the last time you heard someone talk about writing code that’s easy to debug? Never. Yet 90%+ of companies will make you pass arbitrary algorithms tests
The field is dominated by A types who confuse their neuroticism and ego for talent
But why to bring ADHD into this? This I don't understand.
I do understand the fact that everyone is following one or two bandwagons, but I don't have the need of saying that someone must have a mental condition for doing that.
There are MANY more reasons for adopting technologies. Marketing, corporate culture, hiring, trying things out, wrong people in leadership position (as the article shows!), etc etc. It's not just always the minions to do things wrong.
I do understand the fact that everyone is following one or two bandwagons, but I don't have the need of saying that someone must have a mental condition for doing that.
There are MANY more reasons for adopting technologies. Marketing, corporate culture, hiring, trying things out, wrong people in leadership position (as the article shows!), etc etc. It's not just always the minions to do things wrong.
Why? It's a consequence of labelling a sizeable swathe of the IT workforce as "Disordered", having a "Deficit", and a "Mental condition" (which is NOT to be talked about in any context unapproved by the grand council of ADHD).
If you didn't slap derogatory labels on people, just let people who wanted to use amphetamines and methylphenidate, and addressed problems like organisation and hyperactivity independently people would end up with roughly the same quality of life but without the othering and stereotyping. Which is to say, this sort of stereotyping is an inevitable and continual iatrogenic consequence of the sociomedical status quo. When something is labelled a deficit and disorder - then people almost naturally compare that thing to other things they do not like.
If you didn't slap derogatory labels on people, just let people who wanted to use amphetamines and methylphenidate, and addressed problems like organisation and hyperactivity independently people would end up with roughly the same quality of life but without the othering and stereotyping. Which is to say, this sort of stereotyping is an inevitable and continual iatrogenic consequence of the sociomedical status quo. When something is labelled a deficit and disorder - then people almost naturally compare that thing to other things they do not like.
My guess is ADHD people are drawn to the software industry. Open source is probably fueled by it. It’s a spectrum and I think a lot more people are on the spectrum than they realize.
It’s a constant pursuit for new stuff. Sometimes this can be good, but other times it’s just new for the sake of being different.
I think the minions are surprisingly to blame for a lot of the problems with tech. The tech debt excuse is of our own doing.
It’s a constant pursuit for new stuff. Sometimes this can be good, but other times it’s just new for the sake of being different.
I think the minions are surprisingly to blame for a lot of the problems with tech. The tech debt excuse is of our own doing.
Because people with ADHD crave stimulation and pursuing new and exciting technologies is very satisfying.
Does adopting serverless actually accomplish that? With servers you have a bunch of technologies to learn about that are abstracted away with serverless. Every person I know with ADHD in the IT field, not knowing many, is into servers.
What if they know servers really really well? That could be boring for them. Kind of like hiking the same trail every week.
> I don't have ADHD but I find your comment quite offensive.
I don't mean this in a bad way, but people not part of <group> need to dial it down a bit. Too many get offended by things that people in <group> don't see as a problem.
I don't mean this in a bad way, but people not part of <group> need to dial it down a bit. Too many get offended by things that people in <group> don't see as a problem.
I don't think ADHD had anything to do with it.
Young people with little knowledge (because they are young) give a try to every new piece of technology and start projects with them. I've been there.
Older people that eventually found good tools to solve problems have less time and enthusiasm to try new things. They still do but only when they think it's worth it. I'm there now.
Most developers and many CTOs are young.
Young people with little knowledge (because they are young) give a try to every new piece of technology and start projects with them. I've been there.
Older people that eventually found good tools to solve problems have less time and enthusiasm to try new things. They still do but only when they think it's worth it. I'm there now.
Most developers and many CTOs are young.
> Older people that eventually found good tools to solve problems have less time and enthusiasm to try new things. They still do but only when they think it's worth it. I'm there now.
The best thing about this phase is you just get so much shit done instead of screwing around with 10 half-baked over-marketed frameworks.
The best thing about this phase is you just get so much shit done instead of screwing around with 10 half-baked over-marketed frameworks.
> The only things that matters is how easy is something to debug and to change.
These are important, maybe most important, but software exists in an ecosystem. We still use some ASPX with .NET Framework at work and it's old tech but reliable - however, the world is moving on and it's becoming difficult to start new projects in it, not to mention end of life for patching and so on.
Likewise, at a prior role we used Perl and a lot of interfacing was made difficult by the fact that no one bothered making modules for Perl anymore. So while we can file these under easy to change, sometimes the world moves on without you and you have to think about the new shiny thing whether you want to or not. I personally started using Python at home not for any love of the language, but because everyone wrote modules and API examples for it.
These are important, maybe most important, but software exists in an ecosystem. We still use some ASPX with .NET Framework at work and it's old tech but reliable - however, the world is moving on and it's becoming difficult to start new projects in it, not to mention end of life for patching and so on.
Likewise, at a prior role we used Perl and a lot of interfacing was made difficult by the fact that no one bothered making modules for Perl anymore. So while we can file these under easy to change, sometimes the world moves on without you and you have to think about the new shiny thing whether you want to or not. I personally started using Python at home not for any love of the language, but because everyone wrote modules and API examples for it.
I found a similar thing with Node.js. There are tons of under maintained packages while everyone moved to Go and Rust.
A lot of packages you never care what’s underneath if they are maintained and do the job. But when you have to manually patch something and it’s a pile of crap underneath then it sucks.
I much prefer to write things myself if I can now instead of using an existing package or framework. Having my own simple packages in my monorepo that I can instantly change is far better than new third party packages.
I start to wonder if checking in third party modules is the way to go.
A lot of packages you never care what’s underneath if they are maintained and do the job. But when you have to manually patch something and it’s a pile of crap underneath then it sucks.
I much prefer to write things myself if I can now instead of using an existing package or framework. Having my own simple packages in my monorepo that I can instantly change is far better than new third party packages.
I start to wonder if checking in third party modules is the way to go.
I don’t think it’s the ADHD. I have it but I’m very conservative on the stack. And I know many developers who don’t have ADHD, some newbie and some who code longer than 20 years and are going after the latest shiny thing.
I think it’s more of what people assume it’s a status symbol among their peers. I noticed for example, in church, that some people think the more boring is more saint and don’t like more a church with a more contemporary dynamic approach and I think the same effect applies in the software development world.
I think it’s more of what people assume it’s a status symbol among their peers. I noticed for example, in church, that some people think the more boring is more saint and don’t like more a church with a more contemporary dynamic approach and I think the same effect applies in the software development world.
I've often mused about creating a language that put readability, maintainability, logging etc over, say, performance.
You might enjoy Go. Lots of emphasis has been put into the development tooling and ecosystem. There is an impressive commitment to avoiding breaking changes.
We have a legacy Go microservices codebase that’s over a decade old and because the Go maintainers have paid so much attention to avoiding creating breaking changes and a guarantee to never create a “Go 2”, the codebase has aged much better than stuff written at the same time in Python (some of which is still in Python 2 because it is too risky to upgrade, and that functionality will be killed instead because we don’t know how to safely update it). Most of the time we don’t need to make any changes when bumping Go versions, just fixing up things our linter warns is deprecated.
We have a legacy Go microservices codebase that’s over a decade old and because the Go maintainers have paid so much attention to avoiding creating breaking changes and a guarantee to never create a “Go 2”, the codebase has aged much better than stuff written at the same time in Python (some of which is still in Python 2 because it is too risky to upgrade, and that functionality will be killed instead because we don’t know how to safely update it). Most of the time we don’t need to make any changes when bumping Go versions, just fixing up things our linter warns is deprecated.
Performance is absolutely a non-issue when modern "engineering" insists on running everything on underpowered VMs with high-latency network-based storage. Even if your language is truly, actually slow, you can get an instant performance boost just moving that workload to a modern bare-metal server (cloud providers basically ate all the performance improvements of modern hardware), or scale that (most likely stateless) workload horizontally if a single bare-metal machine isn't enough.
Exactly.
It amazes me how some companies still adopt the "build it and they will come" approach, and when confronted with harsh reality, they double down on building (and especially increasing architectural complexity). CEOs, if you can't attract customers with an idea or don't have a vertical that people are eager to pay for, cramming 1000 half-baked features into a microservice architecture and using the latest programming language / paradigm won't save your company!