The products we're building are more complex than they used to be, but I don't know if its true that backend development has inherently become more complex than it was in the past.
I've worked on everything from dinosaurs older than me to new greenfield projects using all the latest tech. I think over-complexity emerges in many ways, notably: too much abstraction, ignorance, dogma, and just straight up carelessness.
I think the abstraction problem is the most common and severe though. In enterprise software, I've spent an entire day in the past combing through several layers of services and repositories just to get the full picture of how a response object is being created for a single controller method.
Let me give another example: think about the great debate over ORM vs No-ORM. It's fundamentally a battle about the balances and costs of abstraction in software development. On one hand, you have a (hopefully) simple interface for defining a db's schema, but do I really want to obfuscate the developer from the implementation details of that `findBy` method and trust that the ORM will be 1) used correctly and 2) generate a performant, sensible query? What about security handling? Should I have my developers concerned about preventing potential SQL injections, or just let the ORM take care of that too? The answer depends on a lot of things.
Anyway, I believe for every abstraction:
- something significantly useful should be added (this seems like a truism as I write this, but i've seen a startling amount of useless abstractions)
- the cost should be carefully calculated
- understand where complexity is being added and subtracted
I believe thinking about abstraction in this way would've cut down a lot of the complexity I've seen in softwares past.
Vendor lock-in can be a real issue as you scale. I work at a company going through a hypergrowth phase, and we're more or less locked into Azure, which has been objectively awful for us
It's been a year since I touched an AWS lambda, but I'd bet money that cold starts are still an issue. There is a common hack that half works: have your function run every minute (you can use eventbridge rules for this); in the function handler, the first thing you should evaluate is whether or not it's a warming event or not, and exit 0 if it is. Your results may vary (mine did lol)
I've worked on everything from dinosaurs older than me to new greenfield projects using all the latest tech. I think over-complexity emerges in many ways, notably: too much abstraction, ignorance, dogma, and just straight up carelessness.
I think the abstraction problem is the most common and severe though. In enterprise software, I've spent an entire day in the past combing through several layers of services and repositories just to get the full picture of how a response object is being created for a single controller method.
Let me give another example: think about the great debate over ORM vs No-ORM. It's fundamentally a battle about the balances and costs of abstraction in software development. On one hand, you have a (hopefully) simple interface for defining a db's schema, but do I really want to obfuscate the developer from the implementation details of that `findBy` method and trust that the ORM will be 1) used correctly and 2) generate a performant, sensible query? What about security handling? Should I have my developers concerned about preventing potential SQL injections, or just let the ORM take care of that too? The answer depends on a lot of things.
Anyway, I believe for every abstraction:
- something significantly useful should be added (this seems like a truism as I write this, but i've seen a startling amount of useless abstractions)
- the cost should be carefully calculated
- understand where complexity is being added and subtracted
I believe thinking about abstraction in this way would've cut down a lot of the complexity I've seen in softwares past.