Modern Applications at AWS(allthingsdistributed.com)
allthingsdistributed.com
Modern Applications at AWS
https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html
42 comments
Yes! Been preaching this for years!
I've untangled more than a couple small eng teams (~ < 20 devs) that were trying to do microservices using this recipe on aws and just horrendously failing at it.
A well-architeched monolith will really go a long way - generally perhaps easily up to 100 or more devs before thinking about microservices.
I've untangled more than a couple small eng teams (~ < 20 devs) that were trying to do microservices using this recipe on aws and just horrendously failing at it.
A well-architeched monolith will really go a long way - generally perhaps easily up to 100 or more devs before thinking about microservices.
- If it doesn't have very different scaling needs from the rest of the application, probably don't make it a microservice.
- If it isn't something you could plausibly imagine using as a 3rd party service (emailer/sms/push messages, authentication/authorization, payments, image processing, et c.) probably don't make it a microservice.
- If you don't have at least two applications at least in development that need the same service/functionality, probably don't make a microservice.
- If the rest of your app will completely fall over if this service fails, probably don't make it a microservice.
- Do write anything that resembles the above, but that you don't actually need to make a microservice yet, as a library with totally decoupled deps from the rest of your program.
- If it isn't something you could plausibly imagine using as a 3rd party service (emailer/sms/push messages, authentication/authorization, payments, image processing, et c.) probably don't make it a microservice.
- If you don't have at least two applications at least in development that need the same service/functionality, probably don't make a microservice.
- If the rest of your app will completely fall over if this service fails, probably don't make it a microservice.
- Do write anything that resembles the above, but that you don't actually need to make a microservice yet, as a library with totally decoupled deps from the rest of your program.
> The most pragmatic approach to microservices is to start with a monolith and shave off the intensive parts to separate services as you need to.
I've seen that attempted several times but I've never seen it succeed even after several years of shaving things off here and there.
The problem with this approach is the same problem that causes all technical debt -- cleaning up technical debt takes time away from developing new features, which is the company's highest priority. Adding more engineers to the team to help tackle debt won't help because management will see that as an opportunity to get more new features.
Now that the dust has settled, it's become clear that the real advantage of microservices is to decouple teams.
Shared codebases become harder to work with as the number of engineers increases, and that hurts developer productivity.
Don't build a monolithic service for the whole company -- have every team manage their own services, on their own development and release schedule, and their own production scale. Their dependencies on other teams should be based on APIs, not code sharing. Whether an individual team chooses a monolith or microservices doesn't really matter.
I've seen that attempted several times but I've never seen it succeed even after several years of shaving things off here and there.
The problem with this approach is the same problem that causes all technical debt -- cleaning up technical debt takes time away from developing new features, which is the company's highest priority. Adding more engineers to the team to help tackle debt won't help because management will see that as an opportunity to get more new features.
Now that the dust has settled, it's become clear that the real advantage of microservices is to decouple teams.
Shared codebases become harder to work with as the number of engineers increases, and that hurts developer productivity.
Don't build a monolithic service for the whole company -- have every team manage their own services, on their own development and release schedule, and their own production scale. Their dependencies on other teams should be based on APIs, not code sharing. Whether an individual team chooses a monolith or microservices doesn't really matter.
It isn’t as black and white as you are portraying it. You can go too far in the microservices direction, and you can go too far towards a monolith. There’s a sweet spot. I currently think that this sweet spot is around one service per “team” (around no more than 10 people). This lets teams operate and deploy their services independently, while still retaining some of the cohesiveness of a monolith. It also lets the company “refactor” engineering teams by moving service ownership around. There are definitely legitimate reasons for a team to own multiple services though (organizational changes, differences in scale, security constraints, etc.).
Monoliths shared across dozens of people are operational nightmares because there’s not a strong sense of operational ownership, so you often end up with a bunch of process and bureaucratic overhead to compensate.
Your advice to start with a monolith is too simplistic. If you know how your service will scale and your engineering teams will grow, then you have enough information a priori to avoid the waste of attempting to split apart a monolith later.
Monoliths shared across dozens of people are operational nightmares because there’s not a strong sense of operational ownership, so you often end up with a bunch of process and bureaucratic overhead to compensate.
Your advice to start with a monolith is too simplistic. If you know how your service will scale and your engineering teams will grow, then you have enough information a priori to avoid the waste of attempting to split apart a monolith later.
Yes, it's awfully fun that they're using the term "Modern" applications to describe applications which I wouldn't touch with a bargepole. Guess I'm just not "Modern".
Then again, I was chuckling in the a similar way a few years ago when not using mongodb et al made me not "web scale".
Then again, I was chuckling in the a similar way a few years ago when not using mongodb et al made me not "web scale".
> To succeed in using application development to increase agility and innovation speed, organizations must adopt five elements, in any order: microservices; purpose-built databases; automated software release pipelines; a serverless operational model; and automated, continuous security.
When I worked at AWS, it used to take us months to change the position of a button in the UI. Things that our competitors did in hours and days, we did in months and years.
I'd say that as a general heuristic, if you want "to increase agility and innovation speed" do the opposite of what Amazon does.
When I worked at AWS, it used to take us months to change the position of a button in the UI. Things that our competitors did in hours and days, we did in months and years.
I'd say that as a general heuristic, if you want "to increase agility and innovation speed" do the opposite of what Amazon does.
An interface even if it's just a user interface is a contract. You can't just go changing it because you want to. Many immature companies do this and think it's innovation. It's not, all your users will have to relearn the new UI change.
I'm guessing the person you're replying to understands the implication of changing a UI. Maybe, instead of telling them why they're wrong, you could ask them if resistance to UI changes was the problem, or something else was? They were there. Were you?
That attitude is what leads to the AWS console.
As someone who uses AWS console every day at job, I must say I find it pretty usable. I especially like the part that whatever buttons I learned stays in the place, so I don't have to waste time figuring out what the hell I have to press this time.
So if AWS's attitude lead to its console, I think they're doing great.
So if AWS's attitude lead to its console, I think they're doing great.
As someone who uses the AWS console once a month, I find it incredibly confusing and I'm always hunting for the right sub-section. This is the old trope: experts prefer complex and efficient systems, while beginners prefer simplicity.
The AWS console is a bit clunky, but would you really prefer it if it was changing all the time?
When I worked there we had continuous deployment with pipelines straight into Lambda in native AWS. There’s so many different teams at Amazon I wouldn’t take anyone’s advice that claim they can speak for all of development there.
Agility is not about deployment speed. It's about being able to react quickly and easily to new information (customer complaints, bugs, new opportunities, etc). You don't need a perspective from the inside to make the claim that AWS's evolution rate of existing products is weak. Innovation speed might be high horizontally (launching a ton of new services), but not vertically (improving existing ones).
Ok, so how do you explain the blistering speed at which AWS (not Amazon) releases new services, updates and features.
Not being smarty pants here; It truly boggles my mind how fast they move.
Not being smarty pants here; It truly boggles my mind how fast they move.
AWS is great at horizontal innovation (launching a lot of brand new stuff). It's vertical innovation where they struggle (improving existing products). Software architecture helps the latter much more than the former.
Excellent point. If they just cycled a lot harder on existing products they would be best in class.
Two cases to your point.
It took a VERY long time for ELB(and thus autoscale/cloudformation) to get the ability to drain connections. This was a show stopper for most serious players on anything customer facing but unless you were, or knew somebody who was, spending close to 500k/mo you would never be able to get a sense of timeframe out of AWS for addressing this.
ALB launched without the ability to route requests based on sub-domain even though this was the only direct upgrade path for most people who had been building SOAs on top of ELB and Route53.. Took way too long to get that added in and of course with no public acknowledgement of time frame.
Now that I think about it.. AWS is NOT very agile at all because they barely include a small fraction of their stakeholders in their product design and iteration process. You'd have better luck trying to squeeze water out of a moon rock than getting AWS to comment on roadmap for a service.
It took a VERY long time for ELB(and thus autoscale/cloudformation) to get the ability to drain connections. This was a show stopper for most serious players on anything customer facing but unless you were, or knew somebody who was, spending close to 500k/mo you would never be able to get a sense of timeframe out of AWS for addressing this.
ALB launched without the ability to route requests based on sub-domain even though this was the only direct upgrade path for most people who had been building SOAs on top of ELB and Route53.. Took way too long to get that added in and of course with no public acknowledgement of time frame.
Now that I think about it.. AWS is NOT very agile at all because they barely include a small fraction of their stakeholders in their product design and iteration process. You'd have better luck trying to squeeze water out of a moon rock than getting AWS to comment on roadmap for a service.
Do they move fast? I'm still waiting for Aurora to offer PostgreSQL 11 from last October.
Hell, I'm still waiting for AWS to offer an upgrade path to PostgreSQL 10 from October 2017.
Hell, I'm still waiting for AWS to offer an upgrade path to PostgreSQL 10 from October 2017.
Or to support any JDK newer than 8 on Elastic Beanstalk.
!!! OpenJDK was EOL'd back in January.
Your last statement is being blunt on Amazon.
Customers come first in AWS culture. With a massive user base, they need to care about not breaking existing customer applications/integrations.
If AWS did not evolve its software processes, they would have already fallen apart by now. The teams move reasonably well for the size of a big company.
Customers come first in AWS culture. With a massive user base, they need to care about not breaking existing customer applications/integrations.
If AWS did not evolve its software processes, they would have already fallen apart by now. The teams move reasonably well for the size of a big company.
> Customers come first in AWS culture
As a customer of AWS, it never seemed that way.
As a customer of AWS, it never seemed that way.
I think from a certain size and number of users any innocent looking change can cause a lot of unintended consequences. I always wonder how Microsoft or Apple make any changes to their OS considering the large number of legacy code that should keep running.
The considerations part of developing an operating system can be viewed pretty well in the mailing lists of Linux Distributions (and other OSes, but less so because of the fewer stake holders).
The decision making processes are influenced by different cultures: Debian aims for a very stable system with no breakages at least within a major version, Windows aims for close to endless backwards compatibility (hence the bloat), while Apple designs its systems with more of a "move fast" attitude (hence the quick proliferation of new features).
The decision making processes are influenced by different cultures: Debian aims for a very stable system with no breakages at least within a major version, Windows aims for close to endless backwards compatibility (hence the bloat), while Apple designs its systems with more of a "move fast" attitude (hence the quick proliferation of new features).
>When I worked at AWS, it used to take us months to change the position of a button in the UI. Things that our competitors did in hours and days, we did in months and years.
From what I understand, the culture in a large company varies from team to team. Was it your experience that all the teams were that slow?
Are you claiming the whole of AWS is that way or that since your team was that way we should disregard any advice about AWS?
If I invert all the advice from the post, would that be better as a development practice? A large monolith, backed by an off the shelf database where there is a central security team looking after every line of code?
From what I understand, the culture in a large company varies from team to team. Was it your experience that all the teams were that slow?
Are you claiming the whole of AWS is that way or that since your team was that way we should disregard any advice about AWS?
If I invert all the advice from the post, would that be better as a development practice? A large monolith, backed by an off the shelf database where there is a central security team looking after every line of code?
It's a company-wide issue, and it can be observed from the outside too. Existing products improve at a rate that is significantly slower than competitors (with a few exceptions). And I'm not referring to feature creep, but fixing obvious gaps and issues that the company would have been aware of for years (or more than a decade in some cases).
> I'd say that as a general heuristic, if you want "to increase agility and innovation speed" do the opposite of what Amazon does.
That’s a pretty vague and useless heuristic. Since you worked at AWS, maybe you could answer why AWS moved slowly on releasing features for existing services? An Amazon-y “five whys” analysis would probably yield a lot more specific and useful heuristics.
That’s a pretty vague and useless heuristic. Since you worked at AWS, maybe you could answer why AWS moved slowly on releasing features for existing services? An Amazon-y “five whys” analysis would probably yield a lot more specific and useful heuristics.
Don’t take it too literally. My point was that AWS doesn’t show agility and rapid innovation on existing products, so when it tries to tell you how to do that, maybe take it with a grain of salt.
Are you comparing to competitors of your size?
I ran an enterprise-backed startup. We built way faster than our enterprise backers when it was our own name. If something carried an association or backing of the enterprise, it took at least 10 times longer. They had a reputation to uphold and needed to ensure everything reflected that.
I ran an enterprise-backed startup. We built way faster than our enterprise backers when it was our own name. If something carried an association or backing of the enterprise, it took at least 10 times longer. They had a reputation to uphold and needed to ensure everything reflected that.
This is because feature verticals build their own UI, right? Seems like there should be a solution to this.
Changing a button was just one example. How about this? https://twitter.com/dvassallo/status/1166700686467166208
You know, instead of linking to your Twitter, you could just embed the content directly here instead.
For people who don't want to go to Twitter since they are here, Tweet is about that AWS changed the limit of ElastiCache names from 20 characters to 40 and author says about it: "If you can make a change like this to your SaaS without any fuss, you can probably outcompete AWS on innovation pace and agility. For AWS, this is a significant accomplishment worthy of the “2019 What’s New” page"
For people who don't want to go to Twitter since they are here, Tweet is about that AWS changed the limit of ElastiCache names from 20 characters to 40 and author says about it: "If you can make a change like this to your SaaS without any fuss, you can probably outcompete AWS on innovation pace and agility. For AWS, this is a significant accomplishment worthy of the “2019 What’s New” page"
This just seems like a pretty dig. The issue of feature verticals/microservices/soa leading to complex UI interaction is a fundamental issue we can actually sink our teeth into. I'm not really interested in the minutia of particular aws failures.
Interesting, the claim is that if you can easily change ElastiCache cluster names "you can probably out compete AWS..."
It will be interesting to see if that happens soon then. I know IBM and the many other Openshift partners are chasing this. Will be interesting to see if they can deliver.
My own view - do not underestimate the depth of services and innovation AWS and GCP can bring to bear. I'm just not hearing folks go - oh, this provider let me use a longer name for my elasticache cluster. Instead its, why shouldn't we just use AWS or GCP.
It will be interesting to see if that happens soon then. I know IBM and the many other Openshift partners are chasing this. Will be interesting to see if they can deliver.
My own view - do not underestimate the depth of services and innovation AWS and GCP can bring to bear. I'm just not hearing folks go - oh, this provider let me use a longer name for my elasticache cluster. Instead its, why shouldn't we just use AWS or GCP.
I have a request for opinions!
I’ve noticed that software quality never seems to enter into conversations about scaling. Do code hygenie, writing clean code, and good documentation not matter when you’re moving fast and growing like crazy? I’ve noticed the approach seems to be to focus on operations rather than the way the code is written. Use CI, monitoring, organize teams around services, etc. From my friends in the bay however, I hear the code itself is mostly written and band aided over and over again with refactoring and rewriting seen as “wasting time” when you could be writing more band aids or improving operations.
Does the band aid and if it’s not broke don’t fix it and do exactly what you need to fix it approaches really yield the same or better results as approaching software as a craftsmen? How do you convince an AWS engineer that these things are important when what’s in prod is working and a rewrite just seems like pot stirring? Or a refactor isn’t critical work to add more features?
I’ve noticed that software quality never seems to enter into conversations about scaling. Do code hygenie, writing clean code, and good documentation not matter when you’re moving fast and growing like crazy? I’ve noticed the approach seems to be to focus on operations rather than the way the code is written. Use CI, monitoring, organize teams around services, etc. From my friends in the bay however, I hear the code itself is mostly written and band aided over and over again with refactoring and rewriting seen as “wasting time” when you could be writing more band aids or improving operations.
Does the band aid and if it’s not broke don’t fix it and do exactly what you need to fix it approaches really yield the same or better results as approaching software as a craftsmen? How do you convince an AWS engineer that these things are important when what’s in prod is working and a rewrite just seems like pot stirring? Or a refactor isn’t critical work to add more features?
For large corporations like Amazon, there are huge opportunities to move to Lambda. Back when I worked there, teams would over-provision their services like crazy. Most of it to scale for the holiday season. Most teams would not give that extra capacity back.
The other way of dealing with this is to put normal services in an autoscaling environment like Kubernetes. You don't have to go as far as Lambda to get the benefits.
Moving to autoscaling is really more of a cultural problem than a technical one. If teams are used to being able to reserve excess capacity they will be wary of autoscaling -- what if they try to scale but there is not capacity to do so because other teams used it up? People need to trust that the system will give them the capacity they need or they will continue to overprovision.
Moving to autoscaling is really more of a cultural problem than a technical one. If teams are used to being able to reserve excess capacity they will be wary of autoscaling -- what if they try to scale but there is not capacity to do so because other teams used it up? People need to trust that the system will give them the capacity they need or they will continue to overprovision.
Cloud applications have become much much more complex than OS based application development.
Don’t take the path of cloud based development if you are aiming to simplify.
Don’t take the path of cloud based development if you are aiming to simplify.
Unfortunately I see a lot of mid and senior devs try to propose these architectures during interviews and when asked questions on integration testing, local environment testing, enforcing contracts, breaking changes, logical refactoring a lot of it falls apart.
There's a lot of pressure for those guys to deliver these sort of highly complex systems and all the DevOps that surround it when they enter a new company or project. Rarely is the project scope or team size considered and huge amounts of time are wasted implementing the bones since they skip the whole monolith stage.
Precious few companies are at the size where they can keep an entire Dev team working on a shopping cart component in perpetuity. For most it's just something that gets worked on for a sprint or two.
AWS made a fundamental assumption in this that monoliths and big balls of mud are the same thing. Monoliths can and should be architected with internal logical separation of concerns and loose coupling. Domain driven design helps achieve that. The other assumption that microservices is a fix to this isn't, because the same poor design can be written but now with a network in between everything.
I think this advice is worth a read, but it's not law. The most pragmatic approach to microservices is to start with a monolith and shave off the intensive parts to separate services as you need to.