How big should a microservice be? (2013)(bovon.org)
bovon.org
How big should a microservice be? (2013)
https://bovon.org/archives/350
47 comments
Avoid microservices entirely and you won't adopt 50 unnecessary cloud services and external SaaS platforms to help you manage those 50 services. If each business unit/domain has their own "monolith" it serves the same goals without the hype and complexity.
But then 10 years down the line you'll be scrambling to implement Master Data Management, where you'll try (unsuccessfully) to merge user profiles and data.
I'm very anti-microservices, and rather than implement them for organizational reasons (read: teams unable to work together), I think that the most important business objects need to be maintained at a single point. Managing user identity at a single point is a valid reasons for microservices; like Google does with gmail/search/drive/storage/cloud services/youtube/android/etc.
I'm very anti-microservices, and rather than implement them for organizational reasons (read: teams unable to work together), I think that the most important business objects need to be maintained at a single point. Managing user identity at a single point is a valid reasons for microservices; like Google does with gmail/search/drive/storage/cloud services/youtube/android/etc.
> where you'll try (unsuccessfully) to merge user profiles and data.
Why? Can't a monolith be refactored?
Why? Can't a monolith be refactored?
I was talking about each silo of data; which is orthogonal to each monolith's code being refactorable.
> If each business unit/domain has their own "monolith"
This furiously looks like accidental microserviced data; which is kind of ironic considering the original comment was about "Avoid microservices entirely".
And yes, important business objects managed by different microservices are a PITA to merge; basically you're going to ask the users themselves to merge profiles, and they'll answer "we're divorced now" / "I like it better having multiple accounts, I get more promo codes" / "Why should I?" / "I don't care" / "Who are you?".
> If each business unit/domain has their own "monolith"
This furiously looks like accidental microserviced data; which is kind of ironic considering the original comment was about "Avoid microservices entirely".
And yes, important business objects managed by different microservices are a PITA to merge; basically you're going to ask the users themselves to merge profiles, and they'll answer "we're divorced now" / "I like it better having multiple accounts, I get more promo codes" / "Why should I?" / "I don't care" / "Who are you?".
Is that a microservice though? That sounds more like a proper service.
Write microservices, but link them in one binary and do normal procedure calls. The important part about microservices are the clear interfaces, the lack of global state blobs, and the independent testability, not the 50 cloud services needed to manage containers.
One advantage of real microservices is that you get better tooling around rate-limiting HTTP APIs than for simple function calls. And it's much easier to see which services creates the most CPU/Database/Network load.
In this context, the measuring aspect would be taken care by profiling/instrumenting.
I guess this is obvious but for the record: once one company using hot_tech_x gets significant funding, several more do. Then you have a bunch of ppl invested (literally) in hot_tech_x's success. hot_tech_x is then the answer to every problem. clogged sink? microservice.
Maybe a question is: what is today's hot tech?
Do we even care as long as we get paid?
Maybe a question is: what is today's hot tech?
Do we even care as long as we get paid?
> Maybe a question is: what is today's hot tech?
React, Kubernetes, Docker, GraphQL, Elixir
You're absolutely right. So few companies are solving actual problems anymore. It's all a cyclical bullshit hype cycle with very little innovation. Once the hot tech company IPOs, VC's force feed the next wave of bullshit tech/ideas.
I hope Fowler is getting compensated well by the cloud companies for his misinterpreted advice.
React, Kubernetes, Docker, GraphQL, Elixir
You're absolutely right. So few companies are solving actual problems anymore. It's all a cyclical bullshit hype cycle with very little innovation. Once the hot tech company IPOs, VC's force feed the next wave of bullshit tech/ideas.
I hope Fowler is getting compensated well by the cloud companies for his misinterpreted advice.
I mean, if that's feasible, sure. The problem is there's plenty of businesses that simply can't or won't justify the work to make a couple monoliths. Often there's dozens of microservices because there's dozens of acquisitions that nobody is willing to invest any money in; the microservices are hipster business glue.
If they were run by smart people they'd realize that the cost of maintenance for dozens of new microservices to wrap around a bunch of shitty legacy acquisitions is far more expensive than just refactoring the legacy apps (or tearing out chunks and shoving them in other products, or making brand new products out of one or two features). But try telling some C-level bozos from an industry that knows nothing about tech that.
If they were run by smart people they'd realize that the cost of maintenance for dozens of new microservices to wrap around a bunch of shitty legacy acquisitions is far more expensive than just refactoring the legacy apps (or tearing out chunks and shoving them in other products, or making brand new products out of one or two features). But try telling some C-level bozos from an industry that knows nothing about tech that.
The monoliths you acquire through acquisitions aren't really microservices :)
I don't really care about service size. I care about service division across logical boundaries, namely encapsulated responsibilities. A properly architected service should be responsible for a particular domain, say payments, messaging, image processing, etc.
It's up to each team to decide where to draw those lines. I think that for most teams, domain-specific services are superior to task-specific microservices, since it strikes the best balance between the benefits of separate services and the infrastructural overhead required to run a distributed system.
That said I agree that you should start with a monolith.
It's up to each team to decide where to draw those lines. I think that for most teams, domain-specific services are superior to task-specific microservices, since it strikes the best balance between the benefits of separate services and the infrastructural overhead required to run a distributed system.
That said I agree that you should start with a monolith.
> domain-specific services are superior to task-specific microservices
Can you elaborate on that? Isn't "messaging" or "image processing" a task?
Can you elaborate on that? Isn't "messaging" or "image processing" a task?
If you go all in on micro-services, image processing may have an ingress service and a scaling service, and a conversion service, and, and, and...
There is such a thing as irreducible complexity, no matter how you implement it, the solution will always have at least that complexity. This means that in a non-trivial system there will be some services that definitely won't fit in your head.
The safest way to build with micro services is still to start with a monolith. When you find a good reason like performance you can split of a micro service.
The safest way to build with micro services is still to start with a monolith. When you find a good reason like performance you can split of a micro service.
Also, if you can't build a maintainable monolith, there's a good chance you won't build a good distributed system.
Certainly, especially given that building a distributed system already is a lot more difficult than building non-distributed one. I have the feeling that most people really underestimate this.
> (2013) (Thoughtworks)
Microservices has certainly proved to be a lucrative trend for consultancies like thoughtworks. I've seen countless projects where a consultancy has been brought in replace some existing monolithic application with microservices. It sure creates a lot of billable work, but I can't say that I've seen much value in the business outcomes.
Microservices has certainly proved to be a lucrative trend for consultancies like thoughtworks. I've seen countless projects where a consultancy has been brought in replace some existing monolithic application with microservices. It sure creates a lot of billable work, but I can't say that I've seen much value in the business outcomes.
I heard of one group that had a consultancy replace their monolith with set of microservices + enterprise service bus - all communicating with a single database on the back end.
The first job of the development team every morning was to review the error logs for locks and timeouts and retrigger failed messages on the highly scalable message queue.
The first job of the development team every morning was to review the error logs for locks and timeouts and retrigger failed messages on the highly scalable message queue.
One thing I ve been painfully working on is upgrades as well. We have, in our bank, 2 system, one to trade with exchange automatically, one to approve short sells based on various inventories.
Both are built as micro services, with various robots talking to each other, optimized at the sub millisecond response time. It works.
But the difference is one has a release/repository organization based on each individual service, the other is deployed as a monolith with all robots all redeployed.
The system with split releases is barely maintanable, bugs appear in one version of a shared dependencies and we cant upgrade all robots without a mind blowingly complex procedure, regression tests etc. It s sometimes impossible and bugs are worked around up or downstream.
The monolithically deployed system is a breeze, we can fix an issue in a dependency and all robots benefit the next week.
There's no difference technically except in the minds and cultures of the team that in one system it s too hard to deploy everything and in the other it s mass redeployed every week.
Both are built as micro services, with various robots talking to each other, optimized at the sub millisecond response time. It works.
But the difference is one has a release/repository organization based on each individual service, the other is deployed as a monolith with all robots all redeployed.
The system with split releases is barely maintanable, bugs appear in one version of a shared dependencies and we cant upgrade all robots without a mind blowingly complex procedure, regression tests etc. It s sometimes impossible and bugs are worked around up or downstream.
The monolithically deployed system is a breeze, we can fix an issue in a dependency and all robots benefit the next week.
There's no difference technically except in the minds and cultures of the team that in one system it s too hard to deploy everything and in the other it s mass redeployed every week.
> all communicating with a single database on the back end
Those aren't exactly what I'd call a microservice then... that's a monolith with tons of type-unsafe code and unnecessary infrastructure glue.
Those aren't exactly what I'd call a microservice then... that's a monolith with tons of type-unsafe code and unnecessary infrastructure glue.
This anti-pattern is called a “distributed monolith”:
https://www.gremlin.com/blog/is-your-microservice-a-distribu...
https://www.gremlin.com/blog/is-your-microservice-a-distribu...
Funny you say that... I'm a PM and inherited a software product that was initially started by a Thoughtworks team. The software was based on microservices from day 1. The company c suite was sold into an MS architecture and gushed about how good it was. So much unnecessary complexity for an early product. What a nightmare.
Distributed monolith[0], much? :)
https://newrelic.com/blog/best-practices/distributed-monolit...
https://newrelic.com/blog/best-practices/distributed-monolit...
My answer would be: as big as can be maintained by a "two pizza" team, a group that can have a daily standup etc..
Conway's law exists for a reason. A highly cohesive group of people can effectively maintain a highly coupled codebase. When your communication channels with another group are lower-bandwidth, your code-level coupling needs to be lower to reflect tha.
Conway's law exists for a reason. A highly cohesive group of people can effectively maintain a highly coupled codebase. When your communication channels with another group are lower-bandwidth, your code-level coupling needs to be lower to reflect tha.
Strongly agree. When functionality is in the same service, you can have logs with a reasonable sequence, a shared clock, stack traces, cache consistency, fewer failure points, one commit atomically updates both ends of an interface, etc. Since modularity is lower overhead, you encourage breaking things into smaller pieces with libraries. You can have softer API contracts which enable deleting old code and faster iteration.
Turning all software into a distributed system and solving the above problems isn't free, and your productivity is better spent elsewhere unless there's a payoff.
Smaller units of deployment, being isolated from implementation changes, etc. are all valuable, but more so across teams than within a team.
Turning all software into a distributed system and solving the above problems isn't free, and your productivity is better spent elsewhere unless there's a payoff.
Smaller units of deployment, being isolated from implementation changes, etc. are all valuable, but more so across teams than within a team.
What's that, 4 people? If something 'micro' needs 4 people to maintain it then it probably isn't 'micro' after all.
Yeah, I agree with that. People take things like the Amazon platform memo (which was absolutely about having the whole work of a team be presented as one or a handful of services) and take that as a reason to have more services than developers, which is pretty backwards IMO.
I like the idea that services should mimic the people organization that maintain them[0].
Obtuse rule - Don't build a service smaller than a team. Probably 2 teams.
[0] http://haacked.com/archive/2013/05/13/applying-conways-law.a...
Obtuse rule - Don't build a service smaller than a team. Probably 2 teams.
[0] http://haacked.com/archive/2013/05/13/applying-conways-law.a...
Services should not mimic the people organization, because the people organization sucks. Don't care how they're organized. No-hierarchy? Sucks. Top-down? Sucks. Balls of people with guns pointing at the other balls of people? Sucks. Two-pizza teams? How fat/skinny are the team members?
Humans are dumb, they're bad at communication, and the bigger the group, the worse both get (it's done been scienced!). Software should not mimic those qualities; and it doesn't have to! We can have really smart people design the software, and have those people tell the organization how to build it and run it in a collaborative way, regardless of how the people are organized. Organization shouldn't dictate how we work together; that's like your filing system dictating how you write letters. What we always seem to forget is that humans are pretty much the most adaptable species on the planet. Give them criteria and they will fit themselves into it, like a cat in a really uncomfortable looking container.
We should of course avoid shitty people organization because of its impact on software design (which is really just poor communication structures put into code). But we should also acknowledge that the design is only impacted by the organization because we're too lazy to reject the concept of "team X can only touch component X". And this isn't just woo-non-hierarchy, there's a simple middle ground: education. Teach your people how all the components work, teach them how to make changes, and unblock them / give them agency. They will flutter around like worker bees just knocking out problems everywhere.
It's when people don't know what's going on with the components they never touch, and don't know how to change them, that they start to make bad decisions that then have knock-on effects. The org structure doesn't have to lead to people not knowing what's going on. It doesn't have to lead to poor communication. It doesn't have to lead to controlling who can do what. We just decided that's what we'd do, probably because standing up and saying "hey, can we do this differently?" causes some kind of social pain. That and of course, the more it changes, the more education is needed. But kind of like with TPS, I think if we focus more on those "boring things" like quality and waste, everything else will be naturally elevated.
Humans are dumb, they're bad at communication, and the bigger the group, the worse both get (it's done been scienced!). Software should not mimic those qualities; and it doesn't have to! We can have really smart people design the software, and have those people tell the organization how to build it and run it in a collaborative way, regardless of how the people are organized. Organization shouldn't dictate how we work together; that's like your filing system dictating how you write letters. What we always seem to forget is that humans are pretty much the most adaptable species on the planet. Give them criteria and they will fit themselves into it, like a cat in a really uncomfortable looking container.
We should of course avoid shitty people organization because of its impact on software design (which is really just poor communication structures put into code). But we should also acknowledge that the design is only impacted by the organization because we're too lazy to reject the concept of "team X can only touch component X". And this isn't just woo-non-hierarchy, there's a simple middle ground: education. Teach your people how all the components work, teach them how to make changes, and unblock them / give them agency. They will flutter around like worker bees just knocking out problems everywhere.
It's when people don't know what's going on with the components they never touch, and don't know how to change them, that they start to make bad decisions that then have knock-on effects. The org structure doesn't have to lead to people not knowing what's going on. It doesn't have to lead to poor communication. It doesn't have to lead to controlling who can do what. We just decided that's what we'd do, probably because standing up and saying "hey, can we do this differently?" causes some kind of social pain. That and of course, the more it changes, the more education is needed. But kind of like with TPS, I think if we focus more on those "boring things" like quality and waste, everything else will be naturally elevated.
Couldn't a team maintain some number of small services? I do see the team size issue with a service that's too big.
It's easy to build and easy to maintain each service individually (most attractive selling points of services).
Where it gets expensive fast is in multi-service issue debugging. It's not as simple as putting a break point and inspecting a bunch of variables.
Specs also can become unwieldy. Does your test suite run all the services it needs to run? Does it connect to a test env?
Everything is a trade off of course, but an obtuse/simple recommendation is the larger the service is the easier issues are to 'figure out'.
If you've ever been part of an organization that drank the microservices koolaid and then had insane time trying to debug issues that spanned 4-5 services, you'll know the pain. It's not always super obvious which service is manipulating the data such that downstream it breaks.
Where it gets expensive fast is in multi-service issue debugging. It's not as simple as putting a break point and inspecting a bunch of variables.
Specs also can become unwieldy. Does your test suite run all the services it needs to run? Does it connect to a test env?
Everything is a trade off of course, but an obtuse/simple recommendation is the larger the service is the easier issues are to 'figure out'.
If you've ever been part of an organization that drank the microservices koolaid and then had insane time trying to debug issues that spanned 4-5 services, you'll know the pain. It's not always super obvious which service is manipulating the data such that downstream it breaks.
Imagine other industries did what the software industry does, so we had articles like "how big should transport vehicles be". So we decided a sedan car is basically the right size, and proceeded to call trucks, trains, airplanes, ships, tankers, motorcycles, bikes and scooters as being the wrong size.
That's how stupid this entire way of thinking is.
It's attempting to take a problem that requires a ton of domain-specific considerations, and dumb it down to something that everyone can read and understand with toddler-level experience and feel smart.
That's how stupid this entire way of thinking is.
It's attempting to take a problem that requires a ton of domain-specific considerations, and dumb it down to something that everyone can read and understand with toddler-level experience and feel smart.
This irritates me. People keep forgetting about costs of maintaining services.
What is a cost of maintaining a service?
What is a cost of maintaining equivalent amount of code as a separate module in a larger service?
The reality is that plopping a piece of code to its own separate module and then maintain communication within the service (ie. call another class) is usually much easier and cheaper than creating new services.
I get that creating a new service might be fun, but if you compare costs of maintaining a thousand services vs cost of maintaining of single application with a thousand modules it is not given that the services are going to be cheaper.
I think there is no rule to decide how big a service should be. How big a module should be?
Rather, services should be split when there is some essential reason to do so. For example, the functionalities might have some intrinsic differences that make it costly to reconcile within a single service. Or you might have a functionality that does not scale linearly and so it might make sense, from performance point of view, to maintain less nodes of that service but fully devoted to executing particular functionality.
Another good reason might be if you have separate teams maintaining the functionality -- separate services might allow the teams to work more independently, make independent technology decisions, etc.
Splitting services just because code size of single service will be smaller is stupid. The total code size is the same -- each part of functionality is still somewhere. Actually, you need more code just for the overhead of having separate services.
If hiding code is your aim, just put it in another folder and squint your eyes so you don't see it. See? How easy?
If defining good APIs is your aim, just do good job defining modules within single service. Do you really need overhead of separate service for that?
So, the good general rule, as with practically anything in life, would be: Split services if you see gains that are bigger than costs and risks.
Actually, much bigger, because your attention is also finite resource. If you spend 50% of your time talking about creating services you are not spending that time focusing on other important stuff.
But don't just split services because they are bigger than your head.
This just irritates me.
What is a cost of maintaining a service?
What is a cost of maintaining equivalent amount of code as a separate module in a larger service?
The reality is that plopping a piece of code to its own separate module and then maintain communication within the service (ie. call another class) is usually much easier and cheaper than creating new services.
I get that creating a new service might be fun, but if you compare costs of maintaining a thousand services vs cost of maintaining of single application with a thousand modules it is not given that the services are going to be cheaper.
I think there is no rule to decide how big a service should be. How big a module should be?
Rather, services should be split when there is some essential reason to do so. For example, the functionalities might have some intrinsic differences that make it costly to reconcile within a single service. Or you might have a functionality that does not scale linearly and so it might make sense, from performance point of view, to maintain less nodes of that service but fully devoted to executing particular functionality.
Another good reason might be if you have separate teams maintaining the functionality -- separate services might allow the teams to work more independently, make independent technology decisions, etc.
Splitting services just because code size of single service will be smaller is stupid. The total code size is the same -- each part of functionality is still somewhere. Actually, you need more code just for the overhead of having separate services.
If hiding code is your aim, just put it in another folder and squint your eyes so you don't see it. See? How easy?
If defining good APIs is your aim, just do good job defining modules within single service. Do you really need overhead of separate service for that?
So, the good general rule, as with practically anything in life, would be: Split services if you see gains that are bigger than costs and risks.
Actually, much bigger, because your attention is also finite resource. If you spend 50% of your time talking about creating services you are not spending that time focusing on other important stuff.
But don't just split services because they are bigger than your head.
This just irritates me.
Microservices are good when one programming language is better at solving something than another. Eg. Python is what data scientists use for machine learning, but I'm not going to build the entire backend in Python just because of that. C# is great for gluing things together and integrating with enterprise systems due to good support for wsdl, and the type system and tooling is excellent, as well as the pool of developers proficient in the language. Postgres/SQL is excellent for business logic, and can act as a kind of monolith. Other languages such as Go are great for a custom reverse proxy when Nginx is insufficient. And having the services seperated means one can crash without affecting the others (Erlang got this right, but I wouldn't build 100% of a system in that because of the reasons just mentioned). I like how microservices allow me to use the right tool for the job.
To me this question is exactly the same as “how big should a program be?” Isn’t a microservice just a program that provides a network interface using standard protocols?
Fortunately we are all engineers and can apply well established methods to answer that question. Well, we can all dream…
Fortunately we are all engineers and can apply well established methods to answer that question. Well, we can all dream…
I think that's the wrong question to ask. There are many instances where it may make sense to have a large microservice just as there are instances where it makes sense to have a 500 line function. This is a territory where there is no right or wrong. The problem with pegging metrics like this is you run risk of coming up against Goodhart's law, where the metric becomes an end in itself. Rather, it is more interesting to think about what problems you are facing in a given situation and patterns of addressing them. Bloat is definitely a signal that may point to something, but if you are not facing a tangible problem to start with, it most certainly does not matter.
[deleted]
This doesn’t need a treatise. You know it when you see it, and if multiple people on your team disagree, for whatever reason, it’s definitely not a microservice.
[deleted]