Build versus Buy(lethain.com)
lethain.com
Build versus Buy
https://lethain.com/build-vs-buy/
95 comments
> Their profit margin is usually in their efficiency in doing it better than you
This is a very nice way of framing the question.
I think there may be at least two scenarios to consider. For some products, cost is primarily driven by R&D; for others, cost of operating the service (COGS) dominates. Of course it's a continuum, but I suspect there are more services near the ends than the middle.
For an R&D-dominated service, if your needs are nontrivial and there is an off-the-shelf product that is a decent fit, you probably want to go off-the-shelf. A commercial provider can amortize the R&D over a large customer base.
For a COGS-dominated service, the commercial provider may still be more efficient but it is less of a slam dunk. Log management (e.g. Splunk) involves a substantial operational component, because you're dealing with large volumes of data.
Disclaimer: I am the founder of Scalyr, a Splunk competitor. Early on we realized that, even though log management is COGS-heavy, we could achieve huge economies of scale, because query workloads are highly bursty. By carefully managing a central pool of resources shared by many clients, we're able to run much more efficiently than a single-tenant homegrown solution [1]. This is one of the key points that we highlight when having build vs. buy discussions with potential customers.
[1] https://www.scalyr.com/blog/searching-1tb-sec-systems-engine...
This is a very nice way of framing the question.
I think there may be at least two scenarios to consider. For some products, cost is primarily driven by R&D; for others, cost of operating the service (COGS) dominates. Of course it's a continuum, but I suspect there are more services near the ends than the middle.
For an R&D-dominated service, if your needs are nontrivial and there is an off-the-shelf product that is a decent fit, you probably want to go off-the-shelf. A commercial provider can amortize the R&D over a large customer base.
For a COGS-dominated service, the commercial provider may still be more efficient but it is less of a slam dunk. Log management (e.g. Splunk) involves a substantial operational component, because you're dealing with large volumes of data.
Disclaimer: I am the founder of Scalyr, a Splunk competitor. Early on we realized that, even though log management is COGS-heavy, we could achieve huge economies of scale, because query workloads are highly bursty. By carefully managing a central pool of resources shared by many clients, we're able to run much more efficiently than a single-tenant homegrown solution [1]. This is one of the key points that we highlight when having build vs. buy discussions with potential customers.
[1] https://www.scalyr.com/blog/searching-1tb-sec-systems-engine...
Hi snewman,
I am currently using logdna as my splunk alternative, but it's not very good.
Why should I try scalyr next, over the other alternatives ?
My main constraint is the solution should be at least half as good as splunk, and cost less than $50 a month (I generate less than 1G a day).
Your pricing seems to be reasonable. But how do you compare to splunk on features ?
Here are the other tools I am considering trying
https://www.humio.com/
https://www.sumologic.com/
https://sematext.com/logsene/
https://www.datadoghq.com/log-management/
https://logz.io/pricing/
https://www.chaossearch.io/
https://www.papertrail.com/
https://www.loggly.com/
https://www.graylog.org/products/enterprise
I am currently using logdna as my splunk alternative, but it's not very good.
Why should I try scalyr next, over the other alternatives ?
My main constraint is the solution should be at least half as good as splunk, and cost less than $50 a month (I generate less than 1G a day).
Your pricing seems to be reasonable. But how do you compare to splunk on features ?
Here are the other tools I am considering trying
https://www.humio.com/
https://www.sumologic.com/
https://sematext.com/logsene/
https://www.datadoghq.com/log-management/
https://logz.io/pricing/
https://www.chaossearch.io/
https://www.papertrail.com/
https://www.loggly.com/
https://www.graylog.org/products/enterprise
Thanks for reaching out. It really depends on what you're looking for, and I don't want to hijack this thread with a long discussion, so feel free to reach out at https://scalyr.com/contact. If you're focused on the power of the query language, then Splunk is the undisputed leader, but we are relatively strong as well; see https://app.scalyr.com/help/power-queries. This iteration of our query language was launched last year and was a lot of fun to build -- the team allowed me to write a fair amount of the code. :-)
You can find a more general overview at https://scalyr.com/product.
You can find a more general overview at https://scalyr.com/product.
Mostly looking for decent query language, decent charting, and API to grab chart data.
Seems like you guys fit the bill. I'll do the 30 day trial, and see how it goes. Thanks for the response.
Seems like you guys fit the bill. I'll do the 30 day trial, and see how it goes. Thanks for the response.
I was an early Scalyr user circa 2014 or so, and have used Splunk at various gigs before and after. My current company is not a Scalyr customer and that makes me pretty sad - even back then, they were snappy and responsive in ways that Splunk has never matched. Despite (maybe due to?) being substantially cheaper, the service we received from the founders and operators was second to none; for example, I remember that someone even hand-wrote a Clojure EDN log-parser for us at no extra charge, just to make sure we were happy.
Steve says elsewhere in the thread that Splunk has the more-flexible query language, and while that may be true I as a product engineer never noticed the delta myself.
TL;DR I think you'd be doing yourself a favor to try Scalyr, and I have no affiliation other than being a happy former customer.
Steve says elsewhere in the thread that Splunk has the more-flexible query language, and while that may be true I as a product engineer never noticed the delta myself.
TL;DR I think you'd be doing yourself a favor to try Scalyr, and I have no affiliation other than being a happy former customer.
Thanks. This is helpful information.
Hey snisarenko, you could use the Gravwell Community Edition which is free up to 2GB/day. 4GB/day if you participate in the current alpha testing program. https://www.gravwell.io/download
Disclaimer: I'm one of the founders. We built it to be a Splunk alternative and I think it does a great job from a data ingest, scalability, and data querying perspective. We're lacking some of the out-of-the-box capability, but not for long. Kits (our "apps") release is coming this quarter.
Disclaimer: I'm one of the founders. We built it to be a Splunk alternative and I think it does a great job from a data ingest, scalability, and data querying perspective. We're lacking some of the out-of-the-box capability, but not for long. Kits (our "apps") release is coming this quarter.
Seeing how an enterprise software chooses to articulate and differentiate itself in product and sales from the internal perspective is super interesting.
Are there other examples of "we chose to go to market like this (x, y, z reasons) and here's why"?
Are there other examples of "we chose to go to market like this (x, y, z reasons) and here's why"?
> Except in the case of Splunk. Their pricing is ridiculous.
Their pricing is ridiculous and what is even stranger is that with the ELK stack there is a good and free alternative. Now Splunk is good, no doubt, but I still wonder how they can be that successful with that pricing.
Their pricing is ridiculous and what is even stranger is that with the ELK stack there is a good and free alternative. Now Splunk is good, no doubt, but I still wonder how they can be that successful with that pricing.
We ran the largest elastic search cluster on the west coast some number of years ago. If you are small enough, elastic search can go a long way. Our data teams and operations folks celebrated the day that that system was turned off. Elastic search does not even hold a candle to the way we leverage splunk at our org, but that could be because we are bigger than some and deal with scale that few others deal with. Splunk costs us a fortune but enables amazing data analysis. It is easily our most expensive service we pay for and worth it. It would be great for our bottom line if it were cheaper, but they can charge what they do because the do it so well.
Can you give us a sense of scale and what issues you faced by elastic?
Splunk is incredibly powerful though, and almost all of that power is available at query time. Replicating the functionality in ELK often means indexing changes, and so when you have a question that isn't answered by the index, you'll forgo the answer unless you really really need it. A very simple example is the 'transaction' command in Splunk, which I absolutely could not live without and often surprise myself with the keys I end up using to research a particular topic.
Lock in and one trick pony trained staff. I'm going to kill it in the next 12 months in our org if it doesn't kill me first.
Splunk is insanely good. To the point that people forget that other tools exist, or that it makes sense to build specialized solutions when needed.
So people start running anything — from dashboards to analytics — from Splunk alone. Their ability to combine and extract data from almost anything in very small amounts of time is unparalleled IMO.
So people start running anything — from dashboards to analytics — from Splunk alone. Their ability to combine and extract data from almost anything in very small amounts of time is unparalleled IMO.
I think people are easy to underestimate the cost of building and maintaining a new piece of software. I mean yeah you can set up an ELK stack (just as an example) on your own hardware / cloud easily enough, but what will it cost in the long run? How many people will spend how much time on it? Or how many people will end up having to spread their attention between the core objective of your business and managing the ELK stack and other services you decided to build / operate yourself?
> My preference in these situations is always to start with buy, and then try to justify build
What if there are significant costs associated with that? Integration, training, friction to move to a new solution later, etc?
When things are commoditized it works OK but for anything complex and strategic, the cost of implementing a potentially bad solution can be absolutely crippling, especially if dependencies on that solution grow rapidly within the organisation.
What if there are significant costs associated with that? Integration, training, friction to move to a new solution later, etc?
When things are commoditized it works OK but for anything complex and strategic, the cost of implementing a potentially bad solution can be absolutely crippling, especially if dependencies on that solution grow rapidly within the organisation.
Then those are all great justifications in the build column!
And Splunk is horrendously slow in my experience.
We self host splunk and it can plow through petabytes of high cardinality data pretty dang fast. If the fields are not indexed and the search is complex, it can take minutes or hours. But usually, I can get live and historic data in a few seconds.
As an example, we have a pipeline of services. I can compute the time spent in each service with multiple levels of percentiles and group that data by high cardinality fields (as in, hundreds of thousands or more values). I just did a search for 4 hours of data across thousands of nodes for half a dozen or so services with multiple eval statements all piped to a timechart doing over a dozen stats operations. Half a billion events. It got done in under a minute.
Splunk charges so much because they are just so dang powerful.
As an example, we have a pipeline of services. I can compute the time spent in each service with multiple levels of percentiles and group that data by high cardinality fields (as in, hundreds of thousands or more values). I just did a search for 4 hours of data across thousands of nodes for half a dozen or so services with multiple eval statements all piped to a timechart doing over a dozen stats operations. Half a billion events. It got done in under a minute.
Splunk charges so much because they are just so dang powerful.
This has not been true in my experience. I run a Splunk server in production and at my data volumes it has been very performant. It's also much easier to setup and maintain than ELK clusters.
In the early days Splunk pricing was exorbitant (we evaluated Splunk 7 years ago and dismissed it), but licensing has changed in recent years and it is now priced by volume ingested (the pricing is transparent and listed on their website now). At low volumes, the pricing is similar to Sumologic, and is pretty accessible now to smaller dev shops. Open-source collectors like fluentd also help to intelligently reduce the ingest volume.
At high volumes, the TCO changes quite a bit.
In the early days Splunk pricing was exorbitant (we evaluated Splunk 7 years ago and dismissed it), but licensing has changed in recent years and it is now priced by volume ingested (the pricing is transparent and listed on their website now). At low volumes, the pricing is similar to Sumologic, and is pretty accessible now to smaller dev shops. Open-source collectors like fluentd also help to intelligently reduce the ingest volume.
At high volumes, the TCO changes quite a bit.
My experience is solely at work where we use Splunk Cloud and it's slow as molasses.
Does the speed matter? How exactly?
I am genuinely curious: at Scalyr we _can_ be very fast but it is a balance with the cost that we want to pass on as price savings. Same with self-hosted Elastic: one can fine-tune it to be fast but minding the cost constraints gets it slower. WDYT?
Same experience. Also splunk forwarders appear to be universally unreliable.
we have extra monitoring for splunk forwarders, and even then we fail to notice them fail from time to time. Sigh.....
Yes same here. I actually monitor the throughput of the network interfaces on our forwarder with prometheus/statsd_exporter and if outbound is smaller than inbound it sets off alerts!
Ours is self-hosted and is plenty fast. This might be something to bring up with the Splunk folks. Maybe their cloud side of things needs tuning.
It's Pareto principle for me. >80% of our code should be off the shelf. The special sauce is 10% of the code. What should we spend that on? What will make us stand out?
If you can answer that, it's all the justification you need for building it. Anything you build that doesn't differentiate you can be bought by a competitor.
If you can answer that, it's all the justification you need for building it. Anything you build that doesn't differentiate you can be bought by a competitor.
As usual, it depends. I know some friends that have profitable websites because they run on a private server and they handcrafted everything. If they had to use cloud services for serving stuff, host the db, provide search and cache things, it would cost too much and they couldn't make a living out of it.
To me, this shows that the actual service/content they're providing has relatively low value, and so they're "making" money not just on the content but also on the services side; eg, they're a cheaper sysadmin than an outsourced one. And they're also possibly ignoring risk; eg, they don't have the data durability that a cloud provider does, but it Probably Won't Matter (TM).
There's absolutely nothing wrong with this way of making money; the lower the margins, the more watching your costs is crucially important, and the difference between making a living and not even breaking even!
There's absolutely nothing wrong with this way of making money; the lower the margins, the more watching your costs is crucially important, and the difference between making a living and not even breaking even!
Possibly, it may also be that the cloud providers are offering things they don't need. Maybe that's uptime guarantees, potential scalability or integration with other services.
Sure.
I just mention it because most people in the world have a job that make them less money than my friends web services.
And I'm assuming most startups, unless heavily founded, may have to think about this trade off as well.
The cost in infra can easily be a factor of ten.
I just mention it because most people in the world have a job that make them less money than my friends web services.
And I'm assuming most startups, unless heavily founded, may have to think about this trade off as well.
The cost in infra can easily be a factor of ten.
If you take SaaS, they are skewed towards buy, because their margin is ridiculously high. Better to hit the market now by using stable products than optimizing on spend.
The bigger you are and the more you can think about optimizing spend.
The bigger you are and the more you can think about optimizing spend.
I think it is quite the opposite. Having worked in startups that were VC funded it seems they consider VC money free money and just spend a lot of it in infrastructure without the need to.
I know for a fact our bill was around 15-20k USD per month for running a webapp that could be ran in digital ocean 40 dollars/month.
For a lifestyle business without VC money to burn it makes all the difference.
I know for a fact our bill was around 15-20k USD per month for running a webapp that could be ran in digital ocean 40 dollars/month.
For a lifestyle business without VC money to burn it makes all the difference.
Oh I've definitely seen that too. My example requires responsible (and smart) leadership. My thought is that optimizing costs engineering hours, which are much more expensive than paying for the service bills.
I agree with the idea, but I also think there are limits.
Let's say you build some kind of product based on some Azure/AWS product. The negative reading of it is: you spend money on innovation and have to pay Azure/AWS for infrastructure. Your product will constantly have to improve to keep up with the competitors, Azure/AWS keeps getting your money. And if your product becomes stable enough, they will make it part of their offering.
Another example would be Apps, with Apple later folding their functionality into the OS and kicking your app off the store.
But judging by my own examples, this migth be more of a platform problem.
Let's say you build some kind of product based on some Azure/AWS product. The negative reading of it is: you spend money on innovation and have to pay Azure/AWS for infrastructure. Your product will constantly have to improve to keep up with the competitors, Azure/AWS keeps getting your money. And if your product becomes stable enough, they will make it part of their offering.
Another example would be Apps, with Apple later folding their functionality into the OS and kicking your app off the store.
But judging by my own examples, this migth be more of a platform problem.
Surely that very much depends on whether your product fits within the current or near future offering of Azure/AWS.
The other side of that coin is that you use a build horizontal integration strategy to replace elements of the products you buy from Azure/AWS if (and only if) it can be done in a way that increase your profit/value.
It can work both ways. Moving from buy to build also has elements of de-risking and economies of scale if you do it at the point where there is measurable success/product to market fit.
Building something you can buy too early runs the risk of premature optimisation if not done for the right reasons.
The other side of that coin is that you use a build horizontal integration strategy to replace elements of the products you buy from Azure/AWS if (and only if) it can be done in a way that increase your profit/value.
It can work both ways. Moving from buy to build also has elements of de-risking and economies of scale if you do it at the point where there is measurable success/product to market fit.
Building something you can buy too early runs the risk of premature optimisation if not done for the right reasons.
Absolutely. This is largely what I'm betting on with my latest project, that the time you spend spinning up all the "standard stuff" for a SaaS project is duplicated - and thus wasted - effort throughout the industry. Why write another user login system?
Though it sets up an interesting situation for me, where the "special sauce" of _my product_ is the 80% "off the shelf" for my customers.
Though it sets up an interesting situation for me, where the "special sauce" of _my product_ is the 80% "off the shelf" for my customers.
Another reason I like to start with buy: It makes scope creep a lot harder. You have a specific feature set, and it lets you do certain things, and not others. You may eventually find there are certain things it has that you can't live without, at which point you may still have to build, but that's offset by the fact that you'll probably also discover there are features you only thought you needed.
By contrast, whenever I've been on a project where the company went straight to building their own, it ended up being a quagmire of scope creep, and the development team would end up stuck on an endless treadmill of implementing features that stakeholders vehemently insist on, but never actually use.
Even for dev tooling and libraries. Some of the worst piles of technical debt I've encountered happened when some rockstar decided that they knew better than everyone else, and dived straight into building what they thought would be a better mouse trap. It's not that the in-house option isn't any more or less likely to solve the problem well (no comment), so much as that the in-house option is going to end up being more deeply tangled with the rest of the codebase. So you're stuck with it, because, even if it doesn't turn out to be all it's cracked up to be, migrating away from it may be next to impossible. Homegrown ORMs seem to always turn out this way.
The one exception is when you know that a quick-and-dirty DIY solution will be cheaper and easier, because it needs to do only one specific thing while the off-the-shelf option has to take on a bunch of extra complexity in order to be general enough for everyone's needs. But even there, I'm only likely to trust that argument if it's being made by someone who's been burned by the complexities of the off-the-shelf solution in the past, and who has a proven track record of obeying the KISS principle.
By contrast, whenever I've been on a project where the company went straight to building their own, it ended up being a quagmire of scope creep, and the development team would end up stuck on an endless treadmill of implementing features that stakeholders vehemently insist on, but never actually use.
Even for dev tooling and libraries. Some of the worst piles of technical debt I've encountered happened when some rockstar decided that they knew better than everyone else, and dived straight into building what they thought would be a better mouse trap. It's not that the in-house option isn't any more or less likely to solve the problem well (no comment), so much as that the in-house option is going to end up being more deeply tangled with the rest of the codebase. So you're stuck with it, because, even if it doesn't turn out to be all it's cracked up to be, migrating away from it may be next to impossible. Homegrown ORMs seem to always turn out this way.
The one exception is when you know that a quick-and-dirty DIY solution will be cheaper and easier, because it needs to do only one specific thing while the off-the-shelf option has to take on a bunch of extra complexity in order to be general enough for everyone's needs. But even there, I'm only likely to trust that argument if it's being made by someone who's been burned by the complexities of the off-the-shelf solution in the past, and who has a proven track record of obeying the KISS principle.
Modularity by purchase. Information hiding by ownership. Interfaces by corporation.
Brooks' software system products (9x the work) is usually done this way in practice. In most usage cases, I would also include open source projects.
It's so hard to get interfaces right and it's helpful to have many users. But I considered your point, of it helping to enforce boundaries.
Brooks' software system products (9x the work) is usually done this way in practice. In most usage cases, I would also include open source projects.
It's so hard to get interfaces right and it's helpful to have many users. But I considered your point, of it helping to enforce boundaries.
A story of Netflix adopting StackStorm for autoremediation is a nice illustration of this approach. They began to build a tool, learned enough to be dangerous, discovered StackStorm and used it, experimented with it like with a breadboard, until finally figured out exactly what works - and reimplemented it. All these time learning the system was working and delivering vale.
Part of this story is here https://netflixtechblog.com/introducing-winston-event-driven...
Disclamer: StackStorm founder here (but not pitching).
Part of this story is here https://netflixtechblog.com/introducing-winston-event-driven...
Disclamer: StackStorm founder here (but not pitching).
This is the core of the whole debate, in my experience:
"When it comes to build versus buy, the frequently repeated but rarely followed wisdom is good advice: if you're a technology company, vendors usually generate significant value if they're outside your company's core competency; within your core competency, they generally slow you down."
Every startup I've ever worked with has had engineers who advocated "build" for tooling that fell both outside the eng team's workflow and our company's core competencies.
I think part of it is that things like marketing automation or customer support software—things that are vital for most teams past a certain scale—don't seem technically "hard" enough to justify the price-tags, particularly to engineers who've been isolated from other parts of the business.
Without fail, every time I've been part of a team building an internal tool that we/another engineering team won't use, it goes poorly. I've spent way too long learning about the ins and outs of email deliverability so that I could fix a bug in a hacky email platform, when we could have just bought a platform that works.
"When it comes to build versus buy, the frequently repeated but rarely followed wisdom is good advice: if you're a technology company, vendors usually generate significant value if they're outside your company's core competency; within your core competency, they generally slow you down."
Every startup I've ever worked with has had engineers who advocated "build" for tooling that fell both outside the eng team's workflow and our company's core competencies.
I think part of it is that things like marketing automation or customer support software—things that are vital for most teams past a certain scale—don't seem technically "hard" enough to justify the price-tags, particularly to engineers who've been isolated from other parts of the business.
Without fail, every time I've been part of a team building an internal tool that we/another engineering team won't use, it goes poorly. I've spent way too long learning about the ins and outs of email deliverability so that I could fix a bug in a hacky email platform, when we could have just bought a platform that works.
Agreed - problem for me, as an engineer, is that I do like building those things that are outside my core competency so I can gain a wider variety of knowledge and experience.
Definitely not worth it from a business perspective, but has resulted in some interesting toy/hobby projects!
Definitely not worth it from a business perspective, but has resulted in some interesting toy/hobby projects!
> I think part of it is that things like marketing automation or customer support software, don't seem technically "hard" enough to justify the price-tags
As someone who's built far too many of those because "Buying is just too expensive" ... good god do I never want to do that ever again. Ever. It's so fucking hard. You will shoot yourself in the foot. You will suffer. And 3 years later you still won't have something half as good as the purpose-built SaaS.
If you do that and enjoy it, just spin it off into its own product. Who knows, you might even get more customers with more expensive problems than the original startup.
As someone who's built far too many of those because "Buying is just too expensive" ... good god do I never want to do that ever again. Ever. It's so fucking hard. You will shoot yourself in the foot. You will suffer. And 3 years later you still won't have something half as good as the purpose-built SaaS.
If you do that and enjoy it, just spin it off into its own product. Who knows, you might even get more customers with more expensive problems than the original startup.
This only makes sense if you have a significant equity stake in the company you're working for. For the vast majority of engineers, they really don't care about the long-term profitability/success of whatever company they're working for. They just want to get paid, and hopefully grow as a dev in the process.
They'll learn more by building something in-house than by bringing in a saas and just writing glue for a living. If you want your engineers to make the decisions a founder would make, give them a significant equity stake.
They'll learn more by building something in-house than by bringing in a saas and just writing glue for a living. If you want your engineers to make the decisions a founder would make, give them a significant equity stake.
It depends. Significant stake helps of course and despite that, I'd prefer solving problems that need solving than spending a bunch of time on problems that don't :)
There's a balance.
There's a balance.
“If it’s a core business function — do it yourself, no matter what.”
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
I've worked at Not-Invented-Here shops and it was annoying that the Powers That Be didn't want to use a COTS solution. As much fun as it would be to reinvent DynamoDB, that's not actually going to help our business.
However, even more annoying is working at a Never-Invented-Here organization. I've been at a place which hired brilliant people, but then didn't let them innovate on pretty much anything. Write a simple TCP listener in Python to read a stream of data and then reply to it? That's crazy talk! We need to write a full Django app and deploy it behind Nginx behind a load balancer so that we can pretend it's highly available!
Don't get me wrong: the off the shelf approach is usually the right thing, but sometimes it just adds complication where it didn't need to be. It's insulting to be told you were hired for your expertise, but then not be allowed to actually use it.
However, even more annoying is working at a Never-Invented-Here organization. I've been at a place which hired brilliant people, but then didn't let them innovate on pretty much anything. Write a simple TCP listener in Python to read a stream of data and then reply to it? That's crazy talk! We need to write a full Django app and deploy it behind Nginx behind a load balancer so that we can pretend it's highly available!
Don't get me wrong: the off the shelf approach is usually the right thing, but sometimes it just adds complication where it didn't need to be. It's insulting to be told you were hired for your expertise, but then not be allowed to actually use it.
It’s shifting the complexity from development to operations ( or “gluing shit” as a sibling has pithily observed.
I’ve found this to be common in places with lower engineering skills. The “engineers” are mortally afraid of building anything novel for fear of failure; the COTS approach can get many clever folks pretty far in their career.
I’ve found this to be common in places with lower engineering skills. The “engineers” are mortally afraid of building anything novel for fear of failure; the COTS approach can get many clever folks pretty far in their career.
NIH and concrete over generic solutions all the way, before you end up in the soulless hamster wheel of constantly upgrading 3rd party software.
“Write a simple TCP listener in Python to read a stream of data and then reply to it? That's crazy talk!”
This has infected my company quite a bit. People get really nervous as soon as you write something that’s little novel. Writing everything from scratch is bad but you also need room for people to innovate.
This has infected my company quite a bit. People get really nervous as soon as you write something that’s little novel. Writing everything from scratch is bad but you also need room for people to innovate.
Except encryption. You don't want to roll your own at all.
I think the advice holds. Cryptographic primitives are vanishingly unlikely to be your core business, even if the things built on them are. Therefore, almost nobody should be building them.
Super important exception. Thanks for mentioning it!
Unless you've spent the last 10+ years working on encryption full time, please don't try it yourself.
I used to think I could write crypto. Then I learned more and realized it needs specialized knowledge I don't have the time to acquire. Then I spent some time trying to acquire it and figured I was OK to try again. Then I watched many other more experienced organizations crash and burn. My current position is that I know enough about crypto to know I never want anything to do with authoring it.
Unless you've spent the last 10+ years working on encryption full time, please don't try it yourself.
I used to think I could write crypto. Then I learned more and realized it needs specialized knowledge I don't have the time to acquire. Then I spent some time trying to acquire it and figured I was OK to try again. Then I watched many other more experienced organizations crash and burn. My current position is that I know enough about crypto to know I never want anything to do with authoring it.
I don't think that's an exception. It's just that encryption isn't actually core functionality for most businesses (even some that think it is).
And if it is core functionality, you should have staff qualified to roll their own, and you should all know why you are doing this (i.e. what differentiation you are going after). This really isn't limited to encryption, it's just a hard domain that bumps up against a lot of other domains, so you see more failures than some others.
Basically if you are going to build a startup or new program in a hard domain, you need to have the resources to do it properly, and you need to know what those are before you start. This is where being a team of smart and motivated individuals may just not be enough.
And if it is core functionality, you should have staff qualified to roll their own, and you should all know why you are doing this (i.e. what differentiation you are going after). This really isn't limited to encryption, it's just a hard domain that bumps up against a lot of other domains, so you see more failures than some others.
Basically if you are going to build a startup or new program in a hard domain, you need to have the resources to do it properly, and you need to know what those are before you start. This is where being a team of smart and motivated individuals may just not be enough.
I'd argue that in a startup, at a very early stage, you can use external services even for core functions, if they work well enough.
Look at Intercom: now it's a juggernaut, but it started as an API mashup with a UI in front of it. If you can hit the market and start getting revenue faster you should. Many services have pay as you go and/or startup discounts.
That's your launch platform, then you start working from there.
Look at Intercom: now it's a juggernaut, but it started as an API mashup with a UI in front of it. If you can hit the market and start getting revenue faster you should. Many services have pay as you go and/or startup discounts.
That's your launch platform, then you start working from there.
Don't take the wrong message from Joel here though. Make sure it's actually a core business function. Is it something that will actually give you a competitive advantage if you do it better than everyone else?
> Is it something that will actually give you a competitive advantage if you do it better than everyone else?
I think you also need to evaluate if it can even be done better than everyone else, and the effort required to do so.
I think you also need to evaluate if it can even be done better than everyone else, and the effort required to do so.
Sure, but that's sort of the entire point of your business.
Joel's quote is more narrow, which is a subset of the dangers of being completely reliant on someone else for your business.
Joel's quote is more narrow, which is a subset of the dangers of being completely reliant on someone else for your business.
I think it’s good to buy something if you can use it immediately without much customization. If you have to do a lot of customization it’s not that clear. For example we have bought a system that requires extensive customization and a lot of people are starting to wonder if the total effort of building from scratch that does wha we need and not more would have been less than the never ending customizations of a system that can do way more than we need.
There is also the question of brain drain. The IT department in my company has outsourced a lot of the tech work to the point that there aren’t many people left who are capable of making informed purchasing decisions. They mostly read sales presentations, white papers and Gardner reports but they have no ability to really assess things. This shows in them repeatedly buying expensive systems that never get really used.
There is also the question of brain drain. The IT department in my company has outsourced a lot of the tech work to the point that there aren’t many people left who are capable of making informed purchasing decisions. They mostly read sales presentations, white papers and Gardner reports but they have no ability to really assess things. This shows in them repeatedly buying expensive systems that never get really used.
That is integration and evolution costs on the article. Those tend to be very hard to estimate, and are the ones that most grow out of control. The article really doesn't do them justice by passing through them like if they were like the others.
There is also an undereported gain that is freeing your team to work on higher impact work.
There is also an undereported gain that is freeing your team to work on higher impact work.
The issue is really about missing requirements... we spent 1,500 dev hours customizing something only to realize we were using the wrong tool for what we wanted... kinda hard to sunk costs that one.
Would these decision makers have produced better outcomes if they were managing software teams?
I think they would produce better outcomes if they understood or used the the systems they are buying or at least listened to actual users. I bet healthcare.gov would have gone better if some high level people had actually have tried to sign up for themselves.
It is a really hard question.
I can attest that the sales presentations tend to be overly optimistic and cheery and if the head guy really wants a given solution, from office politics perspective, you might as well just nod your head, point out the issues you think should be addressed just so that you are covered and roll with it. In practical terms, the decision is not up to you.
In one of my previous roles, manager simply said it is happening ( so it did ), spent some time with the system not fully understanding the underlying logic and was somewhat surprised, when it did not work as advertised. IT had to get involved to smooth some edges. The roll out was still painful. But the contract was already signed and the advertised feature shown during presentation was now extra money, so we were forced to use incomplete product. Annoying.
But then, the real answer is that it varies. Everyone needs to do their due diligence.
What I did not notice is that managers see pretty dashboard and fail to consider the use of the rank and file users. This is the one thing I have learned from all this.
Never underestimate the impact of a pretty dashboard.
I can attest that the sales presentations tend to be overly optimistic and cheery and if the head guy really wants a given solution, from office politics perspective, you might as well just nod your head, point out the issues you think should be addressed just so that you are covered and roll with it. In practical terms, the decision is not up to you.
In one of my previous roles, manager simply said it is happening ( so it did ), spent some time with the system not fully understanding the underlying logic and was somewhat surprised, when it did not work as advertised. IT had to get involved to smooth some edges. The roll out was still painful. But the contract was already signed and the advertised feature shown during presentation was now extra money, so we were forced to use incomplete product. Annoying.
But then, the real answer is that it varies. Everyone needs to do their due diligence.
What I did not notice is that managers see pretty dashboard and fail to consider the use of the rank and file users. This is the one thing I have learned from all this.
Never underestimate the impact of a pretty dashboard.
As someone who helps software vendors overhaul their messaging to attract and convert buyers, I've noticed the following things about the "build vs buy" dilemma (or, from the vendor's point of view: the "build vs buy" objection):
- More senior buyers (CTOs, VPs) are less likely to pit the vendor against a DIY option. I suspect it's because they've been through too many failed DIY projects, or they have great clarity on what is and isn't a good use of their people's time, or they are better at estimating the (usually huge) amount of effort that would go into DIY.
- As the OP points out, a company with huge engineering resources is more likely to consider the DIY option, because, hey, they have all these engineers that need something to do. So if you're a software vendor and are tired of the "build vs buy" conversation, stop trying to sell to other software vendors or VC-backed tech companies.
- If someone chooses the DIY route, just set a calendar reminder to check in two months from now. Chances are they'll be underwhelmed with their progress, and an out-of-the-box solution might sound pretty good by then. (Don't assume they'd reach out on their own in that scenario--they've probably spoken with dozens of other vendors since your last conversation, and forgot about you.)
===
Separately, the part about vendor management is gold:
> To get the full value from vendors, you have to invest in managing vendors well.
I've spent so much time helping evaluate and implement different vendors (usually marketing platforms and similar), only to find out three months later that the software barely gets used.
On the flip side, if you're a software vendor, don't wait until renewal periods to check in on your customers and make sure they're getting the maximum value from your product.
- More senior buyers (CTOs, VPs) are less likely to pit the vendor against a DIY option. I suspect it's because they've been through too many failed DIY projects, or they have great clarity on what is and isn't a good use of their people's time, or they are better at estimating the (usually huge) amount of effort that would go into DIY.
- As the OP points out, a company with huge engineering resources is more likely to consider the DIY option, because, hey, they have all these engineers that need something to do. So if you're a software vendor and are tired of the "build vs buy" conversation, stop trying to sell to other software vendors or VC-backed tech companies.
- If someone chooses the DIY route, just set a calendar reminder to check in two months from now. Chances are they'll be underwhelmed with their progress, and an out-of-the-box solution might sound pretty good by then. (Don't assume they'd reach out on their own in that scenario--they've probably spoken with dozens of other vendors since your last conversation, and forgot about you.)
===
Separately, the part about vendor management is gold:
> To get the full value from vendors, you have to invest in managing vendors well.
I've spent so much time helping evaluate and implement different vendors (usually marketing platforms and similar), only to find out three months later that the software barely gets used.
On the flip side, if you're a software vendor, don't wait until renewal periods to check in on your customers and make sure they're getting the maximum value from your product.
I'm surprised "pricing transparency" isn't included here. A vender that requires you to call a salesperson and do the sort of negotiation on price that is mentioned here is always going to take advantage of you.
My policy is: no pricing on the website, requirement to call a salesperson to talk price? No buy. I'm even willing to pay more to a company with transparent pricing that doesn't require negotiation. It gives me predictability, rather than have to worry that once they've "locked me in" they're going to jack the price and crank all the gears.
My policy is: no pricing on the website, requirement to call a salesperson to talk price? No buy. I'm even willing to pay more to a company with transparent pricing that doesn't require negotiation. It gives me predictability, rather than have to worry that once they've "locked me in" they're going to jack the price and crank all the gears.
If the product fits the bill, giving them a call is no big deal. But they best be flexible.
Tangent:
We had the same issue with splunk. We were blowing through our licenses like a champ.
However the thing that managed to reduce our dependency on splunk wasnt ELK or greylog (they are all poor comparisons to splunk in terms of speed or chopping and changing data) it was grafana(with cloudwatch and graphite, later promethius)
What we realised was that the primary way we were consuming splunk was in graphs. Why spend millions of pounds on generating graphs indirectly, when we could make them directly, in near real time!
Not Tangent:
I think like platform choice, it depends on your company. The company that I was with was utterly incompetent, and unable to see a project through to completion. Therefore if we build a logging system inhouse, we'd get 40% of the way there and rebuild it. Leaving two incomplete systems.
Migrating to splunk cloud was done to save costs, and it actually worked. Had we tried to go inhouse, we'd still have expensive local splunk _and_ a shitty splunk replacement used by two team on a legacy critical system with no dev hours.
We had the same issue with splunk. We were blowing through our licenses like a champ.
However the thing that managed to reduce our dependency on splunk wasnt ELK or greylog (they are all poor comparisons to splunk in terms of speed or chopping and changing data) it was grafana(with cloudwatch and graphite, later promethius)
What we realised was that the primary way we were consuming splunk was in graphs. Why spend millions of pounds on generating graphs indirectly, when we could make them directly, in near real time!
Not Tangent:
I think like platform choice, it depends on your company. The company that I was with was utterly incompetent, and unable to see a project through to completion. Therefore if we build a logging system inhouse, we'd get 40% of the way there and rebuild it. Leaving two incomplete systems.
Migrating to splunk cloud was done to save costs, and it actually worked. Had we tried to go inhouse, we'd still have expensive local splunk _and_ a shitty splunk replacement used by two team on a legacy critical system with no dev hours.
Article felt more like "not buy", not "build vs buy".
I just spent an afternoon helping someone to get us to run on-prem. Phone calls! Installs! Vendor management! We didn't even setup TLS b/c gov stuff is complicated! Working with vendor software is so painful!
Except they were originally thinking they'd have to build what we've spent years not only building, but building as GPU / data / etc. specialists, and we've already adopted many best practices for stuff like TLS and navigated other gov systems for them. (It can still be way better, but imagine if you had none of that stuff, and then had to support it!). Even ignore how we're special -- GPUs are magical if you use them right -- just that we have basic stuff built in and with on-going support. That's so key and adds up over the years. We do it for our graph investigation niche, but same deal for almost everything.
As software has matured, and I see more and more orgs dealing with the reality that they're not Google nor Stripe with thousands of the best-paid software engineers, I'm convinced articles should be explaining from the default of "not build".
I just spent an afternoon helping someone to get us to run on-prem. Phone calls! Installs! Vendor management! We didn't even setup TLS b/c gov stuff is complicated! Working with vendor software is so painful!
Except they were originally thinking they'd have to build what we've spent years not only building, but building as GPU / data / etc. specialists, and we've already adopted many best practices for stuff like TLS and navigated other gov systems for them. (It can still be way better, but imagine if you had none of that stuff, and then had to support it!). Even ignore how we're special -- GPUs are magical if you use them right -- just that we have basic stuff built in and with on-going support. That's so key and adds up over the years. We do it for our graph investigation niche, but same deal for almost everything.
As software has matured, and I see more and more orgs dealing with the reality that they're not Google nor Stripe with thousands of the best-paid software engineers, I'm convinced articles should be explaining from the default of "not build".
In my less charitable moods, I'm inclined to say that you're unlikely to find a whole lot of software developers saying you should build less software for the same reason I've never met a real estate agent who's really on board with my desire to stick to a more modest living situation.
Besides, building in house is inherently more fun. What would you rather do, write new software, or pore over manuals for software that someone else wrote? For my part, there's a reason I became a programmer and not a tech writer.
Besides, building in house is inherently more fun. What would you rather do, write new software, or pore over manuals for software that someone else wrote? For my part, there's a reason I became a programmer and not a tech writer.
Yep! I find that working with management involves planning through outcomes and general resourcing, while working with developers and engineering managers involves planning through the cooler stuff they get to do. ("Instead of reimplmenting some JavaScript UI and Python backend thing for the nth+1 time, we'll zip ahead to the GPU era and focus more time on the data engineering and data science and visual analytics side...")
Folks don't get recognized as heroes for not doing anything, but when they drive the greater story, everyone wins. Understanding the individuals involved in successful software delivery is so important!
Folks don't get recognized as heroes for not doing anything, but when they drive the greater story, everyone wins. Understanding the individuals involved in successful software delivery is so important!
There is so much stuff to build that to pass up an opportunity to build something everyone is building is no big deal. As TFA says: focus on the business, build to its core competency, buy the rest if you can (you can't always).
I'm super surprised to see this article not mention the phrase "opportunity cost".
That's what someone pays to save with something like Splunk. Pay cash to make SWEs more effective towards solving the problems that make up the business's value prop. Cash is much easier to get than the opportunity to grow the business with the people already there. That time never comes back.
That's what someone pays to save with something like Splunk. Pay cash to make SWEs more effective towards solving the problems that make up the business's value prop. Cash is much easier to get than the opportunity to grow the business with the people already there. That time never comes back.
Is it really buy? Or is it "rent"?
I'm guessing for most people, it's actually "rent".
I'm guessing for most people, it's actually "rent".
My assumption when reading it was that buy means both rent and purchase licenses/hardware.
To me the primary thing that matters is how strategic the software is to your core business. This may sound like just evaluating whether it's your core business or not, but it's a nuance on that which is important.
It's easiest described by an example: we use open source Gitlab extensively. Numerous times it has been proposed that we migrate to $expensive alternative because they all have advanced functionality that we don't get from Gitlab (in many cases, even the enterprise version). Yet we are sticking with Gitlab, and making extensive customisations (mostly using the API) and workflows adapted to it.
Is our core business developing software for CI/CD/Issue tracking/source code management? Absolutely not. But is that critical to us? Absolutely. Because Gitlab is open source and because it has an extensive API, we can eek out efficiencies all over the place that aren't possible with closed source software.
So I think it's really important that people use the right frame of mind when evaluating "core business" or not. You don't have to be in the business of a selling particular type of software for it to be worth doing "in-house". You do have to be good at it though.
It's easiest described by an example: we use open source Gitlab extensively. Numerous times it has been proposed that we migrate to $expensive alternative because they all have advanced functionality that we don't get from Gitlab (in many cases, even the enterprise version). Yet we are sticking with Gitlab, and making extensive customisations (mostly using the API) and workflows adapted to it.
Is our core business developing software for CI/CD/Issue tracking/source code management? Absolutely not. But is that critical to us? Absolutely. Because Gitlab is open source and because it has an extensive API, we can eek out efficiencies all over the place that aren't possible with closed source software.
So I think it's really important that people use the right frame of mind when evaluating "core business" or not. You don't have to be in the business of a selling particular type of software for it to be worth doing "in-house". You do have to be good at it though.
When you are looking at paying for an expiring license or SaaS product, it is build versus Rent. And one cost of renting a product is that your vendor's business may change. Perhaps they get bought by Apple and the product is shut down, or maybe they just decide to go in a different direction. Or they raise their prices. Or the service isn't as good as you thought it was going to be. If you have done a lot of integration work (either in software or organizationally) you are either locked in or all that work is wasted.
This is addressed in the article as "risk".
The best way to balance this is to prefer buying/renting Open Source software. That way, if your vendor changes direction, or their service level drops unacceptably, you can choose a new vendor without losing your integrations. Or even switch to a self-supported model.
And any improvements you need, you can contribute to the project so that others can take advantage of them and share the maintenance burden.
This is addressed in the article as "risk".
The best way to balance this is to prefer buying/renting Open Source software. That way, if your vendor changes direction, or their service level drops unacceptably, you can choose a new vendor without losing your integrations. Or even switch to a self-supported model.
And any improvements you need, you can contribute to the project so that others can take advantage of them and share the maintenance burden.
Nice article. I had published my thoughts from a CTOs perspectives on this topic a while ago. Maybe it’s a good addition to your article for anyone:
https://blog.photoeditorsdk.com/build-or-buy-f09785ce1138
https://blog.photoeditorsdk.com/build-or-buy-f09785ce1138
Personally, I (as a software engineer who loves to write software) would advocate buy anywhere and everywhere you can. Even your core business. In my personal experience it's rarely code that's giving you an advantage, it's more likely to be your data, your connections, your efficiency, etc.
Also, try to use the product as off-the-shelf as possible. Little or no customisation. If people in your business are unable to work with a product without customisation consider firing them and replacing them with people who can before you get involved in customisations.
For me the dominating factor here is human resource pool. For whatever we build there are no experts on the system except the ones who already work for us. So aside from all the build, maintenance, etc. costs associated with build, I also can't find any employees on the market who are experts or even competent in the software.
Also, try to use the product as off-the-shelf as possible. Little or no customisation. If people in your business are unable to work with a product without customisation consider firing them and replacing them with people who can before you get involved in customisations.
For me the dominating factor here is human resource pool. For whatever we build there are no experts on the system except the ones who already work for us. So aside from all the build, maintenance, etc. costs associated with build, I also can't find any employees on the market who are experts or even competent in the software.
One thing overlooked here (or not explicitly called out) is talent pool. My last boss didn't want to use anything that he couldn't plug into Dice Or Similar and get dozens of hits for.
If it's not part of your core business/core competency, and you're just reinventing the wheel so you don't have to pay a vendor, you have to weigh against the cost of the off-the-shelf solution NOT just the cost of developing it yourself, but developing it, maintaining it, documenting it, replacing your "irreplaceable engineer" when s/he quits, etc.
If it's not part of your core business/core competency, and you're just reinventing the wheel so you don't have to pay a vendor, you have to weigh against the cost of the off-the-shelf solution NOT just the cost of developing it yourself, but developing it, maintaining it, documenting it, replacing your "irreplaceable engineer" when s/he quits, etc.
The debate as old as tech.
Sometimes, you don’t get a choice. In most large enterprises, it is almost a given that you’re going to buy instead of build, and that thing you bought better come with enterprise support.
Sometimes, you don’t get a choice. In most large enterprises, it is almost a given that you’re going to buy instead of build, and that thing you bought better come with enterprise support.
I think it's all about focus. There is a limit to how many goals/projects you can focus on at once. Every new thing you build distracts your focus. You should minimize what you build in-house to just the things that make you unique. I like Unsplash' blogpost about this: https://medium.com/unsplash/scaling-unsplash-with-a-small-te...
One thing that's commonly overlooked is the risk/vendor lock in associated with building things in-house. Sure external vendors sometimes raise prices or shutdown their product. Those events are pretty rare though and you can typically move to a competing solution. The real risk is building in-house and having the people that create the in-house solution leaving.
> raise prices
Many SaaS have 10x'd their prices over the last few years and it is a significant risk for any tool that you are "renting".
Many SaaS have 10x'd their prices over the last few years and it is a significant risk for any tool that you are "renting".
Moving to another "competing solution" can be extremely expensive.
I guess the real question is how to correctly distinguish between a commodity vs a potential lock-in. For example, if you stick EC, S3 and SES, it feels more like a commodity. However, with more services in the basket you will probably end up in a lock-in situation and your company will hardly survive if e.g. your AWS account gets suspended.
"A company that does this extraordinarily well is Amazon, who issue their vendors quarterly report cards grading their performance against the contract and expectations. Getting great results from vendors requires managing them. If you neglect them and get bad results, that's on you."
- Gives JIRA a report card on them as a vendor to your business.
- Is largely ignored because my business is not Amazon scale with all the power that holds.
tldr; this only really works if you're one of the whales for a vendor.
- Gives JIRA a report card on them as a vendor to your business.
- Is largely ignored because my business is not Amazon scale with all the power that holds.
tldr; this only really works if you're one of the whales for a vendor.
You have an idea, you think it is a good idea based on your early validation.
You start the basic design of how it should work, maybe even based on that early feedback.
If you have no money, and you can build stuff, it doesn't matter if buying is the best advice.
Should you build or should you buy?
Neither. You should survive.
You start the basic design of how it should work, maybe even based on that early feedback.
If you have no money, and you can build stuff, it doesn't matter if buying is the best advice.
Should you build or should you buy?
Neither. You should survive.
[deleted]
When your core existence depends on being the only people able to do a thing, being able to do it with off-the-shelf product begs questions which go to the axium of you being the only people able to do it.
If you understand the role, this doesn't have to be a problem. Hence registry/registrar/delegate separation: the registry is not the delegate. the registrar is not the delegate, and has to meet the demands of systems boundaries implemented by the registry. The delegate has no software limitation and no need to own software because its a role divorced from software. Anything can be out or in source.
Or the CA/RA functional divide in a PKI. Better not to be an amateur implementing crypto, buy off the shelf because your role is to adjudicate what happens, not write systems.
If you understand the role, this doesn't have to be a problem. Hence registry/registrar/delegate separation: the registry is not the delegate. the registrar is not the delegate, and has to meet the demands of systems boundaries implemented by the registry. The delegate has no software limitation and no need to own software because its a role divorced from software. Anything can be out or in source.
Or the CA/RA functional divide in a PKI. Better not to be an amateur implementing crypto, buy off the shelf because your role is to adjudicate what happens, not write systems.
Build, buy, and engage if it's part of a larger project that involves any innovation. Engagement falls under "vendor management," but that seems like a very reduced description.
You are fundamentally buying your time, so buy as much as you can and stay focused on your own business?
Their profit margin is usually in their efficiency in doing it better than you, and at the same time you get the benefit of all the features they add for other customers.
Oftentimes the decision to build usually came down to the difficulty in integration of a commercial product, not the cost. In most cases the cost is set to be competitive against building yourself.
Except in the case of Splunk. Their pricing is ridiculous.