Ask HN: When do you use a library vs. implement something yourself?
70 comments
This is no different to any other "build vs buy" thought process.
Is this a utility-like function for you, or is it defining competitive advantage for your project? Do you need to own the roadmap for this functionality, or are you happy for others to define the roadmap for you? Are you genuinely innovating in this functional or non-functional area - if not, why are you spending time/money/effort on it? What is the TCO of the given solution vs building and maintaining the functionality yourself?
As developers we think the solution to any problem is to develop. It often isn't. If you need something to serve stuff off port 80 at scale, would you build a web server, or install one? If you need crypto, same question.
Spend your time where it counts, be grateful for the decades of work that has gone into the stuff you can take for free out there, use it wisely.
Is this a utility-like function for you, or is it defining competitive advantage for your project? Do you need to own the roadmap for this functionality, or are you happy for others to define the roadmap for you? Are you genuinely innovating in this functional or non-functional area - if not, why are you spending time/money/effort on it? What is the TCO of the given solution vs building and maintaining the functionality yourself?
As developers we think the solution to any problem is to develop. It often isn't. If you need something to serve stuff off port 80 at scale, would you build a web server, or install one? If you need crypto, same question.
Spend your time where it counts, be grateful for the decades of work that has gone into the stuff you can take for free out there, use it wisely.
I'm a library by default kind of guy, but it's always a series of trade-offs. For example:
- how value adding is this functionality?
- how well does the library meet my need?
- is the library still under active development?
- how stable is the library's API? (Building against an unstable API becomes a tedious waste of resources, fast.)
- how good is the documentation?
- is it popular/well-used? If so, this makes it easier to get help on Stackoverflow, or wherever, when I have a problem.
There are times where we've started out with a library, but moved to a homegrown implementation, or vice versa.
For example, we work with a lot of CSV input data. We started out with a library, because properly parsing CSVs is, well, not exacly tricky, but there are a pile of nuances and edge-cases you need to take account of. It was just much easier to use a library for this so that we could focus on adding business value elsewhere.
Anyway, the size of the files we needed to work with grew, so we swapped the original library out for a different, faster, library.
That worked well for a while, but then the size of the files we needed to work with grew again, and the library was too slow, so we swapped in our own faster CSV reader implementation for working with these large files.
We still use the library for other CSVs because our implementation makes certain assumptions about the content of the files that don't hold for the smaller files we work with.
- how value adding is this functionality?
- how well does the library meet my need?
- is the library still under active development?
- how stable is the library's API? (Building against an unstable API becomes a tedious waste of resources, fast.)
- how good is the documentation?
- is it popular/well-used? If so, this makes it easier to get help on Stackoverflow, or wherever, when I have a problem.
There are times where we've started out with a library, but moved to a homegrown implementation, or vice versa.
For example, we work with a lot of CSV input data. We started out with a library, because properly parsing CSVs is, well, not exacly tricky, but there are a pile of nuances and edge-cases you need to take account of. It was just much easier to use a library for this so that we could focus on adding business value elsewhere.
Anyway, the size of the files we needed to work with grew, so we swapped the original library out for a different, faster, library.
That worked well for a while, but then the size of the files we needed to work with grew again, and the library was too slow, so we swapped in our own faster CSV reader implementation for working with these large files.
We still use the library for other CSVs because our implementation makes certain assumptions about the content of the files that don't hold for the smaller files we work with.
I'll use a library if it's at the extreme end of: mature, actively maintained, not buggy, general, and 'standard' for lack of a better word. Anything else I treat as a debt, to be avoided unless I'm desperate. And yes, I absolutely consider it a worse debt than anything I write myself, unless it meets the criteria above. Experience has taught me this a hundred times over.
Huh. Experience has taught me the opposite, that any code written in-house is more likely to be undocumented, untested, or buggy than a library used by many others. Especially in the case of a small team that has grown over 5+ years, where things devs write early on quickly become technical debt that few others can understand without the original context. Contrast that to using an import to a documented library where any new dev can quickly go and read the docs & examples and know exactly what it's for. The few cases where a package upgraded something which broke our code have always been easy to resolve by just downgrading the package.
If this hypothetical library is that good, then it probably meets your version of the general criteria I sketched. One problem is that it isn't always obvious what the quality is at the time you make the decision to take the dependency. I use size/complexity as a proxy for how much trouble it's likely to be.
I don't think this can ever be totally cut and dried.
There is a heuristic when either using external or developing one inhouse is obvious, but if neither applies then finding the best solution requires careful thinking.
The obvious place to develop a library inhouse is when there is no compatible third party solution available.
The obvious place to use a library is when one is available that a) Implements a feature that would requires weeks of work b) implements the specific feature one requires c) on the technology platform of the product d) is compatible financially and licensing wise
If a,b,c and d hold then third party library is a pretty good option. Depending on the complexity of the problem one might be really lenient on c) and do whatever to integrate a MagicalPixieDustGenerator no matter how badly it fits the current tech stack.
Code reuse is one of the silliest arguments ever for librarization. It's totally fine to replicate few lines of code, it's just typing. If you don't have tests to verify functionality you are screwed anyway.
The obvious place to develop a library inhouse is when there is no compatible third party solution available.
The obvious place to use a library is when one is available that a) Implements a feature that would requires weeks of work b) implements the specific feature one requires c) on the technology platform of the product d) is compatible financially and licensing wise
If a,b,c and d hold then third party library is a pretty good option. Depending on the complexity of the problem one might be really lenient on c) and do whatever to integrate a MagicalPixieDustGenerator no matter how badly it fits the current tech stack.
Code reuse is one of the silliest arguments ever for librarization. It's totally fine to replicate few lines of code, it's just typing. If you don't have tests to verify functionality you are screwed anyway.
(1) I am writing a new React web-based LaTeX editor for CoCalc, and needed a PDF viewer. I first tried using the https://github.com/wojtekmaj/react-pdf library, which builds on top of Mozilla's pdf.js. It's useful code, but the library is designed in a way that is fundamentally very inflexible for my use cases. For example, it is critical for me to easily have multiple views of the same document, to easily be able to cache recently loaded PDF's, and to be able to reload a modified PDF with no observable flicker. I found trying to get any of my core requirements to work with this library to be a constant battle involving horrible hacks. So I had to sit down for a very long day and write something similar from scratch directly using pdf.js itself. So my answer: sometimes, at some levels, you have to implement yourself instead of using a library, when the library is not flexible enough and trying to use it gets really hackish.
(2) Another experience I had was when I started the SageMath project. I began with the plan to look at other code and implement everything myself, rather than use existing libraries. That lasted about an hour, since it was way, way too much work (to put it mildly). So I switched to using Python as the user language, then using as much as possible existing C/C++ libraries (like PARI/GP and GAP and Singular) to provide functionality. The result was again a combination of using libraries but also writing a new library.
(2) Another experience I had was when I started the SageMath project. I began with the plan to look at other code and implement everything myself, rather than use existing libraries. That lasted about an hour, since it was way, way too much work (to put it mildly). So I switched to using Python as the user language, then using as much as possible existing C/C++ libraries (like PARI/GP and GAP and Singular) to provide functionality. The result was again a combination of using libraries but also writing a new library.
This is a complex issue that takes days of careful analysis to take a decision.
Long story short we implement something ourself if:
1.It is a differentiator for our company because 2.
2.We can make it better than available libs and 3.
3.Making it better represents a real change on our company or the world, changes like "It will make possible what was not possible" or "will halve the cost of X, because Y".
4.We need to parallelize-make it on hardware.
Implementing something inside is extremely expensive as requires salaries of highly paid people, makes those people unavailable for doing other things, and takes always more time than planned.
On the other hand with your own code you could do magic. You can hardware accelerate(with electronics and FPGAS) it to use 10.000, 50.000 less energy, or GPUs to make it 100-200 faster. You control everything to suit your needs.
We normally start using standard libraries and replace then slowly with our own code if it will make a difference.
We do the same with parallelization: we start with multiCPUs, then GPUs, then FPGAS. Then we can even think on offering investors the option of making ASICS if functionality is proven and very well tested.
Disasters? Yes of course. It is part of experience, they happen like great successes. When you risk you risk disaster, but you give yourself the option to success greatly too.
When I was a kid I learned to ski probably faster than anyone around. But it was because I did not play safe like most did and keep falling(at great speed) until I did not.
In life you could do the same.
Long story short we implement something ourself if:
1.It is a differentiator for our company because 2.
2.We can make it better than available libs and 3.
3.Making it better represents a real change on our company or the world, changes like "It will make possible what was not possible" or "will halve the cost of X, because Y".
4.We need to parallelize-make it on hardware.
Implementing something inside is extremely expensive as requires salaries of highly paid people, makes those people unavailable for doing other things, and takes always more time than planned.
On the other hand with your own code you could do magic. You can hardware accelerate(with electronics and FPGAS) it to use 10.000, 50.000 less energy, or GPUs to make it 100-200 faster. You control everything to suit your needs.
We normally start using standard libraries and replace then slowly with our own code if it will make a difference.
We do the same with parallelization: we start with multiCPUs, then GPUs, then FPGAS. Then we can even think on offering investors the option of making ASICS if functionality is proven and very well tested.
Disasters? Yes of course. It is part of experience, they happen like great successes. When you risk you risk disaster, but you give yourself the option to success greatly too.
When I was a kid I learned to ski probably faster than anyone around. But it was because I did not play safe like most did and keep falling(at great speed) until I did not.
In life you could do the same.
The question is too vague. In what environment? Web? Desktop? Mobile? Embedded? Backend programming is a completely different world than mobile apps when it comes to build size concerns, for instance. As an Android developer, I have to deal with the infamous 64K methods limit imposed by Dalvik. The "I import TONS of libraries" approach (to quote one commenter) is very bad practice here.
I think it depends on how long you want the software to last for.
At one end is a client wanting a quick mock up - yes use libraries.
At the other end is a surgically implanted heart monitor - please dont use a library that depends on an API call.
Using libraries certainly gets the job done quicker (in the short term), but it adds technical debt and brittleness to any project.
At one end is a client wanting a quick mock up - yes use libraries.
At the other end is a surgically implanted heart monitor - please dont use a library that depends on an API call.
Using libraries certainly gets the job done quicker (in the short term), but it adds technical debt and brittleness to any project.
Meta, but, things like pad-left and the other npm things are IMO an issue with the standard library of in this case Javascript. A lot of languages have a richer utility toolbelt atm. (this is also a problem in Java imo, where you basically need a few libraries for convenient string or collection utilities)
Python use a library as they are well built.
Javascript build my own as they are not.
(Joke with a hint of a more serious problem).
Javascript build my own as they are not.
(Joke with a hint of a more serious problem).
I try and stick to the my language's standard library whenever possible. Because I know that the stdlibs will be maintained and it won't be a waste of time to learn them.
If I really need to use some library (for example Graphic toolkits etc.) I tend to try to understand it by looking at its source. I try to make some modifications in it, try to make them run. That way I get somewhat an understanding of the library and the problem domain itself.
Usually you only need a small part of a library. You can look at the implementation at that library and then incorporate it in your program and slowly improve it. That way your dependencies won't get bloated, you'll get a customized solution with little effort and if you encounter problems you'll know where to fix them. That's a trick I've learned over the years.
If I really need to use some library (for example Graphic toolkits etc.) I tend to try to understand it by looking at its source. I try to make some modifications in it, try to make them run. That way I get somewhat an understanding of the library and the problem domain itself.
Usually you only need a small part of a library. You can look at the implementation at that library and then incorporate it in your program and slowly improve it. That way your dependencies won't get bloated, you'll get a customized solution with little effort and if you encounter problems you'll know where to fix them. That's a trick I've learned over the years.
You must be a native programmer. I have found that collegues using managed languages are much less conservative compared to collegues who program using native languages (primarily C/C++).
I think its about which languages have good dependency management.
For example, Go, C and C++ have bad dependency management, so most programmers will try to stick to stdlib solutions, only moving to dependencies if copy-paste is an overkill (I wouldn't copy-paste sqlite wrappers, but I would copy-paste a left-pad like function).
NPM and Rust (which isn't managed) has great dependency management, so they move things out of stdlib into packages.
But, it backfired when no one is sure which packages are official, so Rust made a "nursery" of officially maintained out-of-stdlib standard functions.
For example, Go, C and C++ have bad dependency management, so most programmers will try to stick to stdlib solutions, only moving to dependencies if copy-paste is an overkill (I wouldn't copy-paste sqlite wrappers, but I would copy-paste a left-pad like function).
NPM and Rust (which isn't managed) has great dependency management, so they move things out of stdlib into packages.
But, it backfired when no one is sure which packages are official, so Rust made a "nursery" of officially maintained out-of-stdlib standard functions.
Dependency management isn't a concept in C++. C++ is an ISO standard language with numerous implementations; it's not a software package. A C++ compiler from Microchip for your PIC microcontroller has nothing to do with Microsoft Visual C++ has nothing to do with GNU C++ on your Linux box, etc.
ECMAScript is also an (ECMA) standard language. Yet it has NPM.
Or she is a programmer working with a sensible programming language with sensible standard library (e.g., Python). I would really like to like NodeJS, JavaScript is not that bad language in itself, but missing standard library is a problem for me.
Node has a pretty big standard library really (in fact that's all node.js is; a module system and a ton of modules built on top of a JS engine). What do you find is missing compared to Python's standard library?
The fact that there was a need for the left-pad module is pretty telling on how comprehensive the stdlib in js is.
I wasn't really interested in your opinion based on the lack of one 1-line utility function (and it is one line: `new Array(10 - str.length).fill("0").join("") + str`). I was interested in what significant functionality OP found lacking in node.js' standard library.
Which programming language are you using that has left-pads functionality in the standard library?
Well, python's strings have the ljust and rjust methods which left- and right-pad, and javascript has the padStart and padEnd methods (since ECMAScript 2017).
I'm pretty sure that's talking about all the third-party libraries which exist for Python; that this "antigravity" library is available for Python, not a part of its standard library. (I might be wrong though, I'm not Randall Munroe)
For me, it's whatever I believe would take less time. I can write a left-pad function in less time than it would take to find the npm package, install it, import it and integrate it.
With more complex systems, how much integration/customisation is needed vs building something yourself?
Often I'll look for a library, find a few candidates that don't really suit well and 'resort' to committing to writing something myself. Similar to planning a project, it can take time to work out how much time something is going to take.
With more complex systems, how much integration/customisation is needed vs building something yourself?
Often I'll look for a library, find a few candidates that don't really suit well and 'resort' to committing to writing something myself. Similar to planning a project, it can take time to work out how much time something is going to take.
If I on-the-fly estimate that implementing x feature takes too long* and if this seems like a problem others might have solved, I go for a library pretty fast. Basically I don't want to implement and maintain something if others have done it for me.
Example: I need to parse excel-files in python => straight to looking for a library, I don't even want to start looking into excel-format specs.
Example2: I need to parse JSON in Java. First look if Java have this natively (I don't write too much Java) and then go look for a lib. Others must have had this problem, and they most likely solved it as good as or better than me - the surely devoted more time to it, if the bothered to publish a lib.
I review if the library project seems alive and being updated. I review if it is super easy to use. Then I include it, no more fuss. Most time I end up checking out 2-4 libs to compare ease-of-use and liveliness etc.
I admit that I could do a lot more on security review. Most of the time I fallback to the thinking that this is OSS and/or a widely used lib, and if there was a security issue that I would be able to find, others would have found it by now.
*: What too long is varies I guess, but you always have a gut feeling of how troublesome something is to do. Maybe compared to the value of x feature.
Thats how it is for most all hobby projects. If it is projects for a client, I worry a bit more about security, but the primary driver for decisions are that others must have solved the same problem and most likely better.
And of course in some organisations you'll find it-policies, security policies, dev guidelines etc. that you just need to follow. Then you do that - the client is paying for slow then, by their own choice.
Example: I need to parse excel-files in python => straight to looking for a library, I don't even want to start looking into excel-format specs.
Example2: I need to parse JSON in Java. First look if Java have this natively (I don't write too much Java) and then go look for a lib. Others must have had this problem, and they most likely solved it as good as or better than me - the surely devoted more time to it, if the bothered to publish a lib.
I review if the library project seems alive and being updated. I review if it is super easy to use. Then I include it, no more fuss. Most time I end up checking out 2-4 libs to compare ease-of-use and liveliness etc.
I admit that I could do a lot more on security review. Most of the time I fallback to the thinking that this is OSS and/or a widely used lib, and if there was a security issue that I would be able to find, others would have found it by now.
*: What too long is varies I guess, but you always have a gut feeling of how troublesome something is to do. Maybe compared to the value of x feature.
Thats how it is for most all hobby projects. If it is projects for a client, I worry a bit more about security, but the primary driver for decisions are that others must have solved the same problem and most likely better.
And of course in some organisations you'll find it-policies, security policies, dev guidelines etc. that you just need to follow. Then you do that - the client is paying for slow then, by their own choice.
I am not sure if there would be cut and dry rules regarding this. The rule of the thumb for my startup is to always use libraries for major functionalities and for minor functionalities we either lookup the source and use the required function or we implement it ourselves.
What works best for you might not be the correct answer for someone else.
For us, we initially used libraries for all lat/lon and distance calculations and then refactored to move the functions to our internal code and get rid of the libraries. For algorithms like DBSCAN , even if it's straightforward, it didnt make sense to re-implement it.
I guess it boiled down to the effort required to move inhouse vs the benefits
What works best for you might not be the correct answer for someone else.
For us, we initially used libraries for all lat/lon and distance calculations and then refactored to move the functions to our internal code and get rid of the libraries. For algorithms like DBSCAN , even if it's straightforward, it didnt make sense to re-implement it.
I guess it boiled down to the effort required to move inhouse vs the benefits
Predict whether or not it will save you time. Note that while complexity is a part of it, it isn't everything. If it goes unmaintained and at some point you need to rip it out because your product needs to stay alive, it was unknowingly technical debt to begin with.
I implement business apps for the business. Usually, I organize those apps into libraries.
Beyond that, if it's not core business logic, it gets downloaded.
Beyond that, if it's not core business logic, it gets downloaded.
Interesting question. As you mention, there seem to be a spectrum of potentially valid answers, depending on which individual, team, or even platform you ask / talk about.
Personally, I try to make an informed gut-feel decision about the short term and long term benefits of pulling in a library for anything. The main inputs to the decision are how difficult rolling my own would be, whether there are any complex risks (such as security implications I'm not well versed to handle, or other such topics), and how much use / mileage we will get out of the dependency. Very often this comes down to whether you've done something similar before, and are confident that you can handle it well enough. I think I tend to lean towards rolling my own, or "NIH syndrome" a bit more than the average developer, but this is for a few reasons. Take this with a pinch of salt, and for context, I would of course agree with not rolling your own crypto, etc - I'm mainly referring to the gray area in the middle.
1) It's usually quite difficult to find a library that nails the level of complexity vs pay-off that I want in a solution. Most of the time, libraries that I find are a complete overkill for my usage, or have large surface areas that I don't need / want. If I can roll a custom implementation in under 50 lines of code, I will generally do so.
2) While you often get immediate payoff for pulling in a dependency, that payoff very often turns around and becomes a net loss of time the moment you run into a complex problem. Pad-left is a silly example, but I think we've all encountered situations where we pulled in some library only to run into a strange edge case. Having wasted a day, you're then thinking "I honestly could have done this better, given a few hours".
3) This point tends to depend a lot on the exact environment you're working with, but managing dependencies can be pretty painful. As a simple example, I've had a number of Angular and Ionic projects with a large number of dependencies become extremely non-trivial to upgrade. If you're talking about a small number of such projects which are frequently worked on (say, flagship apps of a startup), I suspect this is not much of an issue. When you're dealing with dozens of apps for a large corporate, some of which may only get attention in a year or so, this can be a painful burden. A hilarious parody of pad-left is "is-thirteen" [1], which drives the point home for me.
4) Rolling your own also tends to architecturally firewall your implementation (if you've done it right). I.e. it is easier to take your custom implementation and just point it at a 3rd party library, than it is to migrate from library A to library B, if you haven't firewalled them nicely.
The number of times I've regretted rolling my own has been far outweighed by the number of times I've pulled in the wrong dependency.
In summary or as examples, I would never willingly pull in "pad-left", would always stick with a well known UI framework rather than roll my own (primarily for various security reasons), but have written simple templating engines, markdown parsers, etc, without regret. On the other end, I would never roll own crypto, and would be highly hesitant to even come up with my own crypto schemes involving well-implemented crypto, as those can be just as weak.
[1] https://github.com/jezen/is-thirteen#is-thirteen
Personally, I try to make an informed gut-feel decision about the short term and long term benefits of pulling in a library for anything. The main inputs to the decision are how difficult rolling my own would be, whether there are any complex risks (such as security implications I'm not well versed to handle, or other such topics), and how much use / mileage we will get out of the dependency. Very often this comes down to whether you've done something similar before, and are confident that you can handle it well enough. I think I tend to lean towards rolling my own, or "NIH syndrome" a bit more than the average developer, but this is for a few reasons. Take this with a pinch of salt, and for context, I would of course agree with not rolling your own crypto, etc - I'm mainly referring to the gray area in the middle.
1) It's usually quite difficult to find a library that nails the level of complexity vs pay-off that I want in a solution. Most of the time, libraries that I find are a complete overkill for my usage, or have large surface areas that I don't need / want. If I can roll a custom implementation in under 50 lines of code, I will generally do so.
2) While you often get immediate payoff for pulling in a dependency, that payoff very often turns around and becomes a net loss of time the moment you run into a complex problem. Pad-left is a silly example, but I think we've all encountered situations where we pulled in some library only to run into a strange edge case. Having wasted a day, you're then thinking "I honestly could have done this better, given a few hours".
3) This point tends to depend a lot on the exact environment you're working with, but managing dependencies can be pretty painful. As a simple example, I've had a number of Angular and Ionic projects with a large number of dependencies become extremely non-trivial to upgrade. If you're talking about a small number of such projects which are frequently worked on (say, flagship apps of a startup), I suspect this is not much of an issue. When you're dealing with dozens of apps for a large corporate, some of which may only get attention in a year or so, this can be a painful burden. A hilarious parody of pad-left is "is-thirteen" [1], which drives the point home for me.
4) Rolling your own also tends to architecturally firewall your implementation (if you've done it right). I.e. it is easier to take your custom implementation and just point it at a 3rd party library, than it is to migrate from library A to library B, if you haven't firewalled them nicely.
The number of times I've regretted rolling my own has been far outweighed by the number of times I've pulled in the wrong dependency.
In summary or as examples, I would never willingly pull in "pad-left", would always stick with a well known UI framework rather than roll my own (primarily for various security reasons), but have written simple templating engines, markdown parsers, etc, without regret. On the other end, I would never roll own crypto, and would be highly hesitant to even come up with my own crypto schemes involving well-implemented crypto, as those can be just as weak.
[1] https://github.com/jezen/is-thirteen#is-thirteen
It's just a judgement call. For existing options you have to score based on: maintainedness (frequency of upkeep, issues answered, etc), invasiveness (cost of removal, definitely a big one here when choosing a framework vs a library), dangerousness (trustworthiness of maintainer), weight (itself and transitive dependencies), quality (quick code review, license, etc), and reimplementation difficulty. That score is weighed against your requirements (e.g. building a lib for others might have stricter dependency reqs than an in-house app). But you can take these metrics and apply them to even broader things such as lang/ecosystem choice, general software choices, and even life choices. No different than entering into any other relationship.
While I try to avoid premature abstraction, on libs that have high volatility or otherwise don't score very high, I'll put a thin interface layer between my use and the true impl. This saved me recently as I had a Scala trait abstract the async Postgres client lib I was using, then I hopped from one to another to another over a couple of years until just recently writing one because the others sucked for various reasons. Just make sure you have integration tests and you'll feel comfy in your choice.
While I try to avoid premature abstraction, on libs that have high volatility or otherwise don't score very high, I'll put a thin interface layer between my use and the true impl. This saved me recently as I had a Scala trait abstract the async Postgres client lib I was using, then I hopped from one to another to another over a couple of years until just recently writing one because the others sucked for various reasons. Just make sure you have integration tests and you'll feel comfy in your choice.
I'd add consistency to the benefits: when I need leftpad and declare a dependency on "big bag of string stuff" for it, chances are high that codevelopers will pick it up when they need rightpad. Because the very first step in every library decision is "do we already have a dependency for that". Bespoke implementations would tend to end up all over the place.
There are a couple reasons I can think of to use a library:
* The code is the best in the world and would take anyone years to compete with its speed. Libraries like gmp, fftw etc come to mind.
* The API is good. A good API extends the language in a way that makes your code better and more enjoyable to write.
I usually try to avoid dependencies where possible because it will always make life easier later on. Introducing a dependency for a single function should be a sackable offence imo. At that point you're just too lazy or incompetent to do your job.
* The code is the best in the world and would take anyone years to compete with its speed. Libraries like gmp, fftw etc come to mind.
* The API is good. A good API extends the language in a way that makes your code better and more enjoyable to write.
I usually try to avoid dependencies where possible because it will always make life easier later on. Introducing a dependency for a single function should be a sackable offence imo. At that point you're just too lazy or incompetent to do your job.
Why waste time rewriting something like leftPad?
Really? It would take ten minutes of my time once, as opposed to risking the idiotic errors that occurred when people depended on that "library" being available.
I'm suppose to make sure stupid shit like that doesn't happen. It's not a waste of my time. It's my job.
I'm suppose to make sure stupid shit like that doesn't happen. It's not a waste of my time. It's my job.
leftPad is an exceptional case because 1° it should have been part of the standard library and 2° it is a library that only contains one function.
Calling it a library is a stretch.
The whole leftPad controverse makes a pretty good case for rewriting. It would have saved many people a lot of trouble.
Also, if you know your langauge, rewriting it is probably faster than finding, installing and using a package.
The whole leftPad controverse makes a pretty good case for rewriting. It would have saved many people a lot of trouble.
Also, if you know your langauge, rewriting it is probably faster than finding, installing and using a package.
The trouble with leftPad was a flaw of NPM which could have happened with any package.
I use the standard library and not much else.
If I want some functionality that's difficult to implement, I'll see about including it verbatim in some branch of the source tree.
If it's a large amount of code, or not license-compatible, or needs to be updated apart from the program, then I'll resort to using a library.
Library updates are the number one reason for code rot, in my opinion. More dependencies is more chance for pain later. I try to pick mature platforms and lower-level libraries which change less frequently.
If I want some functionality that's difficult to implement, I'll see about including it verbatim in some branch of the source tree.
If it's a large amount of code, or not license-compatible, or needs to be updated apart from the program, then I'll resort to using a library.
Library updates are the number one reason for code rot, in my opinion. More dependencies is more chance for pain later. I try to pick mature platforms and lower-level libraries which change less frequently.
I've been thinking about this a long time[1][2]. At the moment, I think "build or reuse?" is the wrong question to ask.
* Reusing someone else's implementation gives you the benefit of an initial kickstart. It also can give you future benefits: a library may contain improvements that you may need in the future, at which point you get them for free. Infinite productivity!
* Building your own implementation has an initial cost, but it reduces your maintenance burden to only the features you actually use. It also insulates you from counterparties whose interests may not align exactly with your own, or who accidentally introduce security risks.
If only we could magically unbundle the benefits of other people's code from their limitations, have features magically appear when we need them, and be magically robust to security holes in features we don't use. Here's the closest I've been able to get so far:
Use libraries by copying them.
Let's see why this is good. For starters, it makes you robust to the leftpad failure mode. The library is literally copied into your repository. Who cares how small it is anymore?! It could literally vanish off the face of the rest of the earth, and your code would continue to work.
A less obvious benefit is that copying a library puts you on a trajectory of taking responsibility for your dependencies. You don't need to understand all of it on day 1, but as you use it and run into issues you should be gradually understanding its internals better, gradually resolving features that you use, and features that you don't. That then gives you the option to rip out stuff you're sure you'll never need. Or to rip out stuff you may need in future, but is easy to put back (this is analogous to deleting code without fear because version control).
I consider an example of exemplary library use to be how I copied the termbox library into Mu (https://github.com/akkartik/mu/commit/5f1285238b), periodically merged commits from upstream (https://github.com/akkartik/mu/commit/9ba313ab7f), gradually cleaned it up to fit better with my project (https://github.com/akkartik/mu/commit/9a31c34f0f), and gradually stripped out code from it that Mu does not require (https://github.com/akkartik/mu/commit/c04baba4f2; https://github.com/akkartik/mu/commit/8e7827dfcf; https://github.com/akkartik/mu/commit/547ec78bf2). In the process I made some wrong turns, deleting features that I later decided I wanted (https://github.com/akkartik/mu/commit/10a3b8cca2) and created bugs for myself (https://github.com/akkartik/mu/commit/3315a7d3bb; https://github.com/akkartik/mu/commit/0c0d1ea5cd). But when it did things I didn't want, I was now empowered to change them (https://github.com/akkartik/mu/commit/ee1a18f050). One of my patches was useful upstream, so I submitted it: https://github.com/nsf/termbox/commit/0730826a07. I would be in no position to submit that patch if I hadn't taken the trouble to understand termbox's internals. And that shows a final benefit of copying and privately forking libraries: it makes you a better citizen of the open source world, because open source depends on eyeballs, and using a library blindly helps nobody except your (extremely short-term) self.
Now you may say, this sounds like a lot of work. I can't do it for all the libraries I use. Start small. Pick one dependency that is pretty central to your project, bring it in. See how you like it. As you gain experience come tell me what you discover. (My email is in my profile.) I didn't have all this thought out when I copied termbox into Mu. Benefits gradually became apparent.
[1] http://akkartik.name/post/libraries
[2] http://akkartik.name/post/libraries2
(Past versions of this comment: http://arclanguage.org/item?id=20213; https://lobste.rs/s/w4auk6/more_shell_less_egg_all_this#c_nv...)
* Reusing someone else's implementation gives you the benefit of an initial kickstart. It also can give you future benefits: a library may contain improvements that you may need in the future, at which point you get them for free. Infinite productivity!
* Building your own implementation has an initial cost, but it reduces your maintenance burden to only the features you actually use. It also insulates you from counterparties whose interests may not align exactly with your own, or who accidentally introduce security risks.
If only we could magically unbundle the benefits of other people's code from their limitations, have features magically appear when we need them, and be magically robust to security holes in features we don't use. Here's the closest I've been able to get so far:
Use libraries by copying them.
Let's see why this is good. For starters, it makes you robust to the leftpad failure mode. The library is literally copied into your repository. Who cares how small it is anymore?! It could literally vanish off the face of the rest of the earth, and your code would continue to work.
A less obvious benefit is that copying a library puts you on a trajectory of taking responsibility for your dependencies. You don't need to understand all of it on day 1, but as you use it and run into issues you should be gradually understanding its internals better, gradually resolving features that you use, and features that you don't. That then gives you the option to rip out stuff you're sure you'll never need. Or to rip out stuff you may need in future, but is easy to put back (this is analogous to deleting code without fear because version control).
I consider an example of exemplary library use to be how I copied the termbox library into Mu (https://github.com/akkartik/mu/commit/5f1285238b), periodically merged commits from upstream (https://github.com/akkartik/mu/commit/9ba313ab7f), gradually cleaned it up to fit better with my project (https://github.com/akkartik/mu/commit/9a31c34f0f), and gradually stripped out code from it that Mu does not require (https://github.com/akkartik/mu/commit/c04baba4f2; https://github.com/akkartik/mu/commit/8e7827dfcf; https://github.com/akkartik/mu/commit/547ec78bf2). In the process I made some wrong turns, deleting features that I later decided I wanted (https://github.com/akkartik/mu/commit/10a3b8cca2) and created bugs for myself (https://github.com/akkartik/mu/commit/3315a7d3bb; https://github.com/akkartik/mu/commit/0c0d1ea5cd). But when it did things I didn't want, I was now empowered to change them (https://github.com/akkartik/mu/commit/ee1a18f050). One of my patches was useful upstream, so I submitted it: https://github.com/nsf/termbox/commit/0730826a07. I would be in no position to submit that patch if I hadn't taken the trouble to understand termbox's internals. And that shows a final benefit of copying and privately forking libraries: it makes you a better citizen of the open source world, because open source depends on eyeballs, and using a library blindly helps nobody except your (extremely short-term) self.
Now you may say, this sounds like a lot of work. I can't do it for all the libraries I use. Start small. Pick one dependency that is pretty central to your project, bring it in. See how you like it. As you gain experience come tell me what you discover. (My email is in my profile.) I didn't have all this thought out when I copied termbox into Mu. Benefits gradually became apparent.
[1] http://akkartik.name/post/libraries
[2] http://akkartik.name/post/libraries2
(Past versions of this comment: http://arclanguage.org/item?id=20213; https://lobste.rs/s/w4auk6/more_shell_less_egg_all_this#c_nv...)
> Has there ever been a case where you've tried to roll your own and it ended in disaster?
As far as I remember, it's always the other way around. But I have enough sense to not implement my own cryptography library and use it in production.
> Or where you'd wished you'd never added a dependency on some library?
That happens a lot, or more frequently it's some library already being used that is overcomplicated and won't work with something else I need it to work, like a build tool, or is in a cross platform language like node but doesn't work on one particular, major OS. But when this happens it's maybe never the case that it would have been reasonable to write it myself. I just have to find a less shitty alternative library.
In general, it's blatantly obvious when you need to do which, you have more much bigger other problems, the chance of this issue even popping up is pretty low, and the resolution is probably fairly straightforward.
As far as I remember, it's always the other way around. But I have enough sense to not implement my own cryptography library and use it in production.
> Or where you'd wished you'd never added a dependency on some library?
That happens a lot, or more frequently it's some library already being used that is overcomplicated and won't work with something else I need it to work, like a build tool, or is in a cross platform language like node but doesn't work on one particular, major OS. But when this happens it's maybe never the case that it would have been reasonable to write it myself. I just have to find a less shitty alternative library.
In general, it's blatantly obvious when you need to do which, you have more much bigger other problems, the chance of this issue even popping up is pretty low, and the resolution is probably fairly straightforward.
Opportunity cost: Can I spend my time on something more important ? Plus the cost of maintaining the code. Sometimes the module is not good enough though, and patching it would take longer then rewriting it.
for things like security, storage, encryption, deployment&operations: use libraries
For everything else usually I have a simple flowchart: is it a core feature and a businnes advantage of the platform? Implement. Otherwise, if exists a library, use the library.
For everything else usually I have a simple flowchart: is it a core feature and a businnes advantage of the platform? Implement. Otherwise, if exists a library, use the library.
If I'm fine with supporting the library when it stops getting developed then I'll usually just use it.
I like this metric because it means I'm saving time just writing the same thing I would have written anyway.
I like this metric because it means I'm saving time just writing the same thing I would have written anyway.
I import TONS of libraries and try to write the fewest lines of code possible.
Not sure why you're being down-voted, but if done intelligently, and assuming good-quality libraries, the goal should be as few lines of code that you are directly responsible for.
Always, always use a library unless you can’t.
The NPM ecosystem is on the extreme end of micro-libraries, so sometimes I will just inline a library's code into my project when the indirection of using a library isn't worth it.
For every library you use, you are now trusting everyone that has publish rights to that library and all of its transitive dependencies, so libraries aren't free.
Also, sheepishly, it's so easy to publish on NPM that I'll sometimes just re-implement a library with my creature features and publish it instead of making a case for them in the existing library's issues. Awful, I know.
For every library you use, you are now trusting everyone that has publish rights to that library and all of its transitive dependencies, so libraries aren't free.
Also, sheepishly, it's so easy to publish on NPM that I'll sometimes just re-implement a library with my creature features and publish it instead of making a case for them in the existing library's issues. Awful, I know.
I almost always opt for a library as long as
1. It is still alive
2. It is on GitHub so I can easily read the source and look for issues
3. People star it
4. It has traction ie >10k downloads preferably >50k
5. It occupies a name that makes sense ie I’d rather trust using “redux-immutable” than “steel-porcupine”, since it unofficially lends itself to the “vetted one true solution”
6. Then I look at the source quickly and make sure it looks well done under the hood
7. Finally, if it’s particularly confusing, I might run a quick google check to see if there’s blogs about it on the internet explaining how to use the advanced features and check the date on the blogs (1-2 years ago = good, it’s mature)
If any of the above are too dicey and there’s no good options, I put the relevant code in a top level folder called “modules” (or otherwise) with the intention that in the future I might get to delete it if the package ecosystem gets more mature in that space. Sometimes I even get to polish those files up and actually publish them.
1. It is still alive
2. It is on GitHub so I can easily read the source and look for issues
3. People star it
4. It has traction ie >10k downloads preferably >50k
5. It occupies a name that makes sense ie I’d rather trust using “redux-immutable” than “steel-porcupine”, since it unofficially lends itself to the “vetted one true solution”
6. Then I look at the source quickly and make sure it looks well done under the hood
7. Finally, if it’s particularly confusing, I might run a quick google check to see if there’s blogs about it on the internet explaining how to use the advanced features and check the date on the blogs (1-2 years ago = good, it’s mature)
If any of the above are too dicey and there’s no good options, I put the relevant code in a top level folder called “modules” (or otherwise) with the intention that in the future I might get to delete it if the package ecosystem gets more mature in that space. Sometimes I even get to polish those files up and actually publish them.
From a decade of software engineering and architecture I'd say these are some excellent guidelines for when to choose a library. Let me follow up with a collection of the cases where I'd absolutely stay away from libraries:
1. Your standard library can do it. Whatever you do, RTFM, learn your language inside out and prefer the standard library for whatever you do, even if it looks a little worse — especially if it's not more than 50% more code than using a library for convencience functions. This already goes 80% of the way to prevent "left-pad"-like disasters.
2. Everything that is your core business logic. Don't succumb to the fascination of rule engines and state machine builders. Just write that damn code yourself, future you will be way happier for it.
3. If you're sure that you'll never need more than 10-20% of a library, but eventually will need something considerably different starting from there. This may be grounds for justifying building your own one. If you do that and it doesn't give your business a unique edge, consider open sourcing it from the start.
4. If in doubt, rather stay away from big, fat frameworks until they give you a really big edge (like in Frontend development) or you tend to do a lot of rinse-and-repeat stuff. They tend to solve a lot of your problems, but rot fast and you have to continuously keep up with the Kardashians.
1. Your standard library can do it. Whatever you do, RTFM, learn your language inside out and prefer the standard library for whatever you do, even if it looks a little worse — especially if it's not more than 50% more code than using a library for convencience functions. This already goes 80% of the way to prevent "left-pad"-like disasters.
2. Everything that is your core business logic. Don't succumb to the fascination of rule engines and state machine builders. Just write that damn code yourself, future you will be way happier for it.
3. If you're sure that you'll never need more than 10-20% of a library, but eventually will need something considerably different starting from there. This may be grounds for justifying building your own one. If you do that and it doesn't give your business a unique edge, consider open sourcing it from the start.
4. If in doubt, rather stay away from big, fat frameworks until they give you a really big edge (like in Frontend development) or you tend to do a lot of rinse-and-repeat stuff. They tend to solve a lot of your problems, but rot fast and you have to continuously keep up with the Kardashians.
Regarding (1), can't say I agree; by all means, know your stdlib well, but if you still wonder if you need a library, and there's one with a lot of traction, I'd just use it.
left-pad is a red herring - the problem there was poor dependency management, not the use of the library itself.
left-pad is a red herring - the problem there was poor dependency management, not the use of the library itself.
Considering traction makes sense, but also remember that every popular library had to start somewhere. If there are no red flags (like a backlog of issues or lack of documentation) and it looks like it could solve your problem, don't be afraid have a look through the code and verify yourself that it's well built, then become an early adopter. I've come across plenty of 10-star libs on Github that solve problems extremely well.
> 3. People star it
> 4. It has traction ie >10k downloads preferably >50k
Why ? Your home-made code probably don't check those boxes.
> 4. It has traction ie >10k downloads preferably >50k
Why ? Your home-made code probably don't check those boxes.
I agree with 3, not necessarily 4.
The amount of people who uses a product does not determine is quality.
However stars can determine to an extend whether it's a project worth looking at or not.
The amount of people who uses a product does not determine is quality.
However stars can determine to an extend whether it's a project worth looking at or not.
My home-made code will probably just do the specific thing I need it to do, and not all of the other things the library has been generically designed for.
Because an Unmaintained library is worse than one you write yourself because bugs are much harder to understand and fix.
You imply that you're still going to understand your code one year later. While this is a favorite fantasy of all programmers, it's usually false. Also, if you're in a team, the person running into the bug is likely not you, so it's foreign code for them anyway.
I was being brief because I was leaving work so I’ll expand a bit.
An unmaintained library is usually more difficult to fix because fixes are no longer committed back or released. At that point you can choose to vendor and maintain your own copy, or replace it with something else. And at either of those points you’re no longer in context.
My point wasn’t really geared at debugging a piece of code. A library, my own, or a coworkers are roughly comparable. The cost to fix is a lot higher in a library though.
It’s not a universal rule, despite the way I originally phrased it.
An unmaintained library is usually more difficult to fix because fixes are no longer committed back or released. At that point you can choose to vendor and maintain your own copy, or replace it with something else. And at either of those points you’re no longer in context.
My point wasn’t really geared at debugging a piece of code. A library, my own, or a coworkers are roughly comparable. The cost to fix is a lot higher in a library though.
It’s not a universal rule, despite the way I originally phrased it.
I think there is also certain negative selection bias in the sense that people are more likely to stop maintaining libraries that were badly designed / problematic to begin with. That's why I tend to be more sceptical than average when it comes to "orphaned" projects.
I've had many a struggle with this.
On the flip side, I have found libraries that are unmaintained and are pleasant enough to understand. I've forked those and used them, happy that:
1. I can quickly fix any bugs that I face
2. Someone else can make use of my maintenance.
On the flip side, I have found libraries that are unmaintained and are pleasant enough to understand. I've forked those and used them, happy that:
1. I can quickly fix any bugs that I face
2. Someone else can make use of my maintenance.
> 1. It is still alive
A mature project that just works does not need new commits.
A mature project that just works does not need new commits.
You can determine that by ie looking at the issues list and eyeballing whether there are pages of issues with no comments on them.
Not necessarily whether there are commits or not.
Not necessarily whether there are commits or not.
[deleted]
True, a project with a bunch of stars and ownloads and whatnot and no open issues and pull requests can be perfectly fine even if the last commit was a few years ago. However, if the last commit was years ago, and issues and pull requests have been piling up without getting any attention, that's probably a bad sign.
That's true, but whether a project is alive or not goes beyond whether people are committing code. E.g.,
- If you file a bug report does the bug get fixed? Do you even get a response?
- If you ask for help from the community, do you get it?
- If you file a bug report does the bug get fixed? Do you even get a response?
- If you ask for help from the community, do you get it?
Use a library when:
1. It would take a significant time (> a few days) to write yourself.
2. The library is widely used.
3. The library has a history of supporting major versions, you don't want breaking changes to come with security and minor fixes.
4. The library is open source. Not always a complete deal breaker, but you need to think of long term costs.
5. You're going to use a significant chunk of the library, don't import a big complicated library for a single function.
1. It would take a significant time (> a few days) to write yourself.
2. The library is widely used.
3. The library has a history of supporting major versions, you don't want breaking changes to come with security and minor fixes.
4. The library is open source. Not always a complete deal breaker, but you need to think of long term costs.
5. You're going to use a significant chunk of the library, don't import a big complicated library for a single function.
When the task at hand requires more than an hour or so of work.
But that is not the rule i work after, It just seams like that is how i work when i work now that i think about it.
But that is not the rule i work after, It just seams like that is how i work when i work now that i think about it.
Answering such questions is probably 50% of the job for a senior-level software engineer. And it's 99% context-dependent.
At another end are things that you should only be implementing yourself in highly unusual circumstances, such as cryptography libraries.
Where on this spectrum do you usually implement the thing yourself, and what is your thought process?
What are the costs of importing a library, and how are those costs different for different projects? Are there security risks? What are the benefits?
Conversely, what are the costs and benefits of implementing something in-house?
Has there ever been a case where you've tried to roll your own and it ended in disaster? Or where you'd wished you'd never added a dependency on some library?
[1] https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/