Ask HN: What is the best code base you ever worked on?
468 comments
The one in my previous job, which was an admin board for a market intelligence application. Ultimately, the reason it was good was because the engineers had zero ego on top of having excellent skills. The team that set the codebase were basically 4 seniors and 3 principals (the client actually did pay for top talent in this case) so not only everything was based on industry standards, written elegantly and organized perfectly, but every time some new requirement came up, these senior / principal engineers would discuss it in the most civilized matter I have ever seen.
E.g, "we need to come up with a way to implement X". Person A gives their idea, person B gives another idea and so on until everybody shared their thoughts. Then someone would say "I think what person C said makes the most sense" and everybody would agree and that was it. 30 minutes to hear everybody out, 3 minutes to discuss who will do it and when and the meeting was over.
I think the biggest testament to this code base was that when junior members joined the team, they were able to follow the existing code for adding new features. It was that easy to navigate and understand the big picture of.
E.g, "we need to come up with a way to implement X". Person A gives their idea, person B gives another idea and so on until everybody shared their thoughts. Then someone would say "I think what person C said makes the most sense" and everybody would agree and that was it. 30 minutes to hear everybody out, 3 minutes to discuss who will do it and when and the meeting was over.
I think the biggest testament to this code base was that when junior members joined the team, they were able to follow the existing code for adding new features. It was that easy to navigate and understand the big picture of.
This is the approach I tried to take as an IC frontend on a building management system dashboard about a year ago. It was basically me and one other frontend, one full-stack, and a QA or 2, plus a manager. My manager I guess had written a lot of the code we'd be interacting with, and I guess was somewhat protective and reluctant to delegate authority over decisions around it. It was a big refactoring project, and I just encouraged my colleagues to take the things they wanted, like rebuilding the state management system. We'd discuss the requirements and why it was necessary, and I'd look for reasons to agree rather than disagree, then we'd be off. Something burnout has taught me is that the marginal differences between one implementation detail or another are not worth getting hung up on unless they pose a real problem, especially when someone else gets to decide that doing it fast is priority (sprints).
> Something burnout has taught me is that the marginal differences between one implementation detail or another are not worth getting hung up on unless they pose a real problem, especially when someone else gets to decide that doing it fast is priority (sprints).
I think this is true. And a self-imposed problem (should a problem arise) is much less frustrating to fix than one that came from a decision imposed by someone else, even if the latter avoided loads of other problems. Sometimes it's better to let people make mistakes (as you believe them to be) and correct them later.
I think this is true. And a self-imposed problem (should a problem arise) is much less frustrating to fix than one that came from a decision imposed by someone else, even if the latter avoided loads of other problems. Sometimes it's better to let people make mistakes (as you believe them to be) and correct them later.
And often, the cost of indecision is much bigger than the margin in utility between the optimal and the second-to-optimal solution.
Yep. I am bad at acting on this true statement.
Otherwise known as "you are not your code" and "you are not your ideas." Essentially, criticism against your code is not a personal attack (and other corollaries). Its definitely something I aspire to, but its surprisingly hard to get right.
Great example as it shows that code base really is a side consideration and a proxy to team quality.
Another reminder that soft skills multiply the effectiveness of hard skills.
Exactly.
What is development if not communication? Between the clients, the coworkers, the users, and the computer
I'd take a mediocre dev with no ego and ok social skills on my project any day over an egotistical genius who can't work with other people or effectively communicate their ideas
What is development if not communication? Between the clients, the coworkers, the users, and the computer
I'd take a mediocre dev with no ego and ok social skills on my project any day over an egotistical genius who can't work with other people or effectively communicate their ideas
AOL's server code was excellent. It was written by some people that had done similar things for BITNET and had a deep understanding of Unix programming and how to use event loops for scalability.
It was in C, but the process was expected to run for months without crashing or running out of memory; if it had an abnormal exit, you'd (the owner as set in some code thing) get an email with the core backtrace. If it had a memory leak, ops would be on you quickly. Ops was considered a primary driver of requirements to dev, and absolutely everything could be reloaded into a running server without restarting it. There was a TCP control port where a TCL interpreter inside the service was exposed and generally you wrote the (very simple, CS 101 style) TCL commands to manage the server. It was a "No Threads Kernel", scaled to the dozens or hundreds of physical machines communicating over a very well managed network, and most 1 process per core, and 1 core for the OS. The 200 or so unix developers (as we were called) had a common understanding of how the framework worked and if you were just writing app code it was basically impossible to write slow services. We had technical writers that would interview the developers and write books that could be handed to outside developers and lead to a successful integration with no developer time spent.
The NTK was primarily for sending msgs over the network - we had a principle to never write to disk (which were pretty slow in the 1990s), so everything was just a server to get network messages and then send out other messages in reponse and then assemble the replies/timeouts and send back a msg to the caller. All done over persistent connections established by the infrastructure, the applications just registered callbacks for msg type 'X' which would present one with the caller information, the msg as a buffer, and potentially a "user word" which would be a way to keep server state around between calls.
The layering, from the main select loop thru different layers of TCP connection, SSL handling (if used, not 100% was SSL in the 90s), thru persistent link handling, application msg handling, timers, memory allocation, etc. was done with such art that I felt it to be a thing of beauty.
It was in C, but the process was expected to run for months without crashing or running out of memory; if it had an abnormal exit, you'd (the owner as set in some code thing) get an email with the core backtrace. If it had a memory leak, ops would be on you quickly. Ops was considered a primary driver of requirements to dev, and absolutely everything could be reloaded into a running server without restarting it. There was a TCP control port where a TCL interpreter inside the service was exposed and generally you wrote the (very simple, CS 101 style) TCL commands to manage the server. It was a "No Threads Kernel", scaled to the dozens or hundreds of physical machines communicating over a very well managed network, and most 1 process per core, and 1 core for the OS. The 200 or so unix developers (as we were called) had a common understanding of how the framework worked and if you were just writing app code it was basically impossible to write slow services. We had technical writers that would interview the developers and write books that could be handed to outside developers and lead to a successful integration with no developer time spent.
The NTK was primarily for sending msgs over the network - we had a principle to never write to disk (which were pretty slow in the 1990s), so everything was just a server to get network messages and then send out other messages in reponse and then assemble the replies/timeouts and send back a msg to the caller. All done over persistent connections established by the infrastructure, the applications just registered callbacks for msg type 'X' which would present one with the caller information, the msg as a buffer, and potentially a "user word" which would be a way to keep server state around between calls.
The layering, from the main select loop thru different layers of TCP connection, SSL handling (if used, not 100% was SSL in the 90s), thru persistent link handling, application msg handling, timers, memory allocation, etc. was done with such art that I felt it to be a thing of beauty.
Is this it?
https://aolserver.github.io/
https://aolserver.github.io/
No that is a webserver that AOL bought. It used TCL but also threads.
Wild seeing "19 years ago" as the last commit time.
https://github.com/aolserver/aolserver
https://github.com/aolserver/aolserver
Can’t be the same thing GP is talking about as the repo says it’s multithreaded and the poster said what they worked on wasn’t
Might have been updated later on.
Sadly, it seems like the moral of this account is that engineering quality is orthogonal to business success.
Good engineering quality can't save a business, but the reverse isn't true, we just usually don't attribute the failure directly to the software. Instead we would say things like "the company ran out of money" - even if the underlying reason is that engineering was expensive with low velocity. Or that a disruptive competitor brought the company down, even if this could have been prevented if the architecture had permitted eg. relatively easily updating to a more modern UI.
I wouldn't say it was a failure. They had a good run in a very quickly evolving time. That they got a slice of business for years during that period isn't bad!
When did they fail? They acquired Time Warner, creating a company that after continued consolidation is now ranked #13 on the Fortune 500 list.
[deleted]
AOL was an incredibly successful business. Nothing lasts forever.
The creative destruction of the business cycle grinds ever onward.
AOL was extremely successful in its day.
They had 40 million customers paying $20/month 30 years ago.
They had 40 million customers paying $20/month 30 years ago.
Orthogonal implies independence, not inverse correlation (which is what I'm guessing you meant).
I bet they meant what they wrote, since it makes more sense than what you're suggesting they meant.
Er, what? AOL was one of the most successful companies… ever. Things moved on, but this is like saying the roman empire was never successful.
AOL's primary business was dialup modems. Those died off after internet became ubiquitous.
They pre-dated Yahoo and Google as an index of the internet.
They pre-dated Yahoo and Google as an index of the internet.
Fascinating, what kind of hardware/OS was used in production?
Started with Tandem and Stratus pre Unix then migrated to HPUX then Solaris then Linux on stock hardware. Also Digital for big machines and early usage of 64 bit. I ported to Mac OSX but only for local dev.
Funny thing about HPUX to Solaris migration is we found a ton of null pointer dereferences which didn't cause SEGV on HPUX but did on Solaris. And Linux migration was to a different endianness so had to fix any copying to/from ints to a buffer.
Funny thing about HPUX to Solaris migration is we found a ton of null pointer dereferences which didn't cause SEGV on HPUX but did on Solaris. And Linux migration was to a different endianness so had to fix any copying to/from ints to a buffer.
I worked at a bank where we used Tandem NonStop on Guardian OS - loved the programming environment using TAL language and the built-in SQL database.
BITNET the IBM academic network? I thought that was mostly IBM mainframes (Not a lot of Unix). A lot of US universities had a mix so I guess people would have exposure to C/Unix.
For me, the most eye-opening codebase of my career was Cocotron, around 2007:
https://github.com/cjwl/cocotron
I was looking for a way to port my native Mac Cocoa apps to Windows. I had been already disappointed by the aimless sprawl of GNUstep.
This one-person project implemented all the essential APIs for both Foundation and AppKit. Reading the code was a revelation: can it really be this simple and at the same time this effortlessly modular for cross-platform support?
I contributed a few missing classes, and successfully used Cocotron for some complex custom GUI apps that needed the dual-platform support.
Cocotron showed me that one person with a vision can build something that will rival or even outgun large teams at the big tech companies. But focus is essential. Architecture astronauts usually never get down from their high orbits to ship something.
https://github.com/cjwl/cocotron
I was looking for a way to port my native Mac Cocoa apps to Windows. I had been already disappointed by the aimless sprawl of GNUstep.
This one-person project implemented all the essential APIs for both Foundation and AppKit. Reading the code was a revelation: can it really be this simple and at the same time this effortlessly modular for cross-platform support?
I contributed a few missing classes, and successfully used Cocotron for some complex custom GUI apps that needed the dual-platform support.
Cocotron showed me that one person with a vision can build something that will rival or even outgun large teams at the big tech companies. But focus is essential. Architecture astronauts usually never get down from their high orbits to ship something.
+1 to this.
I remember my boss asking me to build a MacOS UI we had on Windows with this. I was very sceptical that it would go anywhere.
But no, it worked. I was just open mouth shocked to see the GUI worked pretty flawlessly in Windows.
I remember my boss asking me to build a MacOS UI we had on Windows with this. I was very sceptical that it would go anywhere.
But no, it worked. I was just open mouth shocked to see the GUI worked pretty flawlessly in Windows.
:)
Thank you!
In my case, it was the integration testing framework built for a large Python service.
This was ~10y ago, so my memory might not serve me well. A bit of context:
- proprietary service, written in Python, maaany KLOC,
- hundreds of engineers worked on it,
- before this framework, writing the integration tests was difficult -- you had a base framework, but the tests had no structure, everyone rolled out their own complicated way of wiring things -- very convoluted and flaky.
The new integration tests framework was build by a recently joined senior engineer. TBF, it's wrong to say that it's was a framework, if you think in the xUnit sense. This guy built a set of business components that you could connect & combine in a sound way to build your integration test. Doesn't sound like much, but it significantly simplified writing integration tests (it still had rough edges, but it was 10x improvement). It's rare to see the chaos being tamed in such elegant way.
What this guy did:
- built on top of the existing integration tests framework (didn't rollout something from zero),
- defined a clear semantic for the test components,
- built the initial set of the test components,
- held a strong ownership over the code -- through the code review he ensured that the new components follow semantics, and that each test component is covered by its own test (yep, tests for the test doubles, you don't see that very often).
Did it work well longterm? Unfortunately, no. He stayed relatively short (<2y). His framework deteriorated under the new ownership.
Travis, if you are reading this and you recognized yourself, thank you for your work!
This was ~10y ago, so my memory might not serve me well. A bit of context:
- proprietary service, written in Python, maaany KLOC,
- hundreds of engineers worked on it,
- before this framework, writing the integration tests was difficult -- you had a base framework, but the tests had no structure, everyone rolled out their own complicated way of wiring things -- very convoluted and flaky.
The new integration tests framework was build by a recently joined senior engineer. TBF, it's wrong to say that it's was a framework, if you think in the xUnit sense. This guy built a set of business components that you could connect & combine in a sound way to build your integration test. Doesn't sound like much, but it significantly simplified writing integration tests (it still had rough edges, but it was 10x improvement). It's rare to see the chaos being tamed in such elegant way.
What this guy did:
- built on top of the existing integration tests framework (didn't rollout something from zero),
- defined a clear semantic for the test components,
- built the initial set of the test components,
- held a strong ownership over the code -- through the code review he ensured that the new components follow semantics, and that each test component is covered by its own test (yep, tests for the test doubles, you don't see that very often).
Did it work well longterm? Unfortunately, no. He stayed relatively short (<2y). His framework deteriorated under the new ownership.
Travis, if you are reading this and you recognized yourself, thank you for your work!
I wonder if this is a tragedy of "the org wasn't ready for his solution" or more of a "if I were to go back now, I'd notice it wasn't so good."
Without knowing anything, and before reading this comment, I had the feeling of nr. 2. Strong feeling. I think the reason is „many klocs of Python“. I have developed alergy to big Python programs. I like python for small things, probably wrapping some C code.
What languages do you personally prefer for many KLOCs of code?
Why would KLOC in Python be worse than in any other language?
It's about the different ways the language allows you to shoot yourself in the foot.
I worked on large Python, C++, Java & Go services. I have 10y+ of experience with the first 3. C++ allows you to write incomprehensible code (even to the experienced C++ devs) and justify its existence (because of the performance gains). But you need to be a top expert to write a compileable code of that type. I'm comfortable with diving in any C++ codebase except for the libraries like std, boost, abseil, folly, etc. Most of the code there is absurdly difficult to comprehend.
On the other hand Python leads in the ways a junior dev can introduce hell in the code. Especially if the team doesn't rely on the strict type check. I have seen horrors.
I was bewildered when I realized that working with JavaScript with type checks (Closure compiler) was insanely more productive and smooth than working with Python (before the type checks).
That's why Java won the enterprise world. It takes an effort to make a mess in Java (but people still manage). Go is in a similar place.
I worked on large Python, C++, Java & Go services. I have 10y+ of experience with the first 3. C++ allows you to write incomprehensible code (even to the experienced C++ devs) and justify its existence (because of the performance gains). But you need to be a top expert to write a compileable code of that type. I'm comfortable with diving in any C++ codebase except for the libraries like std, boost, abseil, folly, etc. Most of the code there is absurdly difficult to comprehend.
On the other hand Python leads in the ways a junior dev can introduce hell in the code. Especially if the team doesn't rely on the strict type check. I have seen horrors.
I was bewildered when I realized that working with JavaScript with type checks (Closure compiler) was insanely more productive and smooth than working with Python (before the type checks).
That's why Java won the enterprise world. It takes an effort to make a mess in Java (but people still manage). Go is in a similar place.
Because you're doing more work per line.
A 20 KLOC Python program is not more complicated than the 100 KLOC C program performing the same function. Quite the contrary. But if you're comparing 100 KLOC of C to 100 KLOC of Python, the Python program may seem unwieldy.
A 20 KLOC Python program is not more complicated than the 100 KLOC C program performing the same function. Quite the contrary. But if you're comparing 100 KLOC of C to 100 KLOC of Python, the Python program may seem unwieldy.
Why only those two options? It could be just "the company had other priorities and developers, while they appreciated Travis's work, didn't find it worthwhile to carry the torch themselves". Having a great testing framework is just one of the things that need devs' attention.
Fair point, it's a false dichotomy; though your example is close to "the org wasn't ready for it," which is itself vague.
Maybe the issue is that it easily comes apart without a dedicated censor, so to speak, and nobody wanted to have that role.
Maybe the issue is that it easily comes apart without a dedicated censor, so to speak, and nobody wanted to have that role.
> Did it work well longterm? Unfortunately, no. He stayed relatively short (<2y). His framework deteriorated under the new ownership.
I think the issue is that integ tests are not really a place that sees "development". You write the integ tests, and move on, you are not actively introducing more integ tests.
I think that's a shame that this is the common view of the people that need to fund this. Bad integ tests mean bad dev exp, which then results in increased attrition and dissatisfaction.
I think the issue is that integ tests are not really a place that sees "development". You write the integ tests, and move on, you are not actively introducing more integ tests.
I think that's a shame that this is the common view of the people that need to fund this. Bad integ tests mean bad dev exp, which then results in increased attrition and dissatisfaction.
Worked on a codebase for a large safety-critical system where everything was 100% documented, and the development guide for the project was followed so closely that you couldn't tell, across millions of lines of code, that the whole thing wasn't written by one person. Absolutely impressive levels of attention to detail everywhere, down to not even being able to find typographical errors in comments or documentation (a typo in a comment was treated just as seriously as any other bug).
Let me guess, it was very well funded and there were no fake deadlines and cross-team dependencies, am I correct or am I very correct?
Not only was the funding effectively unlimited, but it was developed over 20+ years (so the whole program was designed to the standard that "the next person touching this code will never have touched this system before, and the last person who touched it isn't available to ask, and it still must be correct in the face of such change."
Nothing moved very quickly, sometimes a change request took a year from initial writeup to implementation and close-out. It wasn't creative, but it worked. Very well.
Nothing moved very quickly, sometimes a change request took a year from initial writeup to implementation and close-out. It wasn't creative, but it worked. Very well.
Cancel the rocket launch people, we have a semantic incongruity! Correct is a binary state.
"When people thought the Earth was flat, they were wrong. When people thought the Earth was spherical, they were wrong. But if you think that thinking the Earth is spherical is just as wrong as thinking the Earth is flat, then your view is wronger than both of them put together."
If you allow for composable statements, the whole can be half correct.
Unless you're working with Quantum Correctness.
I bet you are very correct
I am almost sure this is because the system would have to pass a certification procedure somewhere, and for that they would need this level of clarity. Am I right?
Having worked in a couple of safety critical companies, there are things you definitely HAVE to do, but some companies do it better than others.
Some companies have process and development practices down pat: things go smoothly, and meeting the qualification process objectives is easy, because the work has been done right the whole way.
Other companies have less established or less consistent process. They generally meet the process objectives and deliver a working product, but the development process is often more of a struggle, and there is often a lot of "cleaning up" of missed pieces at the end of the release process.
This is just to say: companies and products in the safety critical space don't necessarily have some intrinsic quality, just a higher minimum bar.
Some companies have process and development practices down pat: things go smoothly, and meeting the qualification process objectives is easy, because the work has been done right the whole way.
Other companies have less established or less consistent process. They generally meet the process objectives and deliver a working product, but the development process is often more of a struggle, and there is often a lot of "cleaning up" of missed pieces at the end of the release process.
This is just to say: companies and products in the safety critical space don't necessarily have some intrinsic quality, just a higher minimum bar.
In my experience there is exactly 0 correlation between certification needed and code Quality. Right now working in a multi billion company doing SW that must pass many certifications. The code is absolut trash. The tooling is terrible. The people confuse make and cmake. All said. And the SW gets certified, because it is all a matter of how much it costs to certify. It is a kind of high level corruption, that is not seen as corruption.
In my experience there is a correlation. All software I tested that had to be certified had the lowest quality by far. Certification only cared for the happy path and lots of very specific error messages. It was extremely important that the correct error codes were returned in the specified cases. Anything unexpected like e.g. an out-of-memory-error was more or less ignored because the specification didn't consider such kind of error. Even if it made the software unusable - all that mattered was the certification.
"Cobra Effect"
https://en.m.wikipedia.org/wiki/Perverse_incentive
https://en.m.wikipedia.org/wiki/Perverse_incentive
Sorry, I was talking about safety critical system certifications, like for avionics. The systems you describe would never pass that for sure.
> The systems you describe would never pass that for sure.
I wouldn't assume that.
I wouldn't assume that.
Right? Boeing would like to have a word.
I'd love to know if working like this is enjoyable or a chore.
Having worked in both kinds personally, it's different.
You spend more time on less topics: the variety of work is much lower, which some people love, and some people are frustrated by. There is often much less "drastic innovation", or when there is, it takes much longer than you might be used to in other industries. Favor is given to proven, predictable technologies and choices, even if that leaves some opportunities on the table.
That being said, it's something I often miss. The ability to have such solid confidence on the things you've built, and the ability to drop into nearly any piece of the system and have everything be consistent and predictable is a quality in itself. It makes debugging (often VERY rare) issues much more tractable, both at a higher systems level, as well as digging deeper into the code itself.
You spend more time on less topics: the variety of work is much lower, which some people love, and some people are frustrated by. There is often much less "drastic innovation", or when there is, it takes much longer than you might be used to in other industries. Favor is given to proven, predictable technologies and choices, even if that leaves some opportunities on the table.
That being said, it's something I often miss. The ability to have such solid confidence on the things you've built, and the ability to drop into nearly any piece of the system and have everything be consistent and predictable is a quality in itself. It makes debugging (often VERY rare) issues much more tractable, both at a higher systems level, as well as digging deeper into the code itself.
> Favor is given to proven, predictable technologies and choices
I feel like this goes two ways too. Sometimes people favor technologies they’ve used before, regardless of how many problems they know it causes.
If it’s predictable that certain tech will cause you issues years down the line, do not choose it again.
I feel like this goes two ways too. Sometimes people favor technologies they’ve used before, regardless of how many problems they know it causes.
If it’s predictable that certain tech will cause you issues years down the line, do not choose it again.
Yep. Sometimes it's "known issues are avoidable issues", and other times it's "hey I wish we didn't spend xx% of our time avoiding (incl. spending time auditing syntax) or doing mitigations for the same limitations/issues over and over and over again".
The wheels do turn, just slowly.
The wheels do turn, just slowly.
Sounds like a well maintained tractor from 1950s that is still chugging along.
After way too many years of working on several sprawling Ruby and Rails projects, I would enjoy the everloving fuck out of working on this.
If lives are at stake, it would probably elicit different emotions altogether.
NetBSD.
The docco, the culture, the clarity and simplicity of design from first principles. The coherency of code across kernel, user-space, accompagnying material. The vibe.
You may know NetBSD (if, at all) as the BSD that wants to be "portable" and may dismiss it against FreeBSD's focus-claim of "performance" and OpenBSD's focus-claim of "security". Interestingly, trying to remain wildly portable requires a cleanliness inside that is very soothing.
The docco, the culture, the clarity and simplicity of design from first principles. The coherency of code across kernel, user-space, accompagnying material. The vibe.
You may know NetBSD (if, at all) as the BSD that wants to be "portable" and may dismiss it against FreeBSD's focus-claim of "performance" and OpenBSD's focus-claim of "security". Interestingly, trying to remain wildly portable requires a cleanliness inside that is very soothing.
Question, I want to get into more lower level programs like compositors or docker daemon or kernels (I do realize they all are different things) but I find tech behind them so intimidating and don't know where to get started. Do you have any advice where I can learn them? I'm about to grad next year
Last year I worked for a client that gave me a lot of time, money and autonomy to lead dev on a critical software rewrite.
We got a small team of competent people, with domain experts to peer code with the devs.
It was wonderful. We could test, document and clean up. Having people who knew the trade and users at hand removed second guessing.
The result was so good we found bugs even in competitors' implementations.
We also got x5 in perfs compared to the system it was replacing and more features.
We got a small team of competent people, with domain experts to peer code with the devs.
It was wonderful. We could test, document and clean up. Having people who knew the trade and users at hand removed second guessing.
The result was so good we found bugs even in competitors' implementations.
We also got x5 in perfs compared to the system it was replacing and more features.
Similar thing.
Had time and autonomy from a client, so took sweet time examining the domain, the existing systems et al. Spent a few months writing the basis and the framework around what will be done, based on years and years of experience I had with bad frameworks and codebases, combined with working on the same domain for their parent company years ago.
And it worked. We delivered features insanely fast, hundreds of forms were migrated, feature generators would create 90% of the boilerplate code and the code was small, readable and neatly clustered. Maintaining it was a piece of cake, leading to us not having enough work after a while so I negotiated our time to half a week for the same money.
After a while, client deemed us too expensive to pay for only 2.5 days of work - after all, how does it make sense - if we are paying them that much, they should work 5 days!
So they cut us out. Two things happened:
1. Devs that got moved to other projects in the company told me they didn't know development could be so smooth and tried to replicate it in future projects, even tho they say it failed a lot of lessons they picked up from the framework were highly relevant in their future careers.
2. The company found a cheaper developer, he said "this is unusable and has to be rewritten" and rewrote it into "clean code", taking longer than the original project took. At least he works 5 days a week now.
Had time and autonomy from a client, so took sweet time examining the domain, the existing systems et al. Spent a few months writing the basis and the framework around what will be done, based on years and years of experience I had with bad frameworks and codebases, combined with working on the same domain for their parent company years ago.
And it worked. We delivered features insanely fast, hundreds of forms were migrated, feature generators would create 90% of the boilerplate code and the code was small, readable and neatly clustered. Maintaining it was a piece of cake, leading to us not having enough work after a while so I negotiated our time to half a week for the same money.
After a while, client deemed us too expensive to pay for only 2.5 days of work - after all, how does it make sense - if we are paying them that much, they should work 5 days!
So they cut us out. Two things happened:
1. Devs that got moved to other projects in the company told me they didn't know development could be so smooth and tried to replicate it in future projects, even tho they say it failed a lot of lessons they picked up from the framework were highly relevant in their future careers.
2. The company found a cheaper developer, he said "this is unusable and has to be rewritten" and rewrote it into "clean code", taking longer than the original project took. At least he works 5 days a week now.
We were going to do this. Get several months to set up a clean base for our new system with only the four most competent people working on it.
Then I went on vacation for a week.
Come back to a system that needs to be delivered to prod next month with 20 randos submitting PR’s…
WTF happened. Did you learn nothing from previous failures?
Then I went on vacation for a week.
Come back to a system that needs to be delivered to prod next month with 20 randos submitting PR’s…
WTF happened. Did you learn nothing from previous failures?
What was the lesson they where supposed to have learned?
9 women cannot have a baby in a month.
There’s a limited amount of people that can work on a system at the start of a project if you want it to be a coherent whole, so that you can have everyone iterate on it in the same style later. If you start out with 20 (junior) engineers, you get a kind of frankenstein where all issues are waved away because ‘we need to deliver in a week’
There’s a limited amount of people that can work on a system at the start of a project if you want it to be a coherent whole, so that you can have everyone iterate on it in the same style later. If you start out with 20 (junior) engineers, you get a kind of frankenstein where all issues are waved away because ‘we need to deliver in a week’
Aka "Brooks's Law" or more broadly, the idea that "adding manpower to a late software project makes it later" from Brook's "Mythical Man-Month"
Good for you. Magical moments in careers are hard to find in my experience but they are so satisfying when you get there.
Glad whomever was over this didnt just drop the "dont rewrite" joel spolsky article and fight making it happen.
Glad whomever was over this didnt just drop the "dont rewrite" joel spolsky article and fight making it happen.
I actually was the one telling them not to rewrite, lol.
But the original code was a mess of matlab spaghetti, they couldn't find a way to hire for that. Not to mention turning it into a web service was already a big hack of java parsing a raw dump of matlab datastructures that nobody dared to touch.
I had to read the matlab code, and it tooks hours to decypher a few lines. Plus the language doesn't have great debugging and error handling capabilities and the tooling is quite terrible.
So rewriting to python won, and for once, I must say it was a good call.
But the original code was a mess of matlab spaghetti, they couldn't find a way to hire for that. Not to mention turning it into a web service was already a big hack of java parsing a raw dump of matlab datastructures that nobody dared to touch.
I had to read the matlab code, and it tooks hours to decypher a few lines. Plus the language doesn't have great debugging and error handling capabilities and the tooling is quite terrible.
So rewriting to python won, and for once, I must say it was a good call.
Joel has since said that that he doesn't really agree with that advice anymore, at least not in the same way. Super annoying that it gets parroted over and over again as though it's the word of the lord.
I agree it shouldn't be parroted as though it's the word of the lord, like any advice it will be more or less applicable on specific situations. I've been on both sides of the let's redesign a thing plenty of times.
The balance is somewhere in the middle, it's valuable advice for what can go wrong when you don't think through the implications of the decision making process, understand how and why the system works the way it does, and what risks exist for what can go wrong with a big redesign. But like anything, if the risks are understood, then those risks can be accepted, mitigated, or rejected if appropriate, or provide guidance on why the redesign investment isn't worth the associated risks.
The balance is somewhere in the middle, it's valuable advice for what can go wrong when you don't think through the implications of the decision making process, understand how and why the system works the way it does, and what risks exist for what can go wrong with a big redesign. But like anything, if the risks are understood, then those risks can be accepted, mitigated, or rejected if appropriate, or provide guidance on why the redesign investment isn't worth the associated risks.
Google3 codebase. It's just so vast and it works. It's a miracle. I feel lucky to have seen it. Everytime you change it, it reruns the dependencies. Everyone has different view concurrently. Commits are efficient immutable snapshots. It's just incredible multiplayer. So massively beyond what can be done with GitHub. Really I feel it's peak codebase. I've not seen what the other big techs do but it blew my mind
+1
I can not understate how much I agree with parent comment.
The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.
The result is a thing of immense beauty and elegance.
I will forever be grateful for the opportunity I had to see this magnificent piece of engineering in action.
I can not understate how much I agree with parent comment.
The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.
The result is a thing of immense beauty and elegance.
I will forever be grateful for the opportunity I had to see this magnificent piece of engineering in action.
I dunno man, I work at google and this comment feels like circlejerk. It's not held to THAT high a standard. We absolutely commit janky MVPs and iterate.
He is referring to the codebase itself, not individual projects or CLs.
He is referring to the codebase not the code?
What does that mean?
What does that mean?
I think they are referring to the developer experience of working in the codebase, rather than the quality of the code.
As a Google outsider, that's not at all the impression I got from the paragraphs.
This:
> It's not held to THAT high a standard. We absolutely commit janky MVPs and iterate.
seems to very directly address this:
> The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.
If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.
So, yanno, when folks on the ground report that individual projects and changesets/PRs/whatever ARE using the "Commit something barely serviceable, test it out, and iterate." process, statements like "We always get the design right before we write even a single line of code!" definitely come off as a circlejerk.
Google's a very large software house, it has a very exclusive hiring process, and it (reportedly) has internal tooling that's very well-adapted for the problems a company of its size faces. But Google is still hiring from the same pool of programmers as everyone else, and the odds that zero of those that it hires will work best with the "Get something out there to field-test, and use the test results to make it more suited for field use." method of development are absolutely zero. Given Google's size, the odds that zero of those will never be able to negotiate to work in the way they work best are ALSO zero.
(And -frankly- I expect that this method of development gets used a lot in the company. You can burn an assload of time on simulators (and what is the "What if?" game, but a in-brain simulator?), but in the realm of software, it's not-infrequently the case that the simulator with the best ROI is real-world deployment.)
This:
> It's not held to THAT high a standard. We absolutely commit janky MVPs and iterate.
seems to very directly address this:
> The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.
If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.
So, yanno, when folks on the ground report that individual projects and changesets/PRs/whatever ARE using the "Commit something barely serviceable, test it out, and iterate." process, statements like "We always get the design right before we write even a single line of code!" definitely come off as a circlejerk.
Google's a very large software house, it has a very exclusive hiring process, and it (reportedly) has internal tooling that's very well-adapted for the problems a company of its size faces. But Google is still hiring from the same pool of programmers as everyone else, and the odds that zero of those that it hires will work best with the "Get something out there to field-test, and use the test results to make it more suited for field use." method of development are absolutely zero. Given Google's size, the odds that zero of those will never be able to negotiate to work in the way they work best are ALSO zero.
(And -frankly- I expect that this method of development gets used a lot in the company. You can burn an assload of time on simulators (and what is the "What if?" game, but a in-brain simulator?), but in the realm of software, it's not-infrequently the case that the simulator with the best ROI is real-world deployment.)
While Google is a monorepo, that doesn't mean that every part of the codebase is equally critical.
People don't yolo submit CLs to core library components for example.
OTOH submitting somewhat hacky code to a new project while you iterate on it doesn't harm the rest of the code base that much since very little will depend on it.
People don't yolo submit CLs to core library components for example.
OTOH submitting somewhat hacky code to a new project while you iterate on it doesn't harm the rest of the code base that much since very little will depend on it.
Okay, sure.
How does this comment address any parts of my statement?
I'll even quote what might be the most important part for you:
> If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.
How does this comment address any parts of my statement?
I'll even quote what might be the most important part for you:
> If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.
I, like many others here, echo this sentiment. While I disliked working in ads, the experience of working with that repo+tooling is unmatched by anything else in my career.
If the tooling is so far beyond anything publicly available, why don't you guys make something like that and make millions?
I strongly suspect it’s not that useful for a lot of businesses.
So many have their code split between a two dozen clouds, BA tools (… does Google put that in the monorepo too? Or is that, which is a lot of the code at most businesses, not “really” code at Google?), vendor platforms, low-code tools you all hate and are trying to deprecate but god damned if you aren’t still spending dev hours on new features in it, et c…
I bet achieving anywhere near the full benefits would first require retooling one’s entire business, including processes, and bringing a whole shitload of farmed-out SaaS stuff in-house at enormous expense, most places.
So many have their code split between a two dozen clouds, BA tools (… does Google put that in the monorepo too? Or is that, which is a lot of the code at most businesses, not “really” code at Google?), vendor platforms, low-code tools you all hate and are trying to deprecate but god damned if you aren’t still spending dev hours on new features in it, et c…
I bet achieving anywhere near the full benefits would first require retooling one’s entire business, including processes, and bringing a whole shitload of farmed-out SaaS stuff in-house at enormous expense, most places.
It’s not just tooling; processes, infrastructure and dedicated teams for centralized infrastructure are what makes Google’s monorepo what it is. FWIW, most of the tools are publicly available or have good publicly available counterparts. What’s likely missing elsewhere is funding for infrastructure teams.
I’d love to see something like that applied to a project like Debian. The tools already exist. The cost of switching is too great, and you’d need everyone to learn your new system before you’d see the benefit.
I wrote a bit about this here: https://blog.williammanley.net/2020/05/25/unlock-software-fr...
I understand that nix have made some progress in this direction, but I don’t know any more than that.
I wrote a bit about this here: https://blog.williammanley.net/2020/05/25/unlock-software-fr...
I understand that nix have made some progress in this direction, but I don’t know any more than that.
Perhaps they make more than that just by using the tooling internally and creating/maintaining other stuff with it — that’s a major competitive advantage
millions sounds like a lot but it's like just 3 employees.
To quote broccoli man, "i've forgotten how to count that low" [1].
[1]: https://www.youtube.com/watch?v=3t6L-FlfeaI
To quote broccoli man, "i've forgotten how to count that low" [1].
[1]: https://www.youtube.com/watch?v=3t6L-FlfeaI
I think skill is also an issue here, in both directions. I have worked with a company that followed the opposite of move fast and it just turned into a 'who is the most correct' and 'what is the most elegant code' competition. We didn't push out a single new feature in the three years i'd worked there at that point. There was so much focus on code that we gave almost no time to business requirements.
The wider implication of this was that the number of tickets we got dropped dramatically, because users knew they'd never be resolved anyway.
Balance is key.
The wider implication of this was that the number of tickets we got dropped dramatically, because users knew they'd never be resolved anyway.
Balance is key.
Cannot overstate.
(Aside). To expand slightly, what robertsdionne is highlighting is the changing usage of this expression. In its original sense, e.g an issue is so important that it is impossible to overstate its importance. It is now increasingly used the other way around.
Old me would have said it’s used wrongly, but this happens all the time with language. Especially things being used in the opposite of their original sense, e.g. inflammable for flammable.
Old me would have said it’s used wrongly, but this happens all the time with language. Especially things being used in the opposite of their original sense, e.g. inflammable for flammable.
In my mind, "cannot overstate" always meant "impossible to overstate", but I think some people interpret/intend "cannot understate" to mean something like "must not understate". I don't know if that's really what they're thinking, but it is how I make sense of it. I have come to just avoid such constructions.
Edit: reminds me of an ancient SNL skit with Ed Asner in which he's a retiring nuclear engineer and as he heads out the door he says to his incompetent co-workers "Just remember, you can't put too much water in a nuclear reactor".
Edit: reminds me of an ancient SNL skit with Ed Asner in which he's a retiring nuclear engineer and as he heads out the door he says to his incompetent co-workers "Just remember, you can't put too much water in a nuclear reactor".
> opposite of their original sense, e.g. inflammable for flammable
Inflammable was never the opposite of flammable. Those word have always been synonyms. The opposite was always non-flammable.
Inflammable was never the opposite of flammable. Those word have always been synonyms. The opposite was always non-flammable.
Google3 codebase very consistently has clean code, but some of the architecture there is very much not great.
Some is great, some not so much.
Some of Verizon's code was much more elegant (though much smaller scope) from an API perspective, and really leaned into advanced type systems in a way Google has not.
Some is great, some not so much.
Some of Verizon's code was much more elegant (though much smaller scope) from an API perspective, and really leaned into advanced type systems in a way Google has not.
I agree but would add that within the codebase there are also vastly different experiences. The search ads serving stack for example is phenomenal, it is >10 years old, very complex and gets contributions from hundreds of engineers every week, yet it is quite easy to understand what is going on anywhere and it has great performance. But I also had to work with other code which had been through 3 rewrites, none of which had ever been totally completed, so there were overlapping paradigms everywhere and working with it was just a nightmare. Tests never caught anything leading to several post mortems.
It was wonderful to work with. :-) It's one of the things I truly miss from Google.
What is Google3?
I would assume Google's monorepo.
This link might help... https://dl.acm.org/doi/10.1145/2854146
The linked pdf has lots of details.
The linked pdf has lots of details.
HTML version in case anyone prefers that to PDF: https://cacm.acm.org/research/why-google-stores-billions-of-...
The replacement of google2
It was certainly great up to like 2015, perhaps later?
You could spend all day finding fun code to read.
I always like the main signal (eg sigsev) handler.
You could spend all day finding fun code to read.
I always like the main signal (eg sigsev) handler.
I honestly can't tell if this comment is sarcastic...
google3 is a quagmire and it's getting worse by the day
google3 is a quagmire and it's getting worse by the day
Well, first impressions:
It is part of our human condition to long for hard lines and clear concepts. When we have them we have to either face the fact that some realities elude them, or else bind ourselves to the inadequacy of the concepts.
Agreed. Once long (10+ years) ago, google3 may have been peak codebase. Nowadays, not so much.
Edit: I guess it also depends at what level of abstraction you work. High: can be easy breezy. Low: oh boy.
Edit: I guess it also depends at what level of abstraction you work. High: can be easy breezy. Low: oh boy.
CouchDB's HTTP API used to fit in just one file (less than 1k lines of Erlang), and it was a great jumping-off point for learning databases and web programming: https://github.com/apache/couchdb/blob/0.8.1/src/couchdb/cou...
It was written by one engineer, and then later refactored by a team to look like this (and many other files): https://github.com/apache/couchdb/blob/main/src/chttpd/src/c...
It's an interesting exercise to see how something grows and changes as it transitions from inspiration to real-world usage.
It was written by one engineer, and then later refactored by a team to look like this (and many other files): https://github.com/apache/couchdb/blob/main/src/chttpd/src/c...
It's an interesting exercise to see how something grows and changes as it transitions from inspiration to real-world usage.
So little blank lines, I need to breathe
One of them stands out, due to being super-productive, over years, and then decades.
A large system that was originally written by only two super-productive engineers (I mean real engineers, both with PhDs in an area of Engineering). And a comparably capable and essential IT person.
The reasons for the super-productivity include one of the developers choosing great technology and using it really well, to build a foundation with "force multiplier" effects, and the other developer able to build out bulk with that, while understanding the application domain.
Another reason was understanding and being pretty fully in control of the code base, so that, as needs grew and changed, over years, someone could figure out how to do whatever was needed.
One of the costs was that most things had to be built from scratch. Over time that also proved to be an advantage, because whenever they needed (put loosely) a "framework" to something it couldn't do, they effectively owned the framework, and could make dramatic changes.
When I said "costs", I mean things like, many times they needed to make a component from scratch that would be an off-the-shelf component in some other ecosystem. So if someone looked closely at how time was sometimes spent, without really understanding it or knowing how that panned out, it would look like a cost that they could optimize away. But if they looked at the bigger picture, they'd see a few people consistently, again and again, accomplishing what you'd think would take a lot more people to do.
It helped that the first programmer also became the director for that area of business, and made sure that smart engineering kept happening.
Someone might look for a reason this couldn't work, and think of bus factor. What I think helped there was the fact that the work involved one of those niche languages that attract way more super programmers than there are jobs. "Gosh, if only we had access to a secret hiring pool of super programmers who were capable of figuring out how to take up where the other person left off, and we had a way to get them to talk with us...")
It was easy to imagine a competitor with 100 developers, not able to keep up, and at many points getting stuck with a problem that none of them were able to solve.
A large system that was originally written by only two super-productive engineers (I mean real engineers, both with PhDs in an area of Engineering). And a comparably capable and essential IT person.
The reasons for the super-productivity include one of the developers choosing great technology and using it really well, to build a foundation with "force multiplier" effects, and the other developer able to build out bulk with that, while understanding the application domain.
Another reason was understanding and being pretty fully in control of the code base, so that, as needs grew and changed, over years, someone could figure out how to do whatever was needed.
One of the costs was that most things had to be built from scratch. Over time that also proved to be an advantage, because whenever they needed (put loosely) a "framework" to something it couldn't do, they effectively owned the framework, and could make dramatic changes.
When I said "costs", I mean things like, many times they needed to make a component from scratch that would be an off-the-shelf component in some other ecosystem. So if someone looked closely at how time was sometimes spent, without really understanding it or knowing how that panned out, it would look like a cost that they could optimize away. But if they looked at the bigger picture, they'd see a few people consistently, again and again, accomplishing what you'd think would take a lot more people to do.
It helped that the first programmer also became the director for that area of business, and made sure that smart engineering kept happening.
Someone might look for a reason this couldn't work, and think of bus factor. What I think helped there was the fact that the work involved one of those niche languages that attract way more super programmers than there are jobs. "Gosh, if only we had access to a secret hiring pool of super programmers who were capable of figuring out how to take up where the other person left off, and we had a way to get them to talk with us...")
It was easy to imagine a competitor with 100 developers, not able to keep up, and at many points getting stuck with a problem that none of them were able to solve.
I assume you avoided identifying (or even hinting at) this "great technology" on purpose, but could you persuaded to divulge what it was?
It started as a bespoke structured data and Web architecture, which did somewhat complex things that would've been a huge headache to build and maintain in the dominant languages of the time, but were viable for one person to figure out and implement in Scheme and PostgreSQL.
That bespoke architecture and implementation language lent itself to a lot of rapid domain-specific functionality, as well as doable architecture changes over time.
That bespoke architecture and implementation language lent itself to a lot of rapid domain-specific functionality, as well as doable architecture changes over time.
Thanks! :) I hoped it'd be a Lisp or Smalltalk - great to hear I almost guessed right, though I never imagined an RDBMS to be in the mix :)
There's actually a bespoke object metamodel mapping atop the RDBMS, and it permits customers who want to use SQL direct to the database to do so.
And, IMHO, PostgreSQL is the default choice for most storage backend purposes, unless one has figured out a good reason something else is better. :)
There were also multiple kinds of large blob storage needs, and changing architecture needs over time (e.g., multi-tenant, changing authn implications of things like browser plugins, scalability, move to cloud), so systems programming skills and controlling your framework comes in handy, for not getting stuck or blocking on vendors, but just solving the problem.
And, IMHO, PostgreSQL is the default choice for most storage backend purposes, unless one has figured out a good reason something else is better. :)
There were also multiple kinds of large blob storage needs, and changing architecture needs over time (e.g., multi-tenant, changing authn implications of things like browser plugins, scalability, move to cloud), so systems programming skills and controlling your framework comes in handy, for not getting stuck or blocking on vendors, but just solving the problem.
What is this niche language attracting super programmers?
Scheme, Common Lisp, Haskell, [edit] Smalltalk, probably more obscure ones...
Also, Erlang, Rust, and Clojure, though those have been rumored to be employable, so no longer get as much of the filter as you get when it was just people caring strongly enough to want to use a particular language despite the unemployability.
Whether that caring happened because of skill (to identify something good), or skill happened because of caring (to invest the time in exploration and thinking and practice), these communities seem to get more than their share of great programmers.
And so you know where to look for them, and you have a reason that they might want to talk to you.
Also, Erlang, Rust, and Clojure, though those have been rumored to be employable, so no longer get as much of the filter as you get when it was just people caring strongly enough to want to use a particular language despite the unemployability.
Whether that caring happened because of skill (to identify something good), or skill happened because of caring (to invest the time in exploration and thinking and practice), these communities seem to get more than their share of great programmers.
And so you know where to look for them, and you have a reason that they might want to talk to you.
>probably more obscure ones.
Apl/BQN worth adding for the recent CUDA/vector language explosion, and has it's fair share of extraordinary programmers!
Forth an interesting one too look, although never properly used.
miniKanren/Prolog another vein to branch down (unification/logic programming)
Apl/BQN worth adding for the recent CUDA/vector language explosion, and has it's fair share of extraordinary programmers!
Forth an interesting one too look, although never properly used.
miniKanren/Prolog another vein to branch down (unification/logic programming)
[deleted]
Postgres. I don't code in C if I can avoid it, since it often feels like an awful lot of extra typing while still having to worry about memory safety. But the Postgres codebase is extraordinarily well organized and respects the humans that work with it with its intelligent handling of memory and judicious use of macros.
I consider the core Postgres codebase to be the gold standard in development even though it's in a language I do not prefer to write in if given the choice.
Shout out to the pgrx folks. You're awesome! https://github.com/pgcentralfoundation/pgrx
I consider the core Postgres codebase to be the gold standard in development even though it's in a language I do not prefer to write in if given the choice.
Shout out to the pgrx folks. You're awesome! https://github.com/pgcentralfoundation/pgrx
That is nice to hear, albeit unsurprising. Their public documentation is some of the best that I have worked with. Postgres is such an impressive project overall.
Not only that, one of the best documented projects out there, clear, comprehensive, consistent and findable.
This codebase is 99.6% Rust according to GitHub.
https://github.com/postgres/postgres
Postgres (core?) seems to be 85% C.
Postgres (core?) seems to be 85% C.
He's referencing pgrx:
https://github.com/pgcentralfoundation/pgrx
To be fair pgrx was mentioned in OP under a positive light, and OP did say he likes avoiding coding in C, so it was something to bring up.
https://github.com/pgcentralfoundation/pgrx
To be fair pgrx was mentioned in OP under a positive light, and OP did say he likes avoiding coding in C, so it was something to bring up.
I laughed at this comment
sorry
sorry
As I said, I don't like coding in C.
The run time library for Turbo Pascal/Delphi for Windows was completely documented, sane, and very easy to work with. The working examples really helped.
The free Pascal RTL seems opaque in comparison. Their reliance on and archaic help file build system keeps contributors away. Thus it's poorly documented at best.
The free Pascal RTL seems opaque in comparison. Their reliance on and archaic help file build system keeps contributors away. Thus it's poorly documented at best.
Was always an Anders fan because of his work in the turbo/delphi community.
Pretty much any internal tool/TUI/CLI/library I've created. If I had to guess I'd say at most 25% of the company projects I've worked on have launched AND have consistent usage. Working hard on something just for it to wither crushes my soul but internal projects are different. They're all skunk works projects. No tickets. No project/board. No PM pushing back on how many points (read: hours) something should be. I'm solving real problems that directly impact the quality of life for myself and my coworkers. The best part is getting real, genuine, feedback. If something sucks they'll tell you and they won't sugarcoat it.
I love this take. What language(s) do you typically use to write CLI programs? I'm also interested in learning about what types of internal TUI tools you have created.
Google's monorepo, and it's not even close - primarily for the tooling:
* Creating a mutable snapshot of the entire codebase takes a second or two.
* Builds are perfectly reproducible, and happen on build clusters. Entire C++ servers with hundreds of thousands of lines of code can be built from scratch in a minute or two tops.
* The build config language is really simple and concise.
* Code search across the entire codebase is instant.
* File history loads in an instant.
* Line-by-line blame loads in a few seconds.
* Nearly all files in supported languages have instant symbol lookup.
* There's a consistent style enforced by a shared culture, auto-linters, and presubmits.
* Shortcuts for deep-linking to a file/version/line make sharing code easy-peasy.
* A ton of presubmit checks ensure uniform code/test quality.
* Code reviews are required, and so is pairing tests with code changes.
* Creating a mutable snapshot of the entire codebase takes a second or two.
* Builds are perfectly reproducible, and happen on build clusters. Entire C++ servers with hundreds of thousands of lines of code can be built from scratch in a minute or two tops.
* The build config language is really simple and concise.
* Code search across the entire codebase is instant.
* File history loads in an instant.
* Line-by-line blame loads in a few seconds.
* Nearly all files in supported languages have instant symbol lookup.
* There's a consistent style enforced by a shared culture, auto-linters, and presubmits.
* Shortcuts for deep-linking to a file/version/line make sharing code easy-peasy.
* A ton of presubmit checks ensure uniform code/test quality.
* Code reviews are required, and so is pairing tests with code changes.
I always find these comments about interesting, having worked at Facebook and Google, I never quite felt this way about Google's Monorepo. Facebook had many of the features you listed and quite performantly if not more so. Compared with working at Facebook where there are no owners owners files and no readability requirements, I found abstraction boundries to be much cleaner at FB. At google, I found there was a ton of cruft in Google's monorepos that were too challenging / too much work for any one person to address.
For non-Googlers like me, here's some background about "readability":
https://www.moderndescartes.com/essays/readability/
https://www.moderndescartes.com/essays/readability/
OWNERS files rarely get in the way - you can always send a code change to an OWNER. They are also good for finding points of contact quickly, for files where the history is in the far past and changes haven't been made recently.
Readability really does help new engineers get up to speed on the style guide, and learn of common libraries they might not have known before. It can be annoying - hell, I'll have to get on the Go queue soon - but that's ok.
Readability really does help new engineers get up to speed on the style guide, and learn of common libraries they might not have known before. It can be annoying - hell, I'll have to get on the Go queue soon - but that's ok.
I think I have heard similar things from other googlers and I think there might be two factors on why I think this:
- I worked on Google Assistant which was responsible for integrating many services. This meant I had to work with other peoples code way more regularly that many at google.
- I moved from FB to google - I'm not really sure how many people have had this experience. I think many of my colleagues at google found it surprising how many of the things they thought were unique to google actually also existed at FB.
At the end of the day, any of these processes have pros/cons but I think the cruft of having APIs that are a couple steps harder to evolve due to finding Readability/Owners for everything you touch just makes things slightly less cohesive and a trickier place to have a "good" codebase.
When I worked at FB, I would frequently rebase my code on Monday and find that, for example, the React framework authors or another smaller infra team had improved the API and had changed *every* callsite in the codebase to be improved. This type of iteration was possible in certain situations but was just much less common at google than at fb.
- I worked on Google Assistant which was responsible for integrating many services. This meant I had to work with other peoples code way more regularly that many at google.
- I moved from FB to google - I'm not really sure how many people have had this experience. I think many of my colleagues at google found it surprising how many of the things they thought were unique to google actually also existed at FB.
At the end of the day, any of these processes have pros/cons but I think the cruft of having APIs that are a couple steps harder to evolve due to finding Readability/Owners for everything you touch just makes things slightly less cohesive and a trickier place to have a "good" codebase.
When I worked at FB, I would frequently rebase my code on Monday and find that, for example, the React framework authors or another smaller infra team had improved the API and had changed *every* callsite in the codebase to be improved. This type of iteration was possible in certain situations but was just much less common at google than at fb.
> I think many of my colleagues at google found it surprising how many of the things they thought were unique to google actually also existed at FB.
Google workers are groomed to believe Google is the best, and hence they are too. A corollary of that, then, is that nobody else has it that good, when in fact, others sometimes have it better.
Google workers are groomed to believe Google is the best, and hence they are too. A corollary of that, then, is that nobody else has it that good, when in fact, others sometimes have it better.
I also made the move from FB to G and echo everything said above. Googlers have a massive superiority complex. In reality, it's naiveté.
My 2 cents: OWNERS is fairly useful, if only as a form of automating code reviewer selection. Readabilty is a massive drag on org-wide productivity. I have had diffs/CLs take MONTHS to be approved by every Tom Dick and Harry whose claws were added to my code and made me re-design whole project approaches, and they were only there because they're supposed to check if my new-lines are in the right spot for that language. I thought about quitting.
My 2 cents: OWNERS is fairly useful, if only as a form of automating code reviewer selection. Readabilty is a massive drag on org-wide productivity. I have had diffs/CLs take MONTHS to be approved by every Tom Dick and Harry whose claws were added to my code and made me re-design whole project approaches, and they were only there because they're supposed to check if my new-lines are in the right spot for that language. I thought about quitting.
People really underestimate how much productivity drain there is in having a bad code review culture. One of the worst things about working at Amazon was that any feedback on a merge request, no matter how small, required you to request a re-review.
It's not culture (organic), it's systems (planned): if you don't, on day zero, agree what code reviews should cover (and what not), then code reviews are a pissing contest first, and a useful tool second.
I noticed a lot of people don't understand both the limitations of code reviews, and what issues they can and should solve. Writing good critique (of anything, not just code) is hard, we don't train people to do it, and usually don't even regard this as something that needs training and understanding.
I noticed a lot of people don't understand both the limitations of code reviews, and what issues they can and should solve. Writing good critique (of anything, not just code) is hard, we don't train people to do it, and usually don't even regard this as something that needs training and understanding.
+1.
Going from FB to $REDACTED to Oculus was a pretty wild ride, there were a lot of different cultures, though I think generally speaking the best qualities filtered through.
(also, howdy former teammate)
Going from FB to $REDACTED to Oculus was a pretty wild ride, there were a lot of different cultures, though I think generally speaking the best qualities filtered through.
(also, howdy former teammate)
(trying to recall unixname to human name mapping.....)
No, we’re quite aware the world outside has been catching up. There’s even a famous doc by a senior director about it…
The system still grooms Googlers to think they're better than though. Until that root cause would be fixed (which would have to be at a huge cost to Google, so no surprise it'd never be), nothing would change.
In Google Cloud at least, we're quite aware we're not the market leader, so there's a pervasive humbleness. We're proud of certain technical achievements (ie: Spanner), but the world can catch up quickly (ie: CockroachDB, Foundation, CosmosDB, etc). This might be a departure from the feeling in older divisions of the company, haha.
Huh? Facebook has a lot of that infra because ex-Googlers built it there. It takes an insane amount of delusion to notice something common between a father and a son and say that the dad inherited it.
QED
[deleted]
This isn't true at all for OWNERS files. If you try developing a small feature on google search, it will require plumbing data through at least four to five layers and there is a different set of OWNERS for each layer. You'll spend at least 3 days waiting for code reviews to go through for something as simple as adding a new field.
3 days for a new change on the biggest service on the planet? Not bad.
I agree that it could be worse! Facebook has significant (if not more) time spent and I found adding features to news feed a heck of a lot easier than adding features that interacted with google search. Generally a lot of this had to do with the number of people needed to be involved to ensure that the change was safe which always felt higher at Google.
I'm only an outside observer in this conversation but could it be that the review process (or the lack thereof) and the ease with which you can add new features has had an impact on the quality of the software?
The thing is, in my experience as a user Facebook (the product, not the former company) is absolutely riddled with bugs. I have largely stopped using it because I used to constantly run into severe UI/UX issues (text input no longer working, scrolling doing weird things, abysmal performance, …), loading errors (comments & posts disappearing and reappearing), etc. Looking at the overall application (and e.g. the quality of the news feed output), it's also quite clear that many people with many different ideas have worked on it over time.
In contrast, Google search still works reasonably well overall 25 years later.
The thing is, in my experience as a user Facebook (the product, not the former company) is absolutely riddled with bugs. I have largely stopped using it because I used to constantly run into severe UI/UX issues (text input no longer working, scrolling doing weird things, abysmal performance, …), loading errors (comments & posts disappearing and reappearing), etc. Looking at the overall application (and e.g. the quality of the news feed output), it's also quite clear that many people with many different ideas have worked on it over time.
In contrast, Google search still works reasonably well overall 25 years later.
There are pretty different uptime and stability requirements for a social product and web search (or other Google products like Gmail). When news feed is broken life moves on, when those products break many people can't get any work done at all.
One of Google's major cultural challenges is imposing the move slow and carefully culture on everything though.
One of Google's major cultural challenges is imposing the move slow and carefully culture on everything though.
It’s not considered ok for newsfeed to break. It would be a massive issue that would command the full attention of everyone.
And yet folks who are on call for it say things like this: https://news.ycombinator.com/item?id=40826497
I have the same background: I find the code quality at G to be quite a lot higher (and test pass-rate, and bug report-rate lower) than News Feed, which was a total shit-show of anything-goes. I still hold trauma from being oncall for Feed. 70 bugs added to my queue per day.
The flip side is of course that I could complete 4 rounds of QuickExperiment and Deltoid to get Product Market Fit, in the time it takes to get to dogfooding for any feature in Google.
The flip side is of course that I could complete 4 rounds of QuickExperiment and Deltoid to get Product Market Fit, in the time it takes to get to dogfooding for any feature in Google.
[deleted]
Huh, also having worked at both I had exactly the opposite experience. Google’s tools looked ugly but just worked. At meta there were actually multiple repos you might have to touch and tools worked unreliably across them. Owners files made sure there was less abandoned code and parent owners woild be found by gwsqueue bots to sign off on big changes across large parts of the repo by just reading these files.
Same and another vote for meta. Meta made the language fit their use case. Go into bootcamp change the search bar text to ‘this is a search bar!’ press F5 and see the change (just don’t ship that change ;D). It’s incredibly smooth and easy.
Googles a mess. There’s always a migration to the latest microservices stack that have been taking years and will take many more years to come.
Like meta just changed the damn language they work in to fit their needs and moved on. Google rewrites everything to fit the language. The former method is better in a large codebase. Meta is way easier to get shit done to the point that google was left in the dust last time they competed with meta.
Googles a mess. There’s always a migration to the latest microservices stack that have been taking years and will take many more years to come.
Like meta just changed the damn language they work in to fit their needs and moved on. Google rewrites everything to fit the language. The former method is better in a large codebase. Meta is way easier to get shit done to the point that google was left in the dust last time they competed with meta.
I think what you're saying is true for www, but not fbcode, and the later starts to look a lot like google3. I agree though, Meta's www codebase has the best developer experience in the industry.
> no owners owners files and no readability requirements
Move fast and break things, right?
Move fast and break things, right?
I don’t think that owners files are the best way to ensure things don’t break.
I’m just one incompetent dev, but I’ll throw this in the convo just to have my perspective represented: every individual part of the google code experience was awesome because everyone cared a ton about quality and efficiency, but the overall ecosystem created as a result of all these little pet projects was to a large extent unmanaged, making it difficult to operate effectively (or, in my case, basically at all). When you join, one of the go-to jokes in their little intro class these days is “TFW you’re told that the old tool is deprecated, but the new tool is still in beta”; everyone laughs along, but hopefully a few are thinking “uhhh wtf”.
To end on as nice of a note as possible for the poor Googs: of all the things you bring up, the one I’d highlight the biggest difference on is Code Search. It’s just incredible having that level of deep semantic access to the huge repo, and people were way more comfortable saying “oh let’s take a look at that code” ad-hoc there than I think is typical. That was pretty awesome.
To end on as nice of a note as possible for the poor Googs: of all the things you bring up, the one I’d highlight the biggest difference on is Code Search. It’s just incredible having that level of deep semantic access to the huge repo, and people were way more comfortable saying “oh let’s take a look at that code” ad-hoc there than I think is typical. That was pretty awesome.
Imho the reason for the deprecated and beta thing is because there is a constant forward momentum.
Best practices, recommendations and tooling is constantly evolving and requires investment in uptake.
I sometimes feel like everything is legacy the moment it's submitted and in a constant state of migration.
This requires time and resources that can slow the pace of development for new features.
The flips side is this actually makes the overall codebase less fractured. This consistency or common set of assumptions is what allows people to build tools and features that work horizontally across many teams/projects.
This constant forward momentum to fight inconsistency is what allows google3 to scale and keep macro level development velocity to scale relative to complexity.
Best practices, recommendations and tooling is constantly evolving and requires investment in uptake.
I sometimes feel like everything is legacy the moment it's submitted and in a constant state of migration.
This requires time and resources that can slow the pace of development for new features.
The flips side is this actually makes the overall codebase less fractured. This consistency or common set of assumptions is what allows people to build tools and features that work horizontally across many teams/projects.
This constant forward momentum to fight inconsistency is what allows google3 to scale and keep macro level development velocity to scale relative to complexity.
That’s all well said, thanks for sharing your perspective! Gives me some things to reflect on. I of course agree re:forward momentum, but I hope they’re able to regain some grace in that momentum with better organization going forward. I guess I was gesturing to people “passing the buck” on hard questions of team alignment and mutually exclusive decisions. Obviously I can’t cite specifics bc of secrecy and bad memory in equal amounts, so it’s very possible that I had a distorted view.
I will say, one of the things that hit me the hardest when the layoffs finally hit was all the people who have given their professional lives to making some seriously incredible dev tools, only to be made to feel disposable and overpaid so the suits could look good to the shareholders for a quarter or two. Perhaps they have a master vision, but I’m afraid one of our best hopes for an ethical-ish megacorp—or at least vaguely pro social—is being run for short term gain :(
However that turns out for society, hopefully it ends up releasing all those tools for us to enjoy! Mark my words, colab.google.com will be shockingly popular 5y from now, if they survive till then
I will say, one of the things that hit me the hardest when the layoffs finally hit was all the people who have given their professional lives to making some seriously incredible dev tools, only to be made to feel disposable and overpaid so the suits could look good to the shareholders for a quarter or two. Perhaps they have a master vision, but I’m afraid one of our best hopes for an ethical-ish megacorp—or at least vaguely pro social—is being run for short term gain :(
However that turns out for society, hopefully it ends up releasing all those tools for us to enjoy! Mark my words, colab.google.com will be shockingly popular 5y from now, if they survive till then
I guarantee you that the master vision is exactly what you wrote.
Google is not a software company, it is an advertising system providing cash-flow to a hedge fund that comprises a large part of every pension and retirement fund in America. It's far too important as simply a financial entity to risk anything on ....product development.
Google is not a software company, it is an advertising system providing cash-flow to a hedge fund that comprises a large part of every pension and retirement fund in America. It's far too important as simply a financial entity to risk anything on ....product development.
My experience with google3 was a bit different. I was shocked at how big things had gotten without collapsing, which is down to thousands of Googlers working to build world-class internal tooling. But you could see where the priorities were. Code Search was excellent - I'd rate it 10/10 if they asked.
The build system always felt more like a necessary evil than anything else. In some parts of google3 you needed three separate declarations of all module dependencies. You could have Angular's runtime dependency injection graph, the Javascript ESM graph, and the Blaze graph which all need to be in sync. Now, the beautiful part was that this still worked. And The final Blaze level means you can have a Typescript codebase that depends on a Java module written in a completely unrelated part of google3, which itself depends on vendored C++ code somewhere else. Updating the vendored C++ code would cause all downstream code to rebuild and retest. But this is a multi billion dollar solution to problems that 99.99% of companies do not have. They are throwing thousands of smart people at a problem that almost everyone else has "solved" by default simply by being a smaller company.
The one tooling I think every company could make use of but doesn't seem to have were all of the little hacks in the build system (maybe not technically part of Blaze?). You could require a developer who updates the file at /path/to/department/a/src/foo.java to simultaneously include a patch to /path/to/department/b/src/bar.java. Many files would have implicit dependency on each other outside of the build graph and a human is needed to review if extra changes are needed. And that's just one of a hundred little tricks project maintainers can employ.
The quality of the code was uniformly at least "workable" (co-workers updating parts of the Android system would probably not agree with that - many critical system components were written by one person poorly who soon after quit).
The build system always felt more like a necessary evil than anything else. In some parts of google3 you needed three separate declarations of all module dependencies. You could have Angular's runtime dependency injection graph, the Javascript ESM graph, and the Blaze graph which all need to be in sync. Now, the beautiful part was that this still worked. And The final Blaze level means you can have a Typescript codebase that depends on a Java module written in a completely unrelated part of google3, which itself depends on vendored C++ code somewhere else. Updating the vendored C++ code would cause all downstream code to rebuild and retest. But this is a multi billion dollar solution to problems that 99.99% of companies do not have. They are throwing thousands of smart people at a problem that almost everyone else has "solved" by default simply by being a smaller company.
The one tooling I think every company could make use of but doesn't seem to have were all of the little hacks in the build system (maybe not technically part of Blaze?). You could require a developer who updates the file at /path/to/department/a/src/foo.java to simultaneously include a patch to /path/to/department/b/src/bar.java. Many files would have implicit dependency on each other outside of the build graph and a human is needed to review if extra changes are needed. And that's just one of a hundred little tricks project maintainers can employ.
The quality of the code was uniformly at least "workable" (co-workers updating parts of the Android system would probably not agree with that - many critical system components were written by one person poorly who soon after quit).
> But this is a multi billion dollar solution to problems that 99.99% of companies do not have.
I know it's trendy for people to advocate for simple architectures, but the honest-to-god truth is that it's insane that builds work ANY OTHER WAY. One of the highest priorities companies should have is to reduce siloing, and I can barely think of a better way to guarantee silos than by having 300 slightly different build systems.
There is a reason why Google can take a new grad SWE who barely knows how to code and turn them into a revenue machine. I've worked at several other places but none of them have had internal infrastructure as nice as the monorepo; it was the least amount of stress I've ever felt deploying huge changes.
Another amazing thing that I don't see mentioned enough was how robust the automatic deployments with Boq/Annealing/Stubby were. The internal observability library would automatically capture RPC traces from both the client and server, and the canary controller would do a simple p-test on whether or not the new service had a higher error rate than the old one. If it did? The rollback CL would be automatically submitted and you'd get a ping.
This might sound meh until I point out that EVEN CONFIG CHANGES were versioned and canaried.
I know it's trendy for people to advocate for simple architectures, but the honest-to-god truth is that it's insane that builds work ANY OTHER WAY. One of the highest priorities companies should have is to reduce siloing, and I can barely think of a better way to guarantee silos than by having 300 slightly different build systems.
There is a reason why Google can take a new grad SWE who barely knows how to code and turn them into a revenue machine. I've worked at several other places but none of them have had internal infrastructure as nice as the monorepo; it was the least amount of stress I've ever felt deploying huge changes.
Another amazing thing that I don't see mentioned enough was how robust the automatic deployments with Boq/Annealing/Stubby were. The internal observability library would automatically capture RPC traces from both the client and server, and the canary controller would do a simple p-test on whether or not the new service had a higher error rate than the old one. If it did? The rollback CL would be automatically submitted and you'd get a ping.
This might sound meh until I point out that EVEN CONFIG CHANGES were versioned and canaried.
I've worked at the majority of FAANG.
Facebook's build system works the same as Googles, because most of FB's infra was made by ex-Googlers around 10-15 years ago. The worst thing I can say about Blaze is basically already pointed out above, sometimes you need to write little notes to the presubmit system to ensure cross-boundary updates. Whatever, it's all text files in the end.
The wildest was at Apple. It's just as you said, 300 build systems. Not only that, but 300 source code repositories! Two teams in the same hall that hang out all the time could be using git and svn, for no good reason besides what someone wrote "init" in 20 years ago. There was no cross team communication, by design, because Steve Jobs was paranoid. Their sync mechanism was to build the entire stack once a night, and force everyone to full-reinstall their OS and toolchain to "rebase". Insane.
Facebook's build system works the same as Googles, because most of FB's infra was made by ex-Googlers around 10-15 years ago. The worst thing I can say about Blaze is basically already pointed out above, sometimes you need to write little notes to the presubmit system to ensure cross-boundary updates. Whatever, it's all text files in the end.
The wildest was at Apple. It's just as you said, 300 build systems. Not only that, but 300 source code repositories! Two teams in the same hall that hang out all the time could be using git and svn, for no good reason besides what someone wrote "init" in 20 years ago. There was no cross team communication, by design, because Steve Jobs was paranoid. Their sync mechanism was to build the entire stack once a night, and force everyone to full-reinstall their OS and toolchain to "rebase". Insane.
I definitely agree most companies should use a monorepo. Most companies don't need Blaze, though.
And the whole rollout system was excellent. I wish that tech was standard but I have a vague idea of how much work that would be to implement and few companies will be able to afford to get that right.
Edit: I forgot to mention - I absolutely hated Cider and all of the included plugins. Sure the code at Google was fine but the code editing experience destroyed all of the fun of coding. Is that function signature correct? You'll find out in 45 seconds when the Intellisense completes! And when I was there Copilot was a thing outside of Google but we were not allowed to use any AI (even Google's own AI) to write code. The whole situation was so bad I wrote a few paragraphs about it in my offboarding survey.
And the whole rollout system was excellent. I wish that tech was standard but I have a vague idea of how much work that would be to implement and few companies will be able to afford to get that right.
Edit: I forgot to mention - I absolutely hated Cider and all of the included plugins. Sure the code at Google was fine but the code editing experience destroyed all of the fun of coding. Is that function signature correct? You'll find out in 45 seconds when the Intellisense completes! And when I was there Copilot was a thing outside of Google but we were not allowed to use any AI (even Google's own AI) to write code. The whole situation was so bad I wrote a few paragraphs about it in my offboarding survey.
> You could require a developer who updates the file at /path/to/department/a/src/foo.java to simultaneously include a patch to /path/to/department/b/src/bar.java.
Could you elaborate on how this worked exactly?
Could you elaborate on how this worked exactly?
There's a configuration directive you put in a plain text file in the monorepo which lets you configure:
* File A's path
* File B's path
* The message shown on the commit's review page if B isn't updated when A is updated
* How a developer can override the alert (Which would be a custom named directive added to a commit message, like "SKIP_FILE_SYNC_ALERT=true")
You then need to either commit a diff to file B when file A is changed or override it in order to get the commit added to HEAD. This is just one of many different "plugins" for the CI system that can be configured with code.
* File A's path
* File B's path
* The message shown on the commit's review page if B isn't updated when A is updated
* How a developer can override the alert (Which would be a custom named directive added to a commit message, like "SKIP_FILE_SYNC_ALERT=true")
You then need to either commit a diff to file B when file A is changed or override it in order to get the commit added to HEAD. This is just one of many different "plugins" for the CI system that can be configured with code.
Thanks!
Interesting to note that almost all of these are to do with tooling _around_ the codebase, not the contents _of_ the codebase!
Just like the man is the product of his genetic code, the codebase is invariably the product of the constraints on its edits enforced by tooling.
so we beat on, commits against the tooling, borne back ceaselessly into the technical debt
Is that true, though? Is the code itself good? Because it is sorely absent from GP's list...
If you are trying to say that people can't make bad code with good tools, I don't agree.
If you are trying to say that people can't make bad code with good tools, I don't agree.
To extend the previous commenter's simile - a man is a _product_ of his genetic code, but is also affected by environment. Bringing it back to the point at hand - yes, you are right that people can make bad code with good tools, but they'll be _much more likely_ to make good code with them (and vice versa).
This Ask HN is not about "the code that should logically be best" but "the best code". There is no need for likelihood, people who have worked on it can report whether it is the case.
And people here seem to praise the tooling exclusively...
I would also point out that good tooling makes for good code, but big scale makes for bad legacy code. It is not at all obvious to me which of those effects should prevail at Google.
And people here seem to praise the tooling exclusively...
I would also point out that good tooling makes for good code, but big scale makes for bad legacy code. It is not at all obvious to me which of those effects should prevail at Google.
Question I've always wondered: Does Google's monorepo provide all its engineers access to ALL its code?
If yes, given the sheer number of developers, why haven't we seen a leak of Google code in the past (disgruntled employee, accidental button, stolen laptop, etc)?
Also how do they handle "Skunkworks" stlye top-secret projects that need to fly under the radar until product launch?
If yes, given the sheer number of developers, why haven't we seen a leak of Google code in the past (disgruntled employee, accidental button, stolen laptop, etc)?
Also how do they handle "Skunkworks" stlye top-secret projects that need to fly under the radar until product launch?
Partial check outs are standard because the entire code base is enormous. People only check out the parts they might be changing and the rest magically appears during the build as needed.
There are sections of the code that are High Intellectual Property. Stuff that deals with spam fighting, for example. I once worked on tooling to help make that code less likely to be accidentally exposed.
Disclaimer: I used to work there, but that was a while back. They probably changed everything a few times since. The need to protect certain code will never go way, however.
There are sections of the code that are High Intellectual Property. Stuff that deals with spam fighting, for example. I once worked on tooling to help make that code less likely to be accidentally exposed.
Disclaimer: I used to work there, but that was a while back. They probably changed everything a few times since. The need to protect certain code will never go way, however.
The very very important stuff is hidden, and the only two examples anyone ever gives are core search ranking algorithms and the self-driving car.
Even the battle-tested hyper-optimized, debugged-over-15-years implementation of Paxos is accessible. Though I’m sure folks could point out other valuable files/directories.
Even the battle-tested hyper-optimized, debugged-over-15-years implementation of Paxos is accessible. Though I’m sure folks could point out other valuable files/directories.
Former employee here. I remember a third example: the anti-DoS code is hidden. I remember this because I needed to do some very complicated custom anti-DoS configuration and as was my standard practice, I looked into how the configuration was being applied. I was denied access.
Fourth example: portions of the code responsible for extracting signals from employees' computers to detect suspicious activity and intrusion. I suspect it's because if an employee wants to do something nefarious they couldn't just read the code to figure out how to evade detection. I only knew about this example because that hidden code made RPC calls to a service I owned; I changed certain aspect of my service and it broke them. Of course they fixed it on their own; I only got a post-submit breakage notification.
Fourth example: portions of the code responsible for extracting signals from employees' computers to detect suspicious activity and intrusion. I suspect it's because if an employee wants to do something nefarious they couldn't just read the code to figure out how to evade detection. I only knew about this example because that hidden code made RPC calls to a service I owned; I changed certain aspect of my service and it broke them. Of course they fixed it on their own; I only got a post-submit breakage notification.
Google3 monorepo source isn't, by policy, supposed to leave the corp network workstations, and can't even be on your corporate provided laptop (except for some edge cases in mobile apps dev). Even during full COVID lockdown WFH we had to remote into our machines. (I worked on embedded stuff and had to compile on my office workstation, scp the binaries home, and flash my device, and repeat. Super slow cycle.)
So, anyways, source code being basically on-premise only and on machines that they can fully audit and control... Would you be stupid enough to "cp -r srccheckout /media/MYUSBSTICK" on such a box?
Also believe it or not they used to have a very open internal culture at Google because the bulk of employees genuinely liked the company and its stated mission and there was a bit of a social contract that seemed to be upheld. Stuff didn't generally leak out of the wide open all hands, even. Past tense.
So, anyways, source code being basically on-premise only and on machines that they can fully audit and control... Would you be stupid enough to "cp -r srccheckout /media/MYUSBSTICK" on such a box?
Also believe it or not they used to have a very open internal culture at Google because the bulk of employees genuinely liked the company and its stated mission and there was a bit of a social contract that seemed to be upheld. Stuff didn't generally leak out of the wide open all hands, even. Past tense.
Google laptops and workstations (anything that can actually access srcfs to get this data) are extremely monitored and controlled.
Very critical stuff (ranking, spam/abuse, etc) can be further protected via silos which lock down sections of the code base (but still allow limited interactions with the build).
Google spent significant engineering $$$ into its development tools and policies (generally building custom with no intent to ever monetize vs buying). I don't see a company today, in this climate, that would emulate that decision.
Very critical stuff (ranking, spam/abuse, etc) can be further protected via silos which lock down sections of the code base (but still allow limited interactions with the build).
Google spent significant engineering $$$ into its development tools and policies (generally building custom with no intent to ever monetize vs buying). I don't see a company today, in this climate, that would emulate that decision.
I don't see a company today, in this climate, that would emulate that decision.
Why not? There are a lot of benefits from owning the tooling and being able to tailor it to do exactly what you want.
Why not? There are a lot of benefits from owning the tooling and being able to tailor it to do exactly what you want.
It's extremely easy to detect a disgruntled employee making a copy of source code. There's no accidental button to leak. There's no source code on laptops as policy doesn't allow it, with limited exceptions only.
But there was a giant leak a long time ago. It was called Operation Aurora done by China. Legend has it that to this date the Chinese search engine Baidu still uses stolen code from Google.
But there was a giant leak a long time ago. It was called Operation Aurora done by China. Legend has it that to this date the Chinese search engine Baidu still uses stolen code from Google.
Within the monorepo there is the notion of "silos" where access to directories can be restricted to groups of people/bots. Though I believe that's exceedingly rare, I've never come across one.
I suspect this is part of the interview process and why it takes so long and so many people.
Character and trustworthiness is extremely important.
Character and trustworthiness is extremely important.
The Google interview process is overly focused on algorithm skills and absolutely does not select for character and trustworthiness. In fact the leaks from Google to the news started circa 2017 and in response the leadership basically neutered internal forums like TGIF and memegen. Remember the Damore incident? While Damore was wrong, it wouldn't be as big of a deal if the incident wasn't leaked to the press. It's clear that Google would be a much better company if its interview process actually accounted for character and trustworthiness.
The old article, Three Years of Misery Inside Google, the Happiest Company in Tech is still the best description of what went wrong inside Google: https://www.wired.com/story/inside-google-three-years-misery...
The old article, Three Years of Misery Inside Google, the Happiest Company in Tech is still the best description of what went wrong inside Google: https://www.wired.com/story/inside-google-three-years-misery...
I have interviewed with Google. It's not explicitly tested for but you can be sure the interviewers will have an opinion of your character.
Also, (in the middle of six hours of interviewing) for lunch I lunched with someone completely outside of the particular group I was interviewing for. Rather genial chap and I'm sure his opinion was sought too.
I'm not downplaying the tech questions, I'm saying there's a meta/side evaluation as well.
Also, (in the middle of six hours of interviewing) for lunch I lunched with someone completely outside of the particular group I was interviewing for. Rather genial chap and I'm sure his opinion was sought too.
I'm not downplaying the tech questions, I'm saying there's a meta/side evaluation as well.
There hasn't been a lunch interview since the pandemic. Everything is online. The old Google is no more.
Also I don't think Damore was wrong .. or right. He was certainly naive.
Edit - I guess there hasn't been zero leaks: https://searchengineland.com/google-search-document-leak-ran...
Code search not just across the entire code base but across all of time.
I recently left Google and knew it was going to be a step down from Google's build ecosystem, but I wasn't prepared for how far a step down it would be. It's the only thing I miss about the place, it' so awesome.
People who have never had it have no concept of how much they are missing. It's so frustrating.
Why do so many people like monorepos?
I tend to much prefer splitting out reusable packages into their own repos with their own packaging and unit tests and tagging to whatever version of that package. It makes it MUCH easier for someone to work on something with minimal overhead and be able to understand every line in the repo they are actually editing.
It also allows reusable components to have their own maintainers, and allows for better delegation of a large team of engineers.
I tend to much prefer splitting out reusable packages into their own repos with their own packaging and unit tests and tagging to whatever version of that package. It makes it MUCH easier for someone to work on something with minimal overhead and be able to understand every line in the repo they are actually editing.
It also allows reusable components to have their own maintainers, and allows for better delegation of a large team of engineers.
Have you ever worked at FB / Google / whatever other company has huge mono repo with great tooling?
I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.
With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.
I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.
With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.
I have worked at a company that has huge monorepos and bad tooling.
Tooling isn't the problem though, the problems are:
- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own
- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?
- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container
- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened
- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it
- ownership of different parts of the codebase is difficult to track; code reviews are a mess
Tooling isn't the problem though, the problems are:
- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own
- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?
- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container
- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened
- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it
- ownership of different parts of the codebase is difficult to track; code reviews are a mess
I think all of your problems are root caused by the phrase "multiple monorepos". This sounds more like "polyrepos" or something, multiple siloed repos that themselves might contained disparate projects, languages and tooling.
Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.
Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.
Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.
Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.
All of these are solved with tooling.
- dont have multiple monorepos
- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.
- other people committing doesn't matter if you are isolating presubmits and similar.
- using a blaze-like you never compile the whole monorepo
- code owners etc. makes this straightforward.
Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.
- dont have multiple monorepos
- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.
- other people committing doesn't matter if you are isolating presubmits and similar.
- using a blaze-like you never compile the whole monorepo
- code owners etc. makes this straightforward.
Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.
> other people committing doesn't matter if you are isolating presubmits and similar.
Could you elaborate on this one?
Could you elaborate on this one?
If you only need to run the tests you affect, and only need to sync and update files touched in your CL, external changes are generally not impactful, sync & submit is a quick process even if people are submitting things elsewhere.
It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.
It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.
If you have a reusable component in a separate repository and need a change, you have to submit that, merge, release, then bump the version in the downstream project to use it. Then if someone else uses the project updates but hits an issue that you introduced they have to go fix it, perhaps a month later, with no context of what changed. Or they just don’t upgrade the version, and reimplement what they need. With a monorepo it would be one change, and your change breaking someone else’s would get flagged and fixed with the code change. I’ve seen the amount of shared code get less and less and more stale with polyrepo.
Up to what scale?
This works well for a couple dozen repos per team in my experience. It’s also my preferred way to work.
It doesn’t scale so well to hundreds of repos per team without significant tooling. At some point anything cross-cutting (build tool updates, library updates, etc) becomes hard to track. Repos are left behind as folks change teams and teams are reorg’d.
I’ve never worked in a monorepo, but I can see the appeal for large, atomic changes especially.
This works well for a couple dozen repos per team in my experience. It’s also my preferred way to work.
It doesn’t scale so well to hundreds of repos per team without significant tooling. At some point anything cross-cutting (build tool updates, library updates, etc) becomes hard to track. Repos are left behind as folks change teams and teams are reorg’d.
I’ve never worked in a monorepo, but I can see the appeal for large, atomic changes especially.
I can change a dependency and my code at the same time and not need to wait for the change to get picked up and deployed separately.
(If they are in the same binary. Still need cross binary changes to be made in order and be rollback safe and all that.)
Google's code, tooling and accompanying practices are developing a reputation for being largely useless outside Google ... and many are starting to suspect it's alleged value even inside Google is mostly cult dogma.
Google used to have a near monopoly on the most expensive, educated, devoted, and conscientiously willful people and imposed very few demands on their time. The lengths to which they were willing to go, to make everything they did with the tools pleasant and elegant, was orders of magnitude beyond anything I'd seen.
Some of us thought that the magic of these people would be imbued in the dev tools that they created, so if enterprises adopted the tools, then they'd reap the benefits of that same magic too. But this simply wasn't true. The tools didn't actually matter; it was the way they used them.
For example, when other companies started adopting tools like Bazel (open source Blaze) they wanted features like being able to launch ./configure scripts inside Bazel, which totally violates the whole point of Bazel, and never would have been allowed or even considered inside Google. The Bazel team was more than happy to oblige, and the users ended up with the worst of all worlds.
Some of us thought that the magic of these people would be imbued in the dev tools that they created, so if enterprises adopted the tools, then they'd reap the benefits of that same magic too. But this simply wasn't true. The tools didn't actually matter; it was the way they used them.
For example, when other companies started adopting tools like Bazel (open source Blaze) they wanted features like being able to launch ./configure scripts inside Bazel, which totally violates the whole point of Bazel, and never would have been allowed or even considered inside Google. The Bazel team was more than happy to oblige, and the users ended up with the worst of all worlds.
If Google open sourced their BUILD files for public libraries, we wouldn’t be able to use workarounds… Migrating something complex like ffmpeg to Bazel is not something trivial.
Bazel is an awesome tool though, I’m very glad it was open sourced and receives constant attention from Google.
Bazel is an awesome tool though, I’m very glad it was open sourced and receives constant attention from Google.
Google's systems were designed to index mountains of low value data at hitherto unseen scale, and they're good at that. But, to-the-second system-wide precision with full audit trails ... not so much.
You keep seeing startups with ex-Googlers that think they can "disrupt" Fintech with Google's "secret sauce" ... this tends to go badly.
I've had to clean up one of these messes where, in all seriousness, even a pre-2000 LAMP stack (never mind Java) implemented by people who understood the finance domain would have worked better.
You keep seeing startups with ex-Googlers that think they can "disrupt" Fintech with Google's "secret sauce" ... this tends to go badly.
I've had to clean up one of these messes where, in all seriousness, even a pre-2000 LAMP stack (never mind Java) implemented by people who understood the finance domain would have worked better.
> Google's code, tooling and accompanying practices are developing a reputation for being largely useless outside Google ...
Not that I don’t believe you, but where do you see this?
Not that I don’t believe you, but where do you see this?
I can vouch for it. It's the main reason I quit: none of the "hard" skills necessary to code at Google were transferrable anywhere outside of Google. It would have been easy enough to skate and use "soft" skills to move up the management ladder and cash big checks, but I wasn't interested in that.
The reason it's not transferrable is that Google has its own version of EVERYTHING: version control, an IDE, build tools, JavaScript libraries, templating libraries, etc, etc. The only thing I can think of that we used that wasn't invented at Google was SCSS, and that was a very recent addition. Google didn't even use its own open-source libraries like Angular. None of the technologies were remotely usable outside Google.
It might sound cool to use only in-house stuff, and I understand the arguments about licensing. But it meant that everything was poorly-documented, had bugs and missing features that lingered for years, and it was impossible to find a SME because whoever initially built a technology had moved on to other things and left a mess behind them.
Some people may be able to deal with the excruciating slowness and scattered-ness, and may be OK with working on a teeny slice of the pie in the expectation that years later they'll get to own a bigger slice. But that ain't me so I noped out as soon as my shares vested.
The reason it's not transferrable is that Google has its own version of EVERYTHING: version control, an IDE, build tools, JavaScript libraries, templating libraries, etc, etc. The only thing I can think of that we used that wasn't invented at Google was SCSS, and that was a very recent addition. Google didn't even use its own open-source libraries like Angular. None of the technologies were remotely usable outside Google.
It might sound cool to use only in-house stuff, and I understand the arguments about licensing. But it meant that everything was poorly-documented, had bugs and missing features that lingered for years, and it was impossible to find a SME because whoever initially built a technology had moved on to other things and left a mess behind them.
Some people may be able to deal with the excruciating slowness and scattered-ness, and may be OK with working on a teeny slice of the pie in the expectation that years later they'll get to own a bigger slice. But that ain't me so I noped out as soon as my shares vested.
12 year current Googler here. You are absolutely correct about "Google has its own version of EVERYTHING". Midway through my current career, I started to get existential dread about the fact that I wasn't "up to date" on any current development practices or frameworks.
Partly, this was assuaged through participating in open source projects in my free time. That's how I learned Docker, Github workflow, React, Vue, Bootstrap, Tailwind, etc.
But at the same time, I think it is a mistake to consider working with tools/languages/frameworks to be the only "hard" skills. Galaxy brain is realizing that anyone can learn a language/framework/workflow in a month or so. The real work is applying sound principles to the design and production of meaningful artifacts within those systems.
Partly, this was assuaged through participating in open source projects in my free time. That's how I learned Docker, Github workflow, React, Vue, Bootstrap, Tailwind, etc.
But at the same time, I think it is a mistake to consider working with tools/languages/frameworks to be the only "hard" skills. Galaxy brain is realizing that anyone can learn a language/framework/workflow in a month or so. The real work is applying sound principles to the design and production of meaningful artifacts within those systems.
Though credit where it’s due, some of their tools really have been years ahead of anything outside of google, e.g. the closure compiler that made javascript development scalable.
Their own IDE? Is it web based? So they don't use, say, IntelliJ/Eclipse for Java projects?
It's basically a fork of VSCode.
That’s a recent development, used to be something else altogether.
I have seen this discussed in hiring decisions. I don't know that it played a large factor in a decision, but lack of experience in the standard tools/practices/terms of software development because of a career at Google was definitely a discussion point.
[deleted]
I haven't worked at google, but this is something I have heard from a few people. Reputation is largely word of mouth, so it checks out for me. I suspect the skills/tools at most large companies are increasingly less transferrable as they continue to grow in scale and scope.
I had a bunch of very tenured teammates that didn’t really know how to use git, so there were only a few of us comfortable enough integrating and interacting with an open source dependency repo.
I’m surprised they didn’t turn that into a product, it sounds great.
Parts have been. Sourcegraph is basically the code search post built by ex-Googlers originally. Bazel is the open source build tool. Sadly, most of these things require major work to set up yourself and manage, but there's an alternate present where Google built a true competitor to GitHub and integrated their tooling directly into it.
I've published my Google proprietary stuff (when we decided to open source it) on GitLab, but they wouldn't let me do it on GitHub.
Building tools for others is a competency that is under rewarded at Google. They would never.
> Entire C++ servers with hundreds of lines of code can be built from scratch in a minute or two tops.
Hundreds, huh? Is this a typo? It makes me wonder if the whole comment is facetious. Or do C++ programmers just have very low expectations for build time?
Hundreds, huh? Is this a typo? It makes me wonder if the whole comment is facetious. Or do C++ programmers just have very low expectations for build time?
That's the beauty of C++, an absurdly slow build is just an include away.
The public version of Google's build tool is Bazel (it's Blaze internally). It has some really impressive caching while maintaining correctness. The first build is slow, but subsequent builds are very fast. When you have a team working on similar code, everyone gets the benefit.
As with all things Google, it's a pain to get up to speed on, but then very fast.
As with all things Google, it's a pain to get up to speed on, but then very fast.
Just wait until you try a “modern” language
Most of those arguments are not about code quality though.
My favourite projects are small, with very focused goals and features.
I have a Laravel project that I have maintained for a customer for seven years. The app is straightforward and allows users to create portals that list files and metadata, such as expiration dates and tags.
Every other year, they ask me to add a new batch of features or update the UI to reflect the business's branding. As the app is so small, I have the opportunity to review every part of the app and refactor or completely rewrite parts I am not happy with.
It is a joy to work on and I always welcome new requests.
I have a Laravel project that I have maintained for a customer for seven years. The app is straightforward and allows users to create portals that list files and metadata, such as expiration dates and tags.
Every other year, they ask me to add a new batch of features or update the UI to reflect the business's branding. As the app is so small, I have the opportunity to review every part of the app and refactor or completely rewrite parts I am not happy with.
It is a joy to work on and I always welcome new requests.
This sounds like wonderful! Maybe one day I can have a few of these “babies” that I can lovingly raise, providing joy to both the dev and the customer.
I've seen a few people say 'google3'.
Q: is it actually the code that you loved, or simply the tooling that exists?
(and if it's tooling, why can't that type of tooling be replicated for other codebases outside of google?)
Q: is it actually the code that you loved, or simply the tooling that exists?
(and if it's tooling, why can't that type of tooling be replicated for other codebases outside of google?)
That type of tooling can be replicated. That's why every Xoogler tries so hard to get Bazel adopted on everything they touch. Sometimes that's appropriate, sometimes it's not, but that's why.
Bazel isn't the whole of everything though, the other piece being exported is kubernetes, which isn't Google's borg, but that's its roots. There's Apache Airflow if you need a workflow engine like Sisyphus, no shortage of databases to choose from, though now we're drifting into the operations side of things.
But basically, Google has invested untold millions of dollars in the form of SWE and other engineering hours into making Google3 operate. If anyone else invested that kind of dough, with enough smarts, they'd also be able to make it a good experience. The problem is few people have that kind of budget, and even fewer invest in that thing, preferring to use free tools instead.
What tool do you use to edit code, and how much did your employer spend on that for you?
Bazel isn't the whole of everything though, the other piece being exported is kubernetes, which isn't Google's borg, but that's its roots. There's Apache Airflow if you need a workflow engine like Sisyphus, no shortage of databases to choose from, though now we're drifting into the operations side of things.
But basically, Google has invested untold millions of dollars in the form of SWE and other engineering hours into making Google3 operate. If anyone else invested that kind of dough, with enough smarts, they'd also be able to make it a good experience. The problem is few people have that kind of budget, and even fewer invest in that thing, preferring to use free tools instead.
What tool do you use to edit code, and how much did your employer spend on that for you?
> That's why every Xoogler tries so hard to get Bazel adopted on everything they touch.
After burning an entire weekend like five years back trying to get a released version of tensorflow to build from source [0], I'm catastrophically disinclined to use Bazel for anything.
I found its diagnostics to be utterly unhelpful, its documentation to say nearly nothing I needed to know, and the various Internet resources for the build system to be somewhere between "as confused and lost as I was" and "total fanboy who is so expert in the system that they are incapable of speaking like anything other than an architecture astronaut".
I'm sure it's legitimately fuckin amazing when you learn it at a company that's big enough to have one or more entire teams dedicated to internal developer tooling (and training for and documentation of the same), but (at least in my experience) for those of us on the outside, it's just bad, bad, bad.
[0] The fucking thing wouldn't even build in the officially-supplied "build tensorflow" Docker image. I was utterly unable to find out why. I get that this indicates that the tensorflow folks fucked up somehow, but the fact that I was utterly unable to figure out how to understand WHAT they fucked up is pretty damning.
After burning an entire weekend like five years back trying to get a released version of tensorflow to build from source [0], I'm catastrophically disinclined to use Bazel for anything.
I found its diagnostics to be utterly unhelpful, its documentation to say nearly nothing I needed to know, and the various Internet resources for the build system to be somewhere between "as confused and lost as I was" and "total fanboy who is so expert in the system that they are incapable of speaking like anything other than an architecture astronaut".
I'm sure it's legitimately fuckin amazing when you learn it at a company that's big enough to have one or more entire teams dedicated to internal developer tooling (and training for and documentation of the same), but (at least in my experience) for those of us on the outside, it's just bad, bad, bad.
[0] The fucking thing wouldn't even build in the officially-supplied "build tensorflow" Docker image. I was utterly unable to find out why. I get that this indicates that the tensorflow folks fucked up somehow, but the fact that I was utterly unable to figure out how to understand WHAT they fucked up is pretty damning.
I drove the adoption of Bazel for my C++ project, which certainly does not have an entire team (nor even an entire person) dedicated to maintaining the build system.
We’re having incredible build performance with it, and I find its language much more readable than CMake or Make. We’re also using Bazel for things like packaging.
I would encourage you to give it another go.
We’re having incredible build performance with it, and I find its language much more readable than CMake or Make. We’re also using Bazel for things like packaging.
I would encourage you to give it another go.
No.
I spent two entire days attempting to understand (let alone correct) whatever was going wrong with the Tensorflow build.
The error messages were inscrutable, as was the debug output. There was a fair bit of official documentation, but little of it was "unhappy path" documentation, and what there was was unhelpful. As far as I could tell (and it was difficult to tell given how I was generally unable to understand the problem), none of the folks talking about this tool on the Greater Internet had ever run into this problem. [0]
I've never run into this problem with CMake, SCons, Make, or any of a dozen or two other build systems I've worked with. To put it another way, I've NEVER had a single problem with a build system be utterly impossible to understand and solve inside of a working day. (Sure, I've had to spend multiple working days solving multiple problems to get a build right, but I've never had any one problem prove to be utterly unsolvable in one day, let alone two.)
How easy it is to get into failure modes and (more crucially) how much support the system and supporting documentation gives you for both identifying the failure mode you're in and getting out of it are much, much more important than being the fastest thing out there. [1][2]
Unless I'm getting paid to use it and am required to use it and nothing else, Bazel is absolutely not worth my time.
[0] This must obviously be not the case. Given the circumstances, whatever I was running into, it's certain that TONS of people must have run into it. But it's telling that I couldn't discover any other people who were running into this.
[1] Yes, I'm sure that you've never had a significant problem with the tool. There are many, many useful tools out there that folks rave about that have burned me and burned the folks I work with badly enough that we won't touch them, ever.
[2] Also, if you learned to use Bazel inside the Googleplex, then that absolutely is a scenario covered by my statement: "I'm sure it's legitimately fuckin amazing when you learn it at a company that's big enough to have one or more entire teams dedicated to internal developer tooling..."
I spent two entire days attempting to understand (let alone correct) whatever was going wrong with the Tensorflow build.
The error messages were inscrutable, as was the debug output. There was a fair bit of official documentation, but little of it was "unhappy path" documentation, and what there was was unhelpful. As far as I could tell (and it was difficult to tell given how I was generally unable to understand the problem), none of the folks talking about this tool on the Greater Internet had ever run into this problem. [0]
I've never run into this problem with CMake, SCons, Make, or any of a dozen or two other build systems I've worked with. To put it another way, I've NEVER had a single problem with a build system be utterly impossible to understand and solve inside of a working day. (Sure, I've had to spend multiple working days solving multiple problems to get a build right, but I've never had any one problem prove to be utterly unsolvable in one day, let alone two.)
How easy it is to get into failure modes and (more crucially) how much support the system and supporting documentation gives you for both identifying the failure mode you're in and getting out of it are much, much more important than being the fastest thing out there. [1][2]
Unless I'm getting paid to use it and am required to use it and nothing else, Bazel is absolutely not worth my time.
[0] This must obviously be not the case. Given the circumstances, whatever I was running into, it's certain that TONS of people must have run into it. But it's telling that I couldn't discover any other people who were running into this.
[1] Yes, I'm sure that you've never had a significant problem with the tool. There are many, many useful tools out there that folks rave about that have burned me and burned the folks I work with badly enough that we won't touch them, ever.
[2] Also, if you learned to use Bazel inside the Googleplex, then that absolutely is a scenario covered by my statement: "I'm sure it's legitimately fuckin amazing when you learn it at a company that's big enough to have one or more entire teams dedicated to internal developer tooling..."
So the development experience Github is marching towards doesn't compare?
E.g., Codespaces, CoPilot, Actions, etc?
(genuinely curious, not trolling)
E.g., Codespaces, CoPilot, Actions, etc?
(genuinely curious, not trolling)
It does! Working like Google is like that but having had decades to build that dream, using the resources that only a megacorp could bring to bear in the problem, and in a monorepo. The fact that it's a monorepo is not to be dismissed. git doesn't work for monorepos.
Codespaces is like CitC; I haven't been a Google since the rise of AI so I can't comment on how the internal equivalent to Copilot is; Actions is very primitive compared to what Google has, but yeah, you can see where it's going. it doesn't compare right now but it could, eventually.
Codespaces is like CitC; I haven't been a Google since the rise of AI so I can't comment on how the internal equivalent to Copilot is; Actions is very primitive compared to what Google has, but yeah, you can see where it's going. it doesn't compare right now but it could, eventually.
The Gemini-powered copilot is fine, but has so far, for me, only done automated like "heres a simple method based on the comment you wrote", or "I patched in a change to the other callsites when you renamed this thing". I have also not used it in 2 months so it's probably better already.
I still mainly do my LLM assisted coding in chat-style interfaces that are more like pair programming by mail.
I think the Gemini code assist is coming out as a consumer product, I forgot the name, because Google is awful at branding.
I still mainly do my LLM assisted coding in chat-style interfaces that are more like pair programming by mail.
I think the Gemini code assist is coming out as a consumer product, I forgot the name, because Google is awful at branding.
> git doesn't work for monorepos.
Why not? Is it related to partial checkouts?
Why not? Is it related to partial checkouts?
Its both. The tooling has a very direct impact on the quality of the code.
I think the reason its not easy replicable is:
1. It takes a ton of initial investment and ongoing maintenance but its worth it when your code base is gigantic.
2. There is a consistent set of top down enforced rules. With the consistency it becomes much, much easier to build tight integrations between tools.
(almost?) everything is buildable by a single build system (blaze). When anyone can consistently build/test/run anything in your codebase it becomes a lot easier to build a whole host of potential tools like code search.
Probably someone can dive deeper than I can. But one thing I learned the most important property for a code base to be maintainable/scalable is consistency.
I think the reason its not easy replicable is:
1. It takes a ton of initial investment and ongoing maintenance but its worth it when your code base is gigantic.
2. There is a consistent set of top down enforced rules. With the consistency it becomes much, much easier to build tight integrations between tools.
(almost?) everything is buildable by a single build system (blaze). When anyone can consistently build/test/run anything in your codebase it becomes a lot easier to build a whole host of potential tools like code search.
Probably someone can dive deeper than I can. But one thing I learned the most important property for a code base to be maintainable/scalable is consistency.
One more thought, is also how much other systems can utilize the same tooling/workflows by just storing things in source code. Things that would probably traditionally stored as application state in a database are often stored instead in google3 as config.
Things like on team rosters, on call rosters, code review permissions, service permissions, feature flags.
All of it stored in google3 and can all utilize the same consistent set of tooling like presubmit test, integration tests, deployment automation, permissions, code search.
Its sort of like Infrastructure as Code but more.
Things like on team rosters, on call rosters, code review permissions, service permissions, feature flags.
All of it stored in google3 and can all utilize the same consistent set of tooling like presubmit test, integration tests, deployment automation, permissions, code search.
Its sort of like Infrastructure as Code but more.
Org as code
I also love the built-in history diffing that this gives you on those files.
> When anyone can consistently build/test/run anything in your codebase it becomes a lot easier to build a whole host of potential tools like code search.
As someone who hasn't worked for Google, how does Google's implementation of this differ from e.g. Guix/nixpkgs? Being able to easily build/test any available package is a big reason I like using tools like those.
As someone who hasn't worked for Google, how does Google's implementation of this differ from e.g. Guix/nixpkgs? Being able to easily build/test any available package is a big reason I like using tools like those.
I'm unfortunately not that familiar with the internals.
I think the public bazel (public version of blaze) docs is probably the best place to look.
https://bazel.build/basics/hermeticity
I think the public bazel (public version of blaze) docs is probably the best place to look.
https://bazel.build/basics/hermeticity
> (and if it's tooling, why can't that type of tooling be replicated for other codebases outside of google?)
The elegance of the tooling from what I hear is that there's tons of different tools maintained by different teams that work seamlessly (and fast) together to produce google3 and all of its supporting pieces.
But to answer your question, sure it can. But good luck building your own. Google has been doing this since the 2000s.
And if you're a big company already, you've already bought into your existing patterns & design choices; things like that are VERY hard to change.
The elegance of the tooling from what I hear is that there's tons of different tools maintained by different teams that work seamlessly (and fast) together to produce google3 and all of its supporting pieces.
But to answer your question, sure it can. But good luck building your own. Google has been doing this since the 2000s.
And if you're a big company already, you've already bought into your existing patterns & design choices; things like that are VERY hard to change.
One that had a sort of improvised facade/adapter pattern (it didn't really follow either) in a clearly cut multilayered and pipelined structure, with actor model bits where it made sense.
The code wasn't simple, at all. It took active training of new arrivals for them to understand it. But it was very well thought out, with very few warts given the complexity, and extremely easy to extend (that was the main requirement, given constant changes in APIs and clients).
We had an API, with multiple concurrent versions, that transformed requests into an intermediate model, on which our business logic operated, later targetted external APIs (dozens of them, some REST, some SOAP, some under NDAs, some also with multiple versions), whose responses turned again into the intermediate model, with more business logic on our end, and a final response through our API. Each transaction got its context serialized so we could effectively have what was an, again improvised, "async/await"-like syntax in what was (trigger warning) C++03 code.
The person who engineered it didn't have formal CS background.
The code wasn't simple, at all. It took active training of new arrivals for them to understand it. But it was very well thought out, with very few warts given the complexity, and extremely easy to extend (that was the main requirement, given constant changes in APIs and clients).
We had an API, with multiple concurrent versions, that transformed requests into an intermediate model, on which our business logic operated, later targetted external APIs (dozens of them, some REST, some SOAP, some under NDAs, some also with multiple versions), whose responses turned again into the intermediate model, with more business logic on our end, and a final response through our API. Each transaction got its context serialized so we could effectively have what was an, again improvised, "async/await"-like syntax in what was (trigger warning) C++03 code.
The person who engineered it didn't have formal CS background.
SecDB at Goldman. There were runtime issues aplenty but the SDLC was top notch and I’ve never heard of or seen better - and it’s 30 years old. Instant deployments and rollbacks globally, you show up to work and commit to prod in the first half day. Within a week you can go on rota. It’s why Goldman was so nimble in the 90’s, and 00’s. It’s still a remarkable system but underinvestment has taken its toll after margins evaporated in 2010.
Although I'm only on job three and have not had that much involvement with open source, I think my current employer (Attio) has one of the best codebases I've seen.
Qualitatively, I experience this in a few ways: * Codebase quality improves over time, even as codebase and team size rapidly increase * Everything is easy to find. Sub-packages are well-organised. Files are easy to search for * Scaling is now essentially solved and engineers can put 90% of their time into feature-focused work instead of load concerns
I think there are a few reasons for this:
* We have standard patterns for our common use cases * Our hiring bar is high and everyone is expected to improve code quality over time * Critical engineering decisions have been consistently well-made. For example, we are very happy to have chosen our current DB architecture, avoided GraphQL and used Rust for some performance-critical areas * A TypeScript monorepo means code quality spreads across web/mobile/backend * Doing good migrations has become a core competency. Old systems get migrated out and replaced by better, newer ones * GCP makes infra easy * All the standard best practices: code review, appropriate unit testing, feature flagging, ...
Of course, there are still some holes. We have one or two dark forest features that will eventually need refactoring/rebuilding; testing needs a little more work. But overall, I'm confident these things will get fixed and the trajectory is very good.
Qualitatively, I experience this in a few ways: * Codebase quality improves over time, even as codebase and team size rapidly increase * Everything is easy to find. Sub-packages are well-organised. Files are easy to search for * Scaling is now essentially solved and engineers can put 90% of their time into feature-focused work instead of load concerns
I think there are a few reasons for this:
* We have standard patterns for our common use cases * Our hiring bar is high and everyone is expected to improve code quality over time * Critical engineering decisions have been consistently well-made. For example, we are very happy to have chosen our current DB architecture, avoided GraphQL and used Rust for some performance-critical areas * A TypeScript monorepo means code quality spreads across web/mobile/backend * Doing good migrations has become a core competency. Old systems get migrated out and replaced by better, newer ones * GCP makes infra easy * All the standard best practices: code review, appropriate unit testing, feature flagging, ...
Of course, there are still some holes. We have one or two dark forest features that will eventually need refactoring/rebuilding; testing needs a little more work. But overall, I'm confident these things will get fixed and the trajectory is very good.
Every code base I have ever worked on was a legacy nightmare. Every "greenfield" project I joined turned into a legacy nightmare within weeks. I have never encountered enjoyable code. I had the displeasure of wading through Spring, Hibernate and Apache HTTP client code before and they were all an incomprehensible mess.
My conclusion: You know the claim "any medication that really has an effect must also have side effects". I would like to adapt that for code: Any code that does a lot of useful and complex things must be an arcane, barely maintainable mess that can only be understood by deep study.
My conclusion: You know the claim "any medication that really has an effect must also have side effects". I would like to adapt that for code: Any code that does a lot of useful and complex things must be an arcane, barely maintainable mess that can only be understood by deep study.
Something in your answer triggered a flash back to when I worked for a phone company, 20 years ago. One team, under the leadership of our questionable chief architect, had produced our new, all encompassing backend system. This was to be the corner stone of all future development and integration, dog slow and complicated as it where. I worked as a .Net developer and had the misfortune to be among the first to integrate with this monster. Try as I might, I could not get .Net to interoperate with these services. Finally I figured out that the SOAP services was using some old deprecated versions. Going back to the architect, I ask "did you build this on Apache Axis, and please say Axis2", but no, it was just Axis, a deprecated version, that would generate webservices not supported by newer .Net version. That wasn't an problem, because: "None of our project have upgraded to those .Net versions yet"... DUDE, we've launched a brand new system based on that .Net version a year ago, and what was you f-ing plan for the future, to redo every single service using Axis2?
This guy had based a brand new system on a framework/library that was no longer maintained, even before the system was launched.
This guy had based a brand new system on a framework/library that was no longer maintained, even before the system was launched.
"Any code that does a lot of useful and complex things must be an arcane, barely maintainable mess that can only be understood by deep study."
"Every code base I have ever worked on was a legacy nightmare. Every "greenfield" project I joined turned into a legacy nightmare within weeks"
I am sorry to say this, but it really sounds like you were either really, really unlucky, or part of the reason why it became a mess.
Complicated things are complicated. Nothing can ever change that and it requires study to understand it.
But it still does matter a lot, how one organizes the whole thing. How it is structured, documented(!), refactored. Are there competent people in charge who understand it all and kick peoples asses if they make even a temporary mess or forget to document, or do random people make changes wherever they see fit, because a deadline is ticking?
Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.
And if one has to ship things, it is not always possible to keep it pure and if the code is not intended to live forever then this is often fine. But if the codebase is supposed to stay, then there needs to be the time to clean up the hacks. Or don't and then you end up scared touching anything.
That being said, the technologies you mentioned, I would not like to touch either..
"Every code base I have ever worked on was a legacy nightmare. Every "greenfield" project I joined turned into a legacy nightmare within weeks"
I am sorry to say this, but it really sounds like you were either really, really unlucky, or part of the reason why it became a mess.
Complicated things are complicated. Nothing can ever change that and it requires study to understand it.
But it still does matter a lot, how one organizes the whole thing. How it is structured, documented(!), refactored. Are there competent people in charge who understand it all and kick peoples asses if they make even a temporary mess or forget to document, or do random people make changes wherever they see fit, because a deadline is ticking?
Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.
And if one has to ship things, it is not always possible to keep it pure and if the code is not intended to live forever then this is often fine. But if the codebase is supposed to stay, then there needs to be the time to clean up the hacks. Or don't and then you end up scared touching anything.
That being said, the technologies you mentioned, I would not like to touch either..
I think that's a bit cruel. Usually mess happens because lots of people with different ideas about the future needs and best structure meet up in the code - and it's hard to develop a consistent culture.
Also codebases get too large for any one person to refactor them into shape in the time they have each day. So you end up needing people who are responsible just for keeping things in shape.
Also codebases get too large for any one person to refactor them into shape in the time they have each day. So you end up needing people who are responsible just for keeping things in shape.
Well, that is why I said with those words it sounds like this to me, not that he or she was in fact responsible. (at least this is what I meant)
"Also codebases get too large for any one person to refactor them into shape in the time they have each day."
Which is why modules, or subprojects were invented. Or however you want to call it, if one person is only responsible for a small part and not everyone for everything. And yes, there is also the non trivial problem of time management.
"Also codebases get too large for any one person to refactor them into shape in the time they have each day."
Which is why modules, or subprojects were invented. Or however you want to call it, if one person is only responsible for a small part and not everyone for everything. And yes, there is also the non trivial problem of time management.
yup, but the style of the modules starts to diverge as different people maintain them.
IMO one does want people whose fulltime job is to looking at a codebase orthogonally to those who are just implementing a feature. People who make sure it builds fast, is secure tries to be consistent to some degree etc.
Many companies call teams/indivuduals like this a "cost centre" and disparage it because they are dolts.
IMO one does want people whose fulltime job is to looking at a codebase orthogonally to those who are just implementing a feature. People who make sure it builds fast, is secure tries to be consistent to some degree etc.
Many companies call teams/indivuduals like this a "cost centre" and disparage it because they are dolts.
If those code checkers are experienced programmers, who do not annoy people with arbitary guidelines and standards, then yes, this might also work. But I do not think they are always necessary as a seperate full time role, if everything works normal otherwise. And with working normal, I mean there is enough time to refactor and document and clean up and is actually done.
> Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.
I think the key aspect is a bit different: understandability. That is, if you open two pages of random code, then can you roughly understand and follow what it roughly does?
Everything in a single package/module/directory can be quite understandable. It doesn't necessarily need to be modular, but it often does help. "Understandability" is more broad though, and also includes things such as not having weird hard to understand functions nested 8 levels deep, and things like that.
It also means not over-engineering things. "Our code is hard to understand, therefore, we need more architecture" is a fallacy I've seen a few times. The code wasn't hard to understand due to lack of modularity or architecture, but because it was just badly written unclear code.
I think the key aspect is a bit different: understandability. That is, if you open two pages of random code, then can you roughly understand and follow what it roughly does?
Everything in a single package/module/directory can be quite understandable. It doesn't necessarily need to be modular, but it often does help. "Understandability" is more broad though, and also includes things such as not having weird hard to understand functions nested 8 levels deep, and things like that.
It also means not over-engineering things. "Our code is hard to understand, therefore, we need more architecture" is a fallacy I've seen a few times. The code wasn't hard to understand due to lack of modularity or architecture, but because it was just badly written unclear code.
"It also means not over-engineering things. "Our code is hard to understand, therefore, we need more architecture" is a fallacy I've seen a few times. The code wasn't hard to understand due to lack of modularity or architecture, but because it was just badly written unclear code."
For sure, if I have 10 tightly separated micro modules, just for the sake of it, then this is not helping clarity.
But if everything is an entangled mess of spaghetti code, then modularisation helps, so I do not have to look into 10 places to get what a certain function is doing.
For sure, if I have 10 tightly separated micro modules, just for the sake of it, then this is not helping clarity.
But if everything is an entangled mess of spaghetti code, then modularisation helps, so I do not have to look into 10 places to get what a certain function is doing.
> Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.
100% agree. The nuance I find is that a lot times, people want to draw the lines of the module along something that's more immediately intuitive, but results in longer-term complexity.
100% agree. The nuance I find is that a lot times, people want to draw the lines of the module along something that's more immediately intuitive, but results in longer-term complexity.
Yes, the hard part is to define what goes inside one module. There are rules of thumb, but lots of exeptions. Taking any advice too literally can also make things awkward.
IMHO library code is especially challenging, as cruft has a tendency to accumulate, historical behavior needs to be preserved, and APIs are set in stone once they’re built.
My past three employers code bases: mono-repos, Bazel, lots ot C++ and Python, thousands of libraries and tools, code generation and modeling tools that are fully integrated into the build, easy cross compilation, large integration tests just one bazel test invocation away, hermetic and uniform dependencies...
Realistically any code base where the engineers had at least a basic understanding of programming. You do not know suffering until you've seen someone hard code basic variables, we're talking about strings all over the place, and then they just copy the function again to replace the strings .
I've legitimately left jobs over bad code. We're talking about code that did nothing in reality. The best code bases have been ones where I've been able to lead the direction. I get to know exactly how things work. I'm privileged to have a job where I essentially created the initial framework right now .
Plus I'm fully remote, life is pretty good.
I've legitimately left jobs over bad code. We're talking about code that did nothing in reality. The best code bases have been ones where I've been able to lead the direction. I get to know exactly how things work. I'm privileged to have a job where I essentially created the initial framework right now .
Plus I'm fully remote, life is pretty good.
> any code base where the engineers had at least a basic understanding of programming
I felt this in my soul as soon as I read it. The number of people writing code who should not even be speaking to others because of how clearly they don't understand anything technical is unfathomably high.
Our industry hires seat fillers and tells them to write software. And then they do, and it's every bit as bad as it sounds.
I felt this in my soul as soon as I read it. The number of people writing code who should not even be speaking to others because of how clearly they don't understand anything technical is unfathomably high.
Our industry hires seat fillers and tells them to write software. And then they do, and it's every bit as bad as it sounds.
Same.
Some genius hired a "software architect" who doesn't know how to write code. When I asked the genius why he would do that, he said it was because regular architects don't know how to build houses, so why would he know how to program?
That was some next-level shit. He showed us a diagram once where the database was calling github which was calling a microservice. These are the days before Github had a CI, so it was pretty funny.
Some genius hired a "software architect" who doesn't know how to write code. When I asked the genius why he would do that, he said it was because regular architects don't know how to build houses, so why would he know how to program?
That was some next-level shit. He showed us a diagram once where the database was calling github which was calling a microservice. These are the days before Github had a CI, so it was pretty funny.
Facebook but for different reasons than most. We called this swapping out the airplane parts mid-flight
A lot of effort goes into language design and tooling to enable continuous migration of code. Rather than re-writing entire repos, existing code is continuously upgraded through semi- & fully-automated code-mods. Every day thousands of patches are landed to upgrade APIs with new type safety, security restrictions, deprecations and other code maintenance.
Most other company repos I worked on had major re-writes outside of the mainline until one day there was a sudden and often troublesome switch-over to the new code.
Code is constantly changing, and unless you have a continuous process for landing and testing those changes, you are going to suffer when you try to apply the changes needed to address the accumulated tech-debt.
A lot of effort goes into language design and tooling to enable continuous migration of code. Rather than re-writing entire repos, existing code is continuously upgraded through semi- & fully-automated code-mods. Every day thousands of patches are landed to upgrade APIs with new type safety, security restrictions, deprecations and other code maintenance.
Most other company repos I worked on had major re-writes outside of the mainline until one day there was a sudden and often troublesome switch-over to the new code.
Code is constantly changing, and unless you have a continuous process for landing and testing those changes, you are going to suffer when you try to apply the changes needed to address the accumulated tech-debt.
> We called this swapping out the airplane parts mid-flight...
I work fairly heavily with Facebook's Graph API, and I think they may have employed too many Boeing engineers for the part swaps lately.
I work fairly heavily with Facebook's Graph API, and I think they may have employed too many Boeing engineers for the part swaps lately.
Most of the compatibility issues over the past few years at least have been due to added privacy restrictions and reduced permissions scoping.
The same few years have seen the developer bug tool closed down in favor of the mostly abandoned “community forums” no one uses.
We had big multi-year bugs in “assigned” state before its closure, too.
We had big multi-year bugs in “assigned” state before its closure, too.
That’s true too. The developer api is no longer a major growth generator as it once was. What remains is the bare minimum to keep partners happy, typically ads vendors
I really wish more people believed in continuous improvement and the malleability of software. Giant heroic refactors or redos are so annoying, but so are ten year old compiler or kernels.
> We called this swapping out the airplane parts mid-flight...
At ksplice we swapped the code out from under the kernel without rebooting, so I know what you mean.
At ksplice we swapped the code out from under the kernel without rebooting, so I know what you mean.
For curious readers, the "building a plane as it flies" metaphor dates back to at least 2000, to these commercials for EDS: https://www.youtube.com/watch?v=L2zqTYgcpfg
I thought it was IBM but I was quite mistaken, so good thing I looked it up lol
I thought it was IBM but I was quite mistaken, so good thing I looked it up lol
It was a 63kB assembly project that compiled down to 384 bytes. However, that project handled phase-locked control of a power dimmer, logged up to 2 days of power-factor stats information into an eeprom, integrated clock timers, and several user inputs.
It was the smallest, and most bizarre interleaved task-structure I'd seen fit into a $0.17 micro-controller. I kind of admired that the code itself formed the delays for other areas of the multiple tasks, and essentially achieved near perfect resource utilization for actual work.
Never saw anything that efficient again for several decades. =3
It was the smallest, and most bizarre interleaved task-structure I'd seen fit into a $0.17 micro-controller. I kind of admired that the code itself formed the delays for other areas of the multiple tasks, and essentially achieved near perfect resource utilization for actual work.
Never saw anything that efficient again for several decades. =3
The latest Go micro-service I have built.
About once a year roughly, for the last couple years, the opportunity has arisen to greenfield a Go micro-service with pretty loose deadlines.
Each time I have come into it with more knowledge about what went well and what I wasn't particularly happy with the last time. Each one has been better than the last.
I've been building software professionally for twenty years, and these micro-services have been one of the few projects in that time that have had clear unified vision and time to build with constant adjustments in the name of code quality.
About once a year roughly, for the last couple years, the opportunity has arisen to greenfield a Go micro-service with pretty loose deadlines.
Each time I have come into it with more knowledge about what went well and what I wasn't particularly happy with the last time. Each one has been better than the last.
I've been building software professionally for twenty years, and these micro-services have been one of the few projects in that time that have had clear unified vision and time to build with constant adjustments in the name of code quality.
Sounds fun. I have a similar policy but only every other year. It means I know what the hell I am talking about when in meetings with people who have slow or inefficient services. It also means my job of mostly telling other people how to solve problems is still a lot of fun for me.
Go micro-service with pretty loose deadlines
Sounds like A Money Printer, congrats.
Sounds like A Money Printer, congrats.
The best codebases I worked on were from startups that failed due to lack of product market fit. Oh man, their code was so beautiful and optimized…
Is this true or a bit of humor? Of the three startups I worked for, the one that had the best exit also had the best code. Ime shitty code hamstrings startups into slow execution during the growth phase, and kills them despite pmf, because competition winds up moving much faster.
Not humor. Just a random observation I realized when I read the title of the topic.
The best codebases have been my own, because I can monopolize the style and design to my own exacting vision of how things should be.
The best work codebases are ones where I may disagree with a lot of the style, but the lack of enforcement at the level of classes and functions gives us creative freedom, builds mutual respect, and enables ego-less, pragmatic, tradeoff-driven discussions about things which are actually important. If these discussions end with a majority but not universal agreement, then its fine, the minority is not offended, and we happily continue with our work.
That might also be because we're all pragmatists and somewhat cynical of shiny things, or the latest fad.
The best work codebases are ones where I may disagree with a lot of the style, but the lack of enforcement at the level of classes and functions gives us creative freedom, builds mutual respect, and enables ego-less, pragmatic, tradeoff-driven discussions about things which are actually important. If these discussions end with a majority but not universal agreement, then its fine, the minority is not offended, and we happily continue with our work.
That might also be because we're all pragmatists and somewhat cynical of shiny things, or the latest fad.
I’m firmly convinced that any codebase written by any 4 people in the same room is better than any written by any higher number or distributed.
You'll be surprised, I think the best codebase I ever contributed to is the Rust compiler, which is being built by a huge distributed team.
Despite it being complex and me being in there for the first time, I could quickly track down a bug in the parser and fix it. As a cherry on top the communication around the pull request was top notch too.
Despite it being complex and me being in there for the first time, I could quickly track down a bug in the parser and fix it. As a cherry on top the communication around the pull request was top notch too.
I’m not trying to say it can’t be good despite that factor, but if the rust compiler were written by the top 4 contributors sitting in the same room, it’d be even better (and possibly not finished yet ;) )
You could at least contribute to the thread instead of grandstanding about an opinion you have.
Name the codebases.
Name the codebases.
It took my company a team of 8 fully-remote devs to fix the clusterfuck of a codebase that the "4 guys in a room" (the CTO and early devs) built up with spit and tree bark,.
I’ve seen plenty of examples of exactly the opposite.
The entire world disagrees, Windows, The Linux Kernel, Anything at Netflix, Google, Facebook etc. Try getting them all in a room.
So NVIDIA must write terrible code?
That would not surprise me. There’s a reason they need all that hardware.
I work at Google on database systems and would still pick the VMware codebase (specifically the virtual machine monitor component).
It was by far the most impressive piece of software engineering I’ve ever had privilege of perusing.
It was by far the most impressive piece of software engineering I’ve ever had privilege of perusing.
[deleted]
Vintage answer
NN (Net News) - it was a text based reader for USENET news, first released in 1984. It's been a minute since then, but I recall it to have been in C. This was before the Internet was highly monetized, and when most projects were open source, written by people who needed (or wanted) a tool, and had the combination of skills, experience, and time to make it happen.
The code was good because it was well designed, and well maintained. Routines were modular and reusable. If you needed to modify something, you could find the right place to make a change, add a feature, and submit it to the repository. Maintainers did not allow crap submissions back into the published code.
So yes to all prompts, with the bonus of having passionate engineers doing something they loved. I'm not sure my contributions ever made it through the "not crap" filter, but they did work beautifully for my employer.
Good code is functional, understandable, flexible, and can be modified without a complete overhaul.
My compliments to all who contributed to NN!
https://www.nndev.org/
NN (Net News) - it was a text based reader for USENET news, first released in 1984. It's been a minute since then, but I recall it to have been in C. This was before the Internet was highly monetized, and when most projects were open source, written by people who needed (or wanted) a tool, and had the combination of skills, experience, and time to make it happen.
The code was good because it was well designed, and well maintained. Routines were modular and reusable. If you needed to modify something, you could find the right place to make a change, add a feature, and submit it to the repository. Maintainers did not allow crap submissions back into the published code.
So yes to all prompts, with the bonus of having passionate engineers doing something they loved. I'm not sure my contributions ever made it through the "not crap" filter, but they did work beautifully for my employer.
Good code is functional, understandable, flexible, and can be modified without a complete overhaul.
My compliments to all who contributed to NN!
https://www.nndev.org/
google3, all the devex tooling was taken care of by other teams. Tons of useful library functions available to import, accumulated over decades.
The best codebase I've worked on was a 3D editor built using the waterfall method (written in Qt and C++).
Most of the engineers were above average.
Everything was planned carefully in advance and documented carefully, down to individual requirements. We even had some nice to have, optional requirements. After literally weeks of just documents and iterating on the architecture, the code for it was written easily in a week basically in auto mode - if we had AI agents back then we could have had the AI agents write the implementation.
During the implementation we realised there were some technical shortcomings (eg. number of lights supported by the engine) and clients wanted some changes, so we had to update our bible documentation and do a few changes. Some were small, some required to swap completely the rendering engine but we managed to keep everything to the same interfaces we planned at the beginning because the architecture was rock solid and we thought carefully about extensibility.
It may have costed more in engineering time than if we threw requirements on a kanban board and iterated on them - but then we would end up with arguably a worse architecture and no documentation.
Most of the engineers were above average.
Everything was planned carefully in advance and documented carefully, down to individual requirements. We even had some nice to have, optional requirements. After literally weeks of just documents and iterating on the architecture, the code for it was written easily in a week basically in auto mode - if we had AI agents back then we could have had the AI agents write the implementation.
During the implementation we realised there were some technical shortcomings (eg. number of lights supported by the engine) and clients wanted some changes, so we had to update our bible documentation and do a few changes. Some were small, some required to swap completely the rendering engine but we managed to keep everything to the same interfaces we planned at the beginning because the architecture was rock solid and we thought carefully about extensibility.
It may have costed more in engineering time than if we threw requirements on a kanban board and iterated on them - but then we would end up with arguably a worse architecture and no documentation.
The best codebase is the one you fully understand. I prefer codebases that are small enough to understand within a week. This is why I like Microservices. Large codebases can be overwhelming and even senior developers working a decade in the company of might not fully understand them. Instead, I prefer maintaining a few Microservices that our team fully comprehends, where the entire codebase fits into a clear mental model. We then interact with other codebases, that have active mental models in other teams, via APIs.
I'm skeptical of this because I think that a well-architected large codebase should be as good as multiple microservices stapled together at the seams. But then I don't do much web dev.
> I think that a well-architected large codebase
I agree with you, but the problem is that such codebases rarely exist in practice. This is partly due to laziness and time pressure. When high standards are not maintained or time becomes a constraint, developers start to make compromises in large codebases, such as placing or connecting functionality in inappropriate parts of the codebase. This is much harder to do when access is restricted by an API (contract) between teams, as it forces communication.
Additionally, if parts of the codebase are completely messed up, refactoring in large codebases can be extremely difficult due to complex interconnections with numerous other components. In contrast, with a microservices architecture, if you have 50 microservices and five are poorly designed, you can rewrite them from scratch while ensuring the new versions maintain the same API. This ensures that the other 45 services adhere to the API contract and cannot interfere in any other way.
I agree with you, but the problem is that such codebases rarely exist in practice. This is partly due to laziness and time pressure. When high standards are not maintained or time becomes a constraint, developers start to make compromises in large codebases, such as placing or connecting functionality in inappropriate parts of the codebase. This is much harder to do when access is restricted by an API (contract) between teams, as it forces communication.
Additionally, if parts of the codebase are completely messed up, refactoring in large codebases can be extremely difficult due to complex interconnections with numerous other components. In contrast, with a microservices architecture, if you have 50 microservices and five are poorly designed, you can rewrite them from scratch while ensuring the new versions maintain the same API. This ensures that the other 45 services adhere to the API contract and cannot interfere in any other way.
This is going to sound insane, but I worked on a firmware codebase at an industrial automation company that was remarkable.
Everything was working against it. No RTOS. Subversion. GCC 5-point-something (I think?).
It was an incredible mass of preprocessor crimes. I'm talking about #including the same file 10 times and redefining some macros each time to make the file do something different.
It used a stackless coroutine library called Protothread, which itself is a preprocessor felony.
And yet? It was brilliant. Compilation was lightning quick. F5, lean back in your chair, and boom, you're running your code. I understand that this kind of thing is normal for web/backend/etc folks, but I yearn for the days of sub-15 second firmware compile times.
It was easy to flip a couple flags and compile it to a Win32-based simulator. Preprocessor felonies are felonies, but when you stick to a small handful of agreed-upon felonies, you can actually reap the benefits of some very sophisticated metaprogramming, while staying close enough to the hardware that it's easy to look at a datasheet and understand exactly what your code is doing.
Everything was working against it. No RTOS. Subversion. GCC 5-point-something (I think?).
It was an incredible mass of preprocessor crimes. I'm talking about #including the same file 10 times and redefining some macros each time to make the file do something different.
It used a stackless coroutine library called Protothread, which itself is a preprocessor felony.
And yet? It was brilliant. Compilation was lightning quick. F5, lean back in your chair, and boom, you're running your code. I understand that this kind of thing is normal for web/backend/etc folks, but I yearn for the days of sub-15 second firmware compile times.
It was easy to flip a couple flags and compile it to a Win32-based simulator. Preprocessor felonies are felonies, but when you stick to a small handful of agreed-upon felonies, you can actually reap the benefits of some very sophisticated metaprogramming, while staying close enough to the hardware that it's easy to look at a datasheet and understand exactly what your code is doing.
> Preprocessor felonies are felonies, but when you stick to a small handful of agreed-upon felonies, you can actually reap the benefits
There's a saying, "Only commit one crime at a time". Knowing when to cheat is a fairly high-risk but pro move.
There's a saying, "Only commit one crime at a time". Knowing when to cheat is a fairly high-risk but pro move.
I haven't contributed to it yet, but one of my personal projects involves hacking NetBSD. Absolutely, legendarily good code base, so thoroughly documented you can actually begin kernel hacking just by reading the man pages. The code itself is really consistent and easy to read; it's pretty clear the NetBSD team enforces a consistent style as well as high standards for thorough documentation. NetBSD is how C code should be done.
The other code base I really liked was QRes, ITA Software's airline reservation system written in Common Lisp. I contracted at Google a few years ago doing maintenance on this code base before it was shuttered for good. Aside from being just high-quality Lisp code, it had its own testing language that allowed new tests to be written in just a few lines or so -- tests that sent XML queries with generated data to the running system and checked for specific results in specific XPath paths. Because the system was Lisp, implementing and extending this testing language was relatively easy. Using it even more so. Truly the only system I was happy to write tests for, rather than seeing it as a necessary chore.
The other code base I really liked was QRes, ITA Software's airline reservation system written in Common Lisp. I contracted at Google a few years ago doing maintenance on this code base before it was shuttered for good. Aside from being just high-quality Lisp code, it had its own testing language that allowed new tests to be written in just a few lines or so -- tests that sent XML queries with generated data to the running system and checked for specific results in specific XPath paths. Because the system was Lisp, implementing and extending this testing language was relatively easy. Using it even more so. Truly the only system I was happy to write tests for, rather than seeing it as a necessary chore.
The first few years at a previous startup I worked on. Java. CI/CD in place when merging PRs. But mostly, the thing that made the code base great was the fact that the code wrapped external libraries in small interfaces that were _owned by the company_, and dependency was used to not depend on implementation interfaces. In short, adhering to the basics on the SOLID principles.
Kinda sounds like abstraction hell. What did you gain by wrapping all external libraries in interfaces?
In my experience, any third-party dependency will inevitably eventually be in-sourced. Third-party dependencies are technical debt - they let you make fast progress initially, but have recurring maintenance costs.
Wrapping the API lets you see, all in one place, the surface area of an external API that is in use, and minimizes the changes required here when reimplementing it.
---
Also, not that Clean Code is any great authority (but it is popular!):
> [Wrapping] third-party APIs is a best practice. When you wrap a third-party API, you minimize your dependencies upon it: You can choose to move to a different library in the future without much penalty. Wrapping also makes it easier to mock out third-party calls when you are testing your own code.
> One final advantage of wrapping is that you aren’t tied to a particular vendor’s API design choices. You can define an API that you feel comfortable with.
Wrapping the API lets you see, all in one place, the surface area of an external API that is in use, and minimizes the changes required here when reimplementing it.
---
Also, not that Clean Code is any great authority (but it is popular!):
> [Wrapping] third-party APIs is a best practice. When you wrap a third-party API, you minimize your dependencies upon it: You can choose to move to a different library in the future without much penalty. Wrapping also makes it easier to mock out third-party calls when you are testing your own code.
> One final advantage of wrapping is that you aren’t tied to a particular vendor’s API design choices. You can define an API that you feel comfortable with.
oasisaimlessly's response says it all.
Lean 4 is the best codebase I've worked on. It's a functional programming language and theorem prover that really stands out. What makes it great is how it combines programming with formal proofs. The code is super clear and well-organized. Even though it's complex, it's easy to read and understand. The documentation is top-notch, with machine-checked proofs for core libraries. It has this cool metaprogramming system that lets you generate code and automate proofs elegantly. Working on Lean 4 has been amazing - it's pushed my skills to new levels. It's rare to find a project where code, math, and logic come together so nicely. Every time I dive into the codebase, I learn something new.
A blank slate, so I can make the first mistake :)
I agree, the best code base is always a zero-line code base.
After that first line, it’s all downhill from there… :-)
So true!
I spend so much time obsessing over how what I am about to write ties in with what has already been written; or fuming over the stupidity of earlier decisions (usually made by myself). A blank slate is incredibly refreshing.
I spend so much time obsessing over how what I am about to write ties in with what has already been written; or fuming over the stupidity of earlier decisions (usually made by myself). A blank slate is incredibly refreshing.
I haven’t contributed code to it, but I’ve read lots of code in it ( for work reasons ) - I really like the golang compiler and library codebase.
About codebases I’ve written code for, the best one strived for simplicity, and was driven by very strong engineers who actively considered code hygiene ( in the broadest possible sense ) a first class citizen.
About codebases I’ve written code for, the best one strived for simplicity, and was driven by very strong engineers who actively considered code hygiene ( in the broadest possible sense ) a first class citizen.
Go, SQLite, Redis are the three that came to mind for me. Simple, readable, fairly flat hierarchy.
A Python version about core features of the Hindley-Milner type inferencer: https://github.com/ethe/typer
It's kindof a dumb answer, but `abcde` (A Better CD Encoder) continues to be my go-to reference for "everything shell".
https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=summ...
https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
...it's just so damned... like... it _is_ 5000 lines of bash, but given the constraints and the problem domain, it's an incredible (and well-organized) feat.
If I ever question "How would I do something like XXX in shell?", I can usually find some clues, inspirations, or answers in the `abcde` codebase.
https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=summ...
https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
...it's just so damned... like... it _is_ 5000 lines of bash, but given the constraints and the problem domain, it's an incredible (and well-organized) feat.
If I ever question "How would I do something like XXX in shell?", I can usually find some clues, inspirations, or answers in the `abcde` codebase.
> https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
Interesting. I was a bit surprised by the use of echo in the usage() function (and not cat and a heredoc) -- but that certainly looks like a very structured shell script.
Thanks for sharing.
Interesting. I was a bit surprised by the use of echo in the usage() function (and not cat and a heredoc) -- but that certainly looks like a very structured shell script.
Thanks for sharing.
Probably the microservices-based one for async video messaging (i.e. Slack for video) for workers in the field. Each service was small enough that we could do a blue-green deploy to prod in about 2 minutes running only the service's tests and a tiny (intentionally limited) set of about 8 system journey tests (can onboard a new user, user can create a contact/group, user can send a common content-type message, user can receive messages, user can react/respond to a message). Every commit to main/master automatically either deployed to prod or broke the CI/CD pipeline and needed to be fixed ASAP. Each service was also well-known by team members that it literally could be rewritten in a week or two if desired to change a key part of its design.
Thinking about what I wrote, I suppose my criteria for a good codebase is one that has the lowest friction to change: a fast edit/run/test/debug loop, a meaningful sense of security (without dogma), and fast automated deployment/revert (via blue/green). Given those a bad codebase can become a good one again (by uncoordinated action of individuals) without everyone having to buy into a large investment.
One that I built and maintained 100% on my own. Nothing like doing things the way you like to without someone else messing everything up.
The one I’m working at right now!
What made it great: the company is 7 years old, I work basically since inception, half year late.
The codebase had some serious problems due to migrating build systems, rapid team growth and lack of proper communication while developing important infrastructure tools, lack of foresight at the start of a crucial requirement for the whole system that became clear 3 years in.
Aaand… Due to unique sequence of events half a year ago we had to start everything from scratch! We could not use a line of code from the old repo !
This brought miracles out! We started designing the right things right. People started talking to each other on the global scale, without the code limiting them!
The old code not only had problems on the engineering level, but it held people and team responsibilities along suboptimal interface boundaries! This is the biggest magic! After the rewrite started, all the people in the team landed exactly where they wanted and their areas of responsibility became clear and highly efficient. Some people started shining like I could not imagine. Because their organizational and communication skills were limited by a huge suboptimal codebase.
Also, it is very empowering to design a system when your whole team has real experience 5 years into the future. Also, everyone turned out to be caring about code simplicity and quality, I finally feel at home.
I could not wish for a better developer experience. The sexiest C++ coding time in my life :)
What made it great: the company is 7 years old, I work basically since inception, half year late.
The codebase had some serious problems due to migrating build systems, rapid team growth and lack of proper communication while developing important infrastructure tools, lack of foresight at the start of a crucial requirement for the whole system that became clear 3 years in.
Aaand… Due to unique sequence of events half a year ago we had to start everything from scratch! We could not use a line of code from the old repo !
This brought miracles out! We started designing the right things right. People started talking to each other on the global scale, without the code limiting them!
The old code not only had problems on the engineering level, but it held people and team responsibilities along suboptimal interface boundaries! This is the biggest magic! After the rewrite started, all the people in the team landed exactly where they wanted and their areas of responsibility became clear and highly efficient. Some people started shining like I could not imagine. Because their organizational and communication skills were limited by a huge suboptimal codebase.
Also, it is very empowering to design a system when your whole team has real experience 5 years into the future. Also, everyone turned out to be caring about code simplicity and quality, I finally feel at home.
I could not wish for a better developer experience. The sexiest C++ coding time in my life :)
Interesting! Seems like you were uniquely well-positioned to tackle the infamous system rewrite in a way that worked out great.
Fixing the interface boundaries is a strong case for it despite all of the downsides (mostly opportunity cost, I'm thinking).
Fixing the interface boundaries is a strong case for it despite all of the downsides (mostly opportunity cost, I'm thinking).
But opportunity cost might not really turn out to be a real factor.
Code is a liability, we only would like to think that our code is worth keeping, but the reality is it drags us down by definition: more code requires more manpower to support, this is objective truth.
In the old system a lot of things were really slow. I know that now half year in we already have a couple of problems solved that required me and a small team’s half a year work to solve and right now Im working on a problem that almost was not solvable at all before, I’m planning it infractructurally solved by the end of the year.
Other teams say they threw away a lot of grabage that grew with the old implementation and kept only “the good parts”.
A lot of work done by the teams is now offloaded to the infrastrucutre.
We have remote IDE for everyone from day one instead of build-inside-docker-do-what -you-want because old build system could not be and never was thrown away overnight.
List goes on
In the old system a lot of things were really slow. I know that now half year in we already have a couple of problems solved that required me and a small team’s half a year work to solve and right now Im working on a problem that almost was not solvable at all before, I’m planning it infractructurally solved by the end of the year.
Other teams say they threw away a lot of grabage that grew with the old implementation and kept only “the good parts”.
A lot of work done by the teams is now offloaded to the infrastrucutre.
We have remote IDE for everyone from day one instead of build-inside-docker-do-what -you-want because old build system could not be and never was thrown away overnight.
List goes on
Considering the biases, the one I wrote for the company I created.
When we have the opportunity to be in this context, keeping in mind what bothered us in the codebases with which we were able to work in the past, we can force ourselves not to reproduce the same errors. Like the unmaintained unit and integration tests, the lack of refactoring, other developers that use fancy technologies instead of simpler concepts more for the opportunity to play with technologies than real need..
And also, I guess, because we are more aware that the code is a reflection of the company that we want to have, that the simpler the better is a key point when we need to debug.
When we have the opportunity to be in this context, keeping in mind what bothered us in the codebases with which we were able to work in the past, we can force ourselves not to reproduce the same errors. Like the unmaintained unit and integration tests, the lack of refactoring, other developers that use fancy technologies instead of simpler concepts more for the opportunity to play with technologies than real need..
And also, I guess, because we are more aware that the code is a reflection of the company that we want to have, that the simpler the better is a key point when we need to debug.
Ruby on Rails.
It is the only framework I have read top to bottom.
Also the FreeBSD kernel, if you want to see a C code base that's quite beautiful (for C).
It is the only framework I have read top to bottom.
Also the FreeBSD kernel, if you want to see a C code base that's quite beautiful (for C).
In this category I would nominate Django as well. It's very well designed (opinionated, but usually for good reasons).
In terms of large C code bases I enjoy reading the PostgreSQL source code.
In terms of large C code bases I enjoy reading the PostgreSQL source code.
whenever I'm stuck on how to structure some code, I ask myself how would Laravel do it? and look up their code and structure mine similarly
My story is similar, but using Laravel as the canonical example of how not to structure things. Even without Facades, the internals are an impenetrable maze of magic methods, with things like Macroable adding methods from dog-knows-where. Even adding phpdoc types doesn’t work when dealing with things like scopes.
libjpeg 6 is pretty good. It's oldschool C so it's pretty spartan, but it's easy enough to read it and follow what it's doing if you have a basic understanding of the concepts. (Discrete cosine transform, Huffman coding, etc)
And it's only had a couple of vulnerabilities, in nearly 30 years of being on the Internet. That's not quite like DJB code, but it's darn close.
My understanding is that it was written by a very small number of people who were experts in the field, as part of the standards process way back in the early 90s.
And it's only had a couple of vulnerabilities, in nearly 30 years of being on the Internet. That's not quite like DJB code, but it's darn close.
My understanding is that it was written by a very small number of people who were experts in the field, as part of the standards process way back in the early 90s.
I worked on an incredible C++ codebase back in the dot.com. We built a mobile SVG renderer and packed into 128kb on a Palm 3 and similar code sizes on Blackberries, Symbian (the worst OS ever), Fujitsu phones, and whatever you threw at us.
I built a few systems. The DOM, a build system that could cross compile to a completely new target phone while the phone executives were at lunch with our sales team (that’s great demo!), and an optimizing build step in Perl that rewrote the C++ to minimize bytes, cycles and joules of energy.
The rendering engine was built by a genius. It was the most efficient and accurate vector graphics renderer I have ever seen.
But of course no one could understand how this system worked.
I mean, Perl that transpiles C++ into more optimized C++ just is abusive. I was not popular with the senior devs.
The main coders including me were coop students who worked furiously at all hours for the fun of it.
But of course the product had no revenue. The wiser devs probably knew this and didn’t care.
Nevertheless, we eventually were let go during the dot.com bust and the remaining main engineers had to throw the code out and try to rewrite it in Java. That didn’t work because in that era Java was too slow to fit on a phone. Or maybe the market was dead.
I learnt a solid lesson.
What’s amazing technically is not an amazing code base.
What is an amazing code base isn’t an amazing product.
What is an amazing product isn’t an amazing business.
Literally every product I have ever seen that is profitable is not a good code base. But if it pays me to code, I love it.
I built a few systems. The DOM, a build system that could cross compile to a completely new target phone while the phone executives were at lunch with our sales team (that’s great demo!), and an optimizing build step in Perl that rewrote the C++ to minimize bytes, cycles and joules of energy.
The rendering engine was built by a genius. It was the most efficient and accurate vector graphics renderer I have ever seen.
But of course no one could understand how this system worked.
I mean, Perl that transpiles C++ into more optimized C++ just is abusive. I was not popular with the senior devs.
The main coders including me were coop students who worked furiously at all hours for the fun of it.
But of course the product had no revenue. The wiser devs probably knew this and didn’t care.
Nevertheless, we eventually were let go during the dot.com bust and the remaining main engineers had to throw the code out and try to rewrite it in Java. That didn’t work because in that era Java was too slow to fit on a phone. Or maybe the market was dead.
I learnt a solid lesson.
What’s amazing technically is not an amazing code base.
What is an amazing code base isn’t an amazing product.
What is an amazing product isn’t an amazing business.
Literally every product I have ever seen that is profitable is not a good code base. But if it pays me to code, I love it.
The ones that were straightforward and close to the business. It starts at the obvious, works in an obvious way and has comment blocks at hard parts.
For this reason I despise most modern [web] projects, which have a weak start, immediately drop into “services” and “components”, do one action per source file per 30-50 lines of code, which are mostly imports and boilerplate, and have hundreds of these files. You can never tell what it does because it does almost nothing except formalities.
I also noted a tendency to use wrong paradigms for a language. E.g. it may have no closures (imagine that in 202x) so they use events as continuations for asynchronicity, which results in a mess. Or it isn’t immutable/functional, but they pretend it is, which results in fragility.
The best projects are both close to their business and written in a paradigm of the language used.
Was there someone enforcing good practices top down?
Natural time pressure is the best bs cleaner, imo. You write effing code, maybe have few hours a week to refactor strange parts. With no time pressure a project naturally becomes massaged by all members into the “likeable” form of their age.
For this reason I despise most modern [web] projects, which have a weak start, immediately drop into “services” and “components”, do one action per source file per 30-50 lines of code, which are mostly imports and boilerplate, and have hundreds of these files. You can never tell what it does because it does almost nothing except formalities.
I also noted a tendency to use wrong paradigms for a language. E.g. it may have no closures (imagine that in 202x) so they use events as continuations for asynchronicity, which results in a mess. Or it isn’t immutable/functional, but they pretend it is, which results in fragility.
The best projects are both close to their business and written in a paradigm of the language used.
Was there someone enforcing good practices top down?
Natural time pressure is the best bs cleaner, imo. You write effing code, maybe have few hours a week to refactor strange parts. With no time pressure a project naturally becomes massaged by all members into the “likeable” form of their age.
Any codebase that I had complete control over.
No, but more seriously, I've found that familiarity with the codebase is more important than having it be perfectly engineered. Once you're really familiar with the codebase, you know where dragons be, and you can make changes more easily. And God (PM) forbid, if you ever find yourself with some extra free time you might even reduce the size of dragons over time.
This brings me to my final point. Any codebase that I really enjoyed working with was the one that was constantly evolving. I don't mean rewriting everything from scratch every few months, but as long as I have permission (and time) to refactor the things that have been bothering me for months as patterns emerge, I'm a happy bee.
No, but more seriously, I've found that familiarity with the codebase is more important than having it be perfectly engineered. Once you're really familiar with the codebase, you know where dragons be, and you can make changes more easily. And God (PM) forbid, if you ever find yourself with some extra free time you might even reduce the size of dragons over time.
This brings me to my final point. Any codebase that I really enjoyed working with was the one that was constantly evolving. I don't mean rewriting everything from scratch every few months, but as long as I have permission (and time) to refactor the things that have been bothering me for months as patterns emerge, I'm a happy bee.
> Any codebase that I had complete control over.
No excuses. Code ownership is important. Sometimes it works for a team, sometimes only for individuals.
But not having to submit to core teams, architects and self-proclaimed experts of all kinds is a blessing.
I now work for an organization that discourages code ownership, and it struggles on many fronts:
No excuses. Code ownership is important. Sometimes it works for a team, sometimes only for individuals.
But not having to submit to core teams, architects and self-proclaimed experts of all kinds is a blessing.
I now work for an organization that discourages code ownership, and it struggles on many fronts:
1. core teams are dysfunctional
2. people find niches and stick to them
3. top talent is leaving, although pay is good and business creates real value for citizens
4. there is virtually no horizontal communication
5. mediocre ones rise to the level of their incompetence and infest the lives of others
6. and so on and so forth...
I’ve had exactly the same issues but because I couldn’t change anything without getting approval from 16.5 code owners on every PR submitted. It’s a real pain if you start modifying your coding for ‘least code owners hit’ instead of ‘best architecture’.
I like how my workplace does it -- there are rigorous codeowners and usually you only need approval from 1-2. if you do need approval from 5+, you can request a select 'super' codeowner review which will approve it for all.
I think core teams being helpful or harmful really comes down to the individuals.
The problem is at this level, most orgs don't have anyone to really judge or contest competency, so they hire the salesmen rather than the doers and when they don't, they tend to cheap out and just get inexperienced people.
Logically it makes a bunch of sense, though.
Why rebuild yet another platform? Why is your central platform bad? Usually it's not self-service, sometimes it's because it's built in cumbersome ways, other times its because it actually enforces good standards on you rather than just giving app your apps admin.
It's difficult for the person who hires the core team to differentiate between those complaints, unless they themselves both have the technical competency and the empathy to really understand the problem. They usually don't.
Point being, done well, it's great, but most folks can't do it well.
The problem is at this level, most orgs don't have anyone to really judge or contest competency, so they hire the salesmen rather than the doers and when they don't, they tend to cheap out and just get inexperienced people.
Logically it makes a bunch of sense, though.
Why rebuild yet another platform? Why is your central platform bad? Usually it's not self-service, sometimes it's because it's built in cumbersome ways, other times its because it actually enforces good standards on you rather than just giving app your apps admin.
It's difficult for the person who hires the core team to differentiate between those complaints, unless they themselves both have the technical competency and the empathy to really understand the problem. They usually don't.
Point being, done well, it's great, but most folks can't do it well.
I also echo the code ownership part.
I have a part-time gig where I maintain accounting software for a former client of mine. It takes up a few months' weekends a year.
I wrote about 60-70% of it when I was working for the owner of the software. It's something where as long as the client's happy, and they get new integrations and updates on time, they could keep using it for a decade longer.
I had almost complete ownership of the architecting of the software. It's broken down into a few microservices (think database, core business logic, reporting, auth, logging etc). The best thing I did at the time was pushing to use gRPC even though management felt it was too new tech.
The UI is in Angular, pain-free periodic upgrades. I've even rewritten some perf-sensitive code in Rust, and everyone's happy with snappier calculations.
The code hygiene is relatively good.
The only downside's that if someone else were to take over the code, they'd struggle (it's one of those things where I'm wearing many hats). I've been fortunate to be a professional accountant who moved into software engineering, so everything makes sense to me.
I have a part-time gig where I maintain accounting software for a former client of mine. It takes up a few months' weekends a year.
I wrote about 60-70% of it when I was working for the owner of the software. It's something where as long as the client's happy, and they get new integrations and updates on time, they could keep using it for a decade longer.
I had almost complete ownership of the architecting of the software. It's broken down into a few microservices (think database, core business logic, reporting, auth, logging etc). The best thing I did at the time was pushing to use gRPC even though management felt it was too new tech.
The UI is in Angular, pain-free periodic upgrades. I've even rewritten some perf-sensitive code in Rust, and everyone's happy with snappier calculations.
The code hygiene is relatively good.
The only downside's that if someone else were to take over the code, they'd struggle (it's one of those things where I'm wearing many hats). I've been fortunate to be a professional accountant who moved into software engineering, so everything makes sense to me.
> And God (PM) forbid, if you ever find yourself with some extra free time you might even reduce the size of dragons over time.
Honest question, what is the company like where you can do that? Everywhere I've worked (only been working in industry for 6 years) has had such rigid agile development that even when I do find myself with free time, there's no flexibility to work on things that haven't been assigned to you and the best I can do is work on profiling/debugging tools.
Honest question, what is the company like where you can do that? Everywhere I've worked (only been working in industry for 6 years) has had such rigid agile development that even when I do find myself with free time, there's no flexibility to work on things that haven't been assigned to you and the best I can do is work on profiling/debugging tools.
Usually at smaller companies with just a handful of developers. Can be a "start-up", or just a smaller business running fine with small teams. Or sometimes in a larger company where there's a small team doing its own thing.
There's up-and-downsides to everything. The pay is usually less, and "fewer processes" often also means fewer processes surrounding HR and such. This can mean you're subject to the whims and mood swings of one person. I was once fired over a pretty far-fetched misunderstanding, but 1) they didn't mention it because "I should know myself", so I didn't really have a chance to clear it up, and 2) when I got fired they agreed it was a complete misunderstanding and that they made a huge mistake, but fired me anyway because that was the road already taken now. Great ... thanks ... It's hard to imagine that happening at a larger company with an actual HR department and stuff.
Also: a Wikipedia-esque "be bold" attitude helps. Obviously refactoring things just for the craic isn't a good idea, but if there's something that can really make an impact then I'd just do it and send a PR, regardless of what I've been assigned. Obviously "refactor everything" on your own initiative isn't a good idea, but usually there's tons of smaller stuff that really helps: "introduce new function/package that helps reduce boiler plate", "split up this difficult to understand function", things like that.
Most PMs will realize they don't have a 100% in-depth understanding of the code, and will trust you when you say it helps you do your work better. Usually it's fine as long as you apply proper judgement and don't spend too much time on it. That said, there are some places where this really isn't appreciated regardless. IMHO that's pretty dysfunctional and when I start looking for somewhere else to work.
There's up-and-downsides to everything. The pay is usually less, and "fewer processes" often also means fewer processes surrounding HR and such. This can mean you're subject to the whims and mood swings of one person. I was once fired over a pretty far-fetched misunderstanding, but 1) they didn't mention it because "I should know myself", so I didn't really have a chance to clear it up, and 2) when I got fired they agreed it was a complete misunderstanding and that they made a huge mistake, but fired me anyway because that was the road already taken now. Great ... thanks ... It's hard to imagine that happening at a larger company with an actual HR department and stuff.
Also: a Wikipedia-esque "be bold" attitude helps. Obviously refactoring things just for the craic isn't a good idea, but if there's something that can really make an impact then I'd just do it and send a PR, regardless of what I've been assigned. Obviously "refactor everything" on your own initiative isn't a good idea, but usually there's tons of smaller stuff that really helps: "introduce new function/package that helps reduce boiler plate", "split up this difficult to understand function", things like that.
Most PMs will realize they don't have a 100% in-depth understanding of the code, and will trust you when you say it helps you do your work better. Usually it's fine as long as you apply proper judgement and don't spend too much time on it. That said, there are some places where this really isn't appreciated regardless. IMHO that's pretty dysfunctional and when I start looking for somewhere else to work.
Counter-point: whitespace PR’s, refactors I didn’t ask for, aren’t attached to a larger initiative, etc actively annoy me as an owner.
Every PR takes time to manage, compounds complexity, and carries risk. We only get so many out a week, and I want them to count.
Maybe it sometimes goes in hand with disfunction, but I’m guessing there’s more going on there.
Every PR takes time to manage, compounds complexity, and carries risk. We only get so many out a week, and I want them to count.
Maybe it sometimes goes in hand with disfunction, but I’m guessing there’s more going on there.
> apply proper judgement
> Once you're really familiar with the codebase, you know where dragons be, and you can make changes more easily.
This is an interesting, and often overlooked, point. A month or two ago someone asked us, the Fossil SCM maintainers, if we'd be open to them refactoring the tree to something which better matches modern sensibilities (i'm paraphrasing here). Despite its many proverbial dragons, the long-time maintainers are comfortable with it and know where those dragons are and how to defeat them (or, in some cases, sneak around them), so, despite our collective geek tendencies to prefer "perfect code," we're happier with the dragons we know than those we don't. (That's not to say that fossil's code is awful, but its structure varies significantly from what one might write today if one was to start such a project from the ground up.)
This is an interesting, and often overlooked, point. A month or two ago someone asked us, the Fossil SCM maintainers, if we'd be open to them refactoring the tree to something which better matches modern sensibilities (i'm paraphrasing here). Despite its many proverbial dragons, the long-time maintainers are comfortable with it and know where those dragons are and how to defeat them (or, in some cases, sneak around them), so, despite our collective geek tendencies to prefer "perfect code," we're happier with the dragons we know than those we don't. (That's not to say that fossil's code is awful, but its structure varies significantly from what one might write today if one was to start such a project from the ground up.)
> Any codebase that I had complete control over.
Anyone other than myself would instantly observe it as the the worst codebase they have ever seen.
Anyone other than myself would instantly observe it as the the worst codebase they have ever seen.
For that, there is a saying: "code as if your kids will maintain it".
But I think it does not convey the right meaning. When I code something I will have to maintain for a long time, I try to make it as simple as possible for my future, older, less motivated and weary self.
The worst codebases are written by people who landed the gig a few months before and do not expect to stay around longer than a year or two.
But I think it does not convey the right meaning. When I code something I will have to maintain for a long time, I try to make it as simple as possible for my future, older, less motivated and weary self.
The worst codebases are written by people who landed the gig a few months before and do not expect to stay around longer than a year or two.
This is very much true. Initially when I joined the industry I used to work for a product from its inception. So I was aware of what section of the code affects what part of the product. Although I hadn't worked on all of those, I kept an eye for the all the changes that were coming in. I knew where I should look immediately a bug is reported even if it is not something related to my line of work.
Recently I switched teams and now I find myself taking up bugs that are only related to my line of work. Not being familiar with the codebase decreases productivity and wants you to rely on other people in the team for most of the time.
Recently I switched teams and now I find myself taking up bugs that are only related to my line of work. Not being familiar with the codebase decreases productivity and wants you to rely on other people in the team for most of the time.
Also nice is when an entire community has agreed to architect a codebase more or less the same. You're basically psychic when that happens.
Your successors just hope you can only remember 3 things at once so you end up with elegant designs rather than one of those monsters who can remember "to add a new wrinkle to this feature, just edit these 25 files."
> Any codebase that I had complete control over.
This is probably one of the pillars of good codebases, or at least decoupling the bits that you don't control as well as you can (this includes external services). I remember needing to write a wrapper around another JWT library, but because it was quite important, I aimed for >95% test coverage, some of the tests acted as documentation for how to use the code, there was also a nice README, there was CI configuration for pushing the build artifacts to Maven and suddenly even managing dependency updates became easy because of the test suite. Years later, it's been integrated in a bunch of services and "just works".
Come to think of it, things always get less pleasant once you add a bunch of complex dependencies and libraries/frameworks. Need to make a few RESTful Web APIs in Java? Something like Dropwizard will probably give you fewer headaches than Spring (or Spring Boot) and all of its inherent complexity, in the case of the former you might even need to do configuration in XML and that has honestly never been pleasant. If you need to integrate with a bunch of other stuff and want something opinionated, going for Spring Boot will make sense then, but for simple use cases it's overkill. Same for ASP.NET, Ruby on Rails, Laravel and many others, while they might be easier to use, updates (especially across major versions) and breaking changes will give you a headache and just add a bunch of churn to the project.
Similarly, if you need a message queue, externalizing that into RabbitMQ might make a lot of sense, same with storing files in an S3 compatible store, using something like Redis or Valkey for key-value storage, as well as not trying to shove too much logic into your RDBMS. Just pick whatever tool feels best for the given task at hand, instead of shaping things into what they're not (using the database for blob storage, for example), unless you have a whole bunch of constraints to contend with. Otherwise, sometimes you just get the worst of both worlds, like needing to use Oracle for a project, not having easily launchable local environments (because Oracle XE doesn't support some features), having to share DB instances with others during development and also running into weird obscure issues like DATABASE LINK queries taking 100 times longer in some cases, even when executing the same SELECT query on the remote DB works without issues.
To not go into a rant, I'd sum it up like this: be in control, isolate the things that you cannot control, pick the correct technologies, do the simplest thing that you can get away with without overengineering and think about the person who'll have to maintain, debug and grow the system in the following months and years (which might also be yourself, sans knowledge about what the code did, if you don't make it explain itself or don't comment the non-trivial stuff).
This is probably one of the pillars of good codebases, or at least decoupling the bits that you don't control as well as you can (this includes external services). I remember needing to write a wrapper around another JWT library, but because it was quite important, I aimed for >95% test coverage, some of the tests acted as documentation for how to use the code, there was also a nice README, there was CI configuration for pushing the build artifacts to Maven and suddenly even managing dependency updates became easy because of the test suite. Years later, it's been integrated in a bunch of services and "just works".
Come to think of it, things always get less pleasant once you add a bunch of complex dependencies and libraries/frameworks. Need to make a few RESTful Web APIs in Java? Something like Dropwizard will probably give you fewer headaches than Spring (or Spring Boot) and all of its inherent complexity, in the case of the former you might even need to do configuration in XML and that has honestly never been pleasant. If you need to integrate with a bunch of other stuff and want something opinionated, going for Spring Boot will make sense then, but for simple use cases it's overkill. Same for ASP.NET, Ruby on Rails, Laravel and many others, while they might be easier to use, updates (especially across major versions) and breaking changes will give you a headache and just add a bunch of churn to the project.
Similarly, if you need a message queue, externalizing that into RabbitMQ might make a lot of sense, same with storing files in an S3 compatible store, using something like Redis or Valkey for key-value storage, as well as not trying to shove too much logic into your RDBMS. Just pick whatever tool feels best for the given task at hand, instead of shaping things into what they're not (using the database for blob storage, for example), unless you have a whole bunch of constraints to contend with. Otherwise, sometimes you just get the worst of both worlds, like needing to use Oracle for a project, not having easily launchable local environments (because Oracle XE doesn't support some features), having to share DB instances with others during development and also running into weird obscure issues like DATABASE LINK queries taking 100 times longer in some cases, even when executing the same SELECT query on the remote DB works without issues.
To not go into a rant, I'd sum it up like this: be in control, isolate the things that you cannot control, pick the correct technologies, do the simplest thing that you can get away with without overengineering and think about the person who'll have to maintain, debug and grow the system in the following months and years (which might also be yourself, sans knowledge about what the code did, if you don't make it explain itself or don't comment the non-trivial stuff).
I can't name a good one, but i have strong
dislike for big project codebases where
even finding one file out of thousands where
the relevant code resides is a challenge:
its never something isolated but acts
like some "component".
The best i can think of is one-person projects
where organization is streamlined as its
actually has to be used by the author,
not like a cog in a giant project.
Illumos, and specifically the SmartOS distribution.
Being ringside for the fork of Sun's OpenSolaris, and watching absolute geniuses steward a thing like that into a useful product and a build process was foundational for my understanding of how and why process is important, as well as how to get things fixed in the real world, while not muddying the idea of the pure design principles. A balance can exist!
Being ringside for the fork of Sun's OpenSolaris, and watching absolute geniuses steward a thing like that into a useful product and a build process was foundational for my understanding of how and why process is important, as well as how to get things fixed in the real world, while not muddying the idea of the pure design principles. A balance can exist!
* medical imaging software to drive a X ray machine
* software was relatively new, and well made. age is probably the main reason why it was good "at the time".
* there were good practices in place, yes, led by an opinionated team lead, which felt like it was the main cause for such a project to work well under medical standards.
I feel like I will never find another job like this one again.
* software was relatively new, and well made. age is probably the main reason why it was good "at the time".
* there were good practices in place, yes, led by an opinionated team lead, which felt like it was the main cause for such a project to work well under medical standards.
I feel like I will never find another job like this one again.
My own, lol. Which just means I need to work on more codebases.
I’m making a tool to convert data schemas to SQL via a UI for lay-users. Just like https://react-querybuilder.js.org/ which is basically a UI based SQL generator. For work.
Except my version extends the idea much further, blending Excel like functionality with functions that can act on Fields and Rules.
What makes it good?
For one, it’s a from scratch project with almost no third party libraries.
For two, I fortunately chose a recursive data structure to represent the data schema, and that has really worked out well. Early on I tried 4 other approaches to represent the data, but went back to recursive feeling it was the best choice.
Furthermore I’m using React, but it’s heavily leveraging reducers. Specifically Groups have a reduced, Rules have a reducer, and Fields have a reducer. The reducers are linked in a chain top to bottom, where changes on a Field change a Rule, and changes on a Rule change a Group. It’s been extremely clean to work with.
Because the base data schema is recursive (Groups contain groups contain groups), most of the functions that manipulate the schema are recursive. There is a real elegance to the code, each recursive function has a very obvious base case, and a very obvious recursive path.
And for the final outcome, walking the query data structure and spitting out the equivalent SQL is also recursive, and feels elegant, coming in at under 40 lines.
Literally as I’ve been writing this codebase, everything somehow perfectly fell into place. I was marvelling near the end that it felt like I chose all the best possible logic paths to build this somehow.
I’m hoping to get the okay from work to open source it (fully open source). The only cruft of the project is the types I’m using, the interface of the code could be improved with generics
I’m making a tool to convert data schemas to SQL via a UI for lay-users. Just like https://react-querybuilder.js.org/ which is basically a UI based SQL generator. For work.
Except my version extends the idea much further, blending Excel like functionality with functions that can act on Fields and Rules.
What makes it good?
For one, it’s a from scratch project with almost no third party libraries.
For two, I fortunately chose a recursive data structure to represent the data schema, and that has really worked out well. Early on I tried 4 other approaches to represent the data, but went back to recursive feeling it was the best choice.
Furthermore I’m using React, but it’s heavily leveraging reducers. Specifically Groups have a reduced, Rules have a reducer, and Fields have a reducer. The reducers are linked in a chain top to bottom, where changes on a Field change a Rule, and changes on a Rule change a Group. It’s been extremely clean to work with.
Because the base data schema is recursive (Groups contain groups contain groups), most of the functions that manipulate the schema are recursive. There is a real elegance to the code, each recursive function has a very obvious base case, and a very obvious recursive path.
And for the final outcome, walking the query data structure and spitting out the equivalent SQL is also recursive, and feels elegant, coming in at under 40 lines.
Literally as I’ve been writing this codebase, everything somehow perfectly fell into place. I was marvelling near the end that it felt like I chose all the best possible logic paths to build this somehow.
I’m hoping to get the okay from work to open source it (fully open source). The only cruft of the project is the types I’m using, the interface of the code could be improved with generics
The best codebase I've ever worked on was when I started software development officially, I had zero knowledge of the stack for real, the codebase was run by just 5 mid level engineers, but based on how well structured and understandable it was, i was able to pick up the stack without reading a tutorial or a documentation, i would later read up on it when i started doing complicating stuff tho, but it's important for engineers to write good code to the extent that another person reading it does not have to look up something just to understand why this does that, and how stuff works in the code
Unreal Engine, by far.
It's not only just extremely well written in general (with your only chance of 'really' learning the engine to go through it, read comments, and the like) but it also defies the wisdom 'everybody knows.' That wisdom being that premature optimization is the root of all evil - like we're supposed to just benchmark things, see where the bottleneck is and then try to work on optimizing spots like that.
Unreal doesn't do this. There are countless thousands of micro-optimizations everywhere. For one striking example there's even a fairly substantial system in place that cache the conversion between a quaternion and a euler rotation. This would never, in a zillion years, be even close to a bottleneck. But with thousands of these little micro-optimizations everywhere you get a final system that just runs dramatically better than any comparable engine.
In more general terms, they've also taken advantage of the 'idiomatic flexibility' that C++ offers to create a sort of Unreal C++ language that is also just lovely to use and feels much closer to something like C# in terms of luxuries like the lack of manual memory management, garbage collection, reflection (!!), and so on. The downsides are that compile times are horrible (even though a cached compile might only take 10 or 15 seconds, it feels like forever when trying to work out one specific issue) and C++ intellisense, especially in a preprocessor heavy environment, is pretty mehhhh.
It's not only just extremely well written in general (with your only chance of 'really' learning the engine to go through it, read comments, and the like) but it also defies the wisdom 'everybody knows.' That wisdom being that premature optimization is the root of all evil - like we're supposed to just benchmark things, see where the bottleneck is and then try to work on optimizing spots like that.
Unreal doesn't do this. There are countless thousands of micro-optimizations everywhere. For one striking example there's even a fairly substantial system in place that cache the conversion between a quaternion and a euler rotation. This would never, in a zillion years, be even close to a bottleneck. But with thousands of these little micro-optimizations everywhere you get a final system that just runs dramatically better than any comparable engine.
In more general terms, they've also taken advantage of the 'idiomatic flexibility' that C++ offers to create a sort of Unreal C++ language that is also just lovely to use and feels much closer to something like C# in terms of luxuries like the lack of manual memory management, garbage collection, reflection (!!), and so on. The downsides are that compile times are horrible (even though a cached compile might only take 10 or 15 seconds, it feels like forever when trying to work out one specific issue) and C++ intellisense, especially in a preprocessor heavy environment, is pretty mehhhh.
> there's even a fairly substantial system in place that cache the conversion between a quaternion and a euler rotation
I hope that does show up on a benchmark though, cause that sounds complex
I hope that does show up on a benchmark though, cause that sounds complex
It definitely would not. The rotation either way is a mixture of up to ~20-30 primitive basic floating point operations, and a few geometric (sin/cos/etc) operations, and you only need to do it in the cases where you're actually going between both systems, and more than once. It's difficult to even ballpark, but it's highly likely that the caching gives anymore than a fraction of a percent of performance boost, like 0.5% or whatever. But now multiply that over hundreds to thousands of similar little 'premature' optimizations.
ClojureScript + Re-frame + GraphQL subscriptions over WebSockets: https://github.com/Vetd-Inc/vetd-app
My open source projects. One in particular I've been working on for about 10 years. The code is consistent and always getting better, even though there is a lot more work that could be done.
One that was written from scratch, by me. All others are terrible.
The problem I have is, I'll revisit code I wrote 1-2 months prior and swear someone else wrote it.
Same.
I know I wrote perfect, maintainable code that can easily live in my brain.
But when I go to work on it again a few months since I wrote it, it seems like some asshole stepped in and overwrote my beautiful code with a bunch of jumbly mish-mash and had the gall to overwrite the git commit history with it, using the exact same commit messages, and my own name!
I know I wrote perfect, maintainable code that can easily live in my brain.
But when I go to work on it again a few months since I wrote it, it seems like some asshole stepped in and overwrote my beautiful code with a bunch of jumbly mish-mash and had the gall to overwrite the git commit history with it, using the exact same commit messages, and my own name!
Right. I even quietly mumble 'asshole' while reading it.
This is the weird thing. I heard it a lot and I don't know if people just over-exaggerate or maybe I'm just forgetting things quicker, but I don't have that. I come to the code after month or two and it is the same as if I came a week or two.
I also don't feel ashamed of it.
I also got compliments of my code in the past.
I do try to make my code readable and avoid being clever. Basically my rule of thumb is that if I need to write a comment about block of code what it does, then that code is too complex.
I also got compliments of my code in the past.
I do try to make my code readable and avoid being clever. Basically my rule of thumb is that if I need to write a comment about block of code what it does, then that code is too complex.
Same. I think it's a meme that people keep repeating, and for some fraction of them it's true.
A month or two is nothing. Go back to it after 5 years.
I used month or two, because that's what OP used and that seemed ridiculous. Almost like OP tried to make a meme that s/he didn't experience themselves.
But anyway, I recently found code that I wrote 15 years ago. And when looking at it actually makes me feel depressed. It looks like something I would write today so I don't think I got much better.
But yes, I do have to look over it again to understand how it works as I forgot everything.
But anyway, I recently found code that I wrote 15 years ago. And when looking at it actually makes me feel depressed. It looks like something I would write today so I don't think I got much better.
But yes, I do have to look over it again to understand how it works as I forgot everything.
Past me is someone else. https://en.wikipedia.org/wiki/Ship_of_Theseus
Heh. Can relate.
Even my own codebases/config. management systems are difficult, if I haven't dealt with them in a while.
However, I am pretty good at understanding old stuff I did, mainly because I leave a legacy for myself.
https://littlegreenviper.com/leaving-a-legacy/
Even my own codebases/config. management systems are difficult, if I haven't dealt with them in a while.
However, I am pretty good at understanding old stuff I did, mainly because I leave a legacy for myself.
https://littlegreenviper.com/leaving-a-legacy/
We were all thinking it, thank you for having The Courage.
One day I got an unexpected email from someone asking permission to use a piece of code on github I forgot I wrote almost 10 years ago without a license clause. I was very surprised and confused because I forgot I wrote that code, and I was not involving coding for quite some years and my github profile was blank without any contact information. I asked. It happened the guy simply made a search and found a post on a forum with my nickname, and email address - newly created recently. At that moment, it clicked and I felt some unusual arrangment. I quickly added an open source license, and the guy, CTO of a decent company, offered to make a donate. That made me a day.
Mine is always terrible and the others are terrible as well.
...unless I haven't looked at it in a few years.
This entire website in a single comment
[deleted]
No name because the project code name changes with each new contract, but the first-level ground processing system for the NRO that turns raw satellite downlinks into some kind of human-intelligible files.
I think a lot was just the restrictions built into the way development happens required a high level of discipline, care, and planning. Also, requirements were pretty tightly coupled to sensor platform capabilities, which are known well in advance and don't change unexpectedly, so waterfall development actually works and you don't have to deal with the chaos of not really knowing what will and won't work and customers constantly changing their minds.
Code base was overwhelmingly C++, some Fortran, a lot of it was very old. It was all developed on airgapped networks, and the difficulty of bringing in external dependencies meant there largely were not any. All of the library functionality we required was mostly developed in-house, so we had extremely well-documented and stable functions available to do damn near anything you could want, with a good chance that whoever first wrote it was still there. All development had always been tied to a ticketing system of some sort that included all of the original discussion, design documents, and that kind of thing might add process overhead upfront, but it means that forever new developers can simply read the history and learn exactly why everything works the way it works.
The system itself was very Unixy. In production, it was meant to be run as a server with many instances striped across high-performance compute nodes, but it did not have to be run that way. Every individual product flow could also be built as its own transient executable, so that working on a single component could easily be done locally. You didn't have to rebuild the world or spin up your own server. Performance requirements were enough that we had our own customized networking stack and filesystem in production, but nothing depended on this for function, so you could still develop and test on plain Linux with ext4.
The culture was terrific. We were part of one of the big five defense contractors, but an acquisition and this program was largely still staffed by employees of the original company that had been acquired. We were de facto exempted from most of the bullshit any other program had to deal with. I don't know if that was part of the original terms of being acquired or just a consequence of having so many long-time developers that couldn't afford to be lost if you subjected them to nonsense. This was the kind of project that people intentionally stayed on and retired from because the experience was so much better than any other project you could get assigned to.
Ironically, it had none of the characteristics that high-performing companies often tout. You work in private. The rest of the company, including your own management chain, doesn't even know what you're working on. You'll never get any recognition or publicity. The pay is mediocre. We weren't attracting the best and brightest from all of the world. You had to be American, have a top-secret clearance, and be geographically close enough to the facility to get there every day, so this was a pretty constrained hiring pool. I still worked with some of the smartest people and best engineers I've ever known. The upside of this kind of environment is you have no mercenaries or publicity hounds. Everyone who sticks around is a person who really loves and cares about what they're working on, and a lot of people did stick around. The sanity and organization of the code was heavily facilitated by having a whole lot of people working on it who'd been working on it for 30+ years.
I think a lot was just the restrictions built into the way development happens required a high level of discipline, care, and planning. Also, requirements were pretty tightly coupled to sensor platform capabilities, which are known well in advance and don't change unexpectedly, so waterfall development actually works and you don't have to deal with the chaos of not really knowing what will and won't work and customers constantly changing their minds.
Code base was overwhelmingly C++, some Fortran, a lot of it was very old. It was all developed on airgapped networks, and the difficulty of bringing in external dependencies meant there largely were not any. All of the library functionality we required was mostly developed in-house, so we had extremely well-documented and stable functions available to do damn near anything you could want, with a good chance that whoever first wrote it was still there. All development had always been tied to a ticketing system of some sort that included all of the original discussion, design documents, and that kind of thing might add process overhead upfront, but it means that forever new developers can simply read the history and learn exactly why everything works the way it works.
The system itself was very Unixy. In production, it was meant to be run as a server with many instances striped across high-performance compute nodes, but it did not have to be run that way. Every individual product flow could also be built as its own transient executable, so that working on a single component could easily be done locally. You didn't have to rebuild the world or spin up your own server. Performance requirements were enough that we had our own customized networking stack and filesystem in production, but nothing depended on this for function, so you could still develop and test on plain Linux with ext4.
The culture was terrific. We were part of one of the big five defense contractors, but an acquisition and this program was largely still staffed by employees of the original company that had been acquired. We were de facto exempted from most of the bullshit any other program had to deal with. I don't know if that was part of the original terms of being acquired or just a consequence of having so many long-time developers that couldn't afford to be lost if you subjected them to nonsense. This was the kind of project that people intentionally stayed on and retired from because the experience was so much better than any other project you could get assigned to.
Ironically, it had none of the characteristics that high-performing companies often tout. You work in private. The rest of the company, including your own management chain, doesn't even know what you're working on. You'll never get any recognition or publicity. The pay is mediocre. We weren't attracting the best and brightest from all of the world. You had to be American, have a top-secret clearance, and be geographically close enough to the facility to get there every day, so this was a pretty constrained hiring pool. I still worked with some of the smartest people and best engineers I've ever known. The upside of this kind of environment is you have no mercenaries or publicity hounds. Everyone who sticks around is a person who really loves and cares about what they're working on, and a lot of people did stick around. The sanity and organization of the code was heavily facilitated by having a whole lot of people working on it who'd been working on it for 30+ years.
Open 3D Engine's codebase is pretty great: https://o3de.org/
Probably WebKit.
I worked on Safari, and while I had to make changes to WebKit very infrequently, the quality of the code, tooling, and team were excellent.
I haven't seen that level of engineering excellence anywhere I've worked since -- it was consistent and pervasive. No one had to enforce that culture from the top because everyone bought into the vision.
I worked on Safari, and while I had to make changes to WebKit very infrequently, the quality of the code, tooling, and team were excellent.
I haven't seen that level of engineering excellence anywhere I've worked since -- it was consistent and pervasive. No one had to enforce that culture from the top because everyone bought into the vision.
I was debugging some issues with Thanos and had pretty good success tweaking the codebase to add additional telemetry.
The code was fairly well organized and more importantly worked out of the box with a Makefile and IDE integration (GoLand). All it took was `git clone` and opening GoLand to get started.
For C (maybe it's C++), fluentbit seemed pretty straight forward (I don't have much experience in C though)
The code was fairly well organized and more importantly worked out of the box with a Makefile and IDE integration (GoLand). All it took was `git clone` and opening GoLand to get started.
For C (maybe it's C++), fluentbit seemed pretty straight forward (I don't have much experience in C though)
Blender is incredible, harfbuzz equally so, pretty much all modern browsers do the impossible so they need to be included as well.
Used: the Boost C++ libraries. In particular ASIO. Variant has also been very useful.
The Mac GUSI BSD socket library is worth a mention too. I built a multithreaded Quark XPress XTension with it that used PowerPlant for UI. Quark even wanted to interview me.
Added to: a pre-Boost cross platform library for Mac (Classic) and Windows NT. It did much of what Boost did way back when.
The Mac GUSI BSD socket library is worth a mention too. I built a multithreaded Quark XPress XTension with it that used PowerPlant for UI. Quark even wanted to interview me.
Added to: a pre-Boost cross platform library for Mac (Classic) and Windows NT. It did much of what Boost did way back when.
Worked on a domain driven design code. It was simple and intelligible. Now, I want to copy that methodology everywhere.
The Xbox codebase was pretty good. We were doing enforced code reviews back in 2011-ish, which was a big thing at the time. There was also a good layer cake going on and a few good architects.
The worst codebase by far was Outlook for Mac. It had code going back to Entourage, and was never properly cleaned up.
The worst codebase by far was Outlook for Mac. It had code going back to Entourage, and was never properly cleaned up.
Office in general is a mess. I think it’s a good example of the fact that the sales team / marketing power of a business is the real driver of success, much as that pains me as a developer.
I feel that Office's trouble is always backward compatibility. Which makes sense.
Ha, if you think that was bad, you should have been there when it was just Entourage ;).
The one I made from scratch after iterating over 3 years. You can see the technologies here [0]
[0] https://gitsell.dev/u/bitofbreeze/r/bitofbreeze/git-sell
[0] https://gitsell.dev/u/bitofbreeze/r/bitofbreeze/git-sell
Not wanting to start a flame war, but the appearance of jquery has been an eye-opener for me. The official API of the Web was instantly outdated thanks to the incredible cleverness of a bunch of developpers. Wow … that was, for me, exactly what open source was all about.
Any code base that doesn't use the advanced features of it's language(s) are always better in my experience. Heavy usage of e.g. meta-programming in Python or perhaps uber's fx (dependency injection) in Go makes projects infinitely harder to get into.
I worked at GOV.UK for a few years on what was effectively specialised CMSs all written in Rails. Mostly basic CRUD stuff. A contractor came along and built the most insane CMS I've ever seen. I found out later it was flavor of the Java Repository Pattern using dependency injection. It became so difficult to work with that the last thing I worked on there was to delete it and rebuild it using standard Rails patterns.
The KISS philosophy exists for a reason and that includes over-using advanced language features just to show off.
The KISS philosophy exists for a reason and that includes over-using advanced language features just to show off.
Besides just KISS, a lot of messes I've seen have been implementing patterns outside the framework or implementing complex patterns that didn't add value.
Besides KISS (or maybe as an extensive), try to keep framework-based codebases as close to the official documented setup as possible. You automatically get to s of free, high-quality documentation available on the Internet.
Besides KISS (or maybe as an extensive), try to keep framework-based codebases as close to the official documented setup as possible. You automatically get to s of free, high-quality documentation available on the Internet.
I've had a consistent experience in Rails where I think for a day or two I've got a legitimate use case for one the whacky things you can do... and then as I work the problem more, it turns out: nope, the simple stuff is still the Right Way.
Someday, Ruby shenanigans, someday...
PS - Being able to pull the shenanigans is super useful during the dev process, usually to skip some yak shaving during exploration, so it's nice to have anyway.
Someday, Ruby shenanigans, someday...
PS - Being able to pull the shenanigans is super useful during the dev process, usually to skip some yak shaving during exploration, so it's nice to have anyway.
Props for gov.uk! I’ve looked at its documentation and design system and see both as peak user experience and clarity.
I disagree with this, but I understand where it's coming from. I think you have a form of whiplash from things like:
- Novices overusing the new shiny.
- Java/C++/etc Jr programmers overusing design patterns.
- Perl programmers solving everything with regexes.
- Small startups with GraphQL, or any other large enterprise tool.
- Metaprogramming, Maco's, Dependency injection, Recursion, etc when a simpler solution is a better fit.
IMHO, a "best codebase" will be just a bit more advanced than I am, with good resources for me to grok it. I want to be able to learn from it. I also don't want to be so far out of my depth that I can't make a reasonable contribution.
IMHO, a "best codebase" will be just a bit more advanced than I am, with good resources for me to grok it. I want to be able to learn from it. I also don't want to be so far out of my depth that I can't make a reasonable contribution.
A pinch of salt can really liven up a dish. Not every dish needs it, but when used appropriately it's almost magic in how much difference it can make.
A lot of salt always make everything disgusting.
A lot of salt always make everything disgusting.
Huh? Salt is one of the most important foundational elements of basically all cooking.
Depends who is providing it.
Django and pydantic meta programming usually make the code easier to deal with.
In shop written meta programming usually sucks.
Django and pydantic meta programming usually make the code easier to deal with.
In shop written meta programming usually sucks.
Such a great point. I audibly groan when I come across Python meta-programming.
While not an advanced feature, I have a similar response when I see lots of decorators in Python. They quickly become a debugging nightmare.
While not an advanced feature, I have a similar response when I see lots of decorators in Python. They quickly become a debugging nightmare.
It would be cool if newer (imperative) languages had a clear protocol to delimit the simple core of the language from the advanced, library-building features. Like Rust has `unsafe` blocks, imagine a `advanced` or `metaprogramming` block that is required in order to use advanced metaprogramming features. Then you could tell a junior to implement $MUNDANE_INTEGRATION without use of `advanced`, and that constraint could be statically verified in PRs, etc.
It seems like it would vastly simplify language evolution. The core language can have rigid limits on changes and follow the Zen of Python or whatever, while the `advanced` language extensions can be a bit looser. Features could graduate from advanced to core without breaking anything. You get a single powerful+expressive language with a "porcelain interface" without necessarily giving juniors ammo to shoot themselves in the foot.
It seems like it would vastly simplify language evolution. The core language can have rigid limits on changes and follow the Zen of Python or whatever, while the `advanced` language extensions can be a bit looser. Features could graduate from advanced to core without breaking anything. You get a single powerful+expressive language with a "porcelain interface" without necessarily giving juniors ammo to shoot themselves in the foot.
I think D language fits your very descriptions.
You have GC by default that's Python like and intuitive programming constructs. Since for the most part D compilation is much faster than C++ and Rust compilation as long as you stick to the core non-CTFE. Heck you can use D as a compiled scripting language with REPL using rdmd [1].
Then if you want to go gung-ho, you can use the other advance features like D's excellent template meta programming, CTFE, bit banging in-line assembly, etc. D has so many advanced features that can be explored later on, even modern C++ is playing catch up with D in every releases. Nowadays D compiler also support C natively [2] and GDC compiler is included inside the venerable GCC eco-system.
There are university that teach software development and engineering class with D due to its managed, on-demand and gradual complexity, not like "in your face" complexity of the big kahuna programming language like C++ and Rust [3]. Like other compiled languages you can build real-time system even firmware device drivers in D unlike Python.
[1] https://dlang.org/areas-of-d-usage.html#academia
[2] Adding ANSI C11 C compiler to D so it can import and compile C files directly:
https://news.ycombinator.com/item?id=27102584
[3] Teaching Software Engineering in DLang [pdf]:
https://news.ycombinator.com/item?id=37620298
You have GC by default that's Python like and intuitive programming constructs. Since for the most part D compilation is much faster than C++ and Rust compilation as long as you stick to the core non-CTFE. Heck you can use D as a compiled scripting language with REPL using rdmd [1].
Then if you want to go gung-ho, you can use the other advance features like D's excellent template meta programming, CTFE, bit banging in-line assembly, etc. D has so many advanced features that can be explored later on, even modern C++ is playing catch up with D in every releases. Nowadays D compiler also support C natively [2] and GDC compiler is included inside the venerable GCC eco-system.
There are university that teach software development and engineering class with D due to its managed, on-demand and gradual complexity, not like "in your face" complexity of the big kahuna programming language like C++ and Rust [3]. Like other compiled languages you can build real-time system even firmware device drivers in D unlike Python.
[1] https://dlang.org/areas-of-d-usage.html#academia
[2] Adding ANSI C11 C compiler to D so it can import and compile C files directly:
https://news.ycombinator.com/item?id=27102584
[3] Teaching Software Engineering in DLang [pdf]:
https://news.ycombinator.com/item?id=37620298
I think one should not use advanced language features just because, but I also think one should not avoid using advanced language features where it is useful.
Why would the code base be worse when advanced language features are used?
Why would the code base be worse when advanced language features are used?
Because unless you hire steadily more intelligent developers you will be headed towards a mass of code that is hard and scary to change.
I can see a few cases where that depends...
Really simple languages: Ruling out meta-programming is really going to limit you in Lua for example. Just being able to do `mySocket:close()` instead of `Socket.close(mySocket)` involves meta-programming.
Older languages: For C++ the "simple" features are going to include raw pointers and macros. Maybe it's not so bad to allow smart pointers and templates to avoid those
Really simple languages: Ruling out meta-programming is really going to limit you in Lua for example. Just being able to do `mySocket:close()` instead of `Socket.close(mySocket)` involves meta-programming.
Older languages: For C++ the "simple" features are going to include raw pointers and macros. Maybe it's not so bad to allow smart pointers and templates to avoid those
Both of these examples are examples of an under-programmed core though. Lua is notorious for lacking batteries, so everyone has to reinvent their own. There’s literally no serious Lua program without some sort of classes, but they still resist adding them into lauxlib.
I worked with someone who insisted on using fx for DI in go. It's so antithetical to the entire Go philosophy, I don't even think it's an "advanced feature". It's just bringing Java cruft to a language where it isn't necessary and making everything worse.
Having just removed a metaclass from my Python code I totally agree.
The Go standard library. The level of detail, the politeness of the reviews, the fact that the features of the language allowed me to put more attention into correct code than "at least it's working" code, the pragmatism and personalism of the core team.
For me best code base I've read so far is Redis code. Feels like a poem to me.
Many of the large Go codebases. Including Kubernetes.
It is super easy to find what I am looking for.
It is super easy to find what I am looking for.
I don't think the 'code' side of code base can be considered in isolation.
What makes a project objectively good (from subjective experience) is a combination of code, design, documentation, and often the humans involved.
What makes a project objectively good (from subjective experience) is a combination of code, design, documentation, and often the humans involved.
V7 Unix Kernel. By far the cleanest most concise code I’ve had had the privilege to work with. Really laid the groundwork for what came after particularly BSD4.3 and beyond.
Litertar.dev and Tornado web .
Litestar code base, architecture and core design is what modern python code should be. Very clean, fully type annotated, DTO, and many cool stuff happening there.
Had to work through a weirdness in kubernetes many years ago when using it on prem for a project (not at google). It stands out as the best code base I have had to go through to figure stuff out.
I tried to add a new SQL function to CockroachDB. I had never seen the source before, but since it was such nice Go code, it only took me a few hours to have a working build, with tests.
Generally code that hasn't been tested commercially. Unencumbered by pesky client driven features, just code for dreamt up features that are fun to code but perhaps will never be used.
> Just being in a group of great engineers
Is such ego. There is almost certainly someone in the group or behind the group enabling this greatness to take place.
Is such ego. There is almost certainly someone in the group or behind the group enabling this greatness to take place.
The LLVM infrastructure.
Other compilers don't even come close.
Other compilers don't even come close.
[deleted]
Laravel, once you get the hang of it everything just works, and using a debug bar to optimize database calls is very satisfying.
[deleted]
Hopefully the last one you were responsible for.
Java, because: Mark Reinhold
He learned from the best.
He learned from the best.
Any code I have ever touched.
Code wise, probably Fuchsia. I worked up and down the Fuchsia stack regularly, from bootloaders to GUI programs and all over in-between. For a sizeable code base with so many layers and concepts, and regularly moving between these without much memory for what's there (as it also moved around quite quickly in my time), it was the lowest average cost of entry to new areas of most other millions of lines of code repositories I've dealt with in my career. Still plenty of bugs came up, lots of polish and more design work to do, it's not at all to over-sell it, but almost everything was approachable with low impediment.
There were also lots of Ruby libraries back in the hayday of the language that were very clean for what they are. I've rarely seen programs structured so well since - some of this is essentially a forced practice by the language - if you don't do it, you'll suffer as there's no types, lsps and whatnot to otherwise save you, but still, it's worth calling out, as I think back occasionally with fond memories of being unimpeded anywhere I needed to go.
Tooling wise it's got to be noted as others have that the Google monorepo was well integrated, but it wasn't what I'd describe as "best" at all. I had a fairly broad time at Google having worked in most of the sub orgs over the years and in doing so saw a lot of the variance even with the chosen integrations. I saw teams who regularly built from no where near head cherry picks out of the repo, I ended up owning stuff that always built off of the main branches (most Googlers don't even know exist) setup before my time to avoid running into otherwise hardline constraints around single version policy that couldn't be adhered to for specific product reasons (hi cloud). Exactly how much and which parts of buganizer teams used varied hugely, which service management tools they used varied, which database systems, etc. The single patch system while also fairly well integrated combined with code review practices that often stymied medium scale refactors (large refactors within a single product area), though making large scale changes that touch every single top level directory, while a bit rough on the edges of the tooling, the tooling existed and worked really well - pushes out hundreds or even thousands of patches and automates pinging owners and informing you of failures, etc. In many ways a lot of smaller systems with far less tooling, and far far less integration were more productive - I've never seen such a bad way of _organizing work to be done_, or such a poor volume management of bugs as I have at Google in various teams/areas, despite all the tooling. Any delivery that involved client programs (desktop programs, mobile programs, etc) was always pretty remarkably awful, and only very very small teams were on the hook for making any of this better. They'd always plaster their new thing with "don't you dare make another or use the deprecated thing" meanwhile they never had enough time to make the new thing ready - a common trope in the company, but really highlights there are edges outside of the "pure saas" side of the business which were pretty bad. The best part of the monorepo and monotooling culture for the saas side is the security properties of it all, which were outstanding, and are among the reasons why I still trust Google as the holder of some main accounts. I hope they never start to lose those properties given the scale and impact they could have.
There were also lots of Ruby libraries back in the hayday of the language that were very clean for what they are. I've rarely seen programs structured so well since - some of this is essentially a forced practice by the language - if you don't do it, you'll suffer as there's no types, lsps and whatnot to otherwise save you, but still, it's worth calling out, as I think back occasionally with fond memories of being unimpeded anywhere I needed to go.
Tooling wise it's got to be noted as others have that the Google monorepo was well integrated, but it wasn't what I'd describe as "best" at all. I had a fairly broad time at Google having worked in most of the sub orgs over the years and in doing so saw a lot of the variance even with the chosen integrations. I saw teams who regularly built from no where near head cherry picks out of the repo, I ended up owning stuff that always built off of the main branches (most Googlers don't even know exist) setup before my time to avoid running into otherwise hardline constraints around single version policy that couldn't be adhered to for specific product reasons (hi cloud). Exactly how much and which parts of buganizer teams used varied hugely, which service management tools they used varied, which database systems, etc. The single patch system while also fairly well integrated combined with code review practices that often stymied medium scale refactors (large refactors within a single product area), though making large scale changes that touch every single top level directory, while a bit rough on the edges of the tooling, the tooling existed and worked really well - pushes out hundreds or even thousands of patches and automates pinging owners and informing you of failures, etc. In many ways a lot of smaller systems with far less tooling, and far far less integration were more productive - I've never seen such a bad way of _organizing work to be done_, or such a poor volume management of bugs as I have at Google in various teams/areas, despite all the tooling. Any delivery that involved client programs (desktop programs, mobile programs, etc) was always pretty remarkably awful, and only very very small teams were on the hook for making any of this better. They'd always plaster their new thing with "don't you dare make another or use the deprecated thing" meanwhile they never had enough time to make the new thing ready - a common trope in the company, but really highlights there are edges outside of the "pure saas" side of the business which were pretty bad. The best part of the monorepo and monotooling culture for the saas side is the security properties of it all, which were outstanding, and are among the reasons why I still trust Google as the holder of some main accounts. I hope they never start to lose those properties given the scale and impact they could have.
redis's C code is incredible
Java SDK 1.0
I just cannot face the thought of calling any of them "best".
Every one had good and bad features though. One or two were OS-sized and I think a codebase that compiles and links to 85GB of output for 20+ devices without being a total disaster inside is harder to do than a neat small python module or whatever.
GOOD FEATURES:
Maintenance of the build and test: I worked on tools that helped builds go faster so I saw a lot of codebases where people were not maintaining the build partly because nobody had that a s a responsibility. There was bad management of dependencies leading to build failures, poor performance, incorrect build output. Android would be a counter example to that - I don't know if people like developing in it but it was always hard to accelerate it as the maintainers fixed performance problems regularly leaving our tools with little to improve.
Using appropriate languages. Writing everything in C++ was a fad at one time. All projects work better, port better, have faster build times, are easier to test etc if they use memory safe "build once" languages to a maximum (e.g. java) and unsafe ones (e.g. C/C++ which have to be rebuilt and tested for each device/os) to a minimum. IMO Android beat Symbian amongst other reasons because it wasn't all C/C++ and that meant a lot of code didn't have to be rebuilt for different devices. This made builds faster and fast builds lead to better quality because of a short dev-test cycle.
Use of faster compilers over "better" compilers. Ultimate code performance and quality depends on a fast development cycle more (IMO) than on having the most optimizing compiler. GCC versus the older ARM compilers for example. Now the ARM compiler is based on LLVM and I know that happened indirectly from a suggestion I made to someone who then made it to ARM who then did it.
The setup and build of one codebase I worked on was as easy as one could expect, the build errored out if you tried to use the wrong tools so you never ended up debugging weird failures because of an incorrect tool in your PATH somewhere. I made this feature happen :-D. With big codebases the tools could be included in the version control system so you knew you had the right compiler, right object dumper etc. This is another strength of Android and yet I was in a project for Symbian to do the opposite because of some utter bonehead who never touched a build in his life who was trying to make a name for himself with his slimy bosses as a "doer" and "reformer."
Codebases (especially big ones) benefit a lot from some kind of codesearch/index where you could find out where some function/class/variable was defined and what version of the source base it was introduced in.
BAD FEATURES:
Exclusively Owned code - we need to know who understands code best and who is best to review it but I don't think anyone should have totally exclusive control. It was a nightmare for me at one job - trying to get another team to make some needed change (like fixing their stupid makefiles to work properly in parallel). We (build team) should have been able to do it ourselves - maybe including them in the PR. Sometimes ownership is entirely theoretical - nobody who wrote it is still employed and nobody among the notional owners understands it and none of them want to approach it within 100 metres in case it blows up and becomes their problem. I simply had to approach such code - no choice - but I kept having to send diffs to people who didn't want to bother to look at them. It was a case of pushing wet spaghetti and took forever to do very simple changes.
Insufficient tests that run infrequently. What else is there to say?
Complicated code with no "why" or "what this is for" type comments. The kind of thing you trawl around in for weeks and cannot make head nor tail of what is going on overall.
Code with so much dependency injection and general SOLID that you have to bounce all over the place to understand a very simple action.
Code where writing tests is an enormous ballache. In one Go codebase the reason was because somone decided that the standard Go practise of an array of test data being run through a kind of "test engine" was the only way anyone should be allowed to write tests. Hence you had to do lots of weird things to make your test cases into data. Generally we use a kind of "religious" approach to try to get consistency out of a group of people but then take it much too far.
codebases without automated reformatting - so everyone wastes time arguing about line spacing or camel-case names or whatever in their PRs.
Every one had good and bad features though. One or two were OS-sized and I think a codebase that compiles and links to 85GB of output for 20+ devices without being a total disaster inside is harder to do than a neat small python module or whatever.
GOOD FEATURES:
Maintenance of the build and test: I worked on tools that helped builds go faster so I saw a lot of codebases where people were not maintaining the build partly because nobody had that a s a responsibility. There was bad management of dependencies leading to build failures, poor performance, incorrect build output. Android would be a counter example to that - I don't know if people like developing in it but it was always hard to accelerate it as the maintainers fixed performance problems regularly leaving our tools with little to improve.
Using appropriate languages. Writing everything in C++ was a fad at one time. All projects work better, port better, have faster build times, are easier to test etc if they use memory safe "build once" languages to a maximum (e.g. java) and unsafe ones (e.g. C/C++ which have to be rebuilt and tested for each device/os) to a minimum. IMO Android beat Symbian amongst other reasons because it wasn't all C/C++ and that meant a lot of code didn't have to be rebuilt for different devices. This made builds faster and fast builds lead to better quality because of a short dev-test cycle.
Use of faster compilers over "better" compilers. Ultimate code performance and quality depends on a fast development cycle more (IMO) than on having the most optimizing compiler. GCC versus the older ARM compilers for example. Now the ARM compiler is based on LLVM and I know that happened indirectly from a suggestion I made to someone who then made it to ARM who then did it.
The setup and build of one codebase I worked on was as easy as one could expect, the build errored out if you tried to use the wrong tools so you never ended up debugging weird failures because of an incorrect tool in your PATH somewhere. I made this feature happen :-D. With big codebases the tools could be included in the version control system so you knew you had the right compiler, right object dumper etc. This is another strength of Android and yet I was in a project for Symbian to do the opposite because of some utter bonehead who never touched a build in his life who was trying to make a name for himself with his slimy bosses as a "doer" and "reformer."
Codebases (especially big ones) benefit a lot from some kind of codesearch/index where you could find out where some function/class/variable was defined and what version of the source base it was introduced in.
BAD FEATURES:
Exclusively Owned code - we need to know who understands code best and who is best to review it but I don't think anyone should have totally exclusive control. It was a nightmare for me at one job - trying to get another team to make some needed change (like fixing their stupid makefiles to work properly in parallel). We (build team) should have been able to do it ourselves - maybe including them in the PR. Sometimes ownership is entirely theoretical - nobody who wrote it is still employed and nobody among the notional owners understands it and none of them want to approach it within 100 metres in case it blows up and becomes their problem. I simply had to approach such code - no choice - but I kept having to send diffs to people who didn't want to bother to look at them. It was a case of pushing wet spaghetti and took forever to do very simple changes.
Insufficient tests that run infrequently. What else is there to say?
Complicated code with no "why" or "what this is for" type comments. The kind of thing you trawl around in for weeks and cannot make head nor tail of what is going on overall.
Code with so much dependency injection and general SOLID that you have to bounce all over the place to understand a very simple action.
Code where writing tests is an enormous ballache. In one Go codebase the reason was because somone decided that the standard Go practise of an array of test data being run through a kind of "test engine" was the only way anyone should be allowed to write tests. Hence you had to do lots of weird things to make your test cases into data. Generally we use a kind of "religious" approach to try to get consistency out of a group of people but then take it much too far.
codebases without automated reformatting - so everyone wastes time arguing about line spacing or camel-case names or whatever in their PRs.
> Code with so much dependency injection and general SOLID that you have to bounce all over the place to understand a very simple action.
I find that happens when people get religious about patterns and methodology; without understanding the "why", the language, and how a computer works.
Case in point: I once worked on a C# project that used a port of Spring for dependency injection: Ultimately, it was near impossible to know when something was constructed, and what was calling what. There were classes that couldn't call themselves through "this" because of certain weird dependency injection features used.
Later, I decided to use dependency injection as a design pattern: Instead of a complicated DI framework, there was just a few files of code. It was very easy for newcomers to understand. It was also easy to swap in mock objects, and easy to swap dependencies based on the target platform. It was also easy to see when a dependency was constructed; because it wasn't hidden behind a giant framework.
I find that happens when people get religious about patterns and methodology; without understanding the "why", the language, and how a computer works.
Case in point: I once worked on a C# project that used a port of Spring for dependency injection: Ultimately, it was near impossible to know when something was constructed, and what was calling what. There were classes that couldn't call themselves through "this" because of certain weird dependency injection features used.
Later, I decided to use dependency injection as a design pattern: Instead of a complicated DI framework, there was just a few files of code. It was very easy for newcomers to understand. It was also easy to swap in mock objects, and easy to swap dependencies based on the target platform. It was also easy to see when a dependency was constructed; because it wasn't hidden behind a giant framework.
[deleted]
NetBSD
I've seen enough code bases that I feel like I have a good spectrum from truly exceptional to horrific.
The best code bases had these things in common:
1. Consistency: they had clear patterns that were followed. It doesn't matter what the pattern was; some were strictly object-oriented, some were purely functional, some used RESTful API, and others leveraged gRPC or GraphQL. The important thing was that it was all consistent across the codebase so even if you were looking at a part you'd never seen before you could still orient yourself and reason about it quickly.
2. Ownership: there was always a single individual person who was ultimately responsible for each section of code. That doesn't mean they wrote all the code themselves; they were the final arbiter on the rare occasion a truly contentious conflict arose. They also had the unilateral power to make changes within their section of the code if they felt it was needed. This was always a rarely exercised power but they could, if they had to, push a change through. There could be many such people spread out across the codebase but for each discrete part the number was always one.
3. Clear Boundaries: it was clear what each part of the codebase's purpose was, and the boundaries were rigidly enforced. Things were never tightly coupled across these boundaries. Business logic was always isolated from things like serialization/deserialization, each system was forced to maintain its own models of the world that it cared about.
The worst code bases had these things in common:
1. Lack of Eng Representation: Product controlling development schedules and insisting everything is high priority and needs done right now! Project managers who always wonder, "yes but what if 9 women could make a baby in a month. Why don't we try adding more resources? Can we just try?" Business types who see software as a cost center not a profit driver. This can also happen if your engineer leadership didn't come up through software engineering but rather QA or IT or started out in academia or is just plain and MBA type with no eng background.
2. "Time saving" Tech: We don't need a database schema, we can just go NoSQL and have JSON blobs, it will save so much time! We can share models between the front end and back end, it will save so much time! Don't write SQL, use this ORM, it will save so much time! Don't think about DevOps, just use this all in one hosting solution, it will save so much time! What about this low code/no code solution? It will save so much time!
3. Misaligned Incentives: This could be because they were contractors with no stake in the company or this could be because it was a large company and there was no realistic way they'd ever be fired. Either way there were no consequences for writing bad code.
The best code bases had these things in common:
1. Consistency: they had clear patterns that were followed. It doesn't matter what the pattern was; some were strictly object-oriented, some were purely functional, some used RESTful API, and others leveraged gRPC or GraphQL. The important thing was that it was all consistent across the codebase so even if you were looking at a part you'd never seen before you could still orient yourself and reason about it quickly.
2. Ownership: there was always a single individual person who was ultimately responsible for each section of code. That doesn't mean they wrote all the code themselves; they were the final arbiter on the rare occasion a truly contentious conflict arose. They also had the unilateral power to make changes within their section of the code if they felt it was needed. This was always a rarely exercised power but they could, if they had to, push a change through. There could be many such people spread out across the codebase but for each discrete part the number was always one.
3. Clear Boundaries: it was clear what each part of the codebase's purpose was, and the boundaries were rigidly enforced. Things were never tightly coupled across these boundaries. Business logic was always isolated from things like serialization/deserialization, each system was forced to maintain its own models of the world that it cared about.
The worst code bases had these things in common:
1. Lack of Eng Representation: Product controlling development schedules and insisting everything is high priority and needs done right now! Project managers who always wonder, "yes but what if 9 women could make a baby in a month. Why don't we try adding more resources? Can we just try?" Business types who see software as a cost center not a profit driver. This can also happen if your engineer leadership didn't come up through software engineering but rather QA or IT or started out in academia or is just plain and MBA type with no eng background.
2. "Time saving" Tech: We don't need a database schema, we can just go NoSQL and have JSON blobs, it will save so much time! We can share models between the front end and back end, it will save so much time! Don't write SQL, use this ORM, it will save so much time! Don't think about DevOps, just use this all in one hosting solution, it will save so much time! What about this low code/no code solution? It will save so much time!
3. Misaligned Incentives: This could be because they were contractors with no stake in the company or this could be because it was a large company and there was no realistic way they'd ever be fired. Either way there were no consequences for writing bad code.
I'm not a software engineer, but the best codebase I worked on was at Quora. I attribute this to them having several years of time to build a base that was solid. That said, I found plenty of bugs, but it was just so nice to work with. Almost everything was Python. DOM, CSS, JS embedding, etc... keeping in mind this was way before Angular/React, but the system worked very similar, doing DOM diffing via a C++ service on the backend and pushing updates out over a long polling JS endpoint.
On top of that the data which was read heavy went through several layers of cache that were very easy to understand (process, system, global - called L1, L2, L3 like a system as a CPU). The front-end had a "Konami code" that let you see the rendering hierarchy from the backend while viewing it from the frontend that let you also understand the caching state of each level of hierarchy that was rendered as well as all the CSS/JS handlers affecting the element, not unlike current "inspect" view of most browsers.
What made this all so good? A very strong foundation that was opinionated, but which was consistent. This was ~ 2012-2014 - they had Flask as an underlying framework, and much was built out on top of Python decorators.
Besides the day-to-day code - they had a bespoke packaging system, auto-scaler, and a number of other services that "just worked" - it was very different from any other startup I've worked for.
On top of that the data which was read heavy went through several layers of cache that were very easy to understand (process, system, global - called L1, L2, L3 like a system as a CPU). The front-end had a "Konami code" that let you see the rendering hierarchy from the backend while viewing it from the frontend that let you also understand the caching state of each level of hierarchy that was rendered as well as all the CSS/JS handlers affecting the element, not unlike current "inspect" view of most browsers.
What made this all so good? A very strong foundation that was opinionated, but which was consistent. This was ~ 2012-2014 - they had Flask as an underlying framework, and much was built out on top of Python decorators.
Besides the day-to-day code - they had a bespoke packaging system, auto-scaler, and a number of other services that "just worked" - it was very different from any other startup I've worked for.
I never got to submitting my own patches but the Linux kernel is nice.
Where I work now I get to work in Haskell. And that’s been a pleasure. Recently rewrote over 1k lines. Deployed to production with one reviewer. No issues.
Just good test coverage and a great type system with no-nonsense code. Refactoring in Haskell is a heady drug.
If this was C++ or even Python there would have been much more intense reviews and probably some fail-forward patches to fix things we may have missed.
But I dunno. Maybe not. Maybe I’m just that good. /s
Where I work now I get to work in Haskell. And that’s been a pleasure. Recently rewrote over 1k lines. Deployed to production with one reviewer. No issues.
Just good test coverage and a great type system with no-nonsense code. Refactoring in Haskell is a heady drug.
If this was C++ or even Python there would have been much more intense reviews and probably some fail-forward patches to fix things we may have missed.
But I dunno. Maybe not. Maybe I’m just that good. /s
I worked with a very experienced engineer who created his own WAF (software-based) using Java.
From a purely architectural perspective, it might not be the best (I might have used an ISAPI filter back then), but the code itself was very efficient, well-written and documented. I used it several times as a teaching example.
>> Ask HN: What is the best code base you ever worked on?
LOL. Obviously. Mine! :)
LOL. Obviously. Mine! :)
Hopefully the most recent one you were responsible for.
A former coworker texted me out of the blue yesterday, saying he missed working on a "<my name> codebase". He specifically appreciated that "doing something simple things were kept simple." Made my day!
Was there someone enforcing good practices top down? Just being in a group of great engineers? Or something else?