The AI applications that I build have a core loop that does most of the work, and then most of the other components are designed to make sure that the output is actually correct and helpful. The components that I use for that -- are they also compounding the failure rate as this article suggests?
I have always felt that my orchestration is supposed to increase the reliability because that is what the pipeline is doing. Checking the work of the main agent and giving feedback. But this article seems to be arguing otherwise. That adding more agents in the pipeline just makes the entire system work less reliably.
This is only true for "development branch" CRs/pull requests. The whole is greater than the sum of the parts. Every small change in the feature that you are building might make complete sense, so every dev-to-feature branch pull request would get approved easily.
But if you not also reviewing the feature-to-main branch pull request, you are just inviting problems. That is a bigger CR that you should review carefully, and there is no way that could be a small CR.
In the maker-checker process, if we are imagining a future where AI will be writing/editing most of the code, the AI-code-review tools will need to integrate within its agentic process.
And the job of a better code-review interface (like the one that I am trying to build) would be to provide a higher level of abstraction to the user so that they can verify the output of the AI code generators more effectively.
What you are saying is true, and this is the feedback I hear every time I talk to a small team of developers (generally fewer than 15 developers).
At this stage, you don't need "another set of eyes" because it is not that big of a problem to break something, as you are not going to lose massive amounts of money because of the mistake.
All these teams need is a sanity check. They also generally (even without the AI code reviewers) do not have a strong code review process.
This is why, in the article, I have clearly mentioned that these are learning based on talking to engineers in Series-B and Series-C startups.
I have had a similar discussion with a fellow On-Deck Founder, and here is where we reached:
- More than being "good enough", it is about taking responsibility.
- A human can make more mistakes than an AI, and they are still the more appropriate choice because humans can be held responsible for their actions. AI, by its very nature, cannot be 'held responsible' -- this has been agreed upon based on years of research in the field of "Responsible AI".
- To completely automate anything using AI, you need a way to trivially verify whether it did the right thing or not. If the output cannot be verified trivially, you are just changing the nature of the job, and it is still a job or a human being (like the staff you mentioned who remotely control Waymos when something goes wrong).
- If an action is not trivially verifiable and requires AI's output to directly reach the end-user without a human-in-the-loop, then the creator is taking a massive risk. Which usually doesn't make sense for a business when it comes to mission-critical activities.
In Waymo's case, they are taking massive risks because of Google's backing. But it is not about being 'good enough'. It is about the results of the AI being trivially verifiable - which, in the case of driving, is true. You just need three yes/no answers: Did the customer reach where they wanted? Are they safe? Did they arrive on time? Are they happy with the experience?
I'm sorry, I didn't mean it to be an ad. I have been interviewing engineering leaders for months, and my startup idea is born out of it. I don't have the product ready yet - it is evolving based on what I am learning.
I just thought it would be a good idea to share what I have learnt.
Wow, this is a very well-written article. I have experienced a lot of this in my own experience as a software developer.
This makes me wonder, the concept of pair programming has been around for a very long time. And yet, pull requests have grown in popularity while the use of pair programming remains pretty limited.
Does that mean that companies want to operate like a bunch of individuals instead of a team? Is independence valued more than speed & collaboration when it comes to software development teams?
I have heard the same as well.
These days, I am thinking about how AI code gen can be affecting this. When AI is writing code, you can't assume yourself to be the pair programmer, because your speed is not even close to the AI's. You are basically reviewing the code that AI has written.
So should people be thinking about pair-reviewing AI code so that they get the benefits of pair programming along with the speed of AI?
In the "Knowledge Sharing" section of the article, the author says that "Distributed Practice" is one of the most effective ways to learn, and yet says that code reviews are not effective ways of knowledge sharing.
Isn't a code review EXACTLY a distributed practice? What am I missing?
And then he goes on to say "underlining and summarization while reading are least effective" -- I don't understand how is that even related to reviewing code.
If 42% of the comments on a pull request are related to increasing the understandability of the code, can we assume that just understanding the proposed code changes consumes the majority of the time spent on reviewing a pull request?
Wow, this looks very interesting. I saw the diagram of your own application and it looks more dynamic than the ones that I am generally used to at work.
I am curious about the motivation behind this project. What experiences triggered you to think that static diagrams are a problem?
Your answer will help me decide whether I'd like to use it for my own documentation or not.
It is good that you are already thinking about it.
My wife and I have separate bank accounts but we operate as a single entity.
We are both completely aware of each other's incomes and expenses (we maintain a common expense manager).
When we got married, there was an immense difference in our incomes (I earned 10x more than her), so we decided that I would bear all the expenses and she could save her income for her future business plans.
Since I left my job to focus full-time on my startup last year, our incomes have been almost equal. Now, I pay the rent and the bills, and she handles all other expenses (groceries, appliances, travel plans, gifting and other purchases).
We transfer money to each other's accounts whenever there is a need (it's mostly from her to mine). Transparency is the key to happiness here.
I am a bit confused about your stage as a product.
When you say "get people to try out your product", I feel that you are building a new product and looking for early feedback - your first design partner.
But then you say "offer long-term deals or any rewards" and "learn GTM for software products", which makes me think that you are done with your pilots proven that your product solves the problem much better than all the other solutions out there.
These two questions are very different. The first one is about validating if the problem is big enough for the person you are talking to. The second one is about communication and growth.
Nice observation.
I still use Google Pub/Sub in my application - I recently also gave a talk on how we use Pub/Sub for our use case in a GCDG event (GCDG stands for Google Cloud Developer Group).
But now that I think about it, we don't use it in the traditional sense. Most of our regular operations work well enough by just using the "async" pattern in our programming (in JS and Rust).
The only place we use Pub/Sub is for communication between our NodeJS backend server and the Rust servers that we deploy on our client's VMs. We didn't want to expose a public endpoint on the Rust server (for security). And there was no need for a response from the Rust servers when the NodeJS server told it to do anything.
We don't fully utilize the features of a messaging queue (like GCP's Pub/Sub), but there just wasn't a better way for our specific kind of communication.