I've been experimenting with AI coding tools daily for the past year with a small engineering team (all senior 15+, FAANG backgrounds). Wrote down everything we learned, the parts that worked and the parts that didn't.
Some things that surprised us:
- AI is slower than writing code yourself on any single task. The win is not speed, it's parallelism. We made our dev environments duplicable and now work on multiple PRs simultaneously.
- AI is dramatically better on clean code than on technical debt. It's like a mirror that shows you how much chaos you've been tolerating.
- Out of the box tools give mediocre results. We invested heavily in configuration (20 AGENTS.md files, 6,553 lines total) and in augmenting the agent with skills: typechecking, test runners, read-only access to tickets, logs, Sentry. The agent self-debugs and fetches its own context now.
- Never trust the output. We built a pr-reviewer agent whose only job is to criticize code changes against the original specs. Strongly typed languages help enormously here, the compiler is your cheapest AI reviewer.
- Mental context-switching is the new must-have skill for engineers. You orchestrate, the AI executes (for now).
Would love to hear what's working for other teams, I'm sure our approach is not the only one.
Author here. Context + details that didn’t fit cleanly in the article:
This was a supervised experiment, not “hands off, let the AI run prod.”
I treated the AI as a junior sysadmin/operator with full read access and explicitly scoped write access, and I stayed accountable for architecture and decisions.
Setup (high level):
Single Raspberry Pi 5 running Debian
Self-hosted services (Nextcloud, finance tools, VPN, backups)
AI accessed the system via SSH, logs, config files, and command output
Tasks were given as concrete operational goals (“diagnose why X stopped working”, “make backups safer”, etc.)
What worked well:
Debugging real issues (DNS, IPv6 quirks, misconfigurations)
Writing small scripts and operational docs
Being extremely fast at “read unfamiliar system → form hypothesis → test”
What did not work well:
Anything involving long-term architectural judgment
Security decisions without explicit constraints
Situations where the right answer was “don’t touch this yet”
The biggest takeaway for me: this didn’t replace judgment, but it dramatically compressed the “thinking + doing” loop for ops work I already understand.
Happy to answer questions about failure modes, security boundaries, or how I constrained access.
Food in the US is just such a pile of shit and ability to distiquish quality from garbage in americans is so non-existent that it doesn't surprise me that there is someone here in the US that really believes you can make Parmiggiano or Feta outside the area they got invented. They just don't get it and I feel sorry for them. I lived decades in Europe and US.
If you are a dairy and have milk, make your own cheese, maybe in 500 years it will be decent, just don't produce knockoff products because "they sell", you are doing a disservice to your fellow americans by depriving them from learning how to taste things.
Some things that surprised us:
- AI is slower than writing code yourself on any single task. The win is not speed, it's parallelism. We made our dev environments duplicable and now work on multiple PRs simultaneously.
- AI is dramatically better on clean code than on technical debt. It's like a mirror that shows you how much chaos you've been tolerating.
- Out of the box tools give mediocre results. We invested heavily in configuration (20 AGENTS.md files, 6,553 lines total) and in augmenting the agent with skills: typechecking, test runners, read-only access to tickets, logs, Sentry. The agent self-debugs and fetches its own context now.
- Never trust the output. We built a pr-reviewer agent whose only job is to criticize code changes against the original specs. Strongly typed languages help enormously here, the compiler is your cheapest AI reviewer.
- Mental context-switching is the new must-have skill for engineers. You orchestrate, the AI executes (for now).
Would love to hear what's working for other teams, I'm sure our approach is not the only one.