This is Atom’s first open source project: Splitter – a coordinator for stateful sharded services. It is used to manage our most critical stateful services: product-oriented (e.g., food order handling) and pure infra services such as our inhouse-built message broker.
Gimbel et al's methodology and limitations are well described. The data is available. And crucially Joshua Kendall's code is available online (https://github.com/Budget-Lab-Yale/AI-Employment-Model). I wish more research was done this way.
The impact is not terrible given active coding takes <25% of engineers time at most companies1. GenAI can't help too much with high level design work, collecting feedback, etc.
I didn't write this post. But I work with the guys
We've been doing data science and ml for years. After iterating on our tooling for a few years, we've finally settled on some tools that we're happy with. A lot of the wins came from spending more time data with DS teams to understand what they really wanted tbh
Doesn't seem a crazy way to start a company. RDS will have scaling problems but is very mature and fairly easy to use early on when you're working on your MVP
I've used CRDB early on at a startup. There was some overhead. You don't get all the nice PSQL features
Although it did save us a giant migration later on
We've been building up CI tooling on top of Nanoscope. The short term goal for this is: make sure that none of our internal libraries (analytics, experimentation, etc) contribute a significant percentage of the time it takes to respond to any tap in the app. We have a lot of internal mobile libraries, so preventing perf regressions in them is a big rock.
If our CI efforts prove stable, then what you're suggesting should also be possible, if I understand it correctly.
> Science is a process and is close to an infallible thing as we have created as humans. There is no problem with science.
Wouldn't you say mathmatics is the most infallible thing human's have come to creating? The level of rigor in mathematics is much higher and there aren't as many repeatability issues.
What is most interesting to me: we prioritized mobile app architecture and saw big wins as a result. Usually its hard to commit time upfront to app architecture. We've made that the norm (ex, 80% of our app code lives inside plugins). And we've seen eng productivity measurably increase over the last year because of these investments. In many cases doubled.
It's definitely a lot of engineers. But we cram a lot of features into the app. I expect that most people will only experience 10% of them. Consider:
1) We exist in a tonne of countries. And different counties often require different product optimizations and payment methods
2) We have our own map provider
3) We experiment with everything
4) We support countries that have terrible networking. That creates lots of challenges.
But the old Uber app had hit its age limit. It was build on top of technology chosen for a small number of features (ex: a single global DI component, lack of typing, a small MVC hierarchy). So we either needed lots of large migrations or a rewrite. The incremental migrations required to fix these issues would have been extremely disruptive, they wouldn't have gotten us an entirely refreshed UI and they wouldn't have given us a number of other benefits.
So we decided to do a rewrite. We had lots of engineers that knew the issues to watch for from the first time we wrote the app. And a handful of us spent months researching/building different architectures, static analysis and tooling that would ensure the rewrites success. We weren't going to repeat the same mistakes twice.
On the very first day the app launched it was more reliable and performant than the version of our app that we had been maintaining for years.
I expect lots of us use it. There is nothing in my email that the NSA would find remotely interesting. Therefore, there is no practical reason not to use it.
You can find more information about Splitter in this blog post: https://techblog.atoms.co/p/easy-as-pie-stateful-services-at
Blog post about our in-house message broker: https://techblog.atoms.co/p/reliable-order-processing
Source code is available in the splitter repository. You can see Splitter in action in splitter-examples (only requires Docker and Bazel). Repo: https://github.com/atoms-co/splitter-examples/tree/main/robo...
We will never intend to sell this. So may as well open source it.