The general idea is we're adopting parts of Kubernetes where we can and working to extend Kubernetes in ways that integrate well with the approach Netflix has taken to Compute. We plan to bring the approaches Netflix takes to Kubernetes to the community over time as makes sense for the broader community. This work on leveraging user namespaces is a good example of something that while unique to Netflix is something we would like to see the community benefit from as well.
I work on this team. We host Netflix compute that while "internal" processes requests from Netflix users and the internet. We use industry standard frameworks and technologies in our workloads. All software ends up having security incidents. We have this lower level security to protect ourselves if the higher level technology has temporary security problems.
This. I continue to hear about hacks of people running ping to keep a single instance warm. But that doesn't cover periodic changes in capacity needs nor spikes. I would think to avoid cold starts all together you'd need a pinger that sent exactly the load difference between peak load and current load. I would love to hear if anyone is keeping Lambdas warm at more than n=1 capacity.
Note: I'm the one in the video, an original engineer on Titus, and now the product/hiring manager of the Titus team at Netflix. Also, I do not actively monitor hacker news, so it is unlikely that I'll follow-on to this conversation (@aspyker on Twitter is an easier way to reach me).
Is Titus Open Source Yet? No.
Why?
Reason #1: When we created Titus, we married the back end of a container execution engine (Titan) with the front end of a scheduling system for stream processing (Mantis). We have a goal at the time to stabilize existing usage at Netflix. We didn't have a goal of removing all code from both sides that wasn't needed after the marriage. We wanted to clean this up before releasing in open source as we would have to spend more time telling people what to ignore in the codebase that is no longer used in Titus. We also didn't have time to create proper interfaces that separate Titus from other important operational systems at Netflix. We have recently been re-implementing the engine to have clean interfaces with minimal code.
Reason #2: When Titus was created it had a user API that grew during initial usage in batch applications. Given the number of clients using this API, it was hard to change it without impacting users. Compounding this, we added service application support without thinking much about making the API well designed. We have recently been working on a new API that was designed to handle both use cases. This API is currently being adjusted based on internal feedback.
Reason #3: The space of container management platforms is truly a crowded space. There are many other great options out there. When we eventually open source Titus, we hope it is clear why Titus is unique in this space specifically for "all-in" Amazon EC2 customers, for NetflixOSS cloud platform adopters (those who use Spinnaker, Eureka, etc.), and supporting the level of complexity we do (VPC IP per container, GPU's, complex fleet and capacity management, etc.). Finally, showing how at scale production battle hardened Titus is. We realize even with these differences, we'll spend a consider amount of time justifying these points to those who look at the space more generally. We would prefer to not spend any time "marketing" our differences.
Reason #4: Our main focus at Netflix is to service Netflix developers providing the easiest to use and most reliable service. Open source is something that is valuable to us with regards to potential external contributions as well hiring and retention of the best engineering talent in the world. This value, while significant, isn't as important as providing our internal Netflix value. As a part time responsibility I help open source at Netflix. Therefore, I am also aware that doing open source well (like NetflixOSS Spinnaker as a shining example) requires more investment for teams than posting the code to github. Our team is small today and we do not believe we can make Titus open source as great as we'd want it to be. That said, we're growing (did I mention I'm the hiring manager - https://jobs.netflix.com/jobs/862432) every day and can see the value of changing this equation.
I am truly honored that someone would great a whole website to ask for Titus. https://www.istitusopensourceyet.com/ made me laugh last night and decide to write this response today. We appreciate your excitement for our container management technology and Netflix open source. As we continue to evaluate this space, we'd love to find other AWS users who are also using the Netflix cloud platform and Spinnaker to consider partnership.
While we would love to eventually open source Titus, we have been unable to invest the time it would take to do this responsibly (invest in helping the community, decoupling it cleanly from our other internal systems, etc.) yet. We have needed to stay focused on growing and supporting Titus for Netflix users. We have no firm plans on when we'd be able to change this position.
We do hope being able to talk about the lessons learned helps the community in the meantime.
You can get a better idea of where we are in our journey to Kubernetes (both already deployed and being developed) from this presentation:
https://www2.slideshare.net/aspyker/herding-kats-netflixs-jo...
The general idea is we're adopting parts of Kubernetes where we can and working to extend Kubernetes in ways that integrate well with the approach Netflix has taken to Compute. We plan to bring the approaches Netflix takes to Kubernetes to the community over time as makes sense for the broader community. This work on leveraging user namespaces is a good example of something that while unique to Netflix is something we would like to see the community benefit from as well.