Tilt: dev environment as code(github.com)
github.com
Tilt: dev environment as code
https://github.com/tilt-dev/tilt
66 comments
Same, being able to depend on CRDs between resources would be very useful, since a lot of my Tilt usage is about making operators or working with operators.
Happy to see new releases of Tilt even if the pace has slowed down. It's a very useful tool.
Happy to see new releases of Tilt even if the pace has slowed down. It's a very useful tool.
As far as I understand, this might be is the best tool for the guy who acquires a SaaS and have no clue how to develop/tweak stuff. Again still have no clue how you enable this or do you still have tilt in production. Marketing is not a big deal for tilt as far as I understand.
You bring up an interesting question -- I think Tilt works best for those that find themselves in an environment where product is delivered using a service-oriented architecture deployed to Kubernetes. It's also easy to get started within a small team in a big company.
To your other point, Tilt is to development as ArgoCD is to deployment. Tilt enables on-demand, reproducible development environments that are sufficiently high-fidelity that you can often replace your shared and/or long-lived testing clusters.
With Tilt, I test my application using the same Kubernetes specs / Kustomizations / Helm Charts that you use to deploy into production. When it comes time to deploy my application, I supply these same specs / kustomizations / charts to ArgoCD.
Because I can reuse the specs for both testing and production, I enjoy far greater testability of my application, improving quality and time to market.
To your other point, Tilt is to development as ArgoCD is to deployment. Tilt enables on-demand, reproducible development environments that are sufficiently high-fidelity that you can often replace your shared and/or long-lived testing clusters.
With Tilt, I test my application using the same Kubernetes specs / Kustomizations / Helm Charts that you use to deploy into production. When it comes time to deploy my application, I supply these same specs / kustomizations / charts to ArgoCD.
Because I can reuse the specs for both testing and production, I enjoy far greater testability of my application, improving quality and time to market.
The pitch is kinda funny to me:
> Modern apps are made of too many services. They're everywhere and in constant communication.
So we made tooling to make it easier for you to make more of them!
> Modern apps are made of too many services. They're everywhere and in constant communication.
So we made tooling to make it easier for you to make more of them!
We need a complexity tariff
Great talk about this... https://www.youtube.com/watch?v=j2AQ9eTZ3-0
Disagree. It’s the same enterprise architecture gibberish that creates the problem in the first place.
Feels like standards xkcd https://xkcd.com/927/
You're always trading off speed with fidelity. Usually, trying to maintain a local integration environment is going to become too slow and expensive. The problem isn't even necessarily Kubernetes, but as dependencies increase it just gets slower and slower to try and run a copy of the world locally.
I like a fast svelte dev environment with something like docker-compose which might require some mocked out dependencies to keep things fast and then using Kubernetes for other environments once I have local tests passing.
I like a fast svelte dev environment with something like docker-compose which might require some mocked out dependencies to keep things fast and then using Kubernetes for other environments once I have local tests passing.
You can always swap out some of those services for always-running prod or dev-in-the-cloud services.
I have a big mix of setups for my projects. I like Vite+Bun (client+server) for smallish projects because the dev servers start instantly. But then I have to remind myself how to actually productionize it when it comes time for that, but it's not too hard.
Then sometimes I need to bring in a database. I don't like running those locally, undockerized because now you're locked into one specific version and it just gets messy. So sometimes if I'm lazy I just connect to my prod database server and just create a separate dev DB on it if needed. Then I don't need to start and stop anything, I just connect. Easy.
For a big complex app you can mix-and-match any of the above. Run the services that you need to iterate heavily on locally, and just farm out the rarely changing stuff to the cloud.
I have a big mix of setups for my projects. I like Vite+Bun (client+server) for smallish projects because the dev servers start instantly. But then I have to remind myself how to actually productionize it when it comes time for that, but it's not too hard.
Then sometimes I need to bring in a database. I don't like running those locally, undockerized because now you're locked into one specific version and it just gets messy. So sometimes if I'm lazy I just connect to my prod database server and just create a separate dev DB on it if needed. Then I don't need to start and stop anything, I just connect. Easy.
For a big complex app you can mix-and-match any of the above. Run the services that you need to iterate heavily on locally, and just farm out the rarely changing stuff to the cloud.
I think that's a fair point -- you're making a tradeoff. And the best part is that you don't need to choose one or the other.
In my case, I find that I prefer having higher fidelity and simpler service code by using Tilt to avoid mocks. It's also nice for frontend development because, using a Kubernetes ingress, you can avoid the need for things like frontend proxies, CORS and other development-only setup.
In my case, I find that I prefer having higher fidelity and simpler service code by using Tilt to avoid mocks. It's also nice for frontend development because, using a Kubernetes ingress, you can avoid the need for things like frontend proxies, CORS and other development-only setup.
I still think "dev environments" really ought to be running tests directly with your languages native tool. e.g. `cargo test`, `bundle exec rspec`, etc. If you make me run a VM that runs Kubernetes that runs a docker container that runs the tests, I will be very, very upset. Doing this properly and reliably can still be a lot of work, possibly more work if not relying on Docker is a design goal (which must be if you want to run natively on macOS).
There seem to be a lot of tools in this space. I wish they wouldn't call themselves tools for "dev environments" when they are really more like tools for "deploying an app to your local machine", which is rather different.
There seem to be a lot of tools in this space. I wish they wouldn't call themselves tools for "dev environments" when they are really more like tools for "deploying an app to your local machine", which is rather different.
I actually have the exact opposite viewpoint: if you’re managing a platform with multiple teams what you are suggesting is way more of a pain than a standardized, container based workflow. You want a language agnostic test runner that runs generic commands. The reason for this is that you want to be able to quickly skill up engineers and have them able to quickly switch codebases since the interface (like tilt) is the same across all of them.
You give up a bit of snappiness, sure, but you can also keep the very small non container based tooling like linting outside of the container.
You give up a bit of snappiness, sure, but you can also keep the very small non container based tooling like linting outside of the container.
> You give up a bit of snappiness, sure, but you can also keep the very small non container based tooling like linting outside of the container.
You give up way more than snappiness. Doing real development work, i.e. compiling, testing, debugging, is very cumbersome in a remote environment.
So where do you want to spend your time? Bandaids to make remote development suck less, or effort to develop locally, natively? There is no free lunch. If you choose "neither", your developer experience is gonna suck. (Most companies choose "neither" by the way, either consciously or unconsciously).
You give up way more than snappiness. Doing real development work, i.e. compiling, testing, debugging, is very cumbersome in a remote environment.
So where do you want to spend your time? Bandaids to make remote development suck less, or effort to develop locally, natively? There is no free lunch. If you choose "neither", your developer experience is gonna suck. (Most companies choose "neither" by the way, either consciously or unconsciously).
> Doing real development work, i.e. compiling, testing, debugging, is very cumbersome in a remote environment.
it really isn’t? Of course it depends on the ecosystem. But for jvm for example you literally just expose your debugging port 5005 out of the container and boom step through, and other live debugging works just as well as outside of the container. And as of course you allude to, if you’re native you are facing a “works on my machine” problem unless you are all in on a hermetic and reproducible solution like bazel or nix. And chances are unless you are having that crack team of 10xers that a good hunk of your dev user base are going to struggle with the complexity and general ecosystem issues with these two solutions.
You’ve probably seen the worst world where people do containers wrong. And a lot of people do them wrong. But it’s pretty easy to learn how to do them right. Someone can study multi stage docker builds for half a day and write perfectly fast, cached first containerized builds. Proper buildkit cached local containers are extremely fast.
There’s other ways of course, each with their own tradeoffs. You can do everything in nix, and now you are spending your time fighting with nix. You can do everything in bazel and now you are spending time fighting with bazel. In the end your stuff is gonna go into a container anyway (for most people). You still need to understand the container technology because of that. so why not both reduce your toolchain sprawl and simultaneously recreate that exact environment on the local machine?
it really isn’t? Of course it depends on the ecosystem. But for jvm for example you literally just expose your debugging port 5005 out of the container and boom step through, and other live debugging works just as well as outside of the container. And as of course you allude to, if you’re native you are facing a “works on my machine” problem unless you are all in on a hermetic and reproducible solution like bazel or nix. And chances are unless you are having that crack team of 10xers that a good hunk of your dev user base are going to struggle with the complexity and general ecosystem issues with these two solutions.
You’ve probably seen the worst world where people do containers wrong. And a lot of people do them wrong. But it’s pretty easy to learn how to do them right. Someone can study multi stage docker builds for half a day and write perfectly fast, cached first containerized builds. Proper buildkit cached local containers are extremely fast.
There’s other ways of course, each with their own tradeoffs. You can do everything in nix, and now you are spending your time fighting with nix. You can do everything in bazel and now you are spending time fighting with bazel. In the end your stuff is gonna go into a container anyway (for most people). You still need to understand the container technology because of that. so why not both reduce your toolchain sprawl and simultaneously recreate that exact environment on the local machine?
Not to mention the developer experience is usually sub-par.
I firmly believe that the primary way of interacting with my tests should be the ability to run them one by one from the IDE, and running the code should be run / attach with breakpoints.
I firmly believe that the primary way of interacting with my tests should be the ability to run them one by one from the IDE, and running the code should be run / attach with breakpoints.
It takes some work, but it's entirely possible to both use Docker and run individual tests with breakpoints (in a Docker container) in your IDE. For example, you can attach VS Code to a running container.
Yes, but it creates a restrictive and fragile happy-path when the aim imo should be closer to a lab/woodshop where you can take it apart however you like and need for the moment.
Shell containers under code are unbearably laggy and crappy.
I simple have a container for each project using my own container-shell
I run my bundles / whatever. Have all the tooling and can use VSCode to attach via ssh (I use orbstack, so I get project hostnames for free)
It’s the best workflow for me. I really wanted to like containers but again, it’s too heavy, buggy, bulky.
http://github.com/jrz/container-shell
I simple have a container for each project using my own container-shell
I run my bundles / whatever. Have all the tooling and can use VSCode to attach via ssh (I use orbstack, so I get project hostnames for free)
It’s the best workflow for me. I really wanted to like containers but again, it’s too heavy, buggy, bulky.
http://github.com/jrz/container-shell
True. I learned this the hard way.
Can't not mention nix-shell: https://nix.dev/manual/nix/2.18/command-ref/nix-shell
If you want to see Tilt in action, our Chroma open-source repo uses it to run the distributed version of the database for development and ci. It's pretty cool - just clone then run `tilt up` and it's working:
https://github.com/chroma-core/chroma
https://github.com/chroma-core/chroma
Thank you for sharing this! I think your Tiltfile just showed me how to solve something that's been bugging me for a while!
I see that you also have docker-compose files -- are those for different tasks or for developer preference?
I'm also curious to understand why you have different build scripts for CI (`buildx`) vs local (regular docker build)? In our team, we use the same build processes for both.
I see that you also have docker-compose files -- are those for different tasks or for developer preference?
I'm also curious to understand why you have different build scripts for CI (`buildx`) vs local (regular docker build)? In our team, we use the same build processes for both.
I don’t think my problem has ever been with setting up a local environment.
Single cluster deployments are very easy.
My problem is that these services we manage in production are deployed across multiple regions (or k8s clusters).
Debugging _distributed_ applications is the issue.
Single cluster deployments are very easy.
My problem is that these services we manage in production are deployed across multiple regions (or k8s clusters).
Debugging _distributed_ applications is the issue.
How does Tilt compare to “skaffold dev“? We use skaffold exactly for that purpose. To develop within a the cluster.
Migrated from Skaffold to Tilt at my last co, found it was much more easier to configure granular rebuild rules, which lead to faster dev loop cycles
Any catch?
Skaffold works but its DX is pretty poor. Too many knobs via yaml- tilt has just enough magic that it doesn't feel like a chore to setup local dev.
I've always appreciated that Tilt chose Starlark instead of YAML. Makes things so much cleaner!
Well, now you've really got me interested. Almost every case of a YAML eDSL would be better served by Starlark instead, in my Bazel-brained opinion.
I think Tilt Extensions highlights the combined power and composability that using Starlarks brings to Tilt: https://docs.tilt.dev/extensions.html.
Want to create a Kubernetes secret? It's as simple as:
Want to create a Kubernetes secret? It's as simple as:
load('ext://secret', 'secret_yaml_generic')
k8s_yaml(secret_yaml_generic(...))
load('ext://vault_client', 'vault_read_secret', 'vault_set_env_vars')
vault_set_env_vars('https://localhost:8200','mytoken')
my_foo = vault_read_secret('path/myfoo', 'value')
my_bar = vault_read_secret('path/mybar', 'foobar')Much more flexible than Skaffold thanks to Starlark config vs. a rigid YAML structure.
Starlark does allow for much more concise and powerful config specification. I am building https://github.com/claceio/clace, which is an application server for teams to deploy internal tools.
Clace uses Starlark for defining apps, instead of something like YAML. https://github.com/claceio/clace/blob/main/examples/utils.st... is a config file which defines around seven apps (apps are downloaded and deployed directly from git).
Clace uses Starlark for defining app level routing rules also. This avoids the need to use a nginx like DSL for routing.
Clace uses Starlark for defining apps, instead of something like YAML. https://github.com/claceio/clace/blob/main/examples/utils.st... is a config file which defines around seven apps (apps are downloaded and deployed directly from git).
Clace uses Starlark for defining app level routing rules also. This avoids the need to use a nginx like DSL for routing.
Isn't this essentially dev containers?
My understanding is that dev containers are more about configuring your development environment with the right toolchains to build and run services.
Tilt is a monitor process that builds and starts your services, with a hot-reload loop that rebuilds and restarts your services when the underlying code changes. The hot reload loop even works for statically compiled languages.
Tilt is a monitor process that builds and starts your services, with a hot-reload loop that rebuilds and restarts your services when the underlying code changes. The hot reload loop even works for statically compiled languages.
Yeah it would be really useful to have in the readme what kind of problems this solves that dev containers don't solve already
I briefly tried Tilt awhile back. Tilt, Garden, maybe a couple others, settled on DevSpace. IIRC, it aligned best with my existing prod infrastructure without having to rewrite everything a different way.
i.e., it worked with my existing kustomize+k8s setup. It adds portforwarding, and fast file sync into the running containers, which is all I really want. Rebuilding an image every time you make a change sucks.
i.e., it worked with my existing kustomize+k8s setup. It adds portforwarding, and fast file sync into the running containers, which is all I really want. Rebuilding an image every time you make a change sucks.
I'm showing my age here, but I legitimately think this classic 2004 progressive anthem from Tilt (Andy Moor project) would have been bang-on perfect for the demo video compared to... whatever stock music that was: https://www.youtube.com/watch?v=0Cm-nrm8H78
i don't get the value of a tool like this.
Do we really struggle bringing up services as containers and applying kube configs?
For my development of services that run in kube, I don't dev with kube, you shouldn't have to. I also use docker-compose for most dev env services.
Perhaps i'm not developing the right kind of software. Whoever finds this type of tool useful, when would you use it?
Do we really struggle bringing up services as containers and applying kube configs?
For my development of services that run in kube, I don't dev with kube, you shouldn't have to. I also use docker-compose for most dev env services.
Perhaps i'm not developing the right kind of software. Whoever finds this type of tool useful, when would you use it?
Our team switched from Docker Compose (without Kubernetes) to Tilt for a distributed systems development environment. (Think platform engineering work on a system that scales from zero to several hundred thousand instances). Our time to go from code change to testable, running code on our laptops went from about a minute to a couple of seconds, using some Tiltfile recipes to do automatic incremental recompilation on our host laptops as we edit source files, and live-reload the new artifacts into running Kubernetes containers. The reload happens so fast that we configured our environment to compile+deploy+run on save, and the new code is already running by the time you reach for the "run tests" button.
I think if you told our team to go back to Docker Compose they'd revolt on the spot haha
I think if you told our team to go back to Docker Compose they'd revolt on the spot haha
tilt might be nice but I do not develop in containers or kubernetes. Tilt to me looks like more glue on top to make updating kube fast. It might make sense for some but it seems pointless to me.
If I can run external dependencies in docker locally, I can setup my app to run entirely from my laptop. That's all docker-compose does, just runs dev deps like DBs and other services i'm not editing code of.
As far as code reloading goes, there is a million tools to do that already. Go already compiles locally much faster than seconds.
All that being said, why are people choosing to develop in containers/kubernetes?
Maybe apps that need to be more tightly integrated with kube would benefit from this?
If I can run external dependencies in docker locally, I can setup my app to run entirely from my laptop. That's all docker-compose does, just runs dev deps like DBs and other services i'm not editing code of.
As far as code reloading goes, there is a million tools to do that already. Go already compiles locally much faster than seconds.
All that being said, why are people choosing to develop in containers/kubernetes?
Maybe apps that need to be more tightly integrated with kube would benefit from this?
If you have sidecar containers that feed your regular containers, or you need to test a Dask KubeCluster, or deploy Helm charts, this kind of lets you work with the real lifecycle. Tilt is kinda better at watching for code changes in all your containers than docker-compose too, and has a nice UI to watch logs and see what succeeded or failed.
Tilt can also help validate things like:
- Service discovery
- Volume mounts
- Ingress and Certificates
- Metrics scraping and configuration
- Dashboards
It’s really quite powerful and replaces the need to mock things out with docker compose. If you’re deploying to Kubernetes, Tilt gives you the option to avoid “development-only” setups like docker compose.
- Service discovery
- Volume mounts
- Ingress and Certificates
- Metrics scraping and configuration
- Dashboards
It’s really quite powerful and replaces the need to mock things out with docker compose. If you’re deploying to Kubernetes, Tilt gives you the option to avoid “development-only” setups like docker compose.
I think if you ever have highly dynamic infrastructure requirements -- think along the lines of a control plane that's spinning up additional workers -- it's really helpful to be able to run your infra provisioning logic locally. There's nothing worse than having to wait on cloud builds to test your iterations.
What would be the difference between eg Tilt and Pulumi for this?
I never used tilt, but it looks very useful for anything that needs kube API to work, like some operator or something that needs to discover configuration from config maps.
otherwise I think it's meant for systems where system that you need for testing is to big to work on your local machine.
otherwise I think it's meant for systems where system that you need for testing is to big to work on your local machine.
Yes, Tilt really shines when you’re testing interactions with Kubernetes, such a APIs. But also things like your services’ ingress configuration and metrics scraping.
By default, Tilt is actually intended for local development using kind, minikube or other similar tooling. It supports developing against a multi-node cluster but it requires extra configuration and slows down iteration time.
By default, Tilt is actually intended for local development using kind, minikube or other similar tooling. It supports developing against a multi-node cluster but it requires extra configuration and slows down iteration time.
How do you make everything reproducible on the dev machines then? And manage versions of all the software?
The value is papering over previous over-engineering. No need to reflect if we might’ve effed up - and we get to keep all the toys!
I hadn't heard of Tilt until this year. I inherited an environment with it integrated. It's a great reliable tool
Been using tilt as a make alternative for years. Great tooling, even as just file watch + pythonic syntax for running tests, etc.
Obvs the real magic is the live syncing patches into remote containers though
Obvs the real magic is the live syncing patches into remote containers though
Love tilt, have found it much more difficult to set up in a docker-compose environment than a cloud native / minikube context
I did a blog post for Linkerd showing some of the benefits of using Tilt https://linkerd.io/2024/12/02/tilt-linkerd-nginx-part-1/.
TL;DR you can run some of your infra in local-dev that provide parity with your production environment.
TL;DR you can run some of your infra in local-dev that provide parity with your production environment.
I'm using Pulumi for this right now (with a dev setup and a prod setup), what would be the benefit for me to use Tilt over this?
Tilt is useful for local-dev where you are going to be modifying code / k8s configs and want live reload when you make changes. That is almost the entire appeal.
Why do not use well organized Makefile instead of invent new junk. Is it a product placement?
Makefiles are bad with state that is not represented as files, right? I remember that I had to create "stamp" files with hashes and things like that to overcome this limitation.
There is also .PHONY, but that would make the rule to always be triggered. Maybe I'm misremembering, it's been a long time :)
There is also .PHONY, but that would make the rule to always be triggered. Maybe I'm misremembering, it's been a long time :)
Makefiles are for building artifacts. They are wholly inadequate at orchestrating environments and task pipelines.
Can you do it? Sure, but somewhere someone is going to be suffering.
Can you do it? Sure, but somewhere someone is going to be suffering.
Better yet, use a well organized "rm -rf" command!
Refactoring advice:
"The cheapest, fastest, and most reliable components are those that aren't there."
— Gordon Bell
I love how Tilt enables creating a local development environment that lets my services run the same in production , test and development. Greatly simplifies my service code and improved my quality.
In particular, I’d love to see Tilt be better around handling things like CRDs (there’s no way to mark a k8s_yaml as depending on a CRD being available, a frequent source of broken tilt up invocations).
Having said that, the first thing I do, when working on any new project, is to get “tilt up” working.
Things I’ve used for testing include: eBPF-based collectors for security and observability, data pipelines, helm chart development, and Kubernetes controllers. It’s very flexible and powerful for a wide range of development.