Starlark Language(github.com)
github.com
Starlark Language
https://github.com/bazelbuild/starlark
50 comments
Lots of positive comments here that are mostly correct, but I will say that if you're working with more junior or just less technically passionate people, Starlark is a lot for them to wrap their heads around. "Python, but not", no Starlark.exe (always hosted in different exes that provide different functionality), poor editor highlighting/intellisense, and the concept of writing configuration in the same language which interprets that configuration are all hurdles that I've found some have difficulty with. DSLs encoded in yaml and JSON have a lot of the same problems, but I still get a lot of feedback that Starlark feels more complicated.
I agree with the difficulty of Starlark. I worked with Starlark a couple years ago to build some non-trivial things, which included writing custom Starlark providers. It was a gigantic pain. Documentation always felt very incomplete and finding existing examples were hard to come by (that has improved somewhat since I worked on that project, I believe).
Something I didn't appreciate before working in a very large tech company was just how much of a difference a good configuration language makes. In my previous company, configuration was a mix of YAML, JSON, and hardcoded values in Python. While this worked fine, deficiencies in YAML/JSON meant that we often had additional tooling around those files to generate more config.
At Google, languages like Starlark are used to build entire software stacks that run at configuration-time. Configuration with a base language, libraries, domain-specific languages, team-specific utilities, conventions, linters, and the whole thing gets built into a package that can be run by some much more general server.
That level of complexity is unnecessary in many places, but the one thing I would take on elsewhere is to pick a good, capable, config language early, and then use it for as much config as possible. Starlark is a good option if you like Python-ish stuff, Jsonnet is also pretty nice.
At Google, languages like Starlark are used to build entire software stacks that run at configuration-time. Configuration with a base language, libraries, domain-specific languages, team-specific utilities, conventions, linters, and the whole thing gets built into a package that can be run by some much more general server.
That level of complexity is unnecessary in many places, but the one thing I would take on elsewhere is to pick a good, capable, config language early, and then use it for as much config as possible. Starlark is a good option if you like Python-ish stuff, Jsonnet is also pretty nice.
Starlark is nice but Google also has GCL...
Google is a mess of configuration languages. Starlark, GCL, piccolo, textproto, yaml, the weird subset of starlark boq uses. From the outside it seems a lot more uniform than it is.
It's a mess, yes, but it's better than most places I've seen that end up using a bunch of YAML/JSON and then poorly defined custom tools to mangle things together.
Also I'd argue that there's a good core in Google's config stack: everything resolving to a proto at the bottom, textproto for "raw" representations, and having one language above that. If we could standardise on that it would be good, but of course that's unlikely.
Does Boq use a subset of Starlark? I thought it was Piccolo. I've always assumed Piccolo predated Starlark and that Starlark was essentially the external-first rewrite with some learnings.
Also I'd argue that there's a good core in Google's config stack: everything resolving to a proto at the bottom, textproto for "raw" representations, and having one language above that. If we could standardise on that it would be good, but of course that's unlikely.
Does Boq use a subset of Starlark? I thought it was Piccolo. I've always assumed Piccolo predated Starlark and that Starlark was essentially the external-first rewrite with some learnings.
From what I remember, Piccolo uses a Python interpreter.
Blaze also used to call a Python interpreter to evaluate BUILD files, but this is something I've changed. Starlark is unrelated to Piccolo, it started as an interpreter written inside Blaze. Starlark was created just before we open-sourced Bazel (because I didn't want to open-source the Python preprocessing).
Edit: indeed, Boq's manifest.bzl is Starlark.
Blaze also used to call a Python interpreter to evaluate BUILD files, but this is something I've changed. Starlark is unrelated to Piccolo, it started as an interpreter written inside Blaze. Starlark was created just before we open-sourced Bazel (because I didn't want to open-source the Python preprocessing).
Edit: indeed, Boq's manifest.bzl is Starlark.
Boq has a lot of restrictions on what you can do (ex. No user functions except a tiny allow list they own). It lives in a manifest.bzl file though, and I'm pretty sure uses the golang starlark interpreter.
> (because I didn't want to open-source the Python preprocessing).
Can you explain why?
I'm not trying to criticize anybody here, I'm just curious about the reasoning behind a decision to open source one but not the other.
Can you explain why?
I'm not trying to criticize anybody here, I'm just curious about the reasoning behind a decision to open source one but not the other.
It was a complete mess, hard to maintain the code, with scalability and performance problems. The semantics were very error-prone because it combined two interpreters (Python interpreter, optional, as a preprocessing step) followed by the interpreter in Bazel.
Bazel started a Python process for each BUILD file to preprocess. If two BUILD files were using the same bzl file, that bzl file would be parsed/evaluated twice. In a graph with lots of dependencies, it was causing a lot of redundant work. It's a bit like #include vs modules in C++: Starlark can evaluate a dependency file once, and provide the result for each BUILD file.
Python can be hard to sandbox, and we got multiple security reports about exploits.
Interestingly, Facebook Buck went through the same way. They originally copied the Python preprocessing approach. When we open-sourced Bazel, the Buck team took our Starlark (aka Skylark) interpreter, and started the migration. See https://buck.build/concept/skylark.html
Bazel started a Python process for each BUILD file to preprocess. If two BUILD files were using the same bzl file, that bzl file would be parsed/evaluated twice. In a graph with lots of dependencies, it was causing a lot of redundant work. It's a bit like #include vs modules in C++: Starlark can evaluate a dependency file once, and provide the result for each BUILD file.
Python can be hard to sandbox, and we got multiple security reports about exploits.
Interestingly, Facebook Buck went through the same way. They originally copied the Python preprocessing approach. When we open-sourced Bazel, the Buck team took our Starlark (aka Skylark) interpreter, and started the migration. See https://buck.build/concept/skylark.html
Google is trying to standardize on a few configuration languages: GCL (because it was too hard to migrate away from it), Starlark (when BCL is not suitable) and Text proto (for things that don't need logic).
It used to be a complete mess, but a plan was written a couple of years ago. So there's hope of improvement.
It used to be a complete mess, but a plan was written a couple of years ago. So there's hope of improvement.
That's good to hear, but sad that I (as a SWE/SRE) haven't heard of the plan! Maybe I've been living under a rock, but knowing that I should be building new projects with these and not, say, Piccolo, would be nice. Updating codelabs to reflect this where there are multiple options would also be nice, IIRC the CDP codelab still uses Piccolo, but could probably use GCL fairly easily?
Interesting to see that GCL still lives on.
Does it have a debugger now?
I worked there a decade ago and we were trying out the "gcl2" reimplementation that was based on an actual formal specification of the languages. The semantics were subtly different and we couldn't switch our complex config to it quite yet back then. I wonder if the experiment succeeded or you're back on to the implementation-defined language semantics
Does it have a debugger now?
I worked there a decade ago and we were trying out the "gcl2" reimplementation that was based on an actual formal specification of the languages. The semantics were subtly different and we couldn't switch our complex config to it quite yet back then. I wonder if the experiment succeeded or you're back on to the implementation-defined language semantics
Plus bcl, ncl, OWNERS, piccolo, SDL, and probably dozens more
Both Starlark and CUE are projects that evolved from former Googlers learnings about BCL and GCL
aiui, Starlark is closer to the internal languages while CUE takes a different philosophical approach (logical language that ensures correctness at scale)
aiui, Starlark is closer to the internal languages while CUE takes a different philosophical approach (logical language that ensures correctness at scale)
Do people actually like jsonnet and that style of embedding a programming language inside a data format? Reminds me of the days of XML programming - it's just awful to work with and barely comprehensible to beginners.
It's much less of an embedding than you think. I think of it more as borrowing enough JS syntax that JSON is valid jsonnet. Apart from that though it's own lazy functional language.
Starlark is awesome. We studied the language quite a lot, Alan Donovan did awesome work on its golang runtime [1] and there are more implementations in rust [2] and others.
Since the language and the backend allow to write deterministic functions, we found it's really suited well for automations written as durable functions. We implemented exactly that [3] over Temporal [4], though we also support Python as well among other runtimes.
[1] https://github.com/google/starlark-go
[2] https://github.com/facebook/starlark-rust
[3] https://autokitteh.com, https://github.com/autokitteh/autokitteh
[4] https://temporal.io
Since the language and the backend allow to write deterministic functions, we found it's really suited well for automations written as durable functions. We implemented exactly that [3] over Temporal [4], though we also support Python as well among other runtimes.
[1] https://github.com/google/starlark-go
[2] https://github.com/facebook/starlark-rust
[3] https://autokitteh.com, https://github.com/autokitteh/autokitteh
[4] https://temporal.io
Having used Starlark somewhat extensively with Buck2 I'm like 32% sure it is a mistake. Vanilla Python or ideally C# should be used instead.
The biggest issue is I want first class debugger support. Language support is fiendishly complex. It's impossible to reason about without a step debugger. printf debugging is insufficient. I should be able to debug as trivially as "attach to process" and F9'ing a breakpoint.
I'm not convinced the restrictions imposed by Starlark actually matter. There's so many compiler toolchains that can introduce non-hermetic behavior that I don't think much is gained from locking down the scripting language. We're professionals here, it's ok to say "don't do that".
Buck2 is pretty rad overall. I kinda wish it was the industry standard. But that will probably never happen because adding support for new languages is so complex it requires a non-trivial team of FAANG engineers to support full-time.
I tried and failed to add Jai support to Buck2. The Starlark rules are too complex and too impossible to understand. Dynamic types are a nightmare to decipher. Compiling programs isn't actually that difficult! Figuring out how to induce the machinery to run the commands I need turned out to be insurmountable. :( Google and Meta are both hamstrung by trying to carry forward a decade+ of old build definitions to new systems. There's significant baggage.
The biggest issue is I want first class debugger support. Language support is fiendishly complex. It's impossible to reason about without a step debugger. printf debugging is insufficient. I should be able to debug as trivially as "attach to process" and F9'ing a breakpoint.
I'm not convinced the restrictions imposed by Starlark actually matter. There's so many compiler toolchains that can introduce non-hermetic behavior that I don't think much is gained from locking down the scripting language. We're professionals here, it's ok to say "don't do that".
Buck2 is pretty rad overall. I kinda wish it was the industry standard. But that will probably never happen because adding support for new languages is so complex it requires a non-trivial team of FAANG engineers to support full-time.
I tried and failed to add Jai support to Buck2. The Starlark rules are too complex and too impossible to understand. Dynamic types are a nightmare to decipher. Compiling programs isn't actually that difficult! Figuring out how to induce the machinery to run the commands I need turned out to be insurmountable. :( Google and Meta are both hamstrung by trying to carry forward a decade+ of old build definitions to new systems. There's significant baggage.
I think it really depends on your use case, or for a build system, the size of your repo. The fact that starlark can efficiently parallelize is a big deal for using bazel in a big monrepo. On the other hand, it's lack of interactive debugging, a repl, and ban on side effects/io make a rather large learning curve for a language that aspires to be similar to Python
Meta also uses (nearly) this for scripting to the search index. Both at index and query time. I think it's a good for that
I just started a new position as a build engineer to help migrate a huge codebase to Bazel at a big automotive company. As there are a lot of custom proprietary tools I’ll probably write quite a bit of Starlark. So far it looks nice, let’s see how my experience will be in a couple of months.
Starlark is one of those things that reminds you about how trends in computing are totally circular. In the early days, people strung together configuration through imperative shell scripts and Lua scripts and the like. Then there was this massive push toward declarative configuration DSLs and configuration-as-code and we got Terraform and a whole mess of YAML in every "cloud-native" open source project. And now we're shifting back to writing configuration imperatively as regular programs with Starlark and similar languages. No doubt another 5 years from now we'll just switch everything back to declarative once programmers get bored of the current trend.
This is preposterous take. Bazel is a hermetic build. Depending on the dependency graph of a change we can build, and deploy 2 monoliths and a suite of microservices in under 5 minutes. Sometimes less than 1 minute. Developers also get access to a shared build cache so local builds are just as fast. This isnt a fad. It solves problems you havent encountered yet.
There's a law that as any discussion on hn grows, the odds that someone chimes in about some new thing being just a rehash of an old thing approaches 1.
Yeah, it's ridiculous how people criticize these things, they solve very relevant problems to massive scale software. Working in a monorepo is also especially efficient if the tooling supports it.
It's still not perfect, and it's very complicated, but people always act like there's no benefit.
It's still not perfect, and it's very complicated, but people always act like there's no benefit.
I never said anything about Bazel, build systems, or build speed. I have no idea what you're trying to refute, perhaps you replied to the wrong comment.
[deleted]
Nonsense. Starlark is hardly a general-purpose programming language. Its whole point is to combine the advantages of something like shell or lua (familiar programming-language-like syntax, easy to reuse common values) with the advantages of something like YAML (safe, inert). That's exactly what real progress looks like.
Yeah, it's far better than a mess of templated yaml...
Technically with Starlark nothing is stopping you from going mostly declarative with an option to add some imperative customization - see Bazel
Fun fact: execution is not only deterministic, but guaranteed to terminate eventually. I.e. no indefinite loops or recursion is allowed. Quite useful for a configuration language. Of course, that means it's not Turing-complete.
It's nice if a language has a static check to prevent infinite loops when they're unnecessary, but this guarantee is less useful in practice if you're going to run the code.
A loop over a billion items (for example, due to a bad SQL statement) can hang your system indefinitely, just like an infinite loop. You still need timeouts, or a progress bar and a way to cancel. If running the code costs money, this is especially important.
A static termination guarantee is most useful in a proof language, when you're not going to run the code, just prove things by compiling it. That's when you really need it. You assume the function returns a value, and if there's any way it could fail, the proof is wrong. But performance is irrelevant if you're not going to run it.
For a configuration language, banning recursion might still make sense, though, since it's another way to write confusing code.
A loop over a billion items (for example, due to a bad SQL statement) can hang your system indefinitely, just like an infinite loop. You still need timeouts, or a progress bar and a way to cancel. If running the code costs money, this is especially important.
A static termination guarantee is most useful in a proof language, when you're not going to run the code, just prove things by compiling it. That's when you really need it. You assume the function returns a value, and if there's any way it could fail, the proof is wrong. But performance is irrelevant if you're not going to run it.
For a configuration language, banning recursion might still make sense, though, since it's another way to write confusing code.
It actually is, because it allows functions as first class objects. That means you can use the fixed-point combinators from lambda calculus (which doesn't have direct recursion either). For instance, the Omega combinator (sort-of):
(this is the starlark-rust repl, which is the only one I have currently installed)
You could easily implement the Y combinator like this, and boom, Turing completeness.
In practice, it doesn't matter, because the way they've been implemented is with limited enough stack space that it's not really relevant. But arguably, that is an implementation detail, and you could implement Starlark with either TCO or growable stacks or whatever. Just like normal Turing complete languages doesn't have infinite memory, Starlark doesn't have infinite stack space. I would strongly argue that the language itself is very much Turing complete.
$> def f(g): g(g)
$> f(f)
Traceback (most recent call last):
* expression:1, in <module>
f(f)
* expression:1, in f
def f(g): g(g)
* expression:1, in f
def f(g): g(g)
* expression:1, in f
def f(g): g(g)
<many stack frames omitted>
* expression:1, in f
def f(g): g(g)
error: Starlark call stack overflow
--> expression:1:11
|
1 | def f(g): g(g)
| ^^^^
|
(this is the starlark-rust repl, which is the only one I have currently installed)
You could easily implement the Y combinator like this, and boom, Turing completeness.
In practice, it doesn't matter, because the way they've been implemented is with limited enough stack space that it's not really relevant. But arguably, that is an implementation detail, and you could implement Starlark with either TCO or growable stacks or whatever. Just like normal Turing complete languages doesn't have infinite memory, Starlark doesn't have infinite stack space. I would strongly argue that the language itself is very much Turing complete.
The Java and the Go implementation of Starlark should detect the recursive call in the example. I'd say it's an implementation bug in the Rust interpreter.
(I agree it doesn't matter that much in practice)
(I agree it doesn't matter that much in practice)
Ah, fair enough. I've only tested the Rust version.
Yeah, important to note, in practice it is always finite, since if you try to do anything "real" using this trick, you blow the stack immediately. This was mostly just a fun fact about how easy it is for Turing-completeness to sneak in. I seem to remember reading something about how Algol-60 was similar, they intended it not to allow recursive procedures, but permitted function pointers (or something) and you could do a similar thing.
Yeah, important to note, in practice it is always finite, since if you try to do anything "real" using this trick, you blow the stack immediately. This was mostly just a fun fact about how easy it is for Turing-completeness to sneak in. I seem to remember reading something about how Algol-60 was similar, they intended it not to allow recursive procedures, but permitted function pointers (or something) and you could do a similar thing.
the go implementation, at least, allows you to enable while loops among others, which makes it turing complete and not guaranteed to complete. obviously you should never turn this on for configuration, but it is useful for general scripting usage.
We used Starlark for all of its properties and built a tool[0] that allows you to spin up containers in an imperative programming way on Docker and Kubernetes through Starlark!
It can be pretty cool as you can pass in arguments and get a different set of services based on what you pass in. I like to think of it is an "exe" for a distributed system.
[0]https://github.com/kurtosis-tech/kurtosis
It can be pretty cool as you can pass in arguments and get a different set of services based on what you pass in. I like to think of it is an "exe" for a distributed system.
[0]https://github.com/kurtosis-tech/kurtosis
Here’s one of my previous attempts at it specifically for k8s: https://github.com/cruise-automation/isopod
I’ve had mixed results trying to adopt this though - some people (usually better coders) get it, some like wallowing in their yaml soup
I’ve had mixed results trying to adopt this though - some people (usually better coders) get it, some like wallowing in their yaml soup
I've been working through a legacy starlark codebase recently. My take is that the language, and the bazel phased execution model, makes it very challenging to debug.
One example: as far as I can tell there is no way to get the Starlark call stack, this is a tool you really want for printf debugging (which is your main debugging tool with Starlark), I can't find it right now but I believe I saw them say that they don't want to give access to the call stack for performance reasons which I find odd - is the bottleneck in slow bazel builds close to be starlark execution time?
Also looking at a non trivial Starlark codebase that didn't mature well, I can't help but wonder if it wasn't better to not give engineers as much flexibility for their build configuration, yes, they'll have some copy pasted configuration snippets that could look tighter with a macro, but you are less likely 5 years later to look at a monster that is slow and hard to debug.
One example: as far as I can tell there is no way to get the Starlark call stack, this is a tool you really want for printf debugging (which is your main debugging tool with Starlark), I can't find it right now but I believe I saw them say that they don't want to give access to the call stack for performance reasons which I find odd - is the bottleneck in slow bazel builds close to be starlark execution time?
Also looking at a non trivial Starlark codebase that didn't mature well, I can't help but wonder if it wasn't better to not give engineers as much flexibility for their build configuration, yes, they'll have some copy pasted configuration snippets that could look tighter with a macro, but you are less likely 5 years later to look at a monster that is slow and hard to debug.
I (Bazel SWE at Google) agree with the too much flexibility being a problem. It's difficult for tools to work on complex Starlark code, so we encourage more copy and paste especially in BUILD files. But I don't think Blaze could have scaled to Google's needs without some amount of programmability. It's not just that it "looks tighter with a macro", when you have a rule that gets used to create millions of targets, you can't just copy and paste. You need to be able to systematically change it.
Not trying to bash, honestly curious; why not just use python?
There's some explicit differences with Python[1]. My understanding is that Starlark was specifically created for Bazel so if I had to guess it's to enforce the immutability of values between contexts.
[1] https://bazel.build/rules/language#differences_with_python
[1] https://bazel.build/rules/language#differences_with_python
The early versions (before it was called Starlark) was indeed just run with a Python interpreter. But in order to have a deterministic and reproducible build, they needed to put some limits on what you can do, and especially what you can accidentally do. Starlark's raison d'etre is to allow a build graph that you can reason about and have a shared cache for.
Embedding the Starlark interpreter into a Rust program took me less than an hour. There's little more to it than adding the crate and calling into it. No futzing with the build process.
If starlark does everything you need (and especially if its limitations are desirable for your use case) then it's the clear choice in my view.
If starlark does everything you need (and especially if its limitations are desirable for your use case) then it's the clear choice in my view.
starlark is easily embeddable, minimal, dependency-free language and python is none of those things
An Overview of the Starlark language - https://news.ycombinator.com/item?id=40573689