Terra: Low-level system programming language meta-programmed from Lua(terralang.org)
terralang.org
Terra: Low-level system programming language meta-programmed from Lua
https://terralang.org/
32 comments
In case anyone is still reading this, I'm here now. Still happy to answer any questions.
I'll also add that we're always looking for new contributors. We're an 100% community run project, so everything is driven by people who care about things stepping in to do them. If you're interested in compiler technology, there's a wide variety of things to work on.
I'll also add that we're always looking for new contributors. We're an 100% community run project, so everything is driven by people who care about things stepping in to do them. If you're interested in compiler technology, there's a wide variety of things to work on.
I'm curious if it a new syntax was ever considered. It seems as though if you dropped Lua compatibility you might be able to make it a bit cleaner, although it's a big undertaking.
We've actually gotten some mileage out of being able to swap one Lua implementation for another. (To date, I think we've tried LuaJIT, Moonjit, and PUC Lua.) E.g., we still rely on Moonjit for PPC64le support, even though it's technically unmaintained.
I've definitely got a long list of gripes with Lua... but they're not really enough to take on the additional maintenance burden.
I've definitely got a long list of gripes with Lua... but they're not really enough to take on the additional maintenance burden.
I've been following this for a while and think it is a gorgeous idea. Particularly nice is the fact that one can still use C and C++ libraries from lua, then dip out into Terra for things that require more speed that don't currently have a library.
Lua tying it all together means you have a tiny, dynamically typed language that can be embedded in lots of popular C based software, which can now be accelerated to (even more) incredible speed. I predict big things and wish them well.
Lua tying it all together means you have a tiny, dynamically typed language that can be embedded in lots of popular C based software, which can now be accelerated to (even more) incredible speed. I predict big things and wish them well.
I remember reading something particularly negative about terra in 2019[1]. Although I haven’t used terra at all since the time I first tried it (not long after I read the above mentioned article), I think it’s relevant to mention that at least at one point, terra has had major problems regarding design. If anyone has used terra in any non-trivial capacity I’d love to hear what your experience with it has been like.
[1] https://erikmcclure.com/blog/a-rant-on-terra/
[1] https://erikmcclure.com/blog/a-rant-on-terra/
Please note: The entire "windows" portion of that article has largely been resolved at this point, because I fixed it myself. Some of the more egregious documentation problems have been fixed, but others remain unresolved.
Unfortunately, we have since discovered even more insane behavior: https://github.com/terralang/terra/issues/460
Unfortunately, we have since discovered even more insane behavior: https://github.com/terralang/terra/issues/460
Thanks for pointing that issue out! I had a fun time figuring out the codegen algorithm by putting printf statements in every function I thought was relevant.
I used terra at least 6 years ago to create a JIT compiler for database queries.
The language itself is performant and pleasant, but there are not enough people using it, so troubleshooting is really painful.
The language itself is performant and pleasant, but there are not enough people using it, so troubleshooting is really painful.
I develop Regent, a programming language for distributed computing based on Terra [1]. We're one of the larger (largest?) Terra codebases and indirect source of users.
(I also help maintain Terra, but this answer is more focused on my capacity as a user.)
Terra is a workhorse and it gets the job done. Overall, compared to a previous iteration of Regent (that was built entirely from the ground up in Python), it's been a good experience. Having first-class code generation capabilities has been really nice, especially with built-in support for things like parsing C header files, vectorization, and CUDA code generation. The language is stable and generally doesn't have many surprises. Unlike, say, C++ or Rust, there aren't as any corners to hide odd behaviors. (Obviously, there still is some room for such things, as you can see in the sibling comments. But I still think it's better than most languages.)
The main issue is, as siblings have noted, the size of the community. While most stuff works, if you find something broken you're likely going to have to fix it yourself. This is an area where we're always looking to grow.
[1]: http://regent-lang.org/
(I also help maintain Terra, but this answer is more focused on my capacity as a user.)
Terra is a workhorse and it gets the job done. Overall, compared to a previous iteration of Regent (that was built entirely from the ground up in Python), it's been a good experience. Having first-class code generation capabilities has been really nice, especially with built-in support for things like parsing C header files, vectorization, and CUDA code generation. The language is stable and generally doesn't have many surprises. Unlike, say, C++ or Rust, there aren't as any corners to hide odd behaviors. (Obviously, there still is some room for such things, as you can see in the sibling comments. But I still think it's better than most languages.)
The main issue is, as siblings have noted, the size of the community. While most stuff works, if you find something broken you're likely going to have to fix it yourself. This is an area where we're always looking to grow.
[1]: http://regent-lang.org/
Also runs in GPU via CUDA. That's it's well known claim to fame.
Appears to be going 1.0[1]!
[1] https://github.com/terralang/terra/commit/1ea278278c104580a0...
Appears to be going 1.0[1]!
[1] https://github.com/terralang/terra/commit/1ea278278c104580a0...
We're working on 1.0. The language is basically stable but needs a little polish. As usual everything depends on what time people have to put in. Always looking for more help!
A Singly Linked List implementation in Terra (just testing out):
https://pastebin.com/xt3Wddtu
Output:
https://pastebin.com/KwSTiN8L
https://pastebin.com/xt3Wddtu
Output:
https://pastebin.com/KwSTiN8L
I wonder how well it handles or rejects syntax-level expansions. E.g. can the outer language produce half an expression for the next level? Do I need to worry about the common pitfalls of macros?
I hope this works out. C++ has been deliberately evolving in the same direction, e.g. using `if constexpr` for conditional compilation. Plus, since the C++ compiler is aware of both levels of the language, it can catch some errors even for compilation paths not taken.
But the C++ syntax for metaprogramming still leaves a fair bit to be desired. So I like that folks are still experimenting in this area.
I hope this works out. C++ has been deliberately evolving in the same direction, e.g. using `if constexpr` for conditional compilation. Plus, since the C++ compiler is aware of both levels of the language, it can catch some errors even for compilation paths not taken.
But the C++ syntax for metaprogramming still leaves a fair bit to be desired. So I like that folks are still experimenting in this area.
Terra macros are hygenic in the sense that they follow the Lua scoping rules. So if you do:
Instead you'd have to write:
You should be able to create towers of languages in Terra. I'm not sure if anyone really does more than 2 levels in practice, but most of the common Terra DSLs are basically written as code generators on top of Terra language expressions.
See e.g. the Regent code generator. Here's an example of some nested metaprogramming---not with a nested language or macros, but with an escape that calls a Lua function to generate more code based on the type of the argument:
https://github.com/StanfordLegion/legion/blob/0cf9ddd60c227c...
local add1 = macro(function()
return `x+1
end)
terra()
var x = 12
add1()
end
Instead you'd have to write:
local add1 = macro(function(y)
return `y+1
end)
terra()
var x = 12
add1(x)
end
You should be able to create towers of languages in Terra. I'm not sure if anyone really does more than 2 levels in practice, but most of the common Terra DSLs are basically written as code generators on top of Terra language expressions.
See e.g. the Regent code generator. Here's an example of some nested metaprogramming---not with a nested language or macros, but with an escape that calls a Lua function to generate more code based on the type of the argument:
https://github.com/StanfordLegion/legion/blob/0cf9ddd60c227c...
Am I understanding this right?
Code inside a `terra` function is statically-typed and executed at runtime, while the rest of the code is just plain Lua and is executed at compiled time.
It's a really interesting idea and I'd be interested to try it for something like an X11 application.
Code inside a `terra` function is statically-typed and executed at runtime, while the rest of the code is just plain Lua and is executed at compiled time.
It's a really interesting idea and I'd be interested to try it for something like an X11 application.
I think it is more like this: the Lua code emits assembly code or LLVM intermediate representation or C source code, then arranges for the assembly code to be assembled (then possibly arranges for the code to be executed, then terminates).
Many years ago, someone wrote a Haskell library along the same lines that lets the library's user mimic assembly language with Haskell code. One thing I particularly liked about this library is that (by deviating a little from the standard way of indenting Haskell code) the user could mimic the 2 columns of assembly language. E.g., in
Many years ago, someone wrote a Haskell library along the same lines that lets the library's user mimic assembly language with Haskell code. One thing I particularly liked about this library is that (by deviating a little from the standard way of indenting Haskell code) the user could mimic the 2 columns of assembly language. E.g., in
foo <- statement
statement2
bar <- statement3
You can interleave Terra and Lua execution. Terra functions are JITted if you call them from Lua. You can call Lua from Terra too, though the language makes you jump through some hoops to encourage code to avoid Lua dependencies in Terra code. So "execution" is more like you'd expect from standard Lua (or Python or whatever), it's just that functions can be compiled if you want.
Of course Terra functions can also be saved out to a file too, and if you do that they're free of any dependency on Lua (i.e., no need to like liblua*.so).
Of course Terra functions can also be saved out to a file too, and if you do that they're free of any dependency on Lua (i.e., no need to like liblua*.so).
Honest question. Why not just use C/subset of C++ with a template language like Python+Jinja2 for meta programming?
Code generation is already popular with popular C++ libraries like protobuf and in game dev.
Code generation is already popular with popular C++ libraries like protobuf and in game dev.
When you use a template language, how do you JIT the code so you can run it in the same process?
How do you parse C/C++ header files so you can introspect their types or functions and take actions based on the results?
How do you write clean abstractions for code generation? Do you build your own C++ AST abstraction entirely from scratch? If not what do you use, and how much power does it expose (and what tradeoffs come with the level of power that you get)?
Terra lets you do all of these things in a clean way with well-defined, predictable semantics that don't leave you scratching your head when you get to 4 nested levels of macros and complex dynamic behavior that depends on program inputs.
How do you parse C/C++ header files so you can introspect their types or functions and take actions based on the results?
How do you write clean abstractions for code generation? Do you build your own C++ AST abstraction entirely from scratch? If not what do you use, and how much power does it expose (and what tradeoffs come with the level of power that you get)?
Terra lets you do all of these things in a clean way with well-defined, predictable semantics that don't leave you scratching your head when you get to 4 nested levels of macros and complex dynamic behavior that depends on program inputs.
What is the advantage here of doing the metaprogramming in a different language from the one you're manipulating?
Different languages help because the different parts of the program expose different needs.
You want the generated code to be low-level, because you want it to be fast, free of dependencies (interpreter, JIT, GC, etc.), and you care less about abstractions. You want the metaprogram to be high-level because you want nice abstractions, and it runs at compile-time anyway so you care less about performance.
Sure, people are working on languages that can do both, but it's easier to make it work with two different languages that can be tuned specifically to their niches.
You want the generated code to be low-level, because you want it to be fast, free of dependencies (interpreter, JIT, GC, etc.), and you care less about abstractions. You want the metaprogram to be high-level because you want nice abstractions, and it runs at compile-time anyway so you care less about performance.
Sure, people are working on languages that can do both, but it's easier to make it work with two different languages that can be tuned specifically to their niches.
It's like Rust's procedural macros, but more flexible because they're interpreted, right?
Well Rust procedural macros can run arbitrary code, so that's not such a big difference. I'd say the larger one is having access to actual AST data structures available if you want them. Rust forces you to parse the token stream yourself and serialize it again. Also, Terra allows user defined keywords so it's a lot easier to create nice looking DSLs (which is my personal use case).
You can think of it as clang, but controlled by a lua engine instead of the CLI. It's not just macros, you have control over generation of build artifacts. By default you're running in JIT mode, but you can at any time bundle up some procedures and spit out an object file or link them into an executable.
You can also call back into Lua from the native code, and use it to generate more native code. You get back some of the cool self-modifying executable techniques of the days gone by, without having to completely YOLO it.
You can also call back into Lua from the native code, and use it to generate more native code. You get back some of the cool self-modifying executable techniques of the days gone by, without having to completely YOLO it.
Is there anything similar for Python?
I help maintain Terra, and also use it to build Regent, my distributed programming language[1].
A bit busy at the moment, but if you leave questions here I'll try to answer them (possibly in some hours).
[1]: regent-lang.org