There's never been an easier time to write your own language (2016)(joshsharp.com.au)
joshsharp.com.au
There's never been an easier time to write your own language (2016)
https://joshsharp.com.au/blog/never-been-an-easier-time-to-write-a-language.html
48 comments
A compiler is an error reporting tool with a code generation side-gig.
I'm not sure that's true in reality - the code generation is what you ship in practice. You can't skip on that feature.
I am being flippant, but what the user of the compiler uses it for the vast majority of the time is to get feedback on whether they've missed something. The vast majority of the code in the Rust compiler is dedicated to error reporting. Efficiency of the generated code is important, but it is a well explored space. Emitting good diagnostics is hard and unless you make it a priority it will always be subpar. Communicating with the user to help them get to that generated binary is at least as important, and in my eyes it is more important.
I also consider parsing and syntax to be the least interesting part of any language, as it is also a solved problem that requires little effort (modulo malformed code recovery) in both design and implementation when put in contrast with the rest of the compiler's functions and language design space.
I also consider parsing and syntax to be the least interesting part of any language, as it is also a solved problem that requires little effort (modulo malformed code recovery) in both design and implementation when put in contrast with the rest of the compiler's functions and language design space.
I've started down the road of language design recently and have committed to roughly these goals:
1. Making "Algol 2020" is only interesting if you also throw person-decades of effort into making it competitive in production. I want to do other things with my time, therefore anything I attempt should not lead to that, which produces these other requirements to create a small yet useful language:
2. The implementation must leverage existing languages in a way that is uncomplicated to the user. Which means that it either compiles to some form of interpreter or to generated source code.
3. The language must focus on fully leveraging a specific data structure or family of data structures. What is and has long been fashionable in PL discussion is to elaborate upon symbolic expression. You do need some symbol definition to have a language, but the preferred orientation we use for many data structures is spatial: "top of stack", "bottom of tree", "traverse the graph", "loop over the array". Engaging with the vocabulary of the data in its context, and simply working to generalize upon that vocabulary and the bookkeeping it needs(iteration counters, selection markers, error cases etc.), rather than the generalities of algorithm definition, leads directly towards a tighter language. We can define many algorithms very well, and we're paying more attention to concurrency lately, but the software we're writing still mostly isn't about algorithms themselves. You don't end up writing one million lines of code because you have a mega-algorithm that's just really hard to express.
Consider regular expressions: a little string matching language, which can be usefully explained in a page or two. The idea of them has been around since the 50's, yet all the hip, popular languages today have implemented some syntax for them, making regex defacto one of the most common and long-lived programming languages in existence, outdoing "big" languages by many measures.
And ideally I'd like to engage in those terms: a language so small you don't really notice except to think "gee, that's handy."
1. Making "Algol 2020" is only interesting if you also throw person-decades of effort into making it competitive in production. I want to do other things with my time, therefore anything I attempt should not lead to that, which produces these other requirements to create a small yet useful language:
2. The implementation must leverage existing languages in a way that is uncomplicated to the user. Which means that it either compiles to some form of interpreter or to generated source code.
3. The language must focus on fully leveraging a specific data structure or family of data structures. What is and has long been fashionable in PL discussion is to elaborate upon symbolic expression. You do need some symbol definition to have a language, but the preferred orientation we use for many data structures is spatial: "top of stack", "bottom of tree", "traverse the graph", "loop over the array". Engaging with the vocabulary of the data in its context, and simply working to generalize upon that vocabulary and the bookkeeping it needs(iteration counters, selection markers, error cases etc.), rather than the generalities of algorithm definition, leads directly towards a tighter language. We can define many algorithms very well, and we're paying more attention to concurrency lately, but the software we're writing still mostly isn't about algorithms themselves. You don't end up writing one million lines of code because you have a mega-algorithm that's just really hard to express.
Consider regular expressions: a little string matching language, which can be usefully explained in a page or two. The idea of them has been around since the 50's, yet all the hip, popular languages today have implemented some syntax for them, making regex defacto one of the most common and long-lived programming languages in existence, outdoing "big" languages by many measures.
And ideally I'd like to engage in those terms: a language so small you don't really notice except to think "gee, that's handy."
Writing a competitive concurrent garbage collector is probably the hardest part these days.
Well, you pretty much can't. A lone hobbyist simply isn't going to throw together a state-of-the-art garbage collector, any more than they're going to outperform LLVM's optimisations and code-generation. The choice is between leveraging existing GCs, such as by compiling to Java bytecode, or making do with an inferior garbage collector (or the choice of several inferior garbage collectors), the way D/Nim/OCaml do.
I on the other hand thought of this:
https://www.explainxkcd.com/wiki/index.php/2309:_X
(I don't try to troll the discussion)(it's just that XKCD is where many my mind runs to on some trigger words)
https://www.explainxkcd.com/wiki/index.php/2309:_X
(I don't try to troll the discussion)(it's just that XKCD is where many my mind runs to on some trigger words)
Just because you can, doesn't mean you should. (On the other hand, just because you can't doesn't mean you shouldn't, and neither does just because you can. They're orthogonal things basically.)
I'm all for writing one's own language. However, anyone who does this should be sentenced to writing a significant program in this language afterwards. Only then can they discover how wonderful or (more likely) horrible their creation actually is. Writing Towers of Hanoi in it doesn't count.
Also, if this language isn't just going to be a toy, they need to write some sort of specification document about it. I'm tired of seeing new language websites announcing version 0.9 of the new language, accompanied by a statement that “we haven't yet written a language manual, but here are some example programs and some obsolete papers about prior versions”.
I used to work in a senior position at a multinational firm. At one point, I went to the site of a company that had been recently purchased by my employers. Their major product was a large online system that was substantially written in a custom language. At that point, since the principals in the original company had left, there was not one employee left who understood this language completely.
Also, if this language isn't just going to be a toy, they need to write some sort of specification document about it. I'm tired of seeing new language websites announcing version 0.9 of the new language, accompanied by a statement that “we haven't yet written a language manual, but here are some example programs and some obsolete papers about prior versions”.
I used to work in a senior position at a multinational firm. At one point, I went to the site of a company that had been recently purchased by my employers. Their major product was a large online system that was substantially written in a custom language. At that point, since the principals in the original company had left, there was not one employee left who understood this language completely.
+1 for LLVM. It's hard to grok at first, but the Kaleidoscope ([0]) tutorial lets you hit the ground running. After that, it's very rewarding to learn LLVM's API by googling, poking around their excellent source code, and having clang emit IR for you (`clang -S -emit-llvm -O0 test.c`)
[0]: https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/index...
[0]: https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/index...
The article takes the approach of gluing together disparate solutions to write a language. With a functional language as your base, writing a new language can be quite elegant and fun! A great series by Stephen Diehl http://dev.stephendiehl.com/fun/ covers the subject in good detail and the fvg Lang https://github.com/Lemmih/fvg is a nice reference.
> The article takes the approach of gluing together disparate solutions to write a language.
But so does Diehl! It's no different!
He pulls together a 'disparate' parser library (actually he implements it himself, but it's a copy of an existing library) and he mentions using literally exactly the same backend (LLVM)!
What do you think the difference is that using a functional language makes?
It looks like exactly the same process to me.
But so does Diehl! It's no different!
He pulls together a 'disparate' parser library (actually he implements it himself, but it's a copy of an existing library) and he mentions using literally exactly the same backend (LLVM)!
What do you think the difference is that using a functional language makes?
It looks like exactly the same process to me.
To be fair, they didn't say that using a functional language made it different from gluing together disparate solutions, just that it made it elegant and fun. Though admittedly, even that is at best subjective.
I am not sure I agree with the claim. What’s really easier with the proposed approach (using a
python port of lex and yacc and generating LLVM) than using yacc and lex (with all their warts) to generate C code for your own language was in 1980?
Yes, if your language doesn’t fit C, your implementation could be more efficient (potentially a lot), but getting there isn’t easier.
I think we got a lot of complexity and options since then that distract from “writing your own language”.
I think a claim that we can do better than lex and yacc nowadays (e.g. using Haskell) would be a much stronger argument for “it’s much easier now”
Yes, if your language doesn’t fit C, your implementation could be more efficient (potentially a lot), but getting there isn’t easier.
I think we got a lot of complexity and options since then that distract from “writing your own language”.
I think a claim that we can do better than lex and yacc nowadays (e.g. using Haskell) would be a much stronger argument for “it’s much easier now”
Our platform lets users create animations in minutes. We use natural language in our scripts.
Eg. Cowboy say "Hi there." Wait for 2 seconds. Cowboy run to Point P2 in 3 seconds.
It has been an eye opening and learning exercise. The advantage is even first time users can start using our language. Disadvantage natural language has such a wide variety of usage styles that it's a challenge.
https://toonclip.com
Eg. Cowboy say "Hi there." Wait for 2 seconds. Cowboy run to Point P2 in 3 seconds.
It has been an eye opening and learning exercise. The advantage is even first time users can start using our language. Disadvantage natural language has such a wide variety of usage styles that it's a challenge.
https://toonclip.com
It was always easy to write your own language. Hard was making that language be a mature one with plenty of libraries for mundane tasks covering all needs people want. And harder was to make said language to be adopted and become mainstream.
Another option that may be useful: transform a custom syntax to an existing language.
And while you are at it, why not try to write a parser to reverse the process? :)
And while you are at it, why not try to write a parser to reverse the process? :)
Use the Forth, Luke!
https://github.com/codr7/gfoo
https://github.com/codr7/gfoo
Don't forget IDE support (syntax highlighting and completion), debugging support, and last but not least marketing.
Marketing may or may not be in scope, depending on one's goals with a language. Some might use it for internal projects and not see a need for external uptake. Others might eventually want more attention but may prefer more organic (word of mouth) growth.
It seems to me that IDE support, at least with regards to syntax highlighting and completion, is also easier than ever. There are great examples in VS Code, Atom, Sublime, JetBrains, and so on.
But why would you do that?
Because every programmer should at least once. Writing a compiler/interpreter is the ultimate exercise for any CS student/professional. that and maybe a file system/database or toy OS or server, because it encompass most aspects of CS.
Anybody that can do this things can probably take on any programming job, because they require a fundamental understanding of how computers and programs work.
Furthermore, It also help when suggesting realistic features to other language developers, or write proposals, or assess languages themselves.
Anybody that can do this things can probably take on any programming job, because they require a fundamental understanding of how computers and programs work.
Furthermore, It also help when suggesting realistic features to other language developers, or write proposals, or assess languages themselves.
Why not?
Programming languages are not a solved problem (or parsing, or diagnostics, or...).
We are today, barely, getting practical solutions to how write safely, fast and ergonomic (for example, rust) and yet:
- Compiling performance is abysmal in most compiled langs (except pascal family)
- And errors...
- And debugging...
- And interactive compilers...
- And ... (thousands of other stuff)
Somebody must try to do this stuff, because if not, forever will be at mercy of C, C++, JS, the shell, unix, etc. Who wanna the cobol effect forever?
Programming languages are not a solved problem (or parsing, or diagnostics, or...).
We are today, barely, getting practical solutions to how write safely, fast and ergonomic (for example, rust) and yet:
- Compiling performance is abysmal in most compiled langs (except pascal family)
- And errors...
- And debugging...
- And interactive compilers...
- And ... (thousands of other stuff)
Somebody must try to do this stuff, because if not, forever will be at mercy of C, C++, JS, the shell, unix, etc. Who wanna the cobol effect forever?
Why is Pascal better at compiling performance?
Nicklaus Wirth cared about not wasting machine resources.
He also wrote “a plea for lean software”:
> “Abstract: Software's girth has surpassed its functionality, largely because hardware advances make this possible. The way to streamline software lies in disciplined methodologies and a return to the essentials. The paper discusses some causes of "fat software" and considers the Oberon system whose primary goal was to show that software can be developed with a fraction of the memory capacity and processor power usually required, without sacrificing flexibility, functionality, or user convenience.”
He also wrote “a plea for lean software”:
> “Abstract: Software's girth has surpassed its functionality, largely because hardware advances make this possible. The way to streamline software lies in disciplined methodologies and a return to the essentials. The paper discusses some causes of "fat software" and considers the Oberon system whose primary goal was to show that software can be developed with a fraction of the memory capacity and processor power usually required, without sacrificing flexibility, functionality, or user convenience.”
Is made to be fast. Also, C/C++/rust is made to be slow.
One recent article that touch it (for rust):
https://pingcap.com/blog/rust-compilation-model-calamity
One recent article that touch it (for rust):
https://pingcap.com/blog/rust-compilation-model-calamity
It seems like one problem with rust is that it is a single threaded compiler.
We’ve reached the ends of fast CPUs. To gain performance improvements, we need to use multiple cores, and process it in parallel.
I wonder if this is an opportunity for github or Microsoft to host a cloud compiler? Just check in your code to github, and the cloud compiler will compile it across thousands of servers, all running in parallel.
Cost: $100/month per engineer.
We’ve reached the ends of fast CPUs. To gain performance improvements, we need to use multiple cores, and process it in parallel.
I wonder if this is an opportunity for github or Microsoft to host a cloud compiler? Just check in your code to github, and the cloud compiler will compile it across thousands of servers, all running in parallel.
Cost: $100/month per engineer.
> It seems like one problem with rust is that it is a single threaded compiler.
That could be a symptom but is not the root cause.Is just that the syntax/features choices (as choiced!) are hostile to fast performance.
Even going parallel, I bet a pascal compiler will beat it overly, because you can give rockets to turtles but are still turtles...
That could be a symptom but is not the root cause.Is just that the syntax/features choices (as choiced!) are hostile to fast performance.
Even going parallel, I bet a pascal compiler will beat it overly, because you can give rockets to turtles but are still turtles...
Many reasons! Languages are tools for solving entire spaces of problems. There are languages for music synthesis, database relations, templating , mathematical proofs, etc - but the list is by no means exhaustive and new languages are always needed, and I bet there are wonderful languages in our future that we didn’t know we needed.
Because why not? It can be a fun an very insightful process, as you'll learn about and have to apply techniques you may not have heard of otherwise.
If nobody made their own language, we wouldn't have any languages, and you'd have to write in machine code.
Yeah, basically a fun exercise in parsing/compiling/optimization.
OTOH what makes a language usable is literally everything else - package manager, IDE support, compile/parsing.
OTOH what makes a language usable is literally everything else - package manager, IDE support, compile/parsing.
> Yeah, basically a fun exercise in parsing/compiling/optimization.
Among other things, sure.
> OTOH what makes a language usable is literally everything else - package manager, IDE support, compile/parsing.
You have parsing/compiling in both categories. It's easier than ever to add IDE support for your homemade language, too, because of things like the LSP. Package management isn’t easy, but if you are building a specialized DSL and bundling the parts key to the domain, or if you are exposing a bridge to an existing ecosystem with strong library availability and decent package management, a language-specific package manager is probably not super important for usability.
Among other things, sure.
> OTOH what makes a language usable is literally everything else - package manager, IDE support, compile/parsing.
You have parsing/compiling in both categories. It's easier than ever to add IDE support for your homemade language, too, because of things like the LSP. Package management isn’t easy, but if you are building a specialized DSL and bundling the parts key to the domain, or if you are exposing a bridge to an existing ecosystem with strong library availability and decent package management, a language-specific package manager is probably not super important for usability.
> OTOH what makes a language usable is literally everything else
But that could be because we haven't had any real innovation in languages for ages, so except for some superficialities and details, the languages are essentially the same.
But that could be because we haven't had any real innovation in languages for ages, so except for some superficialities and details, the languages are essentially the same.
It's a great learning exercise.
like we don't have enough programming languages already?
Please don't
This applied to languages.
https://xkcd.com/927/
https://xkcd.com/927/
Creating a programming language is nothing like building a new standard to replace all the others.
If anything, it's like writing a new novel. Why would anyone do that, when there are so many already written?
If anything, it's like writing a new novel. Why would anyone do that, when there are so many already written?
Can you answer that question? There are more novels than a person can read in a lifetime.
More crime novels alone than a person can read in a lifetime.
Must be approaching that for any given category or niche soon, if not already.
Are they named “novels” because they are novel? If so a novel experience is a distance from where you currently are, and can plausibly be had backwards in time by reading an old book as well as forwards in time reading a new book, or sideways reading a new genre.
More crime novels alone than a person can read in a lifetime.
Must be approaching that for any given category or niche soon, if not already.
Are they named “novels” because they are novel? If so a novel experience is a distance from where you currently are, and can plausibly be had backwards in time by reading an old book as well as forwards in time reading a new book, or sideways reading a new genre.
New novels are not written because the writer thinks people have the need to read that particular story, but because the writer has the need to tell that particular story.
We don’t need yet more languages - it’s like the fall of the Tower of Babel now.
And citing parsing isn't a great example. Parser generators have been around for ages. And they're usually not the hard part anyways. Defining a simple grammar and parsing it, even manually, isn't that terrible of a task. Getting decent error messages and figuring out recovery? That's trickier.
Code generation has certainly gotten easier. But you still need to go through the process of figuring out how to lower your abstractions. My language is still extremely basic but I've still had to map my high level types and control structures down to WebAssembly. LLVM won't do that for you.
There's also more that your average user expects if you want a language that people use. Decent tooling is important, so a language server and some syntax highlighting packages in different editors. Good error messages. Decent type inference. Most of these you can eschew in the first few iterations of your language but eventually you'll need them.
I feel bad criticizing this post because writing a language has been one of the most instructive experiences I've had. I've learned so much about code generation, typechecking, the WASM spec, etc. But it's still a lot of tough work to get to something people can use. I'm not sure parser generators and LLVM make it that much easier.