Don't Write a Programming Language(write.as)
write.as
Don't Write a Programming Language
https://write.as/loke/dont-write-a-programming-language
133 comments
With respect to your first point, I'm reminded of Peter Alvaro's comment "the only good reason to design a programming language is to shape your understanding of the problem". He points out that he was the only user of his first PL, but he had no regrets of about that, because it shaped all his later thought in his research.
From http://lambda-the-ultimate.org/node/5569 - Three Things I Wish I Knew When I Started Designing Languages. Alvaro's talk led to an HN discussion at https://news.ycombinator.com/item?id=19428433
From http://lambda-the-ultimate.org/node/5569 - Three Things I Wish I Knew When I Started Designing Languages. Alvaro's talk led to an HN discussion at https://news.ycombinator.com/item?id=19428433
The lambda the ultimate link doesn't work anymore; is that site still alive??
It's slow, but there. It's just a link to this though in case you can't see it: https://www.infoq.com/presentations/language-design-process/
Here it is on YouTube https://youtu.be/oa0qq75i9oc
> Third, it may turn into something new. If some people don't write a new programming language, then we are stuck with what we have. This advice basically admits that the status quo is good enough... The authors saying that the language, as a project, is a lifetime appointment? Well, this reveals everything. I believe if you want to do a programming language, then you must be willing to invest at least a decade or two.
Sounds like someone should persuade some big-name university to hand out honorary PhDs to people who build new languages that produce demonstrable value (even if the value is not business value).
Seems to be the criteria for a PhD, anyway.
Sounds like someone should persuade some big-name university to hand out honorary PhDs to people who build new languages that produce demonstrable value (even if the value is not business value).
Seems to be the criteria for a PhD, anyway.
To solve a problem about 23 years ago I wrote a programming language + interpreter. It was meant to be a simple language to write phone surveys in, so basically a kind of DSL for a single purpose. The programmers who built the survey taking system that would use my language discovered they could write much of it in the language itself, which surprised me a lot. Was a really fun project. No idea what happened to it.
The problem with writing a _general-purpose_ programming language (as a solo-developer), is the fact that writing the language (i.e defining the syntax+semantics, writing the parser/compiler) is only half the battle - the rest of your time is spent on less technically interesting QoL features, like an IDE, standard library, debugger, syntax highlighting etc. which are nonetheless vital for making the programming language actually usable. (I still think writing your own PL is a useful exercise just to see how the sausage is made, but it should be made clear that there is a world of difference between a toy language and one that people would actually want to use)
It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free, leaving you more time to work on designing the language to suit your needs. If you combine macros with clever uses of GADTs you can even get the host language to "type-check" programs written in your DSL. OCaml and Lisp/Racket are my favourite host languages to use for this approach.
It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free, leaving you more time to work on designing the language to suit your needs. If you combine macros with clever uses of GADTs you can even get the host language to "type-check" programs written in your DSL. OCaml and Lisp/Racket are my favourite host languages to use for this approach.
We are in a golden age for new languages.
You have https://microsoft.github.io/language-server-protocol/ for doing the integration between IDE and your parser.
Also, https://highlightjs.org/ which lets you broadcast your code in a pretty way.
I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.
You have https://microsoft.github.io/language-server-protocol/ for doing the integration between IDE and your parser.
Also, https://highlightjs.org/ which lets you broadcast your code in a pretty way.
I do however feel that your QoL will depend on whether or not you make a compiler or a translator. The key benefit of a translator is that you can co-opt the host language's standard runtime and debugger.
Agree with all of this. Don't do any yak-shaving you don't want to do just for the fun of it; build your language and its tooling on existing systems so you can spend your time on what makes it special, instead of what doesn't.
> It's for this reason that nowadays I rarely write my own "new" programming languages from scratch, but rather embed them as DSLs inside existing programming languages using macros. This way you get the full ecosystem of the host language for free
I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like.
What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.
I don't understand what you mean by this, so I'm hoping you can clarify. What's the advantage of doing this over the alternative way: writing a parser which would compile to code in the host language, or some primitives that you've arranged in the host language that would represent the bytecode of your language interpreter. From there you can leverage whatever in the host language you want, including its type system, packages, language semantics, FFI access to other languages. Really whatever you like.
What you're suggesting is leveraging a macro system to help you write what is essentially a parser/compiler, which is a good win. But I don't see what that buys you past that point. If I do things the other way, by writing a parser that consumes a *.my_lang file and compiles it to primitives or actual source code in the host language, I can still leverage all the benefits of the host language just as you describe doing with a macro system. The advantage of doing things this way is you're not limited to languages with tolerable metaprogramming capabilities.
Ecosystem of packages and language interop (e.g. C bindings) are also criminally underrated.
Exactly, the days that a plain syntax+semantics, with parser/compiler were enough to win the hearts of the market are long gone, decades ago.
It is still a fun exercise, though.
It is still a fun exercise, though.
Yeah it's pretty bad these days. If you tell someone you are writing a PL, they expect:
- a complete parser/compiler and debugger toolchain
- a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc.
- a package manager with a large, secure, and vibrant package ecosystem
- a community with live chat support and a strong Stack Overflow presence
- robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected.
- That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project.
- your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility.
- you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community"
- support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware.
- support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now.
- And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades.
It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
- a complete parser/compiler and debugger toolchain
- a complete language server implementation, and language modes for all popular IDEs including VSCode, VIM, Emacs, etc.
- a package manager with a large, secure, and vibrant package ecosystem
- a community with live chat support and a strong Stack Overflow presence
- robust documentation with copious examples, tutorials, guides, and technical info. These days even a multi-hundred page book with professional editing available for free is expected.
- That you will be responsive to bugs, provide patches promptly, and review/merge community feature requests asap. They will get very mad with you if you don't, and likely write a blog post about how much of a tyrant you are over your project.
- your language is expected to have undergone a full independent security audit, and to be up-to-date on the latest security issues. It's expected to be stable with complete backwards compatibility.
- you have to manage the community and deal with petty interpersonal conflicts or else word gets out that your language has a "toxic community"
- support for all major operating systems and hardware architectures including Mac, Windows, and all distributions of Linux, as well as esoteric hardware.
- support for the web through wasm, so you need experts at that as well as x86 and ARM platforms now.
- And on top of all that, your project needs to be completely open source, with a free and open license that permits royalty free, patent free, commercial use. Oh, and your users both corporate and personal expect your work to be completely free as in beer. They won't pay anything for it. Not even a dollar. And if you ask them to pay something they don't weigh the value of your product versus the competition, they balk immediately and don't even consider it. It's been a non-starter to ask for money for a PL for decades.
It used to be the case that you could write down a spec on paper and call it a PL. Back in the 60s you could call that a "programming language" without even writing an implementation of the parser. Today a programming language is a massive undertaking and the saddest part of it all is that it's been commoditized. I've even seen a sentiment out there that programming languages are done, that we've invented all we need to. All future languages are unnecessary, and all current languages are largely the same. Could you imagine? So as hard as programming languages are, DO write them. That's my message. Just do so responsibly and with clear eyes and expectations.
At very least, because the baton eventually needs to be passed on to younger generations.
Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.
Regarding language innovation, I will oversimplify and assert that due the resistance to modernization (for whatever reasons), we have spent the last decades making Algol 68, Lisp and ML appealing to mainstream, smuggling their capabilities into more acceptable packaging.
This is so strangely sad. Go ahead and write a programming language. Have a weird hobby you clank away on that might not see the light of day. If you burn out on it, that's one thing and that's okay; you've exhausted the medium. Move on to something else you enjoy. Grow for the sake of growing.
Just because something hasn't become big doesn't mean it's a failure. I've labored in dark early hours over paintings and music that will never see the light of day, but it's a vital part of how I express and cultivate myself. My journal.
99% of art is unknown personal projects. Embrace yours. Why be bitter?
Just because something hasn't become big doesn't mean it's a failure. I've labored in dark early hours over paintings and music that will never see the light of day, but it's a vital part of how I express and cultivate myself. My journal.
99% of art is unknown personal projects. Embrace yours. Why be bitter?
For me it's a question of what my goals are. If I have three equally fascinating projects I could spend my free time on, which one advances me towards my goals.
Thank you so much for these words. I have „wasted“ around 5 years designing my own and was recently bitter about that „wasted“ time. Only now I started shifting my perspective that this was a thrilling hobby that I really liked and it wasnt wasted at all. Your words were final nail in that coffin of bitterness. Thank you, kind internet stranger, you changed my life for the better
Back in the 80's when I started my career creating a PL was a right of passage. You got bonus cred if it were compiled and not interpreted! Most people weren't writing general-purpose languages and the languages were very domain-focused - but they were languages nonetheless. It's been kinda sad to see that die. So I agree with you - go write that programming language if that floats your boat and just have fun doing it for the sake of doing it.
FWIW: It didn't come across as bitter to me. More like a general warning from a hardened battle hero.
A warning that many naive young recruits ought to give credence to, before it's too late (when the cost has been sunk, and you're knee deep in it).
A warning that many naive young recruits ought to give credence to, before it's too late (when the cost has been sunk, and you're knee deep in it).
Perhaps this is good advice, but perhaps even better advice is: Don't write a Turing-complete DSL to solve FizzBuzz at a job interview.
(Warning - comic gold, get a tissue) https://aphyr.com/posts/353-rewriting-the-technical-intervie...
(Warning - comic gold, get a tissue) https://aphyr.com/posts/353-rewriting-the-technical-intervie...
Absolutely love Aphyr's interview stories. Here is my all time favourite https://aphyr.com/posts/342-typing-the-technical-interview
There is a sequel to "rewriting the interview" as well. I rather enjoy them as well.
For anyone who doesn't recognize the story by the description, click the link and read it now.
And then read the rest of the stories on the site.
They're all both extremely technically interesting and very good short fiction.
And then read the rest of the stories on the site.
They're all both extremely technically interesting and very good short fiction.
what, they got recommended a strong hire and also received a useful word of advice!
Is that supposed to be legible to a lisper?
I don't know. It looks like Clojure to me.
I have also been working on a programming language, but longer than the author of this piece (since 2011), and the author is correct.
Designing a language consumes you; you never see programming the same way again. Whenever I write C (I'm implementing the interpreter in C), I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
And the design never stops. As mentioned, I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
Designing a language consumes you; you never see programming the same way again. Whenever I write C (I'm implementing the interpreter in C), I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
And the design never stops. As mentioned, I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
> I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language
That is exactly what motivated me to start thinking about my own language and what made me want to keep working on it in the early to mid 1990s. Luckily, I started learning several languages so that I'd have a better idea of the design space. Someone suggested I take a look at a relatively unknown language called Python because it fit how I was thinking about things. He was right and that was the anecdote for me - finding a language that fit how I thought about programming well enough to eliminate enough of the annoyances that were my motivation.
The lesson... If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together. Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly. If you get lucky, you might find a language that stops your descent into the rabbit hole.
The above might only work if you can accept that languages are sufficiently complex that they all have real problems, even yours when it's "done".
That is exactly what motivated me to start thinking about my own language and what made me want to keep working on it in the early to mid 1990s. Luckily, I started learning several languages so that I'd have a better idea of the design space. Someone suggested I take a look at a relatively unknown language called Python because it fit how I was thinking about things. He was right and that was the anecdote for me - finding a language that fit how I thought about programming well enough to eliminate enough of the annoyances that were my motivation.
The lesson... If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together. Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly. If you get lucky, you might find a language that stops your descent into the rabbit hole.
The above might only work if you can accept that languages are sufficiently complex that they all have real problems, even yours when it's "done".
> If you're thinking about designing a language, study as many other languages as you can so that you get a better sense for the design choices involved and how various concepts fit together.
I've written C, C++, Python, Ruby, Lua, Bash, POSIX sh, C#, Java, JavaScript, Common Lisp, Racket, Haskell, bc, dc, Zig, BASIC, Tcl, and some others I'm probably forgetting, and I've studied Rust, D, Ada, Idris, Coq, Isabelle/HOL, HAL/S, OpenGL shaders, Cilk, Cyclone, Ceylon, Scala, Clojure, Kotlin, F#, Scheme Lisp, Jai, Julia, Perl, Nim, and probably even more that I am forgetting.
Does all of that count? :)
> Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly.
Yes, I agree. And I have done all of that.
> If you get lucky, you might find a language that stops your descent into the rabbit hole.
"Lucky" is the operative word. Unfortunately, I'm picky and haven't been that lucky.
I've written C, C++, Python, Ruby, Lua, Bash, POSIX sh, C#, Java, JavaScript, Common Lisp, Racket, Haskell, bc, dc, Zig, BASIC, Tcl, and some others I'm probably forgetting, and I've studied Rust, D, Ada, Idris, Coq, Isabelle/HOL, HAL/S, OpenGL shaders, Cilk, Cyclone, Ceylon, Scala, Clojure, Kotlin, F#, Scheme Lisp, Jai, Julia, Perl, Nim, and probably even more that I am forgetting.
Does all of that count? :)
> Ask people experienced with those languages what they like / dislike about them. Write something challenging enough to get a first hand understanding of some of those pros and cons. If you push on with your own language, this will help your efforts significantly.
Yes, I agree. And I have done all of that.
> If you get lucky, you might find a language that stops your descent into the rabbit hole.
"Lucky" is the operative word. Unfortunately, I'm picky and haven't been that lucky.
It is interesting to see such an extensive list that doesn't include Smalltalk or Forth. The lack of Pascal/Modula-2/Oberon is not a problem given Ada. Not having APL (or J or K) might not hurt if you get the needed ideas from Julia. Other languages such as Cobol (or the microcomputer equivalent such as dBase) most people can probably live without.
Ah, whoops. Smalltalk and Forth are two languages that I have written a bit of code in and studied extensively. I did say that I was probably missing some.
Oh, and Go. I've written some Go.
Oh, and Go. I've written some Go.
> don't write a programming language unless you are prepared to be marked and scarred for life
I can vouch for that! Fortunately, I have a great team to collaborate with.
I can vouch for that! Fortunately, I have a great team to collaborate with.
I was looking for your comment in this thread and I'm not disappointed.
It's fortunate you went ahead despite every thing telling you to do otherwise. And then tricked other people into joining you. A bit evil probably but they consented, in the end. You are probably only partly responsible.
It's fortunate you went ahead despite every thing telling you to do otherwise. And then tricked other people into joining you. A bit evil probably but they consented, in the end. You are probably only partly responsible.
D doesn't need a Reality Distortion Field. People that take the time to learn it find it very pleasant and productive.
It does not require that much time either.
[deleted]
Article author here. The two years mentioned in the post is the time spent actually implementing it. I spent a few more years thinking about it prior, and I thought that implementing the basic ideas would prove, our not, whether it worked, and then I'd be done.
Well, guess what, it didn't actually turn out that way. At least I'm more understanding of people who annoyingly promote their own languages all the time.
Well, guess what, it didn't actually turn out that way. At least I'm more understanding of people who annoyingly promote their own languages all the time.
> So the author is correct: don't write a programming language unless you are prepared to be marked and scarred for life.
I write and wrote internal languages mostly; a bunch of them made it to production inside larger (for my country) companies for years and even a decade. The scarring part is that I always add the features I want and when I go to a mainstream language, I always miss those features. So since 2018 we have been doing a new language and one of them will be publicly launched this or next year so I can at least use that for day to day programming without being annoyed by missing features. Note that I already have that using Racket with a bunch of macros but in my business it is easier to introduce your own language than use Racket...
I write and wrote internal languages mostly; a bunch of them made it to production inside larger (for my country) companies for years and even a decade. The scarring part is that I always add the features I want and when I go to a mainstream language, I always miss those features. So since 2018 we have been doing a new language and one of them will be publicly launched this or next year so I can at least use that for day to day programming without being annoyed by missing features. Note that I already have that using Racket with a bunch of macros but in my business it is easier to introduce your own language than use Racket...
> Note that I already have that using Racket with a bunch of macros but in my business it is easier to introduce your own language than use Racket...
Why is it easier?
Why is it easier?
Most people are allergic for lisp likes. Not sure why but it has since always been blocked. I even did a lispy but with <> instead 20 years ago and that was OK (it was the xml for everything hype) while with () it was rejected. I had a talk with the cto of one of our bigger infra structure partners and he had exactly the same issue, so he replaced the () with [] and suddenly it was fine. So he has had scheme in their products for over 10 years but with [] because people are allergic to ().
I can obviously do the same with racket but if no () then I rather have another syntax all together.
I can obviously do the same with racket but if no () then I rather have another syntax all together.
Ah, it's sad to hear that. People can be so emotional, just because of a bit of syntax.
But since you have full control, you could introduce some extra syntactic sugar as well and desugar it into regular lisp/racket syntax before parsing, no?
As an alternative with a completely different syntax, I found Scala to be extremely nice for building DSLs. Probably the best typesafe language for DSLs out there. It's only really good on the JVM though.
But since you have full control, you could introduce some extra syntactic sugar as well and desugar it into regular lisp/racket syntax before parsing, no?
As an alternative with a completely different syntax, I found Scala to be extremely nice for building DSLs. Probably the best typesafe language for DSLs out there. It's only really good on the JVM though.
Out of interest, what are the features you miss?
> I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language.
What's worse is that PL authors are usually students of PL history, and the history of PLs is littered with "We did things this way out of necessity. We wouldn't have done this if we had more time/resources." Which would be fine but then we are all stuck using those tools with all their arbitrary limitations and quirks for 50 years. If you ever thought "I could write something to make this so much easier", the impulse is 100x worse as a PL designer, because you realize how many things we do are done for no reason at all beyond history and happenstance.
What's worse is that PL authors are usually students of PL history, and the history of PLs is littered with "We did things this way out of necessity. We wouldn't have done this if we had more time/resources." Which would be fine but then we are all stuck using those tools with all their arbitrary limitations and quirks for 50 years. If you ever thought "I could write something to make this so much easier", the impulse is 100x worse as a PL designer, because you realize how many things we do are done for no reason at all beyond history and happenstance.
On the bright side, it probably better equips you to actually argue about where C (or any language) has faults from a language design perspective, which is a relatively rare point of view.
> I've been designing mine since 2011, and I'm only now writing an interpreter after two failed attempts to bootstrap a compiler.
You are lucky (or unlucky) that you have reached that stage. I'm trying since 2004 and had at least 6 failed attempts on design and 3 different implementations that never finished. Fortunately for me, since I had a lot of failed designs (as opposed to implementations) I better appreciate existing designs even with their warts.
You are lucky (or unlucky) that you have reached that stage. I'm trying since 2004 and had at least 6 failed attempts on design and 3 different implementations that never finished. Fortunately for me, since I had a lot of failed designs (as opposed to implementations) I better appreciate existing designs even with their warts.
> I get annoyed because I have to work around C's warts to do what I want, and that is only happening because I think in my language
You "just" have to get to the point where your compiler is written in your own language. This may take a while. Go got there in V1.5:
https://go.dev/doc/go1.5
You "just" have to get to the point where your compiler is written in your own language. This may take a while. Go got there in V1.5:
https://go.dev/doc/go1.5
It's funny/interesting that an 1kloc implementation of minimal self hosting Go exist, or C/pascal/oberon compilers ; perfect hindsight view etc. You theoretically could (well if nobody does, no you can't) create a nice language and its implementation quickly.
Wirth was even more drastic in what concerns writing bootstraped compilers.
I don't recall in what paper from him I have read this, so take the story with a grain of salt, also open to corrections.
He actually wrote the initial version of the compiler directly in Pascal.
How did he perform such thing, one would ask.
By writing it on paper using a Pascal subset good enough as stage 0, and then manually compiling the source code into corresponding Assembly instructions that he would then actually type into the cards.
So when he finally got the compiler done, it was already bootstrapped from the get go.
Additionally, P-Code originally wasn't designed to be an interpreter, rather to repeat the above process in an easier way across computer systems.
He was initially surprised that others took it to write Pascal interpreters instead of a bootstrapping tool.
I don't recall in what paper from him I have read this, so take the story with a grain of salt, also open to corrections.
He actually wrote the initial version of the compiler directly in Pascal.
How did he perform such thing, one would ask.
By writing it on paper using a Pascal subset good enough as stage 0, and then manually compiling the source code into corresponding Assembly instructions that he would then actually type into the cards.
So when he finally got the compiler done, it was already bootstrapped from the get go.
Additionally, P-Code originally wasn't designed to be an interpreter, rather to repeat the above process in an easier way across computer systems.
He was initially surprised that others took it to write Pascal interpreters instead of a bootstrapping tool.
I've never been megalomaniac enough to think I could design a new language. But I did once write a Pascal source-level debugger; I think the compiler was Intel Pascal (long time ago). It taught me a lot about how the compiler worked, and how the language worked.
I've used to suffer quite badly from Impostor Syndrome, and that project did a lot to alleviate my symptoms.
I never shared the project with anyone else; it was never completely finished.
I've used to suffer quite badly from Impostor Syndrome, and that project did a lot to alleviate my symptoms.
I never shared the project with anyone else; it was never completely finished.
Nice story, thanks for sharing.
My hard drive is full of unfinished projects, started to learn about programming language, or try out new design approaches, quickly abandoned after the initial goal was achieved.
Regarding imposter syndrome, two guidelines that have helped me in such situations, are "only repent for paths not taken", "failure is better that not knowing at all".
Obviously those quotes aren't from me.
My hard drive is full of unfinished projects, started to learn about programming language, or try out new design approaches, quickly abandoned after the initial goal was achieved.
Regarding imposter syndrome, two guidelines that have helped me in such situations, are "only repent for paths not taken", "failure is better that not knowing at all".
Obviously those quotes aren't from me.
> It's funny/interesting that an 1kloc implementation of minimal self hosting Go exist
Does it? That's not the technique Go introduced, they started by rebuilding Go from earlier versions written in C - but in fact the bootstrapping process may change soon, see https://github.com/golang/go/issues/44505
Does it? That's not the technique Go introduced, they started by rebuilding Go from earlier versions written in C - but in fact the bootstrapping process may change soon, see https://github.com/golang/go/issues/44505
Sure, go itself didnt build it from scratch - rewriting your implementation each time you change your language might grow tiresome fast, but I was referring that toy/minimal implementations exist : https://benhoyt.com/writings/mugo/
> be marked and scarred
If not, your language will be marked and swept.
If not, your language will be marked and swept.
A programming language is a tool. And like all tools, the situation is very different depending on if you are writing a personal tool that solves your problems or if you are trying to solve other's problems.
I used to lurk in a programming language design group. I joined it because I wanted to "solve everyone's problem" with a new language because I had some success with solving my own problems with a previous language (which is not for everyone; it's a Forth dialect).
For few year I saw some people theorizing and analyzing with little practical results. It's fascinating how people can know about "analysis paralysis" and still fall for it. Meanwhile, I lost interest in my own attempt and returned to my Forth dialect.
Today it powers a few programs that run 24/7. And it is also my goto scripting language. It's not available on Github/Gitlab/whateve because I'm too lazy to do something that will certainly have zero benefits for me ( however curiously, it is half-documented because writing docs helps with spotting silly decisions sometimes).
It is true that you are never done with that sort of program, there's always something to improve or fine tune. Also, since you are the sole user you cannot rely on a ecosystem and just "import their.stuff".
But I think the great mistake is really to start making a tool, a programming language, with no plans on what to use it for; or worse, expect others to use it. The Internet is indeed littered with born-dead or short-lived languages. They are solutions without a problem to solve.
I used to lurk in a programming language design group. I joined it because I wanted to "solve everyone's problem" with a new language because I had some success with solving my own problems with a previous language (which is not for everyone; it's a Forth dialect).
For few year I saw some people theorizing and analyzing with little practical results. It's fascinating how people can know about "analysis paralysis" and still fall for it. Meanwhile, I lost interest in my own attempt and returned to my Forth dialect.
Today it powers a few programs that run 24/7. And it is also my goto scripting language. It's not available on Github/Gitlab/whateve because I'm too lazy to do something that will certainly have zero benefits for me ( however curiously, it is half-documented because writing docs helps with spotting silly decisions sometimes).
It is true that you are never done with that sort of program, there's always something to improve or fine tune. Also, since you are the sole user you cannot rely on a ecosystem and just "import their.stuff".
But I think the great mistake is really to start making a tool, a programming language, with no plans on what to use it for; or worse, expect others to use it. The Internet is indeed littered with born-dead or short-lived languages. They are solutions without a problem to solve.
please clone it to github :) could be interesting for the curious!
It is true: it's a deep pit you'll enter. But the conclusion is wrong. If you feel it's fun -- then do write your own programming language. Even if you won't finish (which is likely). But you'll think about things that are a lot of fun, and you will have more clarity about what you do not like about existing languages, and why. And you'll learn that there are things you need to solve that you did not think about before. That some things cannot be combined, and that you will have to choose.
It is a very good experience, there is a lot to learn. As long as your goal is not to get it done quickly...
It is a very good experience, there is a lot to learn. As long as your goal is not to get it done quickly...
The key to writing a PL and surviving is to have an exit strategy. Figure out ahead of time how long you will spend on this project, otherwise it’s liable to last half a decade or more before your interest wanes.
PL design is like crack for programmers. It is a lot of fun, which is why you see hundreds of little languages posted here every month.
The danger is that some people get don’t have a clear idea. About what it takes to write a pl. They can get quick wins with a parser generator and maybe some easy semantics, which give a good taste of how fun PLs are but also belie the fact that the hardest parts are exponentially harder than that.
Anyone embarking on a PL needs to be clear eyed about a few things.
1. You’ll never make money off of it. No one sells compilers anymore, and there are like 2 big for-profit languages out there out of thousands. If you do make money, let us all know how.
2. The odds are your language will have exactly one user, yourself. Get comfortable with the possibility you will spend years on something and no one will really appreciate it.
3. The real work in creating a language is not technical but community building. Languages need people who speak/write it to thrive, so as much or more work needs to be put into the community effort for the language to succeed. If all you want to do is work on technical problems that’s fine, but technical prowess doesn’t make a language popular.
4. Get ready for criticism. Every programmer has their opinions about languages and they will be sure to tell you their opinion of your language. It will not be good. The only people who will appreciate your language are other language designers. Most everyone else will start by telling you how you’re doing things the wrong way, and they’ll end by asking why you’re eating your time with all this anyway. Get ready for that and steel yourself against it.
PL design is like crack for programmers. It is a lot of fun, which is why you see hundreds of little languages posted here every month.
The danger is that some people get don’t have a clear idea. About what it takes to write a pl. They can get quick wins with a parser generator and maybe some easy semantics, which give a good taste of how fun PLs are but also belie the fact that the hardest parts are exponentially harder than that.
Anyone embarking on a PL needs to be clear eyed about a few things.
1. You’ll never make money off of it. No one sells compilers anymore, and there are like 2 big for-profit languages out there out of thousands. If you do make money, let us all know how.
2. The odds are your language will have exactly one user, yourself. Get comfortable with the possibility you will spend years on something and no one will really appreciate it.
3. The real work in creating a language is not technical but community building. Languages need people who speak/write it to thrive, so as much or more work needs to be put into the community effort for the language to succeed. If all you want to do is work on technical problems that’s fine, but technical prowess doesn’t make a language popular.
4. Get ready for criticism. Every programmer has their opinions about languages and they will be sure to tell you their opinion of your language. It will not be good. The only people who will appreciate your language are other language designers. Most everyone else will start by telling you how you’re doing things the wrong way, and they’ll end by asking why you’re eating your time with all this anyway. Get ready for that and steel yourself against it.
What are the two that make money?
Matlab and Mathematica. There are probably more, and Microsoft makes a ton of money with .Net of course, but I'm thinking more along the lines of pure PL products and not something in a vast enterprise software ecosystem like Microsoft offers.
I started creating a language about fifteen years ago.
It started as a project to learn C: implement an interpreter in C that lets you interactively call C library functions and system calls.
But, little by little it become my own implementation of my own dialect of lisp and the language I use for almost everything.
My only claim for the language is that I like it, I like it a lot.
I guess I will release it at some point, to the indifference, no doubt, of the world, but that's not the point. If you design and build your ideal house, it doesn't matter if no one else builds the same design, or even appreciates it, as long as you love living there.
It started as a project to learn C: implement an interpreter in C that lets you interactively call C library functions and system calls.
But, little by little it become my own implementation of my own dialect of lisp and the language I use for almost everything.
My only claim for the language is that I like it, I like it a lot.
I guess I will release it at some point, to the indifference, no doubt, of the world, but that's not the point. If you design and build your ideal house, it doesn't matter if no one else builds the same design, or even appreciates it, as long as you love living there.
could you share it on github?
Eventually … still need to do the really difficult things like decide on a name and license, and write some documentation …
I disagree with the author's view that any project that isn't used by other people is a waste of time. Does that mean that all homework projects in college are a waste of time? Or that beginner programmers should skip writing simple programs and jump straight into writing a sellable product? Of course not. The point of those projects is to learn, not to make anything that will change the world. If a project is enjoyable to you and you learn something useful from it, then it is not a waste of time.
I guess it depends on what your goal is. If it is simply your own personal learning, then no, it cannot possibly be a waste of time. But if not…
The author mentions that writing a programming language entails writing your own GUI (which language doesn't need an IDE). Surely writing a plugin to an existing IDE is what most people would do in that situation, and I can't see many arguments to the contrary. This alone made me question the author's advice somewhat. If they are such a perfectionist as to care to have a custom IDE for their language perhaps a more reasonable person wouldn't get so deep down the rabbit hole
As someone whose spent the last 9 years writing a programming language, I would say this author is correct. Not necessarily about the GUI in particular, but about the larger point that there are any number of aspects in PL design which will consume an infinite amount of time. For them in was a gui, for others it’s a type system, compiler, package manager, language server, memory management, parallelism, or any number of other infinitely complex subtopics in the field.
I made a comment yesterday to this effect yesterday, so I’ll just leave that here: https://news.ycombinator.com/item?id=30484632
I made a comment yesterday to this effect yesterday, so I’ll just leave that here: https://news.ycombinator.com/item?id=30484632
I did design a language and it is still being used. I did it as part of my job at that time. It started as a simple query language inside the application we were developing. One by one features were added. Some colleague also decided to used it program certain functionality of the application, which caused more and more people to use it.
I implemented it with a parser that I developed myself. So, actually, I fell in the trap of developing a language myself, not a programming language, but a developing my own language developing tools. And I have doing it for more than 20 years now, on and off. I do recognize the self promoting bit. (I am kind of doing it right now.)
I implemented it with a parser that I developed myself. So, actually, I fell in the trap of developing a language myself, not a programming language, but a developing my own language developing tools. And I have doing it for more than 20 years now, on and off. I do recognize the self promoting bit. (I am kind of doing it right now.)
The language I am refering to, is now called the BiZZdesign scripting language. See: https://support.bizzdesign.com/display/knowledge/BiZZdesign+...
The pipe operators are from the original design of the query language. Variables, function, and many other language constructs have been added later.
I think that a big tenant of any modern language should be interoperability, such that you can easily import/export code from another popular language. That not only clears up the issue around libraries, but also makes it a safer bet for usage by people who otherwise wouldn’t be able to use a niche language
These all sound like pros... A project that makes you intimately familiar with a huge breadth of deep, foundational domains and that forces you to make opinions about esoteric concepts, all while pushing you to go deeper.
I'm really confused by it. It's almost like some strange species of humblebrag. "Look at my _____, isn't it horrible?"
I can't understand why the writer seems so bitter about spending so much time on something they clearly love... I mean, what else is there to do? Lots of boring obligatory stuff!
I can't understand why the writer seems so bitter about spending so much time on something they clearly love... I mean, what else is there to do? Lots of boring obligatory stuff!
I totally get where the author is coming from and it's definitely not a humblebrag. The best way I can describe it is that the author feels like the character Cypher from the Matrix, who regrets taking the red pill and waking up to reality, and wished he was warned ahead of time not to do it. That's the spirit in which the author writes this.
The character gained something, which is knowledge, but lost something, which is the ability to be satisfied ever again.
Maybe another good analogy is vampirism. Yes you gain amazing powers like the ability to live forever, shapeshifting, and super strength/healing, who could complain about that? But at what cost? You can't even go outside in the sun anymore or eat garlic knots, so maybe it's not that great.
The character gained something, which is knowledge, but lost something, which is the ability to be satisfied ever again.
Maybe another good analogy is vampirism. Yes you gain amazing powers like the ability to live forever, shapeshifting, and super strength/healing, who could complain about that? But at what cost? You can't even go outside in the sun anymore or eat garlic knots, so maybe it's not that great.
> another good analogy is vampirism
No, that's a great analogy. I can neither condone nor dispute it on rhetorical grounds, but aesthetically it is rad.
No, that's a great analogy. I can neither condone nor dispute it on rhetorical grounds, but aesthetically it is rad.
> You can't even go outside in the sun anymore
Perfectly describes the hero programmer stuck in the rabbit hole.
Perfectly describes the hero programmer stuck in the rabbit hole.
Haha it does, but I think the analogy falls apart at the garlic knots. Aint going to stop a programmer from eating those.
I felt the same. I was waiting for the punchline, the twist, or at least an "All that said it's totally worth it" at the end.
I find writing a library is usually enough for the types of problems I encounter. You can even design it to be language-like using fluent interfaces; take FluentAssertions for example: `actual.Should().StartWith(expected);`.
Beyond that, I tend to reach for:
- A config file. Can the problem be solved declaratively through a config file that a program reads and bases its behavior on?
- Templates, generators, and macros. Along the lines of syntax, would templates or generators solve problems with an existing GPPL?
- A domain-specific language. Does the problem lend itself to composable patterns or statements? Could a DSL help organize and simplify them?
- A transpiler. Is the syntax of an existing GPPL the problem? Would syntactic sugar be enough?
Of course, there are plenty of other reasons to write a new GPPL, like for your own learning or just as a hobby. Just know what you're getting yourself into.
Beyond that, I tend to reach for:
- A config file. Can the problem be solved declaratively through a config file that a program reads and bases its behavior on?
- Templates, generators, and macros. Along the lines of syntax, would templates or generators solve problems with an existing GPPL?
- A domain-specific language. Does the problem lend itself to composable patterns or statements? Could a DSL help organize and simplify them?
- A transpiler. Is the syntax of an existing GPPL the problem? Would syntactic sugar be enough?
Of course, there are plenty of other reasons to write a new GPPL, like for your own learning or just as a hobby. Just know what you're getting yourself into.
Configuration files are such a useful thing to start with - I remember one of my most productive changes was to hack on a local copy of mutt:
* If the configuration file had mode 0755 then execute it, and parse the output.
* Otherwise read as-is.
That was my solution to writing a complex configuration file, which behaved differently upon different hosts - write the thing in perl, and have it "print" blocks and configuration values appropriately.
* If the configuration file had mode 0755 then execute it, and parse the output.
* Otherwise read as-is.
That was my solution to writing a complex configuration file, which behaved differently upon different hosts - write the thing in perl, and have it "print" blocks and configuration values appropriately.
Ok but why not just have separate, simpler configuration files for each host?
Finally a bit of sanity!
The same goes for an OS or a game engine or any other large scope (Or collection of small scope) tool.
The other people who kinda care are those similarly engrossed in their own project, meaning... they do not actually have time to care.
My first priority with computing is to get rid of all the custom stuff. I won't even think about refactoring or any other improvement till I've weeded out all the handwritten reimplementations of library functions, made sure that there's no ad hoc string.split() based data formats that should be JSON, or UDP packets that should be MQTT.
Honestly, I don't even like that macros exist. I understand that in low level languages sometimes one needs metaprogramming, but it's more of an unfortunate reality than a cool feature.
The same goes for an OS or a game engine or any other large scope (Or collection of small scope) tool.
The other people who kinda care are those similarly engrossed in their own project, meaning... they do not actually have time to care.
My first priority with computing is to get rid of all the custom stuff. I won't even think about refactoring or any other improvement till I've weeded out all the handwritten reimplementations of library functions, made sure that there's no ad hoc string.split() based data formats that should be JSON, or UDP packets that should be MQTT.
Honestly, I don't even like that macros exist. I understand that in low level languages sometimes one needs metaprogramming, but it's more of an unfortunate reality than a cool feature.
What does "sanity" mean? Without the curiosity to drive us, we wouldn't have moved out of chilly caves.
Yes, like most initiatives, you are set for failure, if you define success by how popular a creation of yours becomes. You can still learn a lot.
And still, because "success" is such a rare breed, we actually need more people to do challenging stuff, so we'd get more success in absolute numbers.
The OP actually lacks sanity in recognizing that their project did not take off, and they don't have a clear idea of why they are doing their programming language project.
Basically, they are realising it's not a success, and they don't necessarily care about learning much out of it, and yet are still not letting go. If anything, that's a definition of insanity :)
Yes, like most initiatives, you are set for failure, if you define success by how popular a creation of yours becomes. You can still learn a lot.
And still, because "success" is such a rare breed, we actually need more people to do challenging stuff, so we'd get more success in absolute numbers.
The OP actually lacks sanity in recognizing that their project did not take off, and they don't have a clear idea of why they are doing their programming language project.
Basically, they are realising it's not a success, and they don't necessarily care about learning much out of it, and yet are still not letting go. If anything, that's a definition of insanity :)
It's more like addiction than insanity. They aren't doing the same thing and expecting different results exactly... They're doing the same thing just... because it's part of their identity now, even though they aren't all that happy.
Real insanity would be still thinking it was gonna be the next big hit.
I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
There are big wins, but they're incredibly rare and seem very random. And many of them are almost guaranteed to fail.
You'll learn a lot, but what can you actually use that knowledge for? Niche DIY stuff doesn't teach you much about mainstream methods.
The result is 1000 garage tinkerers for every Guido Van Rossum, and sometimes it's even actively detrimental to a project to try to cater to the tinkerers.
Real insanity would be still thinking it was gonna be the next big hit.
I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
There are big wins, but they're incredibly rare and seem very random. And many of them are almost guaranteed to fail.
You'll learn a lot, but what can you actually use that knowledge for? Niche DIY stuff doesn't teach you much about mainstream methods.
The result is 1000 garage tinkerers for every Guido Van Rossum, and sometimes it's even actively detrimental to a project to try to cater to the tinkerers.
The issue I have with your approach is twofold: the 1000 garage tinkerers are harmless, and your approach forgoes the Guido Van Rossums, Rich Hickeys, Chris Lattners and so on.
I've met a bunch of people who do things like write their own typed concatenative language. So I can tell you, first of all, that they tend to spend the week merrily contributing to, say, orchestrating Kube clusters, and second, the time they spend bit banging their pet programming language wasn't going to go towards more orchestrating Kube clusters. Or whatever it is you think is actually important.
I've met a bunch of people who do things like write their own typed concatenative language. So I can tell you, first of all, that they tend to spend the week merrily contributing to, say, orchestrating Kube clusters, and second, the time they spend bit banging their pet programming language wasn't going to go towards more orchestrating Kube clusters. Or whatever it is you think is actually important.
That's fine, but it seems like the target audience of the article is the people who hope to become the next Van Rossum.
It might even be aimed at people deciding whether their team at work should pursue a new language for some specific task.
Maybe this could be solved with a special badge that tells people which of the Three Tribes of Programmers you're writing for...
For purely toy projects, I'm not sure if any code blog advice at all is that helpful, except possibly "Make stuff really small if you want to impress other programmers".
Best Practices don't apply the same way to this kind of thing, since in a real industry project the best practice would probably be "Just don't".
It might even be aimed at people deciding whether their team at work should pursue a new language for some specific task.
Maybe this could be solved with a special badge that tells people which of the Three Tribes of Programmers you're writing for...
For purely toy projects, I'm not sure if any code blog advice at all is that helpful, except possibly "Make stuff really small if you want to impress other programmers".
Best Practices don't apply the same way to this kind of thing, since in a real industry project the best practice would probably be "Just don't".
> I think we'd get more successes if people didn't didn't do this stuff and instead joined existing projects, or started with the business stuff first and tried to start their own foundations and such.
I think that you're working from a very different definition of "success" than your parent comment. "Success" doesn't have to mean "contributed to GDP". For some people "success" means "I had fun, learned stuff, and expanded myself as a person".
I think that you're working from a very different definition of "success" than your parent comment. "Success" doesn't have to mean "contributed to GDP". For some people "success" means "I had fun, learned stuff, and expanded myself as a person".
I'm using the definition the original article seems to imply.
Not so much contributed to the GDP, but "Advanced the state of the art in some way" or "Solved a problem of interest outside the programming community".
I would imagine only the very most dedicated would actually enjoy doing a new language, maybe a third of programmers, and probably only those who fully accepted their thing was probably not going to be an enterprise grade solution.
Not so much contributed to the GDP, but "Advanced the state of the art in some way" or "Solved a problem of interest outside the programming community".
I would imagine only the very most dedicated would actually enjoy doing a new language, maybe a third of programmers, and probably only those who fully accepted their thing was probably not going to be an enterprise grade solution.
I’ve been spending a lot of time myself designing and prototyping my “ideal” language (mainly to address gripes with other languages) and I totally agree with the OP about it being a near total waste of time. But at the same time, albeit being a gargantuan amount of work it helps you learn alot about programming up and down the stack. Good language design requires lots of thought, devotion, trial and error and self moderation to come down to something consistent and coherent that’s not a mess of all the cool little features you thought it would be a good idea to put in.
> Programming language design is at the same time the most interesting technical challenge you can take on, while at the same time being for the most part an utter waste of time.
Yeah that's a hard disagree for me dawg.
But when you write a programming language understand that you're writing it for yourself, so that you can look at other programming languages and understand the tradeoffs.
Just the process of going through "holy shit, this thing I'd like to build is going to be so much harder than I thought" is useful. Particularly if you come to understand that some things you want are at odds with each other and you can't build it the way you thought you could. Then you can look at other languages and understand that to have feature A they had to give up feature B.
Give up pretty quickly on things once you start to get the feel that they're pretty intractable and you start to hate thinking about them. If you enjoy something in particular then jump down the rabbit hole. Understand that you're going to wind up with something only 10% done that nobody will use.
Then maybe when you learn a new programming language go write your language in the new language to learn it. Wash, rinse, repeat to learn new languages. Get good at rewriting your toy language.
Yeah that's a hard disagree for me dawg.
But when you write a programming language understand that you're writing it for yourself, so that you can look at other programming languages and understand the tradeoffs.
Just the process of going through "holy shit, this thing I'd like to build is going to be so much harder than I thought" is useful. Particularly if you come to understand that some things you want are at odds with each other and you can't build it the way you thought you could. Then you can look at other languages and understand that to have feature A they had to give up feature B.
Give up pretty quickly on things once you start to get the feel that they're pretty intractable and you start to hate thinking about them. If you enjoy something in particular then jump down the rabbit hole. Understand that you're going to wind up with something only 10% done that nobody will use.
Then maybe when you learn a new programming language go write your language in the new language to learn it. Wash, rinse, repeat to learn new languages. Get good at rewriting your toy language.
Writing a programming language is easy, when compared to making an actually useful programming language, which again is easy, when compared to actually maintaining and developing a programming language.
My heuristic is that the difference in work required is around three orders of magnitude between creation, usefulness and maintenance.
That being said, I think every programmer would benefit greatly, just by attempting to write a language.
My heuristic is that the difference in work required is around three orders of magnitude between creation, usefulness and maintenance.
That being said, I think every programmer would benefit greatly, just by attempting to write a language.
> Don't Write a Programming Language
Is why we've arrived at using JSON for declarations instead of a nice domain-specific language. So while I would agree that you almost certainly don't need a "programming language", you should definitely consider a DSL over just using JSON.
Is why we've arrived at using JSON for declarations instead of a nice domain-specific language. So while I would agree that you almost certainly don't need a "programming language", you should definitely consider a DSL over just using JSON.
JSON config formats are DSLs—they're embedded in JSON for syntax, but the semantics have to be defined separately.
Yes, for some applications a JSON DSL is inappropriate, and lots of people dislike the syntax. But it's not a choice between "JSON" and "DSL", it's a choice between using an established syntax or defining your own.
Yes, for some applications a JSON DSL is inappropriate, and lots of people dislike the syntax. But it's not a choice between "JSON" and "DSL", it's a choice between using an established syntax or defining your own.
Okay. I probably should have said "Domain Specific Syntax". But DSL is the standard term.
I created a language without lots of trouble. The trick was low expectations. I named it “lila” (limited language), so they were right there in the name. It did very little, it had no compilation phase, but served its didactic purpose. It showed how to build a language using ANTLR.
Started noodling with ANTLR 2.x. Still consider myself a noob.
FWIW, IMHO, two aspects of ANTLR 4.0 dramatically lowered the implementation barriers. Practical stuff that doesn't get much attention.
With L(*) grammars (inspired by packrat, IIRC), parse trees and ASTs can be merged into one. Makes post-processing so much easier.
API now supports interpreting, vs iterating, over the parse tree. Think XML-SAX vs a Visitor. Then you craft your Interpreter to drive a Builder. This simple conceptual change greatly simplifies everything.
FWIW, IMHO, two aspects of ANTLR 4.0 dramatically lowered the implementation barriers. Practical stuff that doesn't get much attention.
With L(*) grammars (inspired by packrat, IIRC), parse trees and ASTs can be merged into one. Makes post-processing so much easier.
API now supports interpreting, vs iterating, over the parse tree. Think XML-SAX vs a Visitor. Then you craft your Interpreter to drive a Builder. This simple conceptual change greatly simplifies everything.
I would rephrase this. First do you want to design a language or implement one? Usually you will want to do both, but where are your motivations and interests? The post starts with a desire to design one but then seems to have spent the most time on implementing it rather than iterating its design. Perhaps it became a sort of vanity project, idk.
Secondly, don't write more than you need to. Look for existing 'narrow waists' that can save having to build much of it. E.g. target a VM ecosystem, use language server tools for editor integration, etc.
It's good to have an idea what you want to do and why, but above all as others say do what you enjoy and learn along the way.
Secondly, don't write more than you need to. Look for existing 'narrow waists' that can save having to build much of it. E.g. target a VM ecosystem, use language server tools for editor integration, etc.
It's good to have an idea what you want to do and why, but above all as others say do what you enjoy and learn along the way.
The same arguments could be applied to almost any field of research. If you get to spend your time doing something that gives you pleasure and challenges you, then it's worth that time. If it results in something that is also of use to others, all the better. There are countless examples in history of how people have worked on something, often despite lack of understanding or even rejection by others, and that very work later became very relevant, often many decades later. You could have spent the same amount of time watching TV instead of writing a programming language. Wouldn't you have found that an "utter waste of time" as well?
I never really got into it that deep: I thought i should
enjoy it, but after writing a small toy language I realised I don’t. What I have realised is I get much more joy from
building things than building the tools but that is just me. Im having much more fun knocking up stuff in nextjs! I am glad there are people who love building the tools and frameworks though such that I am spoilt for choice.
Most people will not need or want to design or implement their own language.
However, it is not as hard as the author suggests. First, there are excellent tutorials, textbooks (Wirth, Dragon book) and tools available (lex, yacc, LLVM, CUP, Jflex, Lemon, ...).
You should give it a try just for the fun of learning.
You will not often _need_ to do it, as most capabilities can perhaps be realized as libraries (including network and mobile functionality mentioned by the author).
Once you design something useful, other people will likely chime in and advertise for your, write documentation etc.
Most complex software system incorporate compiler techniques or Domain Specific Languages (DSLs), for parsing configuration files or as a basis for building higher-level functionality (AutoLISP -> AutoCAD, ELISP -> Emacs).
However, it is not as hard as the author suggests. First, there are excellent tutorials, textbooks (Wirth, Dragon book) and tools available (lex, yacc, LLVM, CUP, Jflex, Lemon, ...).
You should give it a try just for the fun of learning.
You will not often _need_ to do it, as most capabilities can perhaps be realized as libraries (including network and mobile functionality mentioned by the author).
Once you design something useful, other people will likely chime in and advertise for your, write documentation etc.
Most complex software system incorporate compiler techniques or Domain Specific Languages (DSLs), for parsing configuration files or as a basis for building higher-level functionality (AutoLISP -> AutoCAD, ELISP -> Emacs).
Graydon Hoare wrote a really nice intro to the state of compiler technology in 2019: http://venge.net/graydon/talks/CompilerTalk-2019.pdf
He shouts out to Nanopass, a way of writing highly compositional compilers developed in the Scheme world and much ported elsewhere. This kind of approach much reduces the pain of designing and maintaining compilers and can lead to very efficient compilers: it is the technology used in Chez Scheme. It's not got much take-up outside of the Scheme-aware world, despite the existence of many ports of the essential idea to other languages; I note that interesting Nanopass links keep getting submitted to HN without getting commented on: https://hn.algolia.com/?q=nanopass
He shouts out to Nanopass, a way of writing highly compositional compilers developed in the Scheme world and much ported elsewhere. This kind of approach much reduces the pain of designing and maintaining compilers and can lead to very efficient compilers: it is the technology used in Chez Scheme. It's not got much take-up outside of the Scheme-aware world, despite the existence of many ports of the essential idea to other languages; I note that interesting Nanopass links keep getting submitted to HN without getting commented on: https://hn.algolia.com/?q=nanopass
I can feel the author pain, even though I didn't invent a programming language, I just wrote the interpreter for one (specifically a Lua interpreter written in C# - moonsharp).
Why I did it: in 2013-2014 I was SUPER bored and burned-out, I needed a challenge and I picked that one. Oh, I was wrong.
The GOOD: the challenge has been fun, I learned A LOT. I learned how to use ANTLR, and then I learned that ANTLR was getting in the way and now I'm wary of ever using a parser generator at all, for sure not that one. I learned how to write a recursive descent parser, even if I did it by reversing engineering (i.e. mostly copying) the original rd-parser of puc Lua. And I learned that no matter what I do, what I find the hardest is respecting the correct operator priority in math expressions. I know that shouldn't be the hardest thing, but brains are fun. It also ended up getting used. It was used in the OnePassword client for windows, and it's for sure used in the Shenzen IO and TIS-100 videogames. I'm sure there are many more users using that. It's stuck somewhat in between of "nobody uses it" and "it has enough traction that the community will take it forward and you can forget it even existed", and that, turns out, is not a great place to be.
The BAD: I picked C#, as that's the language I was using at the time and I had an interest in seeing it used in Unity. Big mistake. Don't get me wrong: C# is a great language and if you're planning to base your company on it, I don't see big reasons not to (I wouldn't, probably, but every language has its pros and cons). But C#, and in 2014 to boot, was a mistake for such a project. First, however multiplatform C# became, a lot of development is still Windows only (at least, if you want to target the main frameworks). And second, 2014 is precisely the year when C# started changing radically with the core / standard frameworks. The situation now is probably more stable, but in 2014/2015 it was a mess, and half of the reflection frameworks were broken to say the least.
The UGLY: I just didn't consider how much punches life can hit your face with. I changed job, twice. Changed platform from Windows to MacOS, to Linux, to MacOS again, now I program for multiple platforms in Rust most of the time, and my interest in C# is close to non-existent. Family members passed away, new kids were born, situations changed, and life hit me hard enough. Energy got focused on daily jobs, or on other side projects. Every other side project, by the way, has been met with humongous amounts of guilt, because I'm/was leaving moonsharp users (however few they might be) behind.
So where I'm now? I tried to delegate the project to other people, but I'm not effective at it (maybe they are, but in reality, I'm just absent). I tried to get back active into the project but failed a couple of times and now I fear doing it again and creating expectations. Sometimes I feel like contributing something, but I can't even get a proper dev setup running. I don't remember details well enough to not break something when fixing major bugs, nothing major but usually it goes over the amount of energy that I would like to spend. Sometimes I read some question on MoonSharp, and I try to answer, but more often I read a question and before answering I have a guilt trip because I'm answering to this question and left alone all the others and so I simply avoid this one too.
Most of all, the project is not aligned anymore to my interests, so it feels like a chore.
So, if you are reading this: think twice before making the same mistake. If you want a side project, pick one that is either small, or has a clear end to it.
Why I did it: in 2013-2014 I was SUPER bored and burned-out, I needed a challenge and I picked that one. Oh, I was wrong.
The GOOD: the challenge has been fun, I learned A LOT. I learned how to use ANTLR, and then I learned that ANTLR was getting in the way and now I'm wary of ever using a parser generator at all, for sure not that one. I learned how to write a recursive descent parser, even if I did it by reversing engineering (i.e. mostly copying) the original rd-parser of puc Lua. And I learned that no matter what I do, what I find the hardest is respecting the correct operator priority in math expressions. I know that shouldn't be the hardest thing, but brains are fun. It also ended up getting used. It was used in the OnePassword client for windows, and it's for sure used in the Shenzen IO and TIS-100 videogames. I'm sure there are many more users using that. It's stuck somewhat in between of "nobody uses it" and "it has enough traction that the community will take it forward and you can forget it even existed", and that, turns out, is not a great place to be.
The BAD: I picked C#, as that's the language I was using at the time and I had an interest in seeing it used in Unity. Big mistake. Don't get me wrong: C# is a great language and if you're planning to base your company on it, I don't see big reasons not to (I wouldn't, probably, but every language has its pros and cons). But C#, and in 2014 to boot, was a mistake for such a project. First, however multiplatform C# became, a lot of development is still Windows only (at least, if you want to target the main frameworks). And second, 2014 is precisely the year when C# started changing radically with the core / standard frameworks. The situation now is probably more stable, but in 2014/2015 it was a mess, and half of the reflection frameworks were broken to say the least.
The UGLY: I just didn't consider how much punches life can hit your face with. I changed job, twice. Changed platform from Windows to MacOS, to Linux, to MacOS again, now I program for multiple platforms in Rust most of the time, and my interest in C# is close to non-existent. Family members passed away, new kids were born, situations changed, and life hit me hard enough. Energy got focused on daily jobs, or on other side projects. Every other side project, by the way, has been met with humongous amounts of guilt, because I'm/was leaving moonsharp users (however few they might be) behind.
So where I'm now? I tried to delegate the project to other people, but I'm not effective at it (maybe they are, but in reality, I'm just absent). I tried to get back active into the project but failed a couple of times and now I fear doing it again and creating expectations. Sometimes I feel like contributing something, but I can't even get a proper dev setup running. I don't remember details well enough to not break something when fixing major bugs, nothing major but usually it goes over the amount of energy that I would like to spend. Sometimes I read some question on MoonSharp, and I try to answer, but more often I read a question and before answering I have a guilt trip because I'm answering to this question and left alone all the others and so I simply avoid this one too.
Most of all, the project is not aligned anymore to my interests, so it feels like a chore.
So, if you are reading this: think twice before making the same mistake. If you want a side project, pick one that is either small, or has a clear end to it.
I've never written a programming language. I have built a number of constructed languages[1] over the past few decades, which is an equally selfish (and thankless) task.
Less tangentially, I've worked for the past 8 years on building a HTML5 canvas library in Javascript[2]. I don't ever see people talking about JS libraries in terms of computer language design but I think, in a way, that is effectively what a (good) library is doing: building new, sometimes more developer-friendly, ways of conceptualising and creating programs that do stuff using novel APIs, syntax, state, etc.
There's a lot in the OP that resonates with my experience:
- "once you design a programming language you fall down a rabbit hole" - I've lost 8 years of my life to my project; but I've also gained 8 years of learning skills and insights that do translate to my everyday work as a frontend developer. For instance, almost all of my knowledge about accessibility comes from trying out many (many) different approaches to making the canvas element more accessible.
- "You keep adding new features that are really cool" - Guilty! My library doesn't need an integrated reaction-diffusion engine, but somehow it's acquired one along the way. Similar story with the reduced palette filter which, while a lot of fun to build and play with, remains too slow for use in production eg dithering live-stream video.
- "It also affects your communication with the rest of the programming community" - Because of the need for speedy calculations, I've developed an approach to programming Javascript which tends to avoid slower functionality. This sometimes leads to me writing code that could/should be written in a clearer, more concise and understandable way: 99% of JS code doesn't need to be super-fast, just fast enough!
[1] - for example, this one: https://gevey.rikweb.org.uk/
[2] - Scrawl-canvas - https://github.com/KaliedaRik/Scrawl-canvas
Less tangentially, I've worked for the past 8 years on building a HTML5 canvas library in Javascript[2]. I don't ever see people talking about JS libraries in terms of computer language design but I think, in a way, that is effectively what a (good) library is doing: building new, sometimes more developer-friendly, ways of conceptualising and creating programs that do stuff using novel APIs, syntax, state, etc.
There's a lot in the OP that resonates with my experience:
- "once you design a programming language you fall down a rabbit hole" - I've lost 8 years of my life to my project; but I've also gained 8 years of learning skills and insights that do translate to my everyday work as a frontend developer. For instance, almost all of my knowledge about accessibility comes from trying out many (many) different approaches to making the canvas element more accessible.
- "You keep adding new features that are really cool" - Guilty! My library doesn't need an integrated reaction-diffusion engine, but somehow it's acquired one along the way. Similar story with the reduced palette filter which, while a lot of fun to build and play with, remains too slow for use in production eg dithering live-stream video.
- "It also affects your communication with the rest of the programming community" - Because of the need for speedy calculations, I've developed an approach to programming Javascript which tends to avoid slower functionality. This sometimes leads to me writing code that could/should be written in a clearer, more concise and understandable way: 99% of JS code doesn't need to be super-fast, just fast enough!
[1] - for example, this one: https://gevey.rikweb.org.uk/
[2] - Scrawl-canvas - https://github.com/KaliedaRik/Scrawl-canvas
> your language needs an IDE, right?
If it does, it’s a bad language.
(IDEs can be beneficial for many languages. When those benefits become “needs”, that’s a sign that language is pushing complexity onto the editing tools, which has many bad consequences)
If it does, it’s a bad language.
(IDEs can be beneficial for many languages. When those benefits become “needs”, that’s a sign that language is pushing complexity onto the editing tools, which has many bad consequences)
You haven't been bitten hard enough by the PL bug to know what the author is referencing here. The "need" used here is meant to mean desire, not necessity.
Once you start adding all the crazy features to your programming language, the next obvious step is to add IDE support for those features. It's part and parcel of the PL bug; it never stops at the language proper. The feature creep is so overwhelming, and that's what caused the author to write this post.
If anyone is wondering how bad it gets, after the IDE stage of the PL bug, the next stage is to run your language on bare metal, so that opens the whole OS door, as if there weren't enough massive planet sized doors of complexity already open before that. The terminal stage is when you decide to run your PL on custom designed hardware. After that... legend has it you turn into a creature resembling Gollum, and you are driven to live deep with the bowels of the Earth, never to be seen again.
Once you start adding all the crazy features to your programming language, the next obvious step is to add IDE support for those features. It's part and parcel of the PL bug; it never stops at the language proper. The feature creep is so overwhelming, and that's what caused the author to write this post.
If anyone is wondering how bad it gets, after the IDE stage of the PL bug, the next stage is to run your language on bare metal, so that opens the whole OS door, as if there weren't enough massive planet sized doors of complexity already open before that. The terminal stage is when you decide to run your PL on custom designed hardware. After that... legend has it you turn into a creature resembling Gollum, and you are driven to live deep with the bowels of the Earth, never to be seen again.
clojure has pretty good infrastructure for hosting new programming languages that can benefit from standard syntax, data structures and information model (immutability / abstract map reduce), foreign call conventions, macros, js/jvm targets. at hyperfiddle we are developing a multi-tier programming language specialized for server streamed user interfaces and hosting it with clojure lets us inherit a ton of free infrastructure, ecosystem, community and immediate commercial utility (which gets you serious use cases quickly - so important)
> GUI programming (your language needs an IDE, right?)
No language needs an IDE.
No language needs an IDE.
> I'm sure you've all seen the person, that guy who can't help himself from answering programming questions with an explanation how it's done in their own language.
Agree, this can be quite annoying, especially if you're looking for a practical solution to something.
> The person that discusses software optimisation techniques with regards how they implemented it in their compiler.
This on the other hand is often gold. I'd encourage the author and anyone else to share this knowledge whenever they feel the urge!
Agree, this can be quite annoying, especially if you're looking for a practical solution to something.
> The person that discusses software optimisation techniques with regards how they implemented it in their compiler.
This on the other hand is often gold. I'd encourage the author and anyone else to share this knowledge whenever they feel the urge!
> The answer to the Infamous Question is always the same, and if the weasels ever ask it of you, snap back as quickly as possible, "Because I want to know how it works."
Jeff Duntemann
Jeff Duntemann
Literally why I took my first CS course. I just had to know how computers worked. Pure curiosity drove me.
Loved Jeff's book on x86 assembly back in the day. I still remember him (in the book) introducing a DOS interrupt that I could invoke I think through DEBUG that rebooted the computer. And in the text he says something like that: that's the feeling of power. And he was right, that was exactly how I felt.
Loved Jeff's book on x86 assembly back in the day. I still remember him (in the book) introducing a DOS interrupt that I could invoke I think through DEBUG that rebooted the computer. And in the text he says something like that: that's the feeling of power. And he was right, that was exactly how I felt.
FWIW, I would assume this is the language: https://github.com/lokedhs/array
Of course, you can save yourself a ton of trouble by implementing your language on top of the GCC toolchain — the purpose for which it was invented.
Or you can write a translator which converts yourLang into existinglang, then interpret/compile the latter.
Or you can write an interpreter, which is generally less effort than a compiler.
All these are less ambitious undertakings than the full set of subtasks you enumerate.
Or you can write a translator which converts yourLang into existinglang, then interpret/compile the latter.
Or you can write an interpreter, which is generally less effort than a compiler.
All these are less ambitious undertakings than the full set of subtasks you enumerate.
Out of interest, does anyone have any book/article/resource recommendations for writing a programming language? This is something I would like to revisit, not to write a 'good' language, but to revisit all the concepts I was introduced to in college in practice rather than in theory.
I really like crafting interpreters. It walks through 2 implementations of a minimal language. It's also free to read online https://craftinginterpreters.com/
I'll give it a read, thank you!
No, write whatever the hell you feel like writing. Write whatever scratches your itch. End of conversation.
Please do. I have standardized set of solutions and I am lost if the environment does not conform.
Solution: make your own environment on top. https://github.com/timonoko/nokolis.py
Solution: make your own environment on top. https://github.com/timonoko/nokolis.py
don't write a programming language because i couldn't do it. also, don't host your own mail server, because i couldn't do that either. /s
don't listen to these naysayers, do it and learn. "Succeed" or "fail"; however you define these, there is great benefit to be gained.
being able to write a compiler, at whatever level of complexity, is immensely powerful and helpful. don't listen to the naysayers that the compilers class is "hard" -- no matter how far you get, it'll will pay dividends and help you solve more difficult problems more efficiently.
don't listen to these naysayers, do it and learn. "Succeed" or "fail"; however you define these, there is great benefit to be gained.
being able to write a compiler, at whatever level of complexity, is immensely powerful and helpful. don't listen to the naysayers that the compilers class is "hard" -- no matter how far you get, it'll will pay dividends and help you solve more difficult problems more efficiently.
I think folks are misunderstanding the slight sarcasm in this article. The way writing a language will change your brain is a good thing.
Just wanted to say I've been working on mine for a year or so now, and most of this doesn't line up with my experience
Creating a programming language must be both one of the hardest and most unthankful jobs in the world.
You can probably adapt an existing small language to your needs if those are custom needs.
What about writing a DLS for a specific problem at hand?
If you mean a Domain Specific Language, i think that's precisely what the author is advocating for. If you can make a good library or DSL, there is no reason to invent an entirely-new programming language: you'll spend less time reinventing the wheel and more time delivering on high-level features.
This is a really really half-backed article or half-a__ed experience.. Sorry it's getting on my nerves.
Please, do not ever preach someone that if you cannot do something.
Please, do not ever preach someone that if you cannot do something.
This argument makes no sense. Just because I can’t implement new plumbing in my house myself doesn’t mean I can’t complain about my neighbor doing a shit job with his. This board is a Tower of Babel with lots of pointless languages - just because you can doesn’t mean you should.
...but how will we solve new problems?
First, it's technically very difficult, but you will gain deeper insight into the art of the craft. So, if you are a TC chaser or career minded person, then spending half a year writing a language will help you master the coding aspect of the game. I've started many languages since I started college, and each one was instructional. (I'm now 40 and an early retiree)
Second, it's fun.
Third, it may turn into something new. If some people don't write a new programming language, then we are stuck with what we have. This advice basically admits that the status quo is good enough.
The authors saying that the language, as a project, is a lifetime appointment? Well, this reveals everything. I believe if you want to do a programming language, then you must be willing to invest at least a decade or two.
So, here, I am at forty preparing to launch a SaaS around a language that I designed ( http://www.adama-lang.org/ ). The kicker, I believe, is that a project like this requires wandering the desert alone for quite a while.
I'm preparing to launch, and I just started to load test my shiny new production cluster. Low and behold, it sucks. Fortunately, I have a tremendous number of dashboards and isolated it to how I'm interacting with RDS. I've got my work cut out for me which I'll write about.
However, I have a potentially interesting business precisely because I evolved a language which solved a niche use-case. The number of problems that I have had to solve up to this point is not for the faint of heart. Life and reality are harsh mistresses.
So, maybe, yes, you can save yourself some heartache by not writing a language. Perhaps, a better way to think or phase this is "Writing a programming language is a lonely affair that will most likely end in tragedy after a long death march".