Parsing S-expressions in C (2019)(benpaulhanna.com)
benpaulhanna.com
Parsing S-expressions in C (2019)
https://benpaulhanna.com/writing-a-game-in-c-parsing-s-expressions.html
107 comments
I think it is responding to a project as if it were a product.
Even if it were a product there is simply no justification for embedded a second language when all you want to do is read some config files.
I don't known what products you have maintained but I've seen that large software products have a tendency to function as embedded-language magnets.
Sometimes the languages are beautiful domain-specific things that work out of the box and expose a very complex configuration setup in portable format to any platform.
In the worst case they are the most horrible hodge-podge where one party was supposed to do one thing without told what the end result should be, then another party took the existing work and made it work.
I'm using language here in the most wide meaning, thus it includes ad-hoc configuration scripts, inhouse schema parsers, and in-code abstract syntax trees.
This is really beautiful and elegant. First, it does not make my eyes bleed. Second, it does not make me vomit. (I'm only half kidding. The things out in production... shudders).
Third, s-expressions are a succint and extendable way to store configurations. s-expressions are just as portable as XML. Sure, you don't get the xsd schema to verify existing data ... but then again, how many ad-hoc xml configuration files have a schema to go with them.
Sometimes the languages are beautiful domain-specific things that work out of the box and expose a very complex configuration setup in portable format to any platform.
In the worst case they are the most horrible hodge-podge where one party was supposed to do one thing without told what the end result should be, then another party took the existing work and made it work.
I'm using language here in the most wide meaning, thus it includes ad-hoc configuration scripts, inhouse schema parsers, and in-code abstract syntax trees.
This is really beautiful and elegant. First, it does not make my eyes bleed. Second, it does not make me vomit. (I'm only half kidding. The things out in production... shudders).
Third, s-expressions are a succint and extendable way to store configurations. s-expressions are just as portable as XML. Sure, you don't get the xsd schema to verify existing data ... but then again, how many ad-hoc xml configuration files have a schema to go with them.
There's plenty of justification; for example, because you felt like it.
On HN, people don't need to justify their interests or their pleasures. It is the scolds who are at fault.
On HN, people don't need to justify their interests or their pleasures. It is the scolds who are at fault.
Sure, I was imprecise in my language. There is no justification for insisting that one is misguided if they don't embed a full langauge, as many of the early comments suggested.
Sure there is! Shaves a few hours off the initial development time, adds a few milliseconds to execution time (this is scheme we're talking about), adds a few days to later development.
[deleted]
Someone: Here is how I put a sink in my bathroom.
The geniuses of news dot ycombinator dot com: I am scratching my head. If you had simply put a kitchen in your bathroom you would have gotten a sink for free. As a bonus, you would now be able to cook your dinner there.
The geniuses of news dot ycombinator dot com: I am scratching my head. If you had simply put a kitchen in your bathroom you would have gotten a sink for free. As a bonus, you would now be able to cook your dinner there.
What outworlder suggested is not that extreme though. Certain scheme implementations are extremely small and you get a lot of advantages from using one for your config files, just look at emacs for example and compare what you can do with it to the notepad++ config. Surely you can see the advantage of being able to do (bind "C-x" (lambda (state) ...)) compared to (bind "C-x" predefinedaction), or the ability to use functions rather than have to duplicate code all over.
Another positive is that it will make modding/shipping custom levels extremely easy. Just ask these that tried to mod (or config) return to castle wolfenstein or wolfenstein: enemy territory.
Another positive is that it will make modding/shipping custom levels extremely easy. Just ask these that tried to mod (or config) return to castle wolfenstein or wolfenstein: enemy territory.
Yeah embedding a scripting language is not a crazy idea in the context of building a full game but what bothered me enough to make a snarky post was the multiple condescending replies ("I am scratching my head", Greenspun's Tenth Rule, etc.) in response to a small piece about writing a config file parser.
Similar dynamic to StackOverflow (and probably for similar reasons.)
Also why I am exceedingly unlikely to ever post anything I've done or written as a "Show HN".
Also why I am exceedingly unlikely to ever post anything I've done or written as a "Show HN".
[deleted]
Not mentioned: Subtle (possibly unintentional) humblebragging about how you either spend thousands of dollars a year on Amazon kitchens or that you need an Apple Kitchen because your time is so valuable (Both common motifs I see in discussions of said companies)
This is the contrarian dynamic at work. I wrote about this yesterday: https://news.ycombinator.com/item?id=24215601.
Contrarian dynamic in action:
I'm not sure the first part you wrote about "reacting from cache" and "triggered" capture the entire phenomenon.
From personal experience, an empty comments section only leaves the article to comment on with no inspiration for specific content. A poorly written article will obviously provide plenty of fodder (and inspiration!) but hopefully won't end up on the front page. Meanwhile, a well written one (which hopefully does end up on the front page) often doesn't leave me feeling the need to add anything.
Worse, truly brilliant technical articles generally leave me completely out of my depth as far as commenting goes but are really what I come here to read. So the more desirable an article is (to me), the less likely I am to be capable of adding anything of merit.
Assuming the majority are in a similar situation (which could be totally wrong of course), it seems to me that the initial comments will necessarily select for people with less common view points. Of course that includes "triggered" as you say, but I think it's also all sorts of other things - a unique angle, factually incorrect, controversial but correct, a genuine misunderstanding, whatever. (The time constraint would certainly favor "triggered" reactions though.) That would then naturally lead to the rest of what you described.
(Ridiculous idea: Run GPT-3 against each new submission to generate a set of "seed" comments for people to disagree with, elaborate on, make fun of, etc. It would probably just end up derailing things, but who knows?)
I'm not sure the first part you wrote about "reacting from cache" and "triggered" capture the entire phenomenon.
From personal experience, an empty comments section only leaves the article to comment on with no inspiration for specific content. A poorly written article will obviously provide plenty of fodder (and inspiration!) but hopefully won't end up on the front page. Meanwhile, a well written one (which hopefully does end up on the front page) often doesn't leave me feeling the need to add anything.
Worse, truly brilliant technical articles generally leave me completely out of my depth as far as commenting goes but are really what I come here to read. So the more desirable an article is (to me), the less likely I am to be capable of adding anything of merit.
Assuming the majority are in a similar situation (which could be totally wrong of course), it seems to me that the initial comments will necessarily select for people with less common view points. Of course that includes "triggered" as you say, but I think it's also all sorts of other things - a unique angle, factually incorrect, controversial but correct, a genuine misunderstanding, whatever. (The time constraint would certainly favor "triggered" reactions though.) That would then naturally lead to the rest of what you described.
(Ridiculous idea: Run GPT-3 against each new submission to generate a set of "seed" comments for people to disagree with, elaborate on, make fun of, etc. It would probably just end up derailing things, but who knows?)
This is fun and interesting, and I'm not sure what to think of it.
There's an old saying in game dev, which is often good advice for business as well: "Build the game, not a library"[1].
Or as applied to this case, that might mean trying to shoot straight for what you need in the game. Arguably, S-expressions might be the smallest possible thing you can do.
But another approach that is often used: Have your game process a binary file format, and build tools that read text files and compile them into the binary format. It's very common in games to have a tools pipeline that prepares assets for production use. Those tools can safely and easily take whatever dependencies help you get the job done, such as a JSON or YAML parser, or even real LISP if that feels right. This kind of approach also fulfills the laudible goal of keeping the dependencies down (for the production game).
The output of the tool chain could be binary files. Or, depending on your situation, the toolchain could produce code which is then run through your compiler, and then it's ready for direct execution, skipping a parsing step entirely.
[1] https://geometrian.com/programming/tutorials/write-games-not...
There's an old saying in game dev, which is often good advice for business as well: "Build the game, not a library"[1].
Or as applied to this case, that might mean trying to shoot straight for what you need in the game. Arguably, S-expressions might be the smallest possible thing you can do.
But another approach that is often used: Have your game process a binary file format, and build tools that read text files and compile them into the binary format. It's very common in games to have a tools pipeline that prepares assets for production use. Those tools can safely and easily take whatever dependencies help you get the job done, such as a JSON or YAML parser, or even real LISP if that feels right. This kind of approach also fulfills the laudible goal of keeping the dependencies down (for the production game).
The output of the tool chain could be binary files. Or, depending on your situation, the toolchain could produce code which is then run through your compiler, and then it's ready for direct execution, skipping a parsing step entirely.
[1] https://geometrian.com/programming/tutorials/write-games-not...
For what it's worth, I once wrote a game that had a configuration file parser written in bison/flex (I actually finished that game).
I am scratching my head.
If we are parsing S-expressions... why not embed Scheme already? This way, not only you get an S-expression parser for 'free', you can easily manipulate S-expressions, you can add macros and so on. And you also get your scripting language out of this deal!
Something like Racket or Chicken (and possibly others) would work nicely. Chicken specifically has a very nice FFI - which you don't even have to use. As an experiment, I have asked Chicken's compiler to stop at the C code generation, added the generated code to my XCode project, and deployed the whole package to iPhone. Not even cross compilation shenanigans were required, since this was handled by XCode.
As a bonus*2, with a few lines of code, I opened a TCP listener, and fed straight to the REPL. Which meant that I could telnet and have a REPL, which could be used to replace functions on the fly and observe the results immediately, while the app was running.
They would have to rename the project at this point, though :)
If we are parsing S-expressions... why not embed Scheme already? This way, not only you get an S-expression parser for 'free', you can easily manipulate S-expressions, you can add macros and so on. And you also get your scripting language out of this deal!
Something like Racket or Chicken (and possibly others) would work nicely. Chicken specifically has a very nice FFI - which you don't even have to use. As an experiment, I have asked Chicken's compiler to stop at the C code generation, added the generated code to my XCode project, and deployed the whole package to iPhone. Not even cross compilation shenanigans were required, since this was handled by XCode.
As a bonus*2, with a few lines of code, I opened a TCP listener, and fed straight to the REPL. Which meant that I could telnet and have a REPL, which could be used to replace functions on the fly and observe the results immediately, while the app was running.
They would have to rename the project at this point, though :)
Turing-completeness is not just a benefit, it also puts a huge maintenance cost on you. An S-expression is something you can edit in your level editor, process with a script or do a lot of fun stuff with. It's just data. You can even convert it to JSON, XML or binary if you ever get tired of S-expressions.
Once you are no longer dealing with data, but with actual Scheme code that is no longer the case and it becomes virtually impossible to do anything with the code other than directly editing it in a text editor. It's extremely inflexible and every tiny API change will break everything with no automatic way to fix it.
Once you are no longer dealing with data, but with actual Scheme code that is no longer the case and it becomes virtually impossible to do anything with the code other than directly editing it in a text editor. It's extremely inflexible and every tiny API change will break everything with no automatic way to fix it.
[deleted]
> If we are parsing S-expressions... why not embed Scheme already?
I agree. I just thought the goal of the author was to produce a game, not to parse S expressions. It’s possible that parsing S expressions is the most direct route to get there. But there may be more direct and simple ways to get there.
I agree. I just thought the goal of the author was to produce a game, not to parse S expressions. It’s possible that parsing S expressions is the most direct route to get there. But there may be more direct and simple ways to get there.
guile for embedding works really nicely.
Sorry, I keep forgetting about Guile. That thing needs more love indeed.
As a newbie im getting a bit confused. What is the difference between guile and chicken scheme? There's also chez. Im even more confused
They're each different implementations of Scheme, sometimes with different design goals (like embedding or scripting or for standalone programs). Scheme is a pretty simple language (in comparison to many others) and relatively small. There are also several versions (not just implementations). Common ones out there now are R5RS, R6RS (it was considered "big" and was somewhat unpopular), and R7RS. Consequently, different implementations are also conforming to different versions of the spec.
Really, if you're wanting to learn Scheme, just pick one and run with it. For learning purposes they're mostly equivalent if you stick to the more popular ones, but be aware of your learning material and the version of Scheme that they're using.
If you're familiar with other programming languages, it's somewhat like the relationship between different compilers. Though usually they adhere a bit more strictly to the same standards. Since Scheme is such a small language, there's more room for differences between implementations. This mostly doesn't affect you as a learner (most learning materials focus on the core parts of the language which all implementations should be consistent on).
Really, if you're wanting to learn Scheme, just pick one and run with it. For learning purposes they're mostly equivalent if you stick to the more popular ones, but be aware of your learning material and the version of Scheme that they're using.
If you're familiar with other programming languages, it's somewhat like the relationship between different compilers. Though usually they adhere a bit more strictly to the same standards. Since Scheme is such a small language, there's more room for differences between implementations. This mostly doesn't affect you as a learner (most learning materials focus on the core parts of the language which all implementations should be consistent on).
Thank a lot, I decided to give Racket a go. It has a fancy IDE and is battery included
Cool. Racket's a great option, and it's really a multi-lingual environment. Technically not Scheme anymore, but compatible with it in most ways. So any learning material for Scheme will still apply, and you can select different Scheme versions within Racket if the Racket language itself is incompatible with whatever book or similar you're using.
These are different implementations. The language spec they follow is slightly different so they are not completely compatible (although I've been able to use some OpenGL-ES Racket code with no modifications in Chicken by using macros).
Where they will differ the most is things like the packaging system (if they have packages, not all Scheme implementations will), as well as their main use-case.
If you are interested, just pick one and start to learn Scheme, using any implementation. Their differences will become apparent quickly.
Where they will differ the most is things like the packaging system (if they have packages, not all Scheme implementations will), as well as their main use-case.
If you are interested, just pick one and start to learn Scheme, using any implementation. Their differences will become apparent quickly.
Thank you. Quite a few options out there in terms of LISPS and a beginner like me can be quite confused. I started looking at SBCL, Chicken Scheme and Racket and since I'm a newbie I'll stick with racket for the learning phase. I seem to like more the uniformity of Scheme over the cl for example. Do people usually specialize in one or the other or use them interchangeably depending on libraries and such? Another slight barrier has been learning Emacs and that's another reason I chose racket. I like what Emacs is capable but I most likely revisit later.
Racket can be a good first choice. I would nudge you, if I may, and if the lisp glove fits, into taking the deep dive into
That said, you might not always need such large runtimes and other lisps can fit the bill. For example, I usually program in CL (with C FFI) for most projects, but for embedding lisps into existing C programs, Guile can be a joy. Once I was programming servo motor controllers that offered C as one of the programming languages, and I embedded a single C file that gave me a tiny little scheme interpreter and I was then able to prototype motor control algorithms and setting in way less time than before. It's just a game changer for learning about a system when you can get a REPL on it and start rapidly digging and experimenting.
emacs + evil (optional) + paredit + SLIME/SLY + SBCL + Quicklisp
That said, you might not always need such large runtimes and other lisps can fit the bill. For example, I usually program in CL (with C FFI) for most projects, but for embedding lisps into existing C programs, Guile can be a joy. Once I was programming servo motor controllers that offered C as one of the programming languages, and I embedded a single C file that gave me a tiny little scheme interpreter and I was then able to prototype motor control algorithms and setting in way less time than before. It's just a game changer for learning about a system when you can get a REPL on it and start rapidly digging and experimenting.
Thanks. I tried emacs+slime+quicklisp on SBCL and it looks interesting but the learning curve is insane, for now I just want to learn some scheme and take it from there. Racket has some interesting libraries to tot with so Im really excited.
[deleted]
There is a basic explanation that was written by the guile maintainer in 2013: https://www.wingolog.org/archives/2013/01/07/an-opinionated-...
...and this is how we get unnecessarily bloated software with ungodly dependency chains and complex build systems.
Sometimes a purpose-built module that addresses the need and maintains simplicity is preferable over complexity in the interest of some future potential feature that nobody has asked for yet, and may never ask for.
Sometimes a purpose-built module that addresses the need and maintains simplicity is preferable over complexity in the interest of some future potential feature that nobody has asked for yet, and may never ask for.
s7 scheme is pretty trivial to embed in a c project, it consists of an s7.c and s7.h file that you just add to your source tree, no other dependencies. https://ccrma.stanford.edu/software/snd/snd/s7.html
S7 is definitely the way to go if you want to embed Scheme with the absolute minimum of hassle.
Those Scheme interpreters are tiny!
I can guarantee you that a game will need a scripting language. Better get that done already.
That's if you prefer Scheme. You could use Lua too.
I can guarantee you that a game will need a scripting language. Better get that done already.
That's if you prefer Scheme. You could use Lua too.
>I can guarantee you that a game will need a scripting language. Better get that done already.
Myself and thousands of people before and after me have written games without a scripting language.
Myself and thousands of people before and after me have written games without a scripting language.
You _can_ write a game without any scripting language support whatsoever. You could also write the whole game in assembly if you so desired.
However, if every single time you want to change any piece of logic in your game, you have to do the edit/compile cycle, your development will be extremely slow. Especially if you compare with the ability to change code while the game is running.
Not having all the game logic in C or what have you makes the game easier to mod too.
However, if every single time you want to change any piece of logic in your game, you have to do the edit/compile cycle, your development will be extremely slow. Especially if you compare with the ability to change code while the game is running.
Not having all the game logic in C or what have you makes the game easier to mod too.
Fast iteration can be achieved with compiled game logic as well, it all comes down to how things are factored in your project.
An excellent example most with OpenGL experience may be familiar with is supporting hot reloading of GLSL for fast vfx development iteration. It's a compiled language, it gets compiled at runtime without significant delay, and most of us making games have created simple tools to support live editing of GLSL, maybe even supporting it in-game for dev builds.
The demoscene has even turned it into a form of competitive performance art:
https://github.com/Gargaj/Bonzomatic
Edit:
Forgot to mention Casey also goes through support of hot-reloading game logic using C in the Handmade Hero series on YouTube.
An excellent example most with OpenGL experience may be familiar with is supporting hot reloading of GLSL for fast vfx development iteration. It's a compiled language, it gets compiled at runtime without significant delay, and most of us making games have created simple tools to support live editing of GLSL, maybe even supporting it in-game for dev builds.
The demoscene has even turned it into a form of competitive performance art:
https://github.com/Gargaj/Bonzomatic
Edit:
Forgot to mention Casey also goes through support of hot-reloading game logic using C in the Handmade Hero series on YouTube.
>However, if every single time you want to change any piece of logic in your game, you have to do the edit/compile cycle, your development will be extremely slow. Especially if you compare with the ability to change code while the game is running.
1. I'm working on a game in Unity right now that takes about 10 seconds to compile and reload the assemblies for the scripting language (only about 2 seconds of that is actual compilation, the rest is time that would be applicable to an interpreted scripting language).
2. I've built games in C where the entire project took 4 seconds to compile. Incremental compilation and modular architecture can get that down to near instantaneous for most changes.
>Especially if you compare with the ability to change code while the game is running.
Take a look at handmade hero for an example of an architecture that allows you to hot-reload C code while the game is running.
>Not having all the game logic in C or what have you makes the game easier to mod too.
That's completely architecturally dependent. You can make a game easy to mod for non programmers by storing data in a human readable/editable format. If you want to make it easy for players to write code there's a lot more work that needs to be done than just using a scripting language. For most games worrying about allowing modders to easily edit code is premature optimization.
1. I'm working on a game in Unity right now that takes about 10 seconds to compile and reload the assemblies for the scripting language (only about 2 seconds of that is actual compilation, the rest is time that would be applicable to an interpreted scripting language).
2. I've built games in C where the entire project took 4 seconds to compile. Incremental compilation and modular architecture can get that down to near instantaneous for most changes.
>Especially if you compare with the ability to change code while the game is running.
Take a look at handmade hero for an example of an architecture that allows you to hot-reload C code while the game is running.
>Not having all the game logic in C or what have you makes the game easier to mod too.
That's completely architecturally dependent. You can make a game easy to mod for non programmers by storing data in a human readable/editable format. If you want to make it easy for players to write code there's a lot more work that needs to be done than just using a scripting language. For most games worrying about allowing modders to easily edit code is premature optimization.
> For most games worrying about allowing modders to easily edit code is premature optimization.
I was involved in a game project that died extremely quickly a few years ago. The project lead made every decision re: architecture, tooling, etc. somehow involve worrying about modding. As a result, we ended up with nothing to show since everything got derailed by a bizarre obsession by one person with modding before we even had anything to mod. I believe that person doesn't take lead roles on projects any longer...
I was involved in a game project that died extremely quickly a few years ago. The project lead made every decision re: architecture, tooling, etc. somehow involve worrying about modding. As a result, we ended up with nothing to show since everything got derailed by a bizarre obsession by one person with modding before we even had anything to mod. I believe that person doesn't take lead roles on projects any longer...
You could just expose the logic as DLLs though
> I can guarantee you that a game will need a scripting language. Better get that done already.
Is that really so certain?
http://www.youtube.com/watch?v=y2Wmz15aXk0
Is that really so certain?
http://www.youtube.com/watch?v=y2Wmz15aXk0
The issue with scripting engine for game is that for game your almost certainly need scripting engine with completely different semantics than whatever existing implementation will provide. Ie. you either need to save the state or replicate it over network, both with reasonable performance. And looking at various AAA games (from modding standpoint) there is embedded Lua or Python that mostly only changes state of some lower level VM, which more often that not is itself turing-complete.
> I can guarantee you that a game will need a scripting language.
Embedding a scripting language (e.g. Lua) seemed to be very common around 2010, but I get the feeling that it's much less common today.
Has the use of embedded scripting languages in games declined over the last 10 years? Any particular reasons?
Embedding a scripting language (e.g. Lua) seemed to be very common around 2010, but I get the feeling that it's much less common today.
Has the use of embedded scripting languages in games declined over the last 10 years? Any particular reasons?
Yes, preferably chicken scheme as the compiled c code is snappier. I was just looking at chicken scheme on a pocketchip, admittedly I am just just scratching the surface with lisps but this is so cool. For learning I find racket to be quite good but chicken seems a lot better at performance. How long have you been a lisper? Newbie here
Because he needs to parse some config files, not embed a second language.
The author does say the following:
> I'd rather be as library independent as possible, ideally with my SDL being my only cross-platform dependency to worry about.
> I'd rather be as library independent as possible, ideally with my SDL being my only cross-platform dependency to worry about.
All you professional programmer-types seem to have forgotten that solving problems (even if they’ve been solved a thousand times before) is FUN.
I wrote a custom-made script parser... knowing full-well there were many libraries available. (For a goofy program I wrote available at cycell2d.com) Why? Because I enjoyed it!
I wrote a custom-made script parser... knowing full-well there were many libraries available. (For a goofy program I wrote available at cycell2d.com) Why? Because I enjoyed it!
> If we are parsing S-expressions... why not embed Scheme already?
This sort of nonsense is a good example of how some projects die due to absurd feature creep that no one at all can possibly justify. Who in their right mind would ever want to embed a full blown scripting language if your usecase is... Open a file and input data?
If you want to input data following a document format, you parse the document format. That's it.
This sort of nonsense is a good example of how some projects die due to absurd feature creep that no one at all can possibly justify. Who in their right mind would ever want to embed a full blown scripting language if your usecase is... Open a file and input data?
If you want to input data following a document format, you parse the document format. That's it.
> Who in their right mind would ever want to embed a full blown scripting language if your usecase is... Open a file and input data?
Because embedding a mature, well written dependency that fully satisfies all your requirements might well be easier from a technical standpoint. It's certainly likely to reduce your comparative future maintenance burden given that the upstream community is healthy and functional.
Because embedding a mature, well written dependency that fully satisfies all your requirements might well be easier from a technical standpoint. It's certainly likely to reduce your comparative future maintenance burden given that the upstream community is healthy and functional.
> Because embedding a mature, well written dependency
What? Are there no mature well written parsers?
Don't be ridiculous.
What? Are there no mature well written parsers?
Don't be ridiculous.
I thought the comparison was between embedding a Scheme or implementing your own sexp parser from scratch. Of course embedding a good sexp parser is also a valid option to consider, I'm not that dense.
Granted that in this particular case the point seems to have been the exercise of implementing the parser itself so embedding a dependency wasn't a valid approach to begin with.
Granted that in this particular case the point seems to have been the exercise of implementing the parser itself so embedding a dependency wasn't a valid approach to begin with.
> I could telnet and have a REPL, which could be used to replace functions on the fly
I thought you had to restart a scheme’s REPL. Is that just a limitation of Racket?
I thought you had to restart a scheme’s REPL. Is that just a limitation of Racket?
At the top of the page, Ben writes that he's a "Functional Programming Enthusiast", so he'll at least have a vague familiarity with Scheme and similar languages, even if he doesn't use them.
S-expressions are a widely used standard data format and are not exclusive to Lisps. If this is just to read in configuration files, and the resulting data structures never change after they're read in, he can get away with using calloc instead of a garbage collector, and null-terminated char * C-style strings.
Lisps (including Scheme) have more heavyweight implementations of strings and symbols than Ben has here, with an extra level of indirection: a symbol's name slot points to a string struct, which contains a pointer into an array of chars and a length. He can get away with a null-terminated char * and a struct containing an enum to distinguish them. For Lisp programming, the extra complexity is essential but for reading a config file into a game written in C, it's accidental. I don't know what kind of game he's writing but presumably it isn't anything like Abuse or Crash Bandicoot.
In the first paragraph, he writes, "I turned away from XML because I'd rather be as library independent as possible." This implies that he might have considered embedding a Lisp but decided against it for the same reason.
S-expressions are a widely used standard data format and are not exclusive to Lisps. If this is just to read in configuration files, and the resulting data structures never change after they're read in, he can get away with using calloc instead of a garbage collector, and null-terminated char * C-style strings.
Lisps (including Scheme) have more heavyweight implementations of strings and symbols than Ben has here, with an extra level of indirection: a symbol's name slot points to a string struct, which contains a pointer into an array of chars and a length. He can get away with a null-terminated char * and a struct containing an enum to distinguish them. For Lisp programming, the extra complexity is essential but for reading a config file into a game written in C, it's accidental. I don't know what kind of game he's writing but presumably it isn't anything like Abuse or Crash Bandicoot.
In the first paragraph, he writes, "I turned away from XML because I'd rather be as library independent as possible." This implies that he might have considered embedding a Lisp but decided against it for the same reason.
For people not already very familiar with S-expressions, a slightly different way to formulate and format the example from the post is:
(sprites (sprite :x 32
:y 104
(animation :texture "assets/idle.bmp"
:frame-length 2
:frame-width 24
:frame-height 24
:frame-span 5
:loop 0)))
what is the advantage over something more standard like JSON?
I'd say there are a few major benefits:
1. Distinction between symbols and strings.
2. A well-specified number format (integers are integers). In JSON the precision of a number is unspecified, not something you want from an interchange format.
3. It has comments.
4. It's more concise.
5. Easier to parse.
1. Distinction between symbols and strings.
2. A well-specified number format (integers are integers). In JSON the precision of a number is unspecified, not something you want from an interchange format.
3. It has comments.
4. It's more concise.
5. Easier to parse.
For machine reading/writing, not much.
For human reading/writing, sexprs (or Prolog terms) are much nicer as you don't need to quote everything, for example. However, (traditional) sexprs don't have defined literal map or vector/array syntax that's different from a list - for that you need an extension such as EDN format defined by Closure, but usable elsewhere.
Expressions are fun.
In the .NET world we have an Expression library that behaves -almost- like S-Expressions. I'd say some things (i.e. currying) are simplified but for the most part it's the same concept. You can look at every part of an expression and do whatever you want based on it. We often see it used either for advanced reflection, or, for Dynamic Method Generation.
But, having written a few different parsers around it (and writing more LISP than I'd care to admit) I think anyone who is poo-pooing this has just never had the literal pleasure of writing a meaningful expression parser.
EDI transforms and XSLTs don't count, folks. S-Expressions are dead simple
In the .NET world we have an Expression library that behaves -almost- like S-Expressions. I'd say some things (i.e. currying) are simplified but for the most part it's the same concept. You can look at every part of an expression and do whatever you want based on it. We often see it used either for advanced reflection, or, for Dynamic Method Generation.
But, having written a few different parsers around it (and writing more LISP than I'd care to admit) I think anyone who is poo-pooing this has just never had the literal pleasure of writing a meaningful expression parser.
EDI transforms and XSLTs don't count, folks. S-Expressions are dead simple
While you follow Greenspun's tenth rule of programming (Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp) you could embed a Lisp system, like ECL -> Embeddable Common Lisp
https://common-lisp.net/project/ecl/
There are a bunch of applications which include some kind of Lisp implementation in C. From GNU Emacs, Audacity, AutoCAD (and its various clones), and various others.
https://common-lisp.net/project/ecl/
There are a bunch of applications which include some kind of Lisp implementation in C. From GNU Emacs, Audacity, AutoCAD (and its various clones), and various others.
Not a criticism of TFA, but for those who want to parse s-expressions, the algorithm used for common lisp is easy to implement, extensible, and works while only inspecting a single character at at time (no need to mess with formatted input tools like scanf):
http://www.lispworks.com/documentation/HyperSpec/Body/02_b.h...
http://www.lispworks.com/documentation/HyperSpec/Body/02_b.h...
I wrote a Lisp reader for C a long time ago: https://github.com/schani/lispreader
An interesting problem that came up was freeing deeply nested conses without overflowing the stack. I got a bug report about that from a user. They must have generated the structure in memory, because if they were reading it in, the reader would probably also have overflown the stack. In any case, the simple solution was to rotate the structure to be freed so that it can be freed without stack overflow or using an additional queue or stack:
https://github.com/schani/lispreader/blob/master/lispreader....
An interesting problem that came up was freeing deeply nested conses without overflowing the stack. I got a bug report about that from a user. They must have generated the structure in memory, because if they were reading it in, the reader would probably also have overflown the stack. In any case, the simple solution was to rotate the structure to be freed so that it can be freed without stack overflow or using an additional queue or stack:
https://github.com/schani/lispreader/blob/master/lispreader....
The site seems to be down ... here is the referenced page courtesy of the WayBack machine:
https://web.archive.org/web/20200821000640/https://benpaulha...
https://web.archive.org/web/20200821000640/https://benpaulha...
Why not make parsing S-expressions the game? HN crowd would love to play.
[deleted]
Tree is a more natural data structure for s-expressions.
Usually it's called a cons cell in this context, not a tree. The base lisp structure looks something like this:
In c:
data Obj =
| Cons(Obj car * Obj cdr)
| Symbol(string)
| Nil
In c:
typedef enum { Cons, Sym, Nil } NodeType;
typedef struct Node Node;
struct Node {
NodeType type;
union {
struct { Node *car, *cdr; };
const char *symbol;
};
};You're being needlessly pedantic. A cons cell is basically a node of a binary tree. It has fancy names for the left and right child node because of its relation with lisp, but it's nonetheless a tree node.
It is a list node. It's just that you can implement binary trees trivially out of lists.
You got it entirely backwards: a cons cell is a tree node which, as a corner case, can represent branches of a binary tree where the left child node (car) points to a child node value and the right child node (cdr) points to another branch (cons cell or nill).
I indeed got it incorrect but a cons cell is still not a tree node (it can be one though). It is rather just a pair, like (x, y) in haskell or std::pair in C++.
If you want to insist and keep pushing a patently false take then you should at least be aware that you're trying to push a personal definition for a well-established concept that contradicts it's definition, origin, use, and purpose.
https://wiki.c2.com/?ConsCell
https://wiki.c2.com/?ConsCell
(cons 1 2)
Yep, definitely looks like a tree rather than a pair... not. What was the point in linking to this c2 topic even if it does not agree with you?
Yep, definitely looks like a tree rather than a pair... not. What was the point in linking to this c2 topic even if it does not agree with you?
> Yep, definitely looks like a tree rather than a pair...
Node of a binary tree. Either you have problems keeping track of context or you're trolling. Either way I see no point in continuing this conversation. If you have any interest in learning what a cons cell is then there are plenty of resources to go around. If instead you prefer to push personal misconceptions formed in total ignorance then it's fine by me as well.
Node of a binary tree. Either you have problems keeping track of context or you're trolling. Either way I see no point in continuing this conversation. If you have any interest in learning what a cons cell is then there are plenty of resources to go around. If instead you prefer to push personal misconceptions formed in total ignorance then it's fine by me as well.
> Node of a binary tree
std::pair is the node of a binary tree, yeah man. Listen, it is okay to be wrong, just admit it and move on. Pressing on it makes you look like an idiot.
"oh look, a line segment!"
"It's not a line segment, it's a triangle/polygon node"
You can surely see how ridiculous this sounds, right?
std::pair is the node of a binary tree, yeah man. Listen, it is okay to be wrong, just admit it and move on. Pressing on it makes you look like an idiot.
"oh look, a line segment!"
"It's not a line segment, it's a triangle/polygon node"
You can surely see how ridiculous this sounds, right?
The site is not opening for me. I have tried several times. Is anyone experiencing such?
[deleted]
[deleted]
That calloc usage and string copying looks hella broken in the first example.
No mention of the game anywhere else on the blog, and this post is from last year. I wonder what happened?
“Even though I'm never expecting my game to read an empty string in from an s-expression, I didn't want that to be a limitation of the parser.”
This thing is never going to ship. Coming from someone who has written an IDE and wasted a bunch of time on the parser because it was the cool theoretical portion.
That’s fine, I assume learning rather than shipping is the goal here. A fine goal it is.
By the way, the problem is NOT that he wrote his own parser and other homemade tools. This is actually a great thing to do —third party tools usually end up being more trouble than they’re worth. The problem is that he’s ratholing on tiny details that don’t matter.
This thing is never going to ship. Coming from someone who has written an IDE and wasted a bunch of time on the parser because it was the cool theoretical portion.
That’s fine, I assume learning rather than shipping is the goal here. A fine goal it is.
By the way, the problem is NOT that he wrote his own parser and other homemade tools. This is actually a great thing to do —third party tools usually end up being more trouble than they’re worth. The problem is that he’s ratholing on tiny details that don’t matter.
> ratholing on tiny details that don’t matter
That's part of some peoples' journeys though: Learning how to work efficiently. Run into traps and find out again.
I applaud OP with full heart. Especially for stepping up and showing it off.
I don't think I could handle all the negativity you can read here. And so, I'll keep my private projects for myself until I'm done with them. Of course, when I'll be ready to release, nobody will care because I couldn't show it off because of all the negativity.
But on the other hand, I'm not doing this for any of you, only for myself, so I couldn't care less to be frank about what any of you think. I just hope the site being down doesn't mean OP took their server offline just because of all the flak he got for handrolling yet another parser.
That's part of some peoples' journeys though: Learning how to work efficiently. Run into traps and find out again.
I applaud OP with full heart. Especially for stepping up and showing it off.
I don't think I could handle all the negativity you can read here. And so, I'll keep my private projects for myself until I'm done with them. Of course, when I'll be ready to release, nobody will care because I couldn't show it off because of all the negativity.
But on the other hand, I'm not doing this for any of you, only for myself, so I couldn't care less to be frank about what any of you think. I just hope the site being down doesn't mean OP took their server offline just because of all the flak he got for handrolling yet another parser.
And 0 value was added to the actual game. This is exactly how my side projects are never finished
0 value may have been added to the game. A lot of value was added to the author.
Same here, but maybe the goal of his side project is to learn and not ship a product.
In any case, you're not wrong, but I assume most people writing a game in C as opposed to leveraging a tool like Unity are more interested in diving in and learning something new rather than shiping something.
In any case, you're not wrong, but I assume most people writing a game in C as opposed to leveraging a tool like Unity are more interested in diving in and learning something new rather than shiping something.
They're very different attitudes indeed. E.g. compare OP to the level storage of VVVVVV: https://github.com/TerryCavanagh/VVVVVV/blob/cf53de9ed5cd82a...
I love the reference to this because I'm working on a side project and treating it as "something I definitely want to ship" and have found that I have higher tolerance for code that I just copy/paste and slapdash together. I've worked on lots of side projects in the past where I was more deliberate about every line of code and learned that shipping is more important to me now than learning something new.
I will say most of the choices I'm making with respect to when to go "deep" with good design are very much calculated. I'm designing the architecture with a lot of thought, but when it comes to laying out UI elements, I will copy-paste and not refactor even if there's a good opportunity for it. It's just plumbing to me, so if it's ugly, it's not a big deal.
I will say most of the choices I'm making with respect to when to go "deep" with good design are very much calculated. I'm designing the architecture with a lot of thought, but when it comes to laying out UI elements, I will copy-paste and not refactor even if there's a good opportunity for it. It's just plumbing to me, so if it's ugly, it's not a big deal.
Although to be fair I'm sure the author wrote a program that generated that array :-)
I certainly hope so.
With some practice, things like this take between a couple of hours to a couple of days of investment, total. I would say it is a fair price to pay, whenin exchange you get to explore the design space, get to practice some important skills, and get a lot of control over how you want to do things, and get rid of a nasty dependency that might bite you later.
Of course, there are some things that you are not going to do that a solid library gets you for free.
Of course, there are some things that you are not going to do that a solid library gets you for free.
Writing a custom configuration format seems like some prime quality yak shaving :-) Probably flatbuffers would be a good choice for the author needs.
One of the reasons service configurations on OpenBSD (e.g. pf.conf, ipsec.conf, etc) are so convenient and expressive is because they don't shy away from defining and implementing custom configuration syntax using yacc.
At some point somebody needs to implement the configuration interface for the human. Punting things by preferring boilerplate, machinable formats and libraries doesn't actually solve the problem.
At some point somebody needs to implement the configuration interface for the human. Punting things by preferring boilerplate, machinable formats and libraries doesn't actually solve the problem.
C literals const struct files are actually a nice format with no runtime. If you compile in .3s a big C file your compile time wont matter.
Nice, and descriptive. I love the explanation.
It's also an example of Greenspun's tenth rule [0]
"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
[0] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
It's also an example of Greenspun's tenth rule [0]
"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
[0] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
A config file parser is not anywhere close to half of Common Lisp.
It's fine for a personal project, but if you're thinking of writing a custom parser and grammar for configuration for a commercial project: Don't.
There's a lot of flexibility that comes from being able to programmatically generate configs in your scripting language of choice and then ingest them in C++ with little effort. For example, being able to serialize a Python dict as json/ini/yaml/sqlite/whatever and then ingest it in C++ with some library.
It turns your complicated C++ program into a function with a dictionary argument.
I've worked on a few large programs now that have used custom config formats and it sucks. It's a waste of time trying to output correct syntax while fighting through idiosyncratic syntax choices and poorly explained parsing errors in the custom code.
It doesn't create value, it's just a thing you have to do to get to the automation you're trying to accomplish. Avoid it by not making the mistake in the first place.
There's a lot of flexibility that comes from being able to programmatically generate configs in your scripting language of choice and then ingest them in C++ with little effort. For example, being able to serialize a Python dict as json/ini/yaml/sqlite/whatever and then ingest it in C++ with some library.
It turns your complicated C++ program into a function with a dictionary argument.
I've worked on a few large programs now that have used custom config formats and it sucks. It's a waste of time trying to output correct syntax while fighting through idiosyncratic syntax choices and poorly explained parsing errors in the custom code.
It doesn't create value, it's just a thing you have to do to get to the automation you're trying to accomplish. Avoid it by not making the mistake in the first place.
To be fair, S-expressions are not a made up format and in fact predate most other formats you mention.
You're missing the point, S-expression parsers aren't "batteries included" in most scripting languages. The age of the format is irrelevant.
> It's fine for a personal project, but if you're thinking of writing a custom parser and grammar for configuration for a commercial project: Don't.
I would agree. Even more so as parsers for S-expressions have existed... since 1960 or so?
I would agree. Even more so as parsers for S-expressions have existed... since 1960 or so?
Also, the author admits they didn't go to University for CS. You know what you do as part of your CS degree? You write a parser for S-expressions. This is a curious and driven programmer learning CS and sharing their discoveries. No need to berate them for not embedding a scheme....