Is there a reason this guy keeps posting about Rust, especially on articles about other programming languages (ex: also a recent Elixir post)? Overall across posts, I sometimes agree with this guy, but it's getting really obnoxious that people continue to leave such replies every language post.
I really like Rust and use it for major projects, but I fail to see how it's much on topic. I agree with other comments that it gets compared to Go all the time, often without prompting or relationship to articles. Want to do that, why not write an article or do a separate post? Hijacking the comments and/or shilling Rust is getting old.
I have to say that although I love Rust and somewhat despise Go, if it works for you, use it. Go is good at a lot of things - identify what they are and use it because it solves your problem. If another language does that better, use that instead. If you don't know what those things are, you probably need to spend some time not only reading, but writing actual code in that language.
Don't pick one language to solve all your problems and don't look for problems because you like a language or some specific features of a language. Consider the entire scope of your issues, both technical and non-technical. In that sense, I can see why a lot of people choose Go and it is just as valid as any other language choice.
I agree as well. Most of my fastest, "best" code is awful to look at because it has to be. It's full of crazy comments and hacks that were necessary for the hardware, language, and time usually. I think as processing power has increased, the low-level code is actually much less ugly. This is in no small part due to astronomical leaps in game design, engine design, and development time and other resources.
Still, in my experience any codebase that pushes a large amount of technical boundaries tends to be full of huge "WTF is this" moments that require explanation. It is an achievement if you can create a decently powerful codebase that can survive rapid changes and additions. It's much more common, but I've seen and worked on AAA codebases that would break if you blew on them.
Yes, very true, obviously I just mean conceptually poking around. In many ways we definitely have it better now. I am infinitely more productive today. Simultaneously have gone further from teaching the low-level stuff properly.
I spent hours, days, and weeks of my life on things related to graphics/games programming such as:
- Building mouse drivers from scratch or implementing them from alternative vendors
- Reverse engineering consoles to steal processing power from idiotic sources just to render a tiny bit more data or later, a few more polygons
- Spending a huge chunk of cash to throw in a math co-processor into my machine at home
- Debugging code for hours only to realize things like the problems are caused by seemingly unrelated problems such as tape media is at fault, the floppy is corrupt, or the file system doesn't work the way the vendor said it does in the specs
- Rendering a scene and then going home, only to come in the next day and see it is still not done
- Converting from 72 billion formats and interpreting, finding, and/or correcting corrupt data from each one
- Rewriting entire pieces of code bases to squeeze out several more bytes of EMS and XMS
- Implementing 2 or more graphics APIs for the same game. Thank you 3dfx, S3, and many others that pained me, not to mention at a higher-level, OpenGL, DirectX, and so on.
- Doing all my work on 1 platform, then loading it on another. Thank you SGI for taking years off my life.
- Writing matrix operations in pure assembler for the simplest of operations
- Having multiple workstations for reasons such as "this one has the Matrox card in it."
The list goes on. Yeah, I don't miss those days. But I learned a lot, we all did. And the barriers to entry definitely reduced the signal to noise IMO.
Fair enough. Definitely there need to be better resources for getting started in 3D programming, particularly in C/C++, but the barriers to entry are also related to what makes things powerful at a AAA level.
In some ways, just grabbing an engine like Unreal or Unity is a decent alternative to something like JS to learn (some even let you use JS or use other languages that also have traps). Big game engines leave a lot to be desired and at times abstracts too much or makes things like handling shader code annoying. Still, most larger game engines like these two are the closest thing to having LOGO for 3D games programming.
You can at least get stuff on the screen relatively quickly, learn a few things, and then start replacing it from there. I personally learned quite a bit way back when just decompiling or reverse engineering stuff from people much smarter than me.
Sometimes I feel like there's a lack of things like what the C64 provided for younger kids and adults today. I suppose as expectations have risen, so has complexity of getting going.
> you would have to have really compelling reasons to start with C++ and not JavaScript and WebGL. And that's coming from someone who actually likes C++ and used to write it professionally.
As someone working in game programming a long time and now also spending more time in machine learning, I must disagree. I suppose I can see your implied point about getting started quickly or something like that, but there's really few compelling reasons to do graphics programming in JavaScript professionally. As a hobby or for a web-based project like a tutorial site, sure, JavaScript and WebGL make perfect sense. For most professional uses though, I can't agree.
Writing a decent performing and well-architected graphics engine is much more than just using an API like OpenGL, WebGL, or any of the successors or predecessors. Further, writing a graphics engine is not about creating an efficient island, but rather a piece of a larger eco-system. Consider that quite often some of the biggest hinderances to graphics performance are creating the data to actually be rendered (ex: via the game update), sending the data to the GPU efficiently, and still getting all kinds of other program logic to run before you even have a frame or want to display a frame. It would be a long post to go into all the details, so forgive me for oversimplifying. Among many things vital to graphics programming that C and C++ provide that JavaScript doesn't or either fails to do well:
- Control of memory allocations and memory shape/characteristics
- Maximizing cache lines (due to architecture, memory footprint, etc.)
- Fast, efficient, correct math (or possibility thereof)
- Compilation tweaking/assembler output tweaking
- Efficient IO
- Fast compression/decompression algorithms
- Integration with existing toolchains for graphics including middleware, apps, and shader languages
JavaScript has made great strides in some of these areas and certainly things like web asm help. Nevertheless, most JavaScript runtimes people are using are wholly unsuitable for professional graphics programming. This is akin to people who used to say that you can write a professional game in BASIC or later, Visual Basic. Yes, it's true you can, but should you do it or start there? Probably not if your goal is to learn professional graphics programming. Have people taken such routes and become awesome programmers? Sure, but there are just as many people who never learn, grow, and develop nasty habits that are harder to unlearn than learning the difficult stuff.
If your goal is having fun and learning a few things before moving on to C++, dabble in some JS graphics programming, but since you called our professional programming as no longer being compelling to start, I have to point out that this is both wrong and bad advice to someone new. If you are struggling with the concepts, you will struggle in any language. If you personally need instant gratification and a self-esteem boost, from this point of view JS might not be bad for learning, but it won't teach you a lot of vital things and will encourage some horrific practices. The presence of GC and the lack of a proper parallel and concurrent programming model poison the experience quite a bit.
Ironically, other languages that are more friendly to parallelism and concurrency tend to be better than C, C++, and JS. A lot of the realities of graphics programming are simply caused by historical reasons, market dominance of things like x86 and companies such as nVidia, and existing knowledge, code, and toolchains. Functional languages for instance in many ways have the potential to be better suited to take advantage of the direction we're headed in with regard to numbers of cores, Moore's Law (or diminishing of), GPU design, memory architecture, and so on, but this is just hypothetical mostly. Given JavaScript is hardly known for its ability to deal with concurrency and parallelism in most contexts (no, nodejs doesn't count), again I have to find the above dubious considering you mention the future WRT nVidia.
Of course in the end, you can use things like transpiling, engines that may or might compile to JS as a runtime target, and so on and getting a running product that might be decent in the eye of the beholder. Personally, I've dabbled in JS enough to write a high-performance Voxel engine many years ago when I thought about making a game in JS as well as a skeleton of a 3D adventure game engine for a contract. I was productive enough but ran up against walls that weren't worth working around or just made things feel so kneecapped that I wondered why I was even using JS at all in its current state. The old adage holds that in the end, you can write anything in any language, but there is indeed value of selecting the best tool for the job. Even just making something run and at a reduced frame rate typically is something fundamentally different than being THE language for graphics programming.
In truth, when I first saw JS in the 90s, I thought there was no way it would reasonably do anything in 3D. Things are getting better, but I think you are dramatically underselling what people use and exists today in a professional environment. Your comment about 400 lines of code being that impressive and small seems to also hint you haven't dabbled that much in the area professionally. While your links are just fine and the above shader toy link is indeed impressive in the context of now, personally I was much more blown away when I dabbled in the demo scene and saw what people were doing with an Atari ST, Amiga, and PCs with no GPUs and on systems with less power than today's fitness trackers.
Anyway, I can't think of many compelling reasons to start in JS when IMO, it's better to learn to do things the proper way even if they can be a bit rough and punishing. JS will surely be more productive at the beginning, but also skip teaching you vital things you need to know as well as some of the fundamental primitives of graphics programming. This is the real world and no one in a professional environment cares if you are awesome in JS but don't understand the tools and best practices people actually use industry-wide. Moreover, you won't get far if you can't be productive from your first day because you tried to take the easy way out. If anything, just comparing the amount of resources for JS vs. C/C++, especially from professional vendors like nVidia, Microsoft, AMD, and so on makes me think at best, JS as a starting language for graphics programming only holds for web programmers. If anything this seems to makes things an extra layer of difficult.
Now mostly retired, but I've been doing game dev for a couple of decades. I would say based on my career and that of my peers, it is more correct to say C or even assembler is the traditional graphics programming language with regard to games. It just depends on your definition. For that matter, if we consider things like CAD, 3D modeling/animation (using APIs/scripting), movie production, etc. to be graphics programming, again there are some differing results as to which language you use.
It also really depends on what time period we're counting as the "start" and if we value older games or periods of higher but simpler productions vs. now. For instance, I wrote quite a lot of code against NES, SNES, Genesis, Atari, Arcade machines, C64, and so on in assembler. 68000-based systems alone are such a huge chunk of sheer volume of games and most of us wrote quite a bit in pure assembler with regard to graphics.
On other systems that were faster or for different game requirements, we definitely used a ton of C, only moving to C++ later. In the case of C++, it's almost hard to even call things C++ at times. To be honest, most decent game engines I've worked on essentially use C++ to practically rewrite the language and go to great lengths to avoid some of the primary selling points of C++. It's sort of pick and choose some good things about C++ or at least powerful things while avoiding some of the burdens of C with regard to games. Obviously general C++ programming is different, so keep in mind I'm only referring to professional games programming. As things have progressed, people are indeed using more of what the newer C++ standards offer, but most of the same things still hold true. Where one has a tough argument with regard to graphics is the fact that so much now happens on the GPU, which one can argue is its own thing given stuff like shader languages.
Oh didn't think you had anything against Clojurescript, just maybe pro-old school PHP ways or something.
I've been a bit spoiled by ClojureScript and I can't imagine using much else as far as js tooling. I'm less familiar with Elixir and some other newer options. I've only played with Elixir a bit and not used it on a real deployed project, but some people like the experience of that too. For me what puts Clojurescript over the top is it brings most of the sane things about Clojure and allows me to write JavaScript essentially in a sane way. It also does so without stopping me from being able to use JS libraries when I absolutely have no choice. The isomorphic code potential is just a side benefit - most of the time I can just take a Clojure file and rename it cljs or cljc and it just works as long as I'm not doing anything obviously pure Java there.
I agree Clojurescript commits some of the same sins. As far as tooling, I feel at least Clojurescript's is pretty standardized. I agree there are too many ways of doing the same things out there. JavaScript in my eyes is a terrible language that needs to be taken out back and shot - faster VMs and the browser are not an excuse - the browser itself is in a similar position, but that's another discussion. I've been programming a long time and I've been exposed to some pretty terrible things so I don't say that lightly. JavaScript is doubly awful in the browser. And yes, I get it and am aware of the new enhancements to the language long overdue. Clojurescript is the best current compromise for me, but like anything, I'm happy to use something else if it makes sense.
I'd still advise everyone to be wary and there's no one size fits all. In the past, I tried things like GWT and various approaches in .NET to transpiling and found them lacking or inferior, but I understand they work great for some people. Clojurescript is the first time I've found a happy medium, but it's not for the inexperienced programmer. Some people complain about the barriers to entry, but I'm somewhat glad they are there. Trying not to sound like a get off my lawn guy or an elitist, but I'll say that learning a language and to program in general is best first before trying to do things like networked service i.e. web and mobile apps. I understand people sometimes need immediate gratification and feedback, but it's also why we have so much trash just put out there - it's so easy and people who don't really know how to program often speak the loudest. Patience to learn and overcoming adversity are really valuable skills for programmers. For as many steps we've taken forward, there are so many back.
Clojurescript is a good compromise for me. The signal to noise ratio is a bit better vs. other JS-related offerings for now. I don't feel like we should always chase the lowest common denominator, and I wish people would stop the TLDR approach to programming. To that end, I feel being responsive and positive when interacting with new-comers to programming, and not trying to proselytize <insert your favorite one true way> to everyone is a better approach. Moreover, writing good documentation, organizing things cleanly, and so on is valuable to helping people get off the ground easier. I think we've reached a bit of an age where people want to fly the plane before learning what flaps or a landing gear are. I'm all for helping people, just the right way.
So from that point of view, I do somewhat disagree with part of the article's implied premise of wanting to be friendly to non-programmers with regard to Clojurescript. I've heard this promise so many times, from Visual Basic to Smalltalk to Visual Development Environments to various Web Apps - let's just do better with what we have and focus on taking some risks to replace the bad stuff we're so attached to rather than trying to make everyone into a programmer. Many better choices out there and I can't really see Clojurescript itself as being too friendly to non-programmers. I'm also not a believer in free lunches. But what do I know, I'm just another guy that learned the old way with pencil and paper and believes in learning concepts more than syntax.
I don't understand what you are getting at here, can you explain more? How is an interpreted language turned into something compiled? It sounds more like you have an issue with heavy weight frameworks, which I can agree with, but I am not sure if this is what you are really trying to say. I also agree that feedback loops and debugging trails are important.
Clojurescript tends to do well with 2 of the 3 you listed. Feedback loops are quick with hot code reloading, side-by-side development, browser tools integrations (ex: with chrome dev tools) and so on. The Clojure/Clojurescript REPL itself is probably one of the best feedback loops I can think of that I've used other than proper Lisp machines or good Smalltalk images. Add in working with a data-oriented approach cljs encourages and using immutable data structures, EDN, transit, and things like that, and again things are a fast feedback loop predictable, and easy to debug.
The actual debug trails however in Clojure and Clojurescript leave much to be desired but are getting better. I personally haven't had many issues but I grew up on assembler, Lisp, C, C++, and that kind of stuff. After a few days, most of the more obscure things are at least predictable and you understand what they are. Given you can attach with a debugger from Emacs, Intellij, Vim, etc. it's even easier to figure out what's going on. Clojurescript and Clojure tend to discourage big monolithic things that aren't composable via the standard library, 3rd party libraries, and language design. You simply will shoot yourself in the foot if you start creating giant framework blobs and things that can't compose, so to me that's a good thing. For instance, using things like map, reduce, cat, transduce, apply, update, etc. become infinitely harder. Of course like any language, anyone can still do dumb stuff but it's nice that things are pretty simple from the functional point of view that common sense hopefully sets off alarm bells for people.
Anyway, Clojurescript in general for me is a superior experience compared to the old days when I messed around with Perl, CGI scripts, PHP, ASP classic (COM+ add-ons!), and so on to get a dynamic website going for something. Refresh, print line, and approaches like that are and have always been garbage. I've never had better visibility into an app than with Clojurescript, and given that most things are idempotent and pure that I write where possible, testing is also even easier. In most SPA frameworks in Clojurescript, you also have a single source of truth and visibility into it is as simple as running a built-in function to grab a map key or just printing the map if you want the old school approach.
> It's a useful model, but I've found that it can break down a bit on really large, complex apps. If a piece of state is being passed to multiple places via props (either manually or using something like Redux), it can be difficult to see at a glance every place in an app that a single bit of state is being used.
This is why a UI app state query-based approaches (ex: om next) on a single app state tend to be a bit easier, but still can suffer from what you describe (especially if queries are dynamically built by a user or something). At a minimum, you can either re-use the same query or at least know through the query syntax where your code is touch some bit of state as it should be available for analysis by an IDE and easily searchable as plain text. Still, there's not really a perfect solution I've seen and eventually things can get messy if you aren't careful and as you increase the number of developers touching code.
In general, updating a UI only based on changes is a great approach. The challenge has always been identifying what changed and minimizing the tracking of those changes in scalable way. I saw many old approaches use things like dirty flags everywhere or doing field comparisons one by one. Things like react frameworks in Clojurescript make this so much easier today because you can do a very fast identity check that is cheaper using immutable structures like those in Clojure. If your check itself is expensive, the benefits of a delta-based approach are limited vs. giving up and doing full re-renders. I've hit this in game programming either way, but usually the change approach wins unless there's some very specialized case or design issue or simply just tons of raw power it's not an issue anyway.
Where events themselves suck is predictability. This is doubly so for systems that introduce event hierarchies i.e. inheritance-like constructs for events. I strongly prefer deterministic approaches to rendering when possible. That's not to say you re-render at a fixed interval, but rather attempt to re-render changes only if they exist. It makes debugging, optimizing, and understanding the system so much easier.
From a performance and debugging point of view, events or signals and slots tend to cause situations where you're jumping so much over the code you lose all kinds of CPU cache or GPU benefits (buffers, batching, pipelining, etc.) depending on what you are doing. Also some event systems use a lot of objects and in systems requiring allocations on the heap and/or garbage collection, this can become really ugly if the app is running for awhile. Events do make things easy for small projects, but tend to create spaghetti for larger ones, even with a single event bus. Approaches using loops, deltas and possibly queues/mailboxes tend to scale a lot better for games, and also for apps that have performance issues. A side benefit is your state tends to become a functional reduction, which has its own benefits such as making undo/redo, logging, and error handling easier.
Sometimes I'm rather confused by all the UI issues created in app dev. I understand them, but as someone who has many decades of experience doing both game and app development, I'm like wtf app programmers. React and things like it are or can be at least a bit closer to loop, pipelined, time approach that's used in modern game architectures precisely to achieve consistent performance and do sane things in the face of GPUs.
Again, you're welcome to disagree. We're talking about game development in particular, not Lua in general. I've used Lua and almost always look to it for scripting in game engines I have built.
It's telling that all the major game engines and a large portion of papers and talks disagree with you. If Lua was the solution to data-driven development, people wouldn't bend over backwards looking for solutions like entity component systems. Lua is nice, but not suitable for most professional game development as a primary language. As such, it makes it completely unsuitable because Lua data, especially things that depend on runtime state are utterly useless.
Your shipping comment also makes no sense in this context. Tons of other languages have more games shipped than Lua. There aren't too many people relying on Lua for composition and data-driven development in games, because it doesn't work. Even Lisp doesn't work in this context putting speed and other things aside, because of so many reasons I've already listed. Code in the context of game development is not data by the definition of data I am referencing.
I started a prototype project first to try it out and used Datomic. With Datomic, most things are entierly automatic.
In the second, I'm using a bit of Datomic mainly to save UI-related and small bits of user state. Mainly I use Cassandra, Redis, Kafka, and Elasticsearch. I should add in this project, I'm also using untangled.
I communicate a lot over websockets as well, so between that and Cassandra I have to do a lot of mapping and transformations on my own, however once you get it setup the rest is mostly magic. Most of my reads and writes over the socket are happening with consumers and producers to Kafka so mapping om-next data here is just standard Clojure, nothing fancy.
When I do query Cassandra directly, the queries in particular by nature need to be somewhat restrictive which it turns out play nicely with om-next since mostly for reads I just need to worry about the "select" columns. Redis I just use for LRU caching and to buffer some video-like blocks of data before the user tries to access them. Elasticsearch is just used for searches and some type of filtering, but gets populated through Cassandra data anyway so again no problem with om-next.
I'd say the most annoying thing here was as I hinted, getting it all setup. Things are clearly built for Datomic, but there are integration points all over the place and just using standard functions in Clojure to manipulate vectors and maps is easy. If you're not experienced with datalog or datomic pull syntax, things might be a bit harder at the start for some people.
Regarding API calls, I should add since I am using untangled, I use the default websocket lib which is just sente. I have rigged things up with aleph and yada, but also played around with standard compojure api and other offerings. Long-term I'd like to stick with yada, but I've encountered some documented bugs that are annoying so I've put my focus elsewhere in the short-term when needed.
I have used reframe to great success, as well as vanilla reagent. I've currently switched to om-next for a variety of reasons.
I have my concerns about om-next, but overall since I'm writing a complex side-project, it's worth it. If I had to ship something tomorrow, I'd go with reagent still, but for longer projects where you can spend more time and get over some learning curve and alpha/beta type bugs, om-next is a good fit for large projects. I found reframe to be generally the best way of working with reagent, but not as good when I had lots of chained events and complex queries. I spent a lot of time optimizing around some ways that reagent/reframe does things where as in om-next, I could tweak things more to my needs. Om-next also offers some additional features many people don't need, but fit my use-cases well (ex: server-story, pulling only what you need, normalizing client app db data).
To summarize, reframe is a nice framework and really great for small projects or projects that need to ship today. It works and is the best approach I've seen on top of reagent (and react). Om-next is good for people that can afford to take some risks and have complex requirements but still want a proper framework on top of react. Om-next will make you do a lot more work up-front and has a lot more ceremony, but there is a reason it exists. Generally, I have found om-next powerful once you are able to grasp the way it wants you to do things. For people who want something like reframe like om-next, there is the "untangled" framework. Anyway, I wouldn't recommend om-next or reframe/reagent over the other, rather I'd suggest either and match your requirements accordingly.
Much more productive after the initial learning curve as some said. It's pretty easy to call standard js from Clojurescript which makes things play well.
I find having the syntax, tools, and power of Clojure to be a much better experience than using vanilla js. There are so many things that just work, and having immutable data structures as the default is a plus, and a huge win. Further, I find that it makes things a lot easier to use with the backend, especially if it is also Clojure.
As far as working with the backend, I often am able to rename a clj file to cljs and it just works. I can also share the same exact code literally and insert the occasional reader conditional and cljc file. This is a huge win as well given both Clojure and Clojurescript encourage data-first approaches. Most of my code tends to be just maps, sets, lists, and vectors, with the occasional custom data structure. Sending this back and forth between client and server is infinitely easier and nicer than vanilla js. Adding things like EDN, transit, and libs that are built for dual use in Clojure and Clojurescript save a ton of time here as well.
Regarding tools, I find if you just use decent emacs plugins or an IDE like IntelliJ with the Cursive plugin, the syntax checking and dev experience is great. I tend to do a lot in the REPL which I feel is a productivity win for me over just working in chrome dev tools alone or some other js tools. Figwheel for example is another great one I use for doing hot code reloads/side-by-side changes.
For more specific and complex cases, I get even more benefit using reactjs from Clojurescript. There are multiple good frameworks in cljs that simplify the react experience and in some cases, even makes it run a bit faster (unless you replicated more or less the same immutable experience from js).
As for bad things, here's a few worth noting:
- Initial project setup can be a lot more of a pain, especially with lein. Boot can be an improvement but isn't as widely supported and doesn't always makes things easier. The larger point is there is a lot of ceremony getting new projects setup for non-trivial things. For trivial things, lein templates and various git repos make it quick.
- Some situations using vanilla js arrays can lead to better performance. It just depends because the opposite can be true. In both Clojure and Clojurescript I use transients where possible for a slight boost in some use-cases.
- Debugging can be a bit of a pain at times if you really have something deep. Since most things are data though, I don't really care that much as I normally solve problems using the REPL and/or dumping app state. In reagent and other frameworks, you tend to have a single atom i.e. source of truth which makes it easier than most typical approaches in js.
- Every once in awhile I hit something weird in Google Closure or a weird bug with Clojurescript itself. These tend to get fixed though and it's not hard to reach the actual devs.
- Clojurescript tends to make me hate using non-Clojurescript libraries. Interfacing is not difficult, rather the quality bar and extra dependencies that many npm projects pull in just make me insane. It's more of a personal and elitist thing that happens to a lot of people I know who use Cljs, but not really a problem, just pointing it out.
As a whole, I feel like Clojurescript makes Javascript not suck so bad. I know a lot of people love js, and I've been using it since the beginning. It sucks and is a horrible language, but we're stuck with it. Fortunately Clojurescript and Elm are at least ways of making the experience less bad. The biggest criticism is any time you get into transpiling or further from an original language, there are always a few places where things leak or become more confusing, but I'd say given js in general, it's worth it for a solid set of libraries that just work and the dev experience that goes with it.
The term component here would be incorrect, rather "system" would be a better term.
There are many variations and approaches to what I am talking about so I'll stick to one very common approach used by a lot of studios and engines today. A typical more cache-friendly update cycle goes like this, calling "update" as a function of time/state on each system:
System A -> System B -> System C -> System D -> etc. (loop and repeat each frame)
This cycle sometimes skips certain systems for either performance or sim reasons. Each system tends to have inner loops and the overall system loop is often broken up a bit into sub-loops or separate loops, ex: render vs. physics for purposes of separate sim-time looping. The idea here is you do a single operation over and over again on the same type of data, avoiding lots of pointer chasing, v-tables, and so on along the way. The code branches very predictably and it becomes easier to avoid loading data into the cache that will blow it up or not fit into a page.
What makes this friendly to a data-driven approach is if you're working mostly with systems that operate on pure data, your update each frame from a purist point of view approaches a functional reduction. In practice, there are certain things that don't fit perfectly into this paradigm, but the idea most engines take is this approach + cut corners when necessary.
I am contending that your point data == code is wholly false. Again, this is well-known in computer science to the point where your argument only stands if you're looking at code from a very low-level point of view. Again, all current game development trends refute what you are saying - almost every year there's one or more GDC speaker sessions about this topic, see the GDC vaults, see Unreal Engine, Unity, CryEngine, etc.
Yes, to a degree code can be data, but it becomes exponentially difficult to manage and process vs. simple representations that tend to be language, runtime, and network agnostic - i.e. data structures. Lua tables and other things inside Lua are not the same (at least not until serialized to a more language agnostic format), especially in the face that in most real game dev situations, you're passing data between tools and languages, and in several different flavors of run and design time. I've done quite a bit of code as data approaches to know that even when it can be done, it has many drawbacks - see object databases, Smalltalk continuation serialization, code gen tools, etc.
Functional languages and several more fringe languages tend to be closer to code == data, the most wildly known and regarding being Lisp-like languages. Typically these languages have a minimal amount of abstractions and power things like first-class macro systems. Even in the face of macros, there are many drawbacks to dealing with code this way in general, and even more in a game. For instance, restoring state exactly as it was and is, and using that state in the face of updates to the game itself are paramount to the point where few games can survive without it otherwise. Data makes this easy.
What you are asserting as more power is exactly the opposite and I'm not sure how I can make this any clearer. Your code can freely operate on data any way it wishes, not the other way around. Code operating on code is generally a bad idea which is why clever meta-programming techniques and code generation are things you would rarely see in any game engine unless it was a hack or a bad implementation.
You are losing little by making things a data representation and gaining much in terms of what I already listed. All the benefits of data are far more valuable for game development and I am not sure what cannot be achieved by forward-loading or pipelining data into an asset pipeline and operating on that data doesn't achieve that this does. Beyond performance, the other benefits like editors, network transparency, easy loading and saving of state, and so on make it easier and faster to develop even small games.
What you are proposing is a solution looking for a problem. I'm not denying it's a fun thing to do, especially if we're talking about a regular "app" rather than a game. Personally, I like coroutines, fibers, CSP, continuations, multiple dispatch, and other fun CS tools but not for game programming in most of the time, and certainly not at the level of an important game system. I am asserting that in the game dev world, treating code as data is a really bad idea, which several other people have also pointed out here. Your solution adds very little, while adding unnecessary complexity to check a "cool" computer science box.
Please re-read what I wrote. I said 1 or more of these items, not all of these items. Additionally, please read my reply elsewhere in this thread about speed in dialog engines where I listed out why, where, and when speed does indeed matter for dialog engines (thus cache lines). Moreover, the last thing you would want is for dialog of all things to murder your cache when it one of the easiest places to optimize for the cache in an entire game engine.
Lua is indeed used to fix many things in game engines, but not really most of what I listed. The top reasons scripting languages tend to be used on top of other languages like C/C++ in games is to avoid lengthy recompilations, provide an entry point for user-editing/plug-ins, serve to some degree as a more RAD tool, and build a DSL for scripting events.
You are correct data-driven is interesting, but your conclusion is 100% wrong. The entire entity component system approach and the general trend towards functional paradigms couldn't be more at odds with you. I can't think of what might be the best resources off the top of my head for you to read, but there's plenty out there on ECS, functional-style C/C++ as related to games, and low-level engine design stuff on older blogs like the one BitSquid wrote years ago.
The entire industry is moving to trying to make games as data-driven as possible. This is as you point out the anti-thesis of what you are saying. Further, your point is really bad advice for any game programmer. Making things less data-driven in general makes it infinitely harder to debug, cache, build editors around, send over a network, save/load, update/migrate, share between threads, and be consistent.
What you are proposing would make you fail an interview with any decent game developer or get you fired. You would drive a tool designer insane as well. Huge chunks of games are built by creating tools with code to manipulate data, not code. That data often lives external to the code entirely, whether it is an excel sheet, custom binary format, custom database, or something else. You want as little as possible in the code and nothing that will affect compilation time. Moreover, you need to be able to also do things like translations and internationalization, not to mention add new assets, hot-swap things during dev, and so on.
In other words, no, building a DSL in Lua is not more powerful as you describe. It could be powerful if it was data-driven, anything else simply would not get used in a professional setting with decent developers. I don't understand why sometimes on HN people want to graft general programming or web dev on to games. That's the only way I can explain this kind of thinking, and even data-driven approaches are becoming more prevalent in these areas.
Not sure what your point is here, could you elaborate? It seems to me maybe you are missing the point.
As I said, this is about programming games, primarily in C/C++ or a related language (though applies mostly to others), not programming in general. If you enter into other high-performance domains, you will encounter the same issues. Conversely, other domains such as general application programming do not adhere to this.
Regarding try/catch, I am well aware of low-level implementations in various languages. Did you know that in most game engines, you don't use that much try/catch for this exact reason? If you're littering your game code with try/catch everywhere, you are doing something wrong. Not to say you don't have any, but there are other error handling and return methods or you just let things bubble-up and handle them there. What would you actually do in most try/catch situations deep inside a game loop outside of things where resources can leak horribly like IO?
When you're working on a game, there are of course areas you can compromise. It also as I mentioned depends on the nature of your game. I'm speaking strictly of properly designed, professional games that have to be competitive in terms of features and run at a smooth frame rate while doing many things (i.e. your average AAA console game).
In case you still doubt what I am saying, go look at GDC talks, game dev papers, etc. and you'll see the same thing. As far as someone people around here might be more familiar with as a source, I seem to recall there were multiple times Jonathan Blow mentioning exactly what I did (maybe a talk he gave at Berkley?). Anyway, the point is that as programmers we learn all these cool and fascinating things, but in game dev, you tend to have to let go of "cool" things and spin your thinking a bit towards things like predictability, performance, stability, transparency, and composeability.
No, this couldn't be further from the truth and is the reason why things like entity component systems exist - to graft data on to code and make data-driven programming easier. In Lisp-like languages, this is more true. In C++ which most games are written in and even Lua which is a common scripting language, this isn't really true. There is a wealth of information about how code is not data all over the web - google it please.
A few reasons why in C++, C, and Lua, and in general in most game programming languages code is not the same as data:
- Tends not to be editor friendly
- Requires heavy meta programming to achieve the same as other formats
- Requires compilation and/or interpretation
- Is not necessarily idempotent depending on compiler options and platform. The same data should be the same cross-platform, and code is almost definitely not because at a minimum level, it can generate vastly different assembly depending on compiler flags or processor targets.
- Purely composite. I would argue that as much as we try, it's hard to compose a lot of code properly and without considerable effort in games. Conversely, composing data can be quite easy.
- Cannot on its own be serialized/deserialized easy while in a runtime state
- Not network transparent
- Cannot be compressed/decompressed in real-time as easily
See my points in other posts in this thread. I am assuming you downvoted me, which is ridiculous and childish, especially given your brief response without any reasoning or evidence despite being contrary to all computer science research.
Speed is not always a secondary concern for dialog. Certainly in many games where the dialog is very fixed, sure. However more and more, dialog systems actually need to be pretty fast. The simplest reason is you generally have only a few or even 1 frame to process dialog and you certainly don't want it tying up your frame's update cycle. It's easy enough most of the time as a place where you can shave off time with some good optimizations anyway for simpler cases.
An example I see a lot where it needs to happen fast is spatial queries or AI-related dialog. For instance if I walk by someone or something, that has to trigger a query on the dialog engine for someone to say something to the player. It's not just the dialog, but the sound assets, playback, etc. that need to happen and they need to happen in a way that isn't annoying - i.e. player is far away already before dialog starts or has to "wait" in place.
Sometimes as I mentioned, AI also gets thrown into the mix in different ways. The simplest is that you want some piece of dialog to be triggered by some other game states - ex: if player killed dragon, say this, otherwise say that. It seems simple, but when you start throwing in lots of world states like in open world games and want some decent variation and natural dialog that isn't awful and repetitive, it gets harder. Now add on spatial and other requirements for a dialog query and things start to suck. To this end, it's why a lot of dialog systems suck. It's not just laziness, rather sometimes people actually can't get their dialog systems to work fast enough during update cycles and end up dropping features because of the cost vs. benefit for most games.
Dialog seems simple, but is surprisingly complicated in the greater ecosystem of other game systems. But again, this depends on the game and if you just have a side-scroller where you repeat the same 5 catch phrases, then yeah, dialog is simple and speed is not a problem.
Agreed. I've seen things like this tried in Smalltalk images and it was usually a disaster long-term, meaning that the state was simply thrown out and a new image spun up because you're SOL. Either that, or you need tons of logic to handle the old program state and it becomes a convoluted mess.
Most of the games and engines I've ever worked on that have dialog systems expressly avoid encoding things into the program state as much as reasonably possible. This is not only for restoring state, but also making for a saner experience editing and changing things during development. Very rarely does the format you encode things like dialog stay the same during a dev cycle unless you're using a canned engine and don't make any serious changes.
Using Lua program state itself seems like a way to drive yourself mad by the end of the dev cycle, and make it incredible unfriendly to work with writers and other content creators. People who work with asset pipelines often do a lot of cruel and unusual data conversions and munging to bridge how different people and systems need to work, but eventually things in the actual game engine are best kept as simple data that can be read in and reproduced at any time. Moreover, as a developer, I should be able to load up any state of my game at any point in the game just by providing the necessary input data as possible, and like a pure function, it should produce the same game state every time if possible. Let's also not forget other things like internationalization and localization.
As far as interfacing with Lua, I've mostly done it in the last few decades in a simple way where we handed entity component-style data that is just pure data, i.e. no function pointers and crazy stuff, and either let Lua do the rest with that state, or send pure data back to the C++ when needed. As for the data format itself, I've seen everything from XML to custom binary formats to CSV and Excel used for dialog. I think I'd sooner murder someone if game dialog lived in code, even in a Lua script.
I can see how it might be fun to do something like this for a blog post or simple project, but in a real game this just doesn't work for many reasons, most listed here already.
There are tons of ways of doing dialog system. Mostly it comes down to your specific game and I highly recommend against generic dialog systems. Rather, the most you can do at a generic level is try to come up with a good way to store dialog and retrieve it in the ways you need. This isn't the best thing I've seen or worked with, but here's an easy to understand and documented method and presentation about it form Valve:
Anyway, I've been programming games and game engines for several decades, and coroutines are always in the list of things that you can throw in the junk pile along with other cool things like continuations, various theoretical data structures, fancy dispatch mechanisms, events/signals, application-wide functional purity, etc. Of course all of these things are useful, but they tend to have problems that make them of limited use in professional game development.
Here's a brief checklist of show-stoppers off the top of my head I tend to go through before introducing anything "creative" into a game, and especially a game engine. Most of the aforementioned items fail one or more of these.
- Performs awful in common cases or when applied to actual game-like conditions vs. theoretical
- Murders cache lines
- Hard or impossible to debug
- Hard to save, load, serialize, deserialize, or snapshot
- Doesn't work over networks
- Doesn't play nice with libraries or data coming from other libraries or languages (ex: 3rd party physics lib)
- Unpredictable execution start or end times
- Non-determinant memory usage or allocation patterns
- Risk of stack-overflows
- Non-portable or problems on specific target architectures
- Requires "religion" meaning it infects all your code or forces you to write all of your code a certain way, everywhere
- Fragments memory
- Long blocking or uninterruptable processing
Conversely, besides the obvious opposites of most of the above, I look for the following when introducing some major data structure or conceptual item like coroutines to a game or game engine:
- Leads to predictable, consistent code and resource usage
- Data-driven
- Editor friendly (as a corollary to the above)
- Clear debugging story
- Both simple and easy if possible. No, they are not the same.
- Plays well with the world around it
- Portable
- Fast
- Loved by the CPU, GPU, and/or compiler, and produces reasonable assembly on target hardware
- Easy for someone else to jump in and understand the flow and how it works, assuming they are not a moron.
Pretty much these two lists mean that games tend to be made of meat and potatoes things. Simple data structures like arrays, custom allocators, predictable branching and execution patterns, up-front memory allocation, and CPU and GPU loving goodness. Pretty much anything stashing things away and building up massive states or jumping around all over the place, pointer chasing, and so-on should always be a no-go.
Anyway, if you're building a truly simple game, as in a 2 day sort of affair, do whatever you want. If you want to build something even remotely complex that you will work on for awhile, it's best to do things the right way which means detach yourself from everything you think you know about most of computer science and think in terms of pipelines, CPU instructions, caches, and predictability among other things. This often means programming against the grain of your language a bit, throwing out sugar, std libraries, and all kinds of things. It really sucks, but that's the essence of good game programming. I hope one day it will be different, but as long as there's still need to push gameplay and graphical boundaries, professional developers almost always need to squeeze out what they can.
I really like Rust and use it for major projects, but I fail to see how it's much on topic. I agree with other comments that it gets compared to Go all the time, often without prompting or relationship to articles. Want to do that, why not write an article or do a separate post? Hijacking the comments and/or shilling Rust is getting old.
I have to say that although I love Rust and somewhat despise Go, if it works for you, use it. Go is good at a lot of things - identify what they are and use it because it solves your problem. If another language does that better, use that instead. If you don't know what those things are, you probably need to spend some time not only reading, but writing actual code in that language.
Don't pick one language to solve all your problems and don't look for problems because you like a language or some specific features of a language. Consider the entire scope of your issues, both technical and non-technical. In that sense, I can see why a lot of people choose Go and it is just as valid as any other language choice.