Automated testing in Bevy(chadnauseam.com)
chadnauseam.com
Automated testing in Bevy
https://chadnauseam.com/coding/gamedev/automated-testing-in-bevy/
86 comments
Riot Games also has an automated testing framework: https://technology.riotgames.com/news/automated-testing-leag...
I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
What a lot of people don’t realize about QA is that it’s also a point of Empathy to your end users. Some might think it’s the PMs job to think as an end user but in my experience PMs are so bogged down with other sales adjacent duties that they rarely if ever can provide that pillar.
This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
Recently Inherited a code base with almost 20000 tests. Many very detailed. Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.
> Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.
But what's the alternative here, when your code depends on other services and other logic that might return complex data structures, or need to do dozens of database calls?
Should you go for primarily integration tests, which might not only slow everything down, but also force you to think about the possibility of either needing to mirror your current environment per test run (e.g. launch a database, run all the migrations, generate needed test data, launch your app against the database instance, run tests against it, clean everything up), or being limited to a single test run at a time with no parallelism?
But what's the alternative here, when your code depends on other services and other logic that might return complex data structures, or need to do dozens of database calls?
Should you go for primarily integration tests, which might not only slow everything down, but also force you to think about the possibility of either needing to mirror your current environment per test run (e.g. launch a database, run all the migrations, generate needed test data, launch your app against the database instance, run tests against it, clean everything up), or being limited to a single test run at a time with no parallelism?
It depends on the codebase I think, some codebases are spaghetti mess that really benefit from being well integration tested. Other codebases are really well organized, atomic and have really clearly separated concerns. I think you can afford to run smaller less integrated unit tests. I don't think there's a right or wrong answer, just a balance of the two depending on your project.
Wonder what their coverage is in this - or if they even still use it. There are many occurrences of "how does this ship" bugs in League where damage numbers don't add up or status's have no actual effect, etc - sometimes hanging around for many patches. Not that I don't understand how hard it is to test the big ball of mud that (10+ years old!) game engines end up being.
I strongly agree with this. You need someone around that actually uses what you develop, be it dogfooding or structured QA.
I wonder if there is any books on game/software QA akin to Masters of Doom or Stay Awhile and Listen? Those pretty much skip over QA, it's usually just the devs playing their own game (which anecdotally is usually biased and of limited use).
I wonder if there is any books on game/software QA akin to Masters of Doom or Stay Awhile and Listen? Those pretty much skip over QA, it's usually just the devs playing their own game (which anecdotally is usually biased and of limited use).
Here's also a good overview how Croteam (creators of Serious Sam and The Talos Principle) approach automated testing: https://m.youtube.com/watch?v=YGIvWT-NBHk&feature=emb_logo
The talk is from 2018 and references some of the trends in the industry towards more automated testing.
The talk is from 2018 and references some of the trends in the industry towards more automated testing.
Fantastic games. Also, multiplatform sounds like a nightmare to manage.
The industry is changing towards more automated testing, I heard of a lot of AAA studios implementing/creating testing frameworks. It was very uncommon back in the days, they just threw a bunch of badly paid and overworked QAs at new builds. Games are very complex, covering a wide range of behaviours in environments with a lot of randomness is difficult. With automation, the predictable behaviours (for example, testing that climbing a ladder works) can be offloaded from QA and they can focus on finding more random issues with playthroughs. I guess there was also a push from engineers without a gaming background (and being more used to TDD) that have been recruited in the industry throughout the years.
Based on many of the comments here, it seems like there’s a lot of misunderstanding of what an ECS is - unfortunately the name does this no favors as it should probably be written as entity/component/system paradigm or renamed entirely. It’s not just a system that contains entities and components (that’s not what “system” refers to in the ECS name).
ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries.
I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS.
[0]: https://learn.unity.com/tutorial/entity-component-system
ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries.
I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS.
[0]: https://learn.unity.com/tutorial/entity-component-system
This Rust conference talk, of all things, is still my favorite ECS explanation, because she builds the concept piece by piece starting from megaclass-hell to full entities-as-IDs with reasons for each step. Also she's a pretty entertaining speaker IMO. https://youtu.be/aKLntZcp27M
This is also one of my favorite talks, because it showed me that Rust is mostly recommended by people who don't know what they're talking about made me not waste any time on it (besides the horrendous compile times in real projects even compared to cpp).
For example, she spends a significant amount of time there explaining how to implement her component placement strategy while working around the borrow checker and at the end proclaiming how great it now is compared to a cpp implementation...
...not realizing that what she's just done is implement a custom allocator (not literally allocating memory, but the operations are the same: give me space for a component, free it, memory is latter accessed by an address instead of being some opaque thing hidden behind language semantics, etc) that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic.
For example, she spends a significant amount of time there explaining how to implement her component placement strategy while working around the borrow checker and at the end proclaiming how great it now is compared to a cpp implementation...
...not realizing that what she's just done is implement a custom allocator (not literally allocating memory, but the operations are the same: give me space for a component, free it, memory is latter accessed by an address instead of being some opaque thing hidden behind language semantics, etc) that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic.
Even assuming you're right about the content of this talk,
> showed me that Rust is mostly recommended by people who don't know what they're talking about
You do realize that this is, like, colossally, stereotypically bad reasoning, right? It's ad hominem and improper generalization simultaneously, which is really quite impressive and indicates a singular degree of motivated reasoning.
Yes, lots of Rust evangelists are idiots (not Catherine West, u/spoiler did a decent job showing that). No, that doesn't significantly affect the technical properties of the language that make it useful or worth learning about. There are good reasons to ignore Rust, yours is not one of them.
> showed me that Rust is mostly recommended by people who don't know what they're talking about
You do realize that this is, like, colossally, stereotypically bad reasoning, right? It's ad hominem and improper generalization simultaneously, which is really quite impressive and indicates a singular degree of motivated reasoning.
Yes, lots of Rust evangelists are idiots (not Catherine West, u/spoiler did a decent job showing that). No, that doesn't significantly affect the technical properties of the language that make it useful or worth learning about. There are good reasons to ignore Rust, yours is not one of them.
Wow... Okay. I see you didn't even bother watching the video in full.
You somehow managed to ignore all the points of the talk. Like literally every one of them, and I'm not even exaggerating. And then you somehow both managed to focused on irrelevant details and to talk out of your ass (like, at least be right about the details?), all to make a tangential point about how "sUpErIoRlY sMaRt CpP dEvs ArE" because they know this really basic shit.
But, I'll respond in case your misinformed comment manages to deter people from watching it. Let's start.
First you talk about "borrow checker issues" she's demonstrating in the OOO design. The point here was to illustrate the amount of coupling and "explosion" in size with this approach. The point was "OOO is probably not a good design paradigm for game dev". Then she continues how Rust surfaces this bad design paradigm much earlier than it would've happened in other languages/ecosystems. That's all.
More, what you call a custom allocator with tombstoning/generations are just things that exist with a lot of ECS implementations (regardless of language), and it's just become "tribal wisdom" at this point that this table-esque storage backend work really well with ECS (I dunno which one came first; older game devs might know more about ECS history/evolution than I do).[1]
Also not sure what you mean by "that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic"... Maybe I misunderstand what you mean, but it sounds to me like you're comparing working with `Option` as being the same failure mode as working with `void*`. That's just utterly ludicrous nonsense.
Her ECS implementation is... Well, it was made to fit inside a few slides for a 40-minute talk. So, yeah it's not great; there's plenty of small issues with it, but none of them are with Rust, or with ECS itself... Did you expect a production ready ECS library you can copy off of slides and use them in your next game, lol? If anything, her implementation has too much passing resemblance with what a C++ implementation would look like (which is something she's very familiar with and her starting point) than a Rust one. So, I'm just bewildered why this is the bone you decided to pick.
If you're interested in what a production-ish ECS implementation in Rust looks like, check out Amethyst's Legion or Bevy's ECS. Although, Bevy takes takes ECS a few... staircases (rather than steps) further with its own ECS. I really encourage everyone to read up on Bevy's ECS and check out the unofficial Bevy cheatsheet book; I'm certain at leat UE game devs will know how to appreciate how beautiful and ergonomic the design is, but others should as well). Aside: her ECS implementation is still safer than most C/C++ implementations used in published games, but that's a bit besides the point since that's the borrow checker doing its job.
Anyway, the irony is that she much better explains the pitfalls of her own implementation details that you do and she also explains why they're "fine" sometimes. And it was kinda implied that it's fine (from what I remember) because it's no worse than what you end up doing in C++ or Java when implementing ECS, but at least it's safe in Rust. She acknowledges that there's probably better ways to do it.
Here's a few links on some of the stuff I mentioned:
- https://github.com/amethyst/legion
- https://bevyengine.org/learn/book/getting-started/ecs/
Edit: formatting
[1] Edit 2: I kinda phrased this poorly. I'm not saying the best way to implement tables or just ECS is this approach, but it's common enough approach used to avoid memory bandwidth/allocations in games.
You somehow managed to ignore all the points of the talk. Like literally every one of them, and I'm not even exaggerating. And then you somehow both managed to focused on irrelevant details and to talk out of your ass (like, at least be right about the details?), all to make a tangential point about how "sUpErIoRlY sMaRt CpP dEvs ArE" because they know this really basic shit.
But, I'll respond in case your misinformed comment manages to deter people from watching it. Let's start.
First you talk about "borrow checker issues" she's demonstrating in the OOO design. The point here was to illustrate the amount of coupling and "explosion" in size with this approach. The point was "OOO is probably not a good design paradigm for game dev". Then she continues how Rust surfaces this bad design paradigm much earlier than it would've happened in other languages/ecosystems. That's all.
More, what you call a custom allocator with tombstoning/generations are just things that exist with a lot of ECS implementations (regardless of language), and it's just become "tribal wisdom" at this point that this table-esque storage backend work really well with ECS (I dunno which one came first; older game devs might know more about ECS history/evolution than I do).[1]
Also not sure what you mean by "that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic"... Maybe I misunderstand what you mean, but it sounds to me like you're comparing working with `Option` as being the same failure mode as working with `void*`. That's just utterly ludicrous nonsense.
Her ECS implementation is... Well, it was made to fit inside a few slides for a 40-minute talk. So, yeah it's not great; there's plenty of small issues with it, but none of them are with Rust, or with ECS itself... Did you expect a production ready ECS library you can copy off of slides and use them in your next game, lol? If anything, her implementation has too much passing resemblance with what a C++ implementation would look like (which is something she's very familiar with and her starting point) than a Rust one. So, I'm just bewildered why this is the bone you decided to pick.
If you're interested in what a production-ish ECS implementation in Rust looks like, check out Amethyst's Legion or Bevy's ECS. Although, Bevy takes takes ECS a few... staircases (rather than steps) further with its own ECS. I really encourage everyone to read up on Bevy's ECS and check out the unofficial Bevy cheatsheet book; I'm certain at leat UE game devs will know how to appreciate how beautiful and ergonomic the design is, but others should as well). Aside: her ECS implementation is still safer than most C/C++ implementations used in published games, but that's a bit besides the point since that's the borrow checker doing its job.
Anyway, the irony is that she much better explains the pitfalls of her own implementation details that you do and she also explains why they're "fine" sometimes. And it was kinda implied that it's fine (from what I remember) because it's no worse than what you end up doing in C++ or Java when implementing ECS, but at least it's safe in Rust. She acknowledges that there's probably better ways to do it.
Here's a few links on some of the stuff I mentioned:
- https://github.com/amethyst/legion
- https://bevyengine.org/learn/book/getting-started/ecs/
Edit: formatting
[1] Edit 2: I kinda phrased this poorly. I'm not saying the best way to implement tables or just ECS is this approach, but it's common enough approach used to avoid memory bandwidth/allocations in games.
She also wrote a long-form essay which covers the same material but goes into much more detail: https://kyren.github.io/2018/09/14/rustconf-talk.html
It gets particularly confusing because the standard Unity paradigm is still one of entities and components, as I believe was first promoted by Dungeon Siege at GDC 2002[0]. But it's not quite the same as what is called an ECS.
[0] https://www.gamedevs.org/uploads/data-driven-game-object-sys...
[0] https://www.gamedevs.org/uploads/data-driven-game-object-sys...
We could all probably benefit from some slightly better naming, so that EC and ECS don't refer to almost completely different things. Otherwise you get some of the confusion as Java and JavaScript might have had back in the day. If the concepts are so different, they should be named more distinctly.
I think maybe the problem with explaining ECS is that the database-like part you mention is just given a passing mention, rather than being a critical part of the explanation. Obviously ECS is not a database which I think is why people avoid it, but it really helps for understanding it.
Entities are basically tables (just the table, not the data in the table), components are fields/data in them and systems are the SQL you write for them. Its obviously not a perfect representation, especially for components, but it gets the rough idea out there.
Maybe CSE is a better name for it. Its a longshot, but maybe entity being first in the list tricks people into thinking they play a big part, whereas actually they are, as you say, just an ID. Components are probably most important, then systems, then entities
Entities are basically tables (just the table, not the data in the table), components are fields/data in them and systems are the SQL you write for them. Its obviously not a perfect representation, especially for components, but it gets the rough idea out there.
Maybe CSE is a better name for it. Its a longshot, but maybe entity being first in the list tricks people into thinking they play a big part, whereas actually they are, as you say, just an ID. Components are probably most important, then systems, then entities
I would say Systems are Tables, Components are rows, and Entities are Primary Keys.
Entities as primary keys is very good! Though as systems generally handle the data transformation, Im not sure they equate to tables
IMO most analogies kinda fall short, but kinda makes sense in my head is:
Entity as ID; Components as Columns, but only kinda; Systems as Application code or functions
Tables in this database/ECS analogy are just tables here too, or memory rather. The data is often conceptually (and implementation wise) laid out as tables in ECS, too
Entity as ID; Components as Columns, but only kinda; Systems as Application code or functions
Tables in this database/ECS analogy are just tables here too, or memory rather. The data is often conceptually (and implementation wise) laid out as tables in ECS, too
So your saying everything is a entity, and that entities have components, and the systems operate on entities (possibly based on their components). Do I understand that correctly? That understanding is definitely worth the time it took to read.
> is a entity, and that entities have components, and the systems operate on entities (possibly based on their components)
In an object-based design, objects are monoliths, with fields. You get a reference to player, and each Player instance has attributes. When you work on a player, you get the Player instances as a whole.
In an ECS design, you'd create a player entity; the main differences are that the Entity is just a reference, but most importantly, you access each field (called "component") separately.
The bulk of querying is performed on components, which, importantly are all global; entities are typically retrieved starting from a component instance they own. This makes the radical difference with a traditional object-based design; example:
- in OB, you have, say, players and enemies; when you draw, you'll cycle the players, and enemies, and draw them.
- in ECS instead, you'll query something like "drawable sprite" components, and draw them
It's a very interesting design, as it forces to reason more generically. It tends to get messy very easily though, since it's harder to think in terms of location of a system (in the project).
The best book for beginners to understand ECS is (IMO) Hands-on Rust (https://pragprog.com/titles/hwrust/hands-on-rust).
> So your saying everything is a entity
Not sure about other ECSs, but Bevy has also the concept of "resources", which are globals; they're monolithic, and there's only one isntance of each type.
In an object-based design, objects are monoliths, with fields. You get a reference to player, and each Player instance has attributes. When you work on a player, you get the Player instances as a whole.
In an ECS design, you'd create a player entity; the main differences are that the Entity is just a reference, but most importantly, you access each field (called "component") separately.
The bulk of querying is performed on components, which, importantly are all global; entities are typically retrieved starting from a component instance they own. This makes the radical difference with a traditional object-based design; example:
- in OB, you have, say, players and enemies; when you draw, you'll cycle the players, and enemies, and draw them.
- in ECS instead, you'll query something like "drawable sprite" components, and draw them
It's a very interesting design, as it forces to reason more generically. It tends to get messy very easily though, since it's harder to think in terms of location of a system (in the project).
The best book for beginners to understand ECS is (IMO) Hands-on Rust (https://pragprog.com/titles/hwrust/hands-on-rust).
> So your saying everything is a entity
Not sure about other ECSs, but Bevy has also the concept of "resources", which are globals; they're monolithic, and there's only one isntance of each type.
Ah, so what we used to call a database in third normal form, before it was rediscovered and given a different name so it could generate revenue. Of course, the term database has also been redefined to refer to commercial RDBMS products, but it does make it hard to keep up with the underlying technology whenever a new set of salepeople come along looking to make commission.
Sarcasm is not actually correct. An ECS is not an RDBMS, because in the latter is based on row-based storage, while the former is column-based.
The storage part of ECSs could actually be considered a form of columnar db.
For example, locking is at column level, so you can have as many mutable accesses to a single entity (like a row) as there are columns (in reality, it's more complicated, due to access via iterators, but that's the principle).
The storage part of ECSs could actually be considered a form of columnar db.
For example, locking is at column level, so you can have as many mutable accesses to a single entity (like a row) as there are columns (in reality, it's more complicated, due to access via iterators, but that's the principle).
This makes the cache locality excellent and is great for game performance.
It also makes saving and loading state a breeze... you're basically just doing a database dump/restore. Coming across ECS was one of the few "holy shit what" moments of my time as a programmer.
With the caveat that it’s actually quite a lot of work on top of this basic concept to actually make this happen.
That’s my understanding too.
Think of super Mario.
Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns.
Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives.
System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.
Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns.
Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives.
System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.
I would not describe it quite that way.
Functionally, an Entity is an ID. A Component is Data. A System is a function that operates on a set of Components.
In a sense "that's it". Having the hard separation between data (components) and code (functions) leads to some interesting consequences. For example the ability to more easily multi-thread System updates because each System clearly defines what Component type it needs const or mutable access to.
Functionally, an Entity is an ID. A Component is Data. A System is a function that operates on a set of Components.
In a sense "that's it". Having the hard separation between data (components) and code (functions) leads to some interesting consequences. For example the ability to more easily multi-thread System updates because each System clearly defines what Component type it needs const or mutable access to.
I think it's worth noting that when we talk about a system operating on a set of components, it literally means a function which either reads and creates new components or entities or mutates existing components.
Most "things" don't exist in ECS. There's no singular "Mario" or "gravity". If you feel like you ought to split it into multiple identifiers for your systems (for example, treating each part of Mario like a different sprite), then you do so.
The hardest part of ECS IMO, is that dependencies between systems are super loosely defined since it's meant to be modular and the dependencies can only be tracked by components. The semantics of ECS systems are also highly technical (do you have hard sync points, how do you deal with recursive entities etc...) which means it can be quite fiddly to work with.
Most "things" don't exist in ECS. There's no singular "Mario" or "gravity". If you feel like you ought to split it into multiple identifiers for your systems (for example, treating each part of Mario like a different sprite), then you do so.
The hardest part of ECS IMO, is that dependencies between systems are super loosely defined since it's meant to be modular and the dependencies can only be tracked by components. The semantics of ECS systems are also highly technical (do you have hard sync points, how do you deal with recursive entities etc...) which means it can be quite fiddly to work with.
[deleted]
This is essentially correct, yes.
The post contains a common fallacy that you will know of if you have tested substantially sized games:
What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy".
ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.
What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy".
ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.
I think you're reading something that isn't being argued in the post, though. The author is not arguing that you will have no bugs in your game if you use ECS and add tests, it's that testing in this way can make it easier for development because you reduce the possibility of introducing a bug inadvertently in one of your systems and then have to rely on QA to find it later (components too I guess, but I would guess component "tests" can be covered by types since they're just data and the systems are the ones that interact with the data). The kinds of bugs you mention are of course different and some of those kinds of bugs aren't worth the trouble of even trying to test, but if your game is shaped in a way that it is basically a collection of code modifying data that represents the game, you can eliminate the need for a lot of manual QA testing by just automating it.
And it's not just games, this trend of moving code into data. I've been in several companies where their fundamental logic was a configuration interpreter, very complex, tied to a gigantic set of xml or json "configurations" which themselves had interdependent logic written as structure.
So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of.
So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.
So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of.
So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.
> ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.
New startup idea. Probably applies to any "hard to test" thing.
New startup idea. Probably applies to any "hard to test" thing.
I've been looking into Unity and game development in my spare time.
It seems writing tests for game code is incredibly dependent on the style used to develop the game.
If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests.
Using Unity is like a weird mix of no code tools and code.
If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests.
Using Unity is like a weird mix of no code tools and code.
I never understand why people always seem to think 'test flakiness' should/can be fixed by saying "it's OK if test works 1/10 times" -- if it's so dependent on the RNG then something is bound to fail for users as well, so either the test is not representative of the game or something is rotten.
I guess time to fix tests is expensive...
I guess time to fix tests is expensive...
It depends entirely on the reason for that 1/10 times.
E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues.
Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well.
Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.
E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues.
Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well.
Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.
Agreed - at a previous job lots of time was invested in setting up test retries, and splitting test runs out so that you could retrigger only the subsets that failed instead of the whole test suite.
That's a really good point. It would be interesting to run the tests 10 times and have the CI fail if it doesn't pass all 10 times.
> The only big-budget one I know of that has publicly put a large effort into it is Sea of Thieves.
Another big big title that makes use of automated testing is Riot’s League of Legends.
https://technology.riotgames.com/news/automated-testing-leag...
Another big big title that makes use of automated testing is Riot’s League of Legends.
https://technology.riotgames.com/news/automated-testing-leag...
For the last few years I've been trying to add unit tests to all of my personal projects.
I do a bit of game modding in my spare time and those are the last projects that ended up getting unit tests, but they were also the ones I most wanted to add tests for because they're the most complicated and also the most time-consuming to test manually.
The biggest hurdle to start with was that the Visual Studio projects are all set up in such a way that integrating Microsoft's C++ unit test framework either requires you to (probably) recreate the project the right way, or to hack around with scripts to create a working test runner. Setting up Boost tests might be less painful. But once that is done, my next hurdle was mocking out the game engine. Not overly difficult as the engine exposes a nice API but still time consuming. Now that that's done it's been good start adding tests, though I haven't added any tests for anything too complicated yet.
>But in videogames, correctness matters barely at all.
This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position.
Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.
This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position.
Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.
>the benefit of automated testing is that it scales in O(1) with respect to number of code changes, while playtesting is O(n)
This isn't true. Running more tests takes more time even with automated tests.
This isn't true. Running more tests takes more time even with automated tests.
True, although the constant factor can be pretty low depending on how many cores you have :P
I have often thought that GUIs, including browsers, are hard to test because the monstrous frameworks that they are make no accommodation for testing, so... of course they're hard to test. Not only are you writing a test harness around your own code, you also must rather laboriously and unreliably harness up their code too, and from the "outside" not designed to be used the way you're trying to use it to boot. Now, I don't necessarily fault GUI designers for not thinking about how to expose testability easily to their users, they're big enough projects as it is, enough to choke even a major multibillion dollar corporation, but it doesn't mean I'm any less frustrated while using them.
Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable?
I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation.
(I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)
Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable?
I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation.
(I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)
Writing automated tests for video games is some of the most fun programming I do.
> Bevy uses the ECS pattern to organize game code, and it occurred to me that ECS substantially alleviates the spaghetti-code problem that makes testing in Unity and Unreal so terrible.
Both Unreal and Unity uses ECS.
Both Unreal and Unity uses ECS.
This is where the unfortunately-chosen name "Entity-Component-Systems" (ECS) comes back to bite us. Unity's GameObjects are Entity-Component (EC): you have an Entity (GameObject), onto which you attach Components (MonoBehavior). Unreal is similar: you have an Entity (UActor), onto which you attach Components (UActorComponent), though you can also have plenty of code living outside the UActorComponent system in Unreal. Entities are objects, and Components are code/data.
Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data.
Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.
Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data.
Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.
Unreal 5 added true ECS called MASS and its really quite great.
It uses different terminology such as fragments. Traits and others but its incredibly powerful and scalable.
This is a separate system then the ec of actors and components.
It uses different terminology such as fragments. Traits and others but its incredibly powerful and scalable.
This is a separate system then the ec of actors and components.
In unity the methods "Update", "Start" and so on are the system part of ECS. That you write their implementations alongside the data doesn't really matter.
Sorry, you're wrong on this one.
ECS has come to mean a very particular style of architecture. It's not a great name, but it's what the industry has converged on. Naming things is hard.
Unity's MonoBehavior framework uses GameObjects and Components. You don't have to squint hard to see that it contains Entities, Components, and arguably Systems. However ECS has come to mean an architecture that is quite different from MonoBehavior. So different infact that Unity is writing an ECS framework called DOTS!
Both DOTS and MonoBehaviors containing entities, components, and functions that operate on them. However the two Unity systems are extremely different. MonoBehaviors are not an ECS.
ECS has come to mean a very particular style of architecture. It's not a great name, but it's what the industry has converged on. Naming things is hard.
Unity's MonoBehavior framework uses GameObjects and Components. You don't have to squint hard to see that it contains Entities, Components, and arguably Systems. However ECS has come to mean an architecture that is quite different from MonoBehavior. So different infact that Unity is writing an ECS framework called DOTS!
Both DOTS and MonoBehaviors containing entities, components, and functions that operate on them. However the two Unity systems are extremely different. MonoBehaviors are not an ECS.
ECS is fundamentally about having id references and components being attached to entities via such references instead of pointers or inheritance or other clunkier systems. This lets you write systems that can query components and entities. Both Unity and Unreal uses this architecture for their standard workflows.
Then some people started talking about a single implementation style of ECS as if that was all ECS is, Unity DOTS is an example of that, but the base Unity is still ECS at its core.
Then some people started talking about a single implementation style of ECS as if that was all ECS is, Unity DOTS is an example of that, but the base Unity is still ECS at its core.
What you're describing is an Entity-Component (EC) architecture. ECS specifically requires that third noun, Systems.
But yes, everyone admits the names are confusing. But at no point does Unity or Unreal say their architecture is an ECS one.
But yes, everyone admits the names are confusing. But at no point does Unity or Unreal say their architecture is an ECS one.
Strong disagree. I suspect the number of people in the world who agree with your definition is vanishingly small.
Unreal's framework is a mix of inheritance and components. It's clunky AF. AActor I'm looking at you!
Neither Unreal nor Unity provide a mechanism to cleanly query such as "give me all gameobjects in the world contain this set of components". And they definitely don't provide a mechanism to perform that query quickly.
Unity will let you search from components within a GameObject, in children, or in scene. These queries are also slow as fuck and should generally be avoided.
The type of code you write in Unity and in a Bevy-style ECS is quite different. I think ECS is overrated in some ways as it's actually pretty difficult and non-obvious how to accomplish a lot of tasks.
I don't love the name ECS. It's just not a great name. However I've come to terms with it. ECS has colloquially come to mean something that is extremely distinct from MonoBehaviors. You're welcome to keep fighting the community accepted definition if you want. I don't think it's useful or helpful to do so.
Unreal's framework is a mix of inheritance and components. It's clunky AF. AActor I'm looking at you!
Neither Unreal nor Unity provide a mechanism to cleanly query such as "give me all gameobjects in the world contain this set of components". And they definitely don't provide a mechanism to perform that query quickly.
Unity will let you search from components within a GameObject, in children, or in scene. These queries are also slow as fuck and should generally be avoided.
The type of code you write in Unity and in a Bevy-style ECS is quite different. I think ECS is overrated in some ways as it's actually pretty difficult and non-obvious how to accomplish a lot of tasks.
I don't love the name ECS. It's just not a great name. However I've come to terms with it. ECS has colloquially come to mean something that is extremely distinct from MonoBehaviors. You're welcome to keep fighting the community accepted definition if you want. I don't think it's useful or helpful to do so.
The industry has redefined the terms to refer to a system nobody uses instead of the system widely used? That's odd.
No, ECS is a relatively new term. It wasn't used 15 or even 10 years ago. Unity never once used ECS to refer to their MonoBehavior framework.
"Component Systems" were popularized ~20 years ago. I'd consider Unity MonoBehaviors a typical Component implementation. Scott Bilas's 2002 GDC presentation "A Data-Driven Game Object System" helped evangelize the idea.
ECS is a new term that implies a handful of implementation details. The name is somewhat confusing, but it's not redefining a previously popular term.
I believe the first time ECS made an appearance at GDC was Blizzard's stellar 2017 talk "Overwatch Gameplay Architecture and Netcode". The term went mainstream in the years after this.
"Component Systems" were popularized ~20 years ago. I'd consider Unity MonoBehaviors a typical Component implementation. Scott Bilas's 2002 GDC presentation "A Data-Driven Game Object System" helped evangelize the idea.
ECS is a new term that implies a handful of implementation details. The name is somewhat confusing, but it's not redefining a previously popular term.
I believe the first time ECS made an appearance at GDC was Blizzard's stellar 2017 talk "Overwatch Gameplay Architecture and Netcode". The term went mainstream in the years after this.
I'm only a hobbyist game dev, but I've always understood EC separately from ECS.
EC was the answer to God objects and deep, inflexible inheritance hierarchies.
Among other things, ECS is partially a performance play - it's more cache friendly. I'm not sure what all the tradeoffs are though, I've only really used EC.
EC was the answer to God objects and deep, inflexible inheritance hierarchies.
Among other things, ECS is partially a performance play - it's more cache friendly. I'm not sure what all the tradeoffs are though, I've only really used EC.
[deleted]
[deleted]
[deleted]
The major difference between a true ECS paradigm and MonoBehaviours (EC minus S) is that Systems aren’t local, ie you can select and aggregate various entities and change their states based on the results of your aggregations.
A proper system, for example, would be able to add a new enemy entity every 5 seconds and switch the flag that the level is cleared once the total count of present enemies is 0. In fact, it would be quite trivial to implement it. But in Unity you would need some sort of an architectural solution to bypass inherent limitations of EC-S.
A proper system, for example, would be able to add a new enemy entity every 5 seconds and switch the flag that the level is cleared once the total count of present enemies is 0. In fact, it would be quite trivial to implement it. But in Unity you would need some sort of an architectural solution to bypass inherent limitations of EC-S.
Unity at least isn’t a true ECS architecture, more just Entity-Component. You can write game logic in an entity with Unity, whereas in ECS that would have to live in a System
Unity is working on a full ECS system as part of a larger DOTS initiative, but it is still pre-alpha last I knew.
What is ECS? Assuming it's not AWS Elastic Container Service :>
Around 2007 I believe the game industry came up with a semi standard way of building games that somehow alleviated a lot of issues in producing complex games... aka spaghetti loops of hell.
There are a few ECS approach and to this day it's still vague who does it best (depends on your goals).
One I found interesting is the "sparse entity component system".
There are a few ECS approach and to this day it's still vague who does it best (depends on your goals).
One I found interesting is the "sparse entity component system".
it's more to avoid god classes and inheritance messes.
Entity component system
[deleted]
Seems like the author never truly worked with these engines? Also, I never had problem with unit testing on Unity in pure C#. Even when it comes to complicated scenario there is always a way to separate your GameObjects from game logic.
I think this comment and its parent have misunderstood what ECS is.
I have spent way too much time in Unity. (I was a Unity developer for pay for several years.) I suppose I didn't do a good job explaining the differences between what Unity does and an ECS approach in my post, but if you'd like to get a feel for what an ECS you could try out DOTS. (Although DOTS is kindn of unpleasant to use compared to Bevy.)
Thanks, I rolled my eyes and had an issue with the author's statement.
My guess is what he meant was that the Bevy approach is cleaner to test upon.
My guess is what he meant was that the Bevy approach is cleaner to test upon.
Can you point me to what part of Unreal uses an ECS approach? I’ve never seen this. I know Unity DOTS added ECS features, but that still lives in a hybrid world.
This surely isn't what was meant in previous conversation, but Unreal Engine 5 added MassEntity which is uh... an Entity Fragment Processor approach. (Because Components are already an object that exists, it probably would have been an extra special kind of confusing if it adopted typical ECS naming.)
https://docs.unrealengine.com/5.0/en-US/overview-of-mass-ent...
https://docs.unrealengine.com/5.0/en-US/overview-of-mass-ent...
Based on the definitions in this thread, it wouldn't be a stretch to pair Unreal's `AActor` as the entities and `UActorComponent` as the components. You could technically go a step further and pair `UWorldSubsystem` as the system.
I realize that there's the whole "grouping" and "batching" concept in ECS paradigms, but I'm failing to see how that's much different from using the functions in `UWorld` that can query both components and actors of a certain class.
I realize that there's the whole "grouping" and "batching" concept in ECS paradigms, but I'm failing to see how that's much different from using the functions in `UWorld` that can query both components and actors of a certain class.
This kind of reminds me of making UIs in SwiftUI which requires a way to render in the editor so you need to provide test data and I thought it was neat because it sort of acts like a test.
Really enjoyable read!
A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!
A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!
Thank you! It doesn't look like that when you make the window small in Chrome so I didn't catch it.
You can use dev tools to simulate various mobile devices!
https://developer.chrome.com/docs/devtools/device-mode/
https://developer.chrome.com/docs/devtools/device-mode/
https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62