Re serialization: We have custom binary serialization that essentially dumps the game state into a binary stream. No allocations, no copies, no conversions. Our saves can be big, >100 MB uncompressed, so there is no room for waste.
The big advantage is that it reads data directly from game's classes, so there is no boilerplate needed, no prorobufs, no schema. And it supports versioning, adding or removing members mostly without limitations.
I think it's a cool system, maybe I should write a blog post about it :)
Sure, we could use Burst to speed up some strategic parts, but that would not help with the core of the game.
To give some context, things are very complex in our game, we have fully dynamic terrain with terrain physics (land-slides), advanced path-finding of hundreds of vehicles (each entity has its own width and height clearance), trains, conveyors and pipes carrying tens or even hundreds of thousands of individual products, machines, rockets, ships, automated logistics, etc. There is no one thing that could be bursted to get 3x gain. At this point, we'd have to rewrite the entire game in C++.
So what's the reason we use C#? Productivity, ease of debugging and testing, and resilience to bugs (e.g. null dereference won't kill the program). Messing with C++ or even burst would cost us more time and to be honest, the game would possibly not even exist at that point.
Could you share some details about your custom thread pool that got 3x speedup? What was the speedup from? It is highly unlikely that a custom thread pool would have any significant impact on the benchmark in our case. As you can see from Figure 3, threaded tasks run for about 25% of the total time and even with Mono, all tasks are reasonably well balanced between threads. Threads utilization is surely over 90% (there is always slight inefficiency towards the end as threads are finishing up, but that's 100's of ms). An "oracle" thread pool could speed tings up by 10% of 25%, so that is not it.
Vectorization could help too but majority of the code is not easily vectorizable. It's all kinds of workloads, loading data, deserialization, initialization of entities, map generation, precomputation of various things. I highly doubt that automatic vectorization from code generated by IL2CPP would bring more than 20% speedup here. The speedup from burst would mostly come from elimination of inefficient code generated by Mono's JIT, not from vectorization.
For now, we are accepting the Mono tax to be more productive. But I am hoping that Unity will deliver on the CoreCLR dream. In the meantime, my post was meant raise awareness and stir up some discussion, like this one, which is great. I've read lots of interesting thoughts in this comments section.
> Going to completely disagree that Burst and HPC# are unnecessary and messy.
Making a managed code burst-compatible comes with real constraints that go beyond "write performant C#". In Burstable code, you generally can't interact with managed objects/GC-dependent APIs, so the design is pushed towards unmanaged structs in native collections. And this design spreads. The more logic is to be covered by Burst, the more things has to be broken down to native containers of unmanaged structs.
I agree that designing things in data-oriented way is good, but why to force this additional boundary and special types on devs instead of just letting them write it in C#? Writing burstable code can increase complexity, one has to manage memory/lifetimes, data layout, and job-friendly boundaries, copying data between native and managed collections, etc., not just "writing fast C#".
In a complex simulation game, my experience is that there are definitely things that fit the "raw data, batch processing" model, but not all gameplay/simulation logic does. Things like inheritance, events, graphs, AI (the dumb "game" version, no NN), UI, exceptions, etc. And on top of it all, debugging complications.
Wouldn't you be relieved with announcement: "C# is now as fast as Burst, have fun!"? You'd be able to do the same data-oriented design where necessary, but keep all the other tings handy standing by when needed. It's so close, yet, so far!
> The fragmentation you mention
What you say makes sense. I've actually spent a lot of time debugging this and I did find some "leaks" where references to "dead objects" were keeping them from being GC'd. But after sorting all these out, Unity's memory profiler was showing that "Empty Heap Space" was the culprit, that one kept increasing after every iteration. My running theory is that the heap is just more and more fragmented, and some static objects randomly scattered around it are keeping it from being shrunk. ¯\_(ツ)_/¯
> You can also install new DLLs and force the player to restart if you want downloadable mod support.
I am not aware of an easy way to load (managed) mods as DLLs to IL2CPP-compiled game. I am thinking about `Assembly.LoadFrom("Mod.dll")`.
Can you elaborate how this is done?
> there are better, more performant alternatives for that use case, in my experience.
We actually use reflection to emit optimal code for generic serializers that avoid boxing and increase performance.
There may be alternatives, we explored things like FlatBuffers and their variants, but nothing came close to our system in terms of ease of use, versioning support, and performance.
If you have some suggestions, I'd be interested to see what options are out there for C#.
> FieldOffset is supported by IL2CPP at compile time
You are right, I miss-remembered this one, you cannot get it via reflection, but it works.
Author here, thanks for your perspective. Here some thoughts:
> approach of separating the simulation and presentation layers isn't all that uncommon
I agree that some level of separation is is not that uncommon, but games usually depend on things from their respective engine, especially on things like datatypes (e.g. Vector3) or math libraries. The reason I mention that our game is unique in this way is that its non-rendering code does not depend on any Unity types or DLLs. And I think that is quite uncommon, especially for a game made in Unity.
> Most games don't ship on the mono backend, but instead on il2cpp
I think this really depends. If we take absolute numbers, roughly 20% of Unity games on Steam use IL2CPP [1]. Of course many simple games won't be using it so the sample is skewed is we want to measure "how many players play games with IL2CPP tech". But there are still many and higher perf of managed code would certainly have an impact.
We don't use IL2CPP because we use many features that are not compatible with it. For example DLC and mods loading at runtime via DLLs, reflection for custom serialization, things like [FieldOffset] for efficient struct packing and for GPU communication, etc.
Also, having managed code makes the game "hackabe". Some modders use IL injection to be able to hook to places where our APIs don't allow. This is good and bad, but so far this allowed modders to progress faster than we expected so it's a net positive.
> In modern Unity, if you want to achieve performance, you'd be better off taking the approach of utilizing the burst compiler and HPC#
Yeah, and I really wish we would not need to do that. Burst and HPC# are messy and add a lot of unnecessary complexity and artificial limitations.
The thing is, if Mono and .NET were both equally "slow", then sure, let's do some HPC# tricks to get high performance, but it is not! Modern .NET is fast, but Unity devs cannot take advantage of it, which is frustrating.
By the way, the final trace with parallel workers was just C#'s workers threads and thread pool.
> Profiling the editor is always a fools errand
Maybe, but we (devs) spend 99% of our time in the editor. And perf gains from editor usually translate to the Release build with very similar percentage gains (I know this is generally not true, but in my experience it is). We have done many significant optimizations before and measurements from the editor were always useful indicator.
What is not very useful is Unity's profiler, especially with "deep profile" enabled. It adds constant cost per method, highly exaggerating cost of small methods. So we have our own tracing system that does not do this.
> I've seen a lot of mention around GC through this comment section, and professional Unity projects tend to go out of their way to minimize these at runtime
Yes, minimizing allocations is key, but there are many cases where they are hard to avoid. Things like strings processing for UI generates a lot of garbage every frame. And there are APIs that simply don't have an allocation-free options. CoreCLR would allow to further cut down on allocations and have better APIs available.
Just the fact that the current GC is non-moving means that the memory consumption goes up over time due to fragmentation. We have had numerous reports of "memory" leaks where players report that after periodic load/quit-to-menu loops, memory consumption goes up over time.
Even if we got fast CoreCLR C# code execution, these issues would prevail, so improved CG would be the next on the list.
From my experience, performance gains seen in Debug builds in Unity/C#/Mono nearly always translate in gains in Release mode. I know that this is not always true, but in this context that's my experience.
Setting up release benchmarks is much more complex and we develop the game in Debug mode, so it is very natural to get the first results there, and if promising, validate them in Release.
Also, since our team works in Debug mode, even gains that only speed things up in Debug mode are valuable for us, but I haven't encountered a case where I would see 20%+ perf gain in Debug mode that would not translate to Release mode.
MaFi Games | Senior ASP.NET (full-stack) | Contract | Remote
Hi, I'm the co-founder of MaFi Games, the indie studio behind Captain of Industry. We're looking for an experienced full-stack ASP.NET engineer to help us grow our community website, including features like a modding database, blog, and forum.
Some reasons you'd enjoy working with us:
- A multicultural, collaborative, and innovative work environment where your voice is heard.
- Fully remote job with flexible working hours and vacation schedule.
- High quality C# code base, code reviews, tests.
- High work satisfaction, work with a talented team on a popular video game with a wonderful community.
MaFi Games | Senior SWE/Game dev | Contract preferred | Remote | C#
Hi, I’m the co-founder of MaFi Games – an indie studio behind the game Captain of Industry. We are a small but passionate team who gave up their jobs at Google and Nvidia to pursue building the best factory simulation game possible, and we need more hands!
We are looking for an experienced software engineer to grow the team and accelerate our progress. We strongly prefer candidates with a background in game development, experience with 3D graphics, and performance optimizations.
Some reasons you’d enjoy working with us:
* A multicultural, collaborative, and innovative work environment where your voice is heard.
* Fully remote job with flexible working hours and vacation schedule.
* Long-term full-time collaboration (not a fixed-term contract).
* High quality C# code base, code reviews, tests.
* High work satisfaction, work on a popular video game with a wonderful community.
Thanks! We use custom engine written in C# for simulation and Unity 3D for rendering. The game simulation can run without Unity.
Our "team" is tiny, it's just two of us. We've been working on it past 6 years as a hobby project over the weekends. Last year we decided to quit our tech jobs and focus on this full-time. We work with freelancers who do 3D art and music for us.
That's cool! I was actually thinking of something similar, a game mode where you could take a control over a vehicle, like an excavator, and just have fun with it in the game, mine, dump, call trucks, etc.
This could be even cooler in multiplayer where many people could be controlling different parts of the factory, different vehicles, etc. But this sounds like a little too much work to get it "right".
Thank you! That trailer is actually nearly 1 year old now, it's from the pre-alpha stage of the game. Unfortunately, we had no time to update the trailer with all the new things. We are way to busy with all the release preparations. Last time it took us around 10 days (2 people full time) to make it.
While being more realistic than other games, I'd say it is not realistic enough to simulate real open-pit mining. You would not recommend to plan real mining operations based on the results from the game.
For example, the way how terrain collapses during mining is balanced to make a fun game rather than trying to be super realistic. We don't take into account weather effects (esp. rain). Also, in reality, hard rock needs to be blasted, but we don't have this feature (yet). Refilling of vehicles is mostly automatic, given that they have fuel available somewhere reachable. Etc...
On the other had, similar to other sim games, you will certainly need to think and plan your mine/factory well in order to be successful.
I love factory simulation games (think Factorio, Satisfactory, Banished, DSP, ...) and one thing that I was always missing was a game with better simulation of raw material mining. Most simulation games have you just place a "mine" on a resource and that's it.
I wanted to manage an open pit mine myself. Have excavators that mine ore and trucks move it for processing, but as they do, the shape of the terrain changes, leaving deep holes behind. Maybe even compromising your factory as the mining operation expands.
And as it sometimes go, when you want something and that does not exist, you try to make it, and that was my case here. Together with a fried we attempted to make such game. It's called Captain of Industry in case anyone is interested: https://store.steampowered.com/app/1594320/Captain_of_Indust...
Can you share how large is the team responsible for .NET Modernization?
> migrating legacy native C++ code to C# backed by CoreCLR.
Yes please! Surely things like Quaternion.Lerp don't have to be C++ code under CoreCLR.
Feel free to get in touch in case I could be of any help :)