Building multiplayer games with Socket.io and HTML5 Canvas(github.com)
github.com
Building multiplayer games with Socket.io and HTML5 Canvas
https://github.com/sgoedecke/socket-io-game/blob/master/BLOG.md
105 comments
Those are some very good points, but I think some of them are wrong or do not apply for most uses cases.
1) "Do not serialize JSON, or use msgpack or protobuf - write your own protocol." - Why should you write your own protocol insead of protobuf? If your game has more than 10-20 types of messages/objects than it becomes very hard to implement your own binary encoding for each object. Plus with protobuf is easy to use and the packets generated are really smaller (most likely smaller than your own protocol).
2) "Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view." - Profiling is something that you should always do, with both synthetic and real loads. In your case, my guess is that your diffing code was inefficient or that the load was so small that both computing the delta and sending the message were almost instant, so comparing them would give inaccurate results.
3) "Prefer fixed entity speeds." - I think this is a game design decision and you shouldn't limit your game to this if you don't have too. Also, I don't think there's any issue with entities being able to have a different speed each tick.
And, a question. What VPS are you using? Do you recommend any specific VPS for hosting Node.js game servers?
1) "Do not serialize JSON, or use msgpack or protobuf - write your own protocol." - Why should you write your own protocol insead of protobuf? If your game has more than 10-20 types of messages/objects than it becomes very hard to implement your own binary encoding for each object. Plus with protobuf is easy to use and the packets generated are really smaller (most likely smaller than your own protocol).
2) "Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view." - Profiling is something that you should always do, with both synthetic and real loads. In your case, my guess is that your diffing code was inefficient or that the load was so small that both computing the delta and sending the message were almost instant, so comparing them would give inaccurate results.
3) "Prefer fixed entity speeds." - I think this is a game design decision and you shouldn't limit your game to this if you don't have too. Also, I don't think there's any issue with entities being able to have a different speed each tick.
And, a question. What VPS are you using? Do you recommend any specific VPS for hosting Node.js game servers?
1) Demonstrably not so, in my case. Devising and implementing a protocol flexible enough for me was, along with test-driven implementation of that particular aspect, a joy and more performant solution than any other out of the box. Yes, I've tested against msgpack/protobuf (and I use protobuf in production on other projects) and found it to be less optimal, for both implementation (you're never as flexible as with blank board) and runtime resource usage, across board.
2) Sure, I might go the way of the delta one of these days - it does still smell like good design. That's why I was underwhelmed to see it underperform first time around.
3) Yeah, design and implementation are never separate. Neither is content and presentation. :)
DigitalOcean works for me now. Wouldn't mind slightly better CPUs, as even High Compute droplets are not all they're cracked to be.
2) Sure, I might go the way of the delta one of these days - it does still smell like good design. That's why I was underwhelmed to see it underperform first time around.
3) Yeah, design and implementation are never separate. Neither is content and presentation. :)
DigitalOcean works for me now. Wouldn't mind slightly better CPUs, as even High Compute droplets are not all they're cracked to be.
Just curious about your own binary protocol. If your object has an int32 value, but the value could actually be represented with only 12 bits, do you send the entire int value (32 bits) or only the 12 bits needed to reconstruct the number?
I also used DO for a low-tickrate game (2ticks/second), but it doesn't seem like the servers are able to handle multiple game servers running at 60FPS.
If you were to implement the game again, would you rethink your networking so you don't have 2TB of data transfer for only 200 players?
I also used DO for a low-tickrate game (2ticks/second), but it doesn't seem like the servers are able to handle multiple game servers running at 60FPS.
If you were to implement the game again, would you rethink your networking so you don't have 2TB of data transfer for only 200 players?
I pick smallest needed value type, and some values encode multiple fields (i.e. type/state/frame are all in uint8). Node.js uses cast to unsigned 32-bit integers for bitwise operations on numbers, so bundling doesn't pay out as much in CPU as does in bandwidth savings.
2TB of data is really not that much. Say you run at 30fps, streaming only 2D position (say, 2 x uint32) of 100 entities to 200 clients...
(64 * 100 * 200 * 30 * 60 * 60 * 24 * 30 ) / 8 / 1024^4 ~= 11TB per month.
You bring some sanity to that with less frames, less entities, naive delta not sending updates for entities, etc. It's really not surprisingly big number.
I'll definitely explore world state deltas in next game, but this works good enough for now.
2TB of data is really not that much. Say you run at 30fps, streaming only 2D position (say, 2 x uint32) of 100 entities to 200 clients...
(64 * 100 * 200 * 30 * 60 * 60 * 24 * 30 ) / 8 / 1024^4 ~= 11TB per month.
You bring some sanity to that with less frames, less entities, naive delta not sending updates for entities, etc. It's really not surprisingly big number.
I'll definitely explore world state deltas in next game, but this works good enough for now.
>>> A question to all reading - I'm considering adding an option to mine Monero in my next game. It'd probably be a flip-switch between ads and mining right there on the home screen, with some intelligent throttle/thread decisions depending on the kind of device game is running on. I'd appreciate any feedback regarding the sentiment on that.
Mining => You WILL murder the battery of your users. You WILL cause overheating of the devices. You will also grind the browser to a halt on all hardware of moderate speed or already busy for reading a youtube video in the background.
Personally. I'd consider that any website that start mining when opened should go right into an adblock list, if not blocked as malicious entirely.
Mining => You WILL murder the battery of your users. You WILL cause overheating of the devices. You will also grind the browser to a halt on all hardware of moderate speed or already busy for reading a youtube video in the background.
Personally. I'd consider that any website that start mining when opened should go right into an adblock list, if not blocked as malicious entirely.
This is the reason why I'm considering only enabling that feature on Chrome (because of WASM and battery API) and Desktop PC (because of the battery/heat issues).
For experiment, I've turned on a miner in one tab of my brother's 7-year old modest PC setup. He went for several hours without noticing it, while doing blender rendering as well as somewhat intensive photoshoping.
For experiment, I've turned on a miner in one tab of my brother's 7-year old modest PC setup. He went for several hours without noticing it, while doing blender rendering as well as somewhat intensive photoshoping.
Just because the user doesn't notice or doesn't understand what happens doesn't make it acceptable.
Chrome is going toward more and more aggressive CPU throttling of tabs in the background. If mining becomes a thing, we're going to a world where background tabs will stop being rendered at all.
Chrome is going toward more and more aggressive CPU throttling of tabs in the background. If mining becomes a thing, we're going to a world where background tabs will stop being rendered at all.
> Just because the user doesn't notice or doesn't understand what happens doesn't make it acceptable.
I did this on my brother, for data gathering. It'd most certainly not be implemented non-transparently in the end-product.
> Chrome is going toward more and more aggressive CPU throttling of tabs in the background.
That's why you mine during active game. The more people love it, the more they mine it. It can certainly be made mostly win-win scenario, which is why I aim on, again, transparently experimenting with it.
I did this on my brother, for data gathering. It'd most certainly not be implemented non-transparently in the end-product.
> Chrome is going toward more and more aggressive CPU throttling of tabs in the background.
That's why you mine during active game. The more people love it, the more they mine it. It can certainly be made mostly win-win scenario, which is why I aim on, again, transparently experimenting with it.
Is mining in a browser remotely cost efficient? Doesn't it end up costing the user more in electricity than just buying the coins directly?
There are many tradeoffs one makes taking any route. Electricity-wise? I'm not sure. We'll see.
> ...just buying the coins directly?
Realistically, there's never such a thing such as "just buying coins". You need payment processor / security / refunds / authorization / ...
There is, however, something close to "just mining coins". This is one of the fundamental reasons cryptocurrency is attractive in the first place.
> ...just buying the coins directly?
Realistically, there's never such a thing such as "just buying coins". You need payment processor / security / refunds / authorization / ...
There is, however, something close to "just mining coins". This is one of the fundamental reasons cryptocurrency is attractive in the first place.
Is this not already in the BTC situation, where you burn more in cost of electricity than you gain from mining? Granted, in this case it would be somebody else's electricity, but that feels a bit ethically grey, as well as perhaps environmentally irresponsible.
Don't get me wrong, I've considered similar things, but I'm pretty uncomfortable with the idea for the above reasons. Plus, whatever people might tell you, even the absolute best JavaScript engine is slow by comparison with other languages.
Example: in the last few weeks due to #reasons I've ported a measure calculation engine for market research data from JavaScript to C#. At first, in many ways, it was a like for like conversion, in that I preserved the data model and general mechanism and flow of calculations, albeit that I applied a much stronger separation of concerns. Nevertheless, with no optimisation, it was roughly 2x the performance of the JS version (V8, of course) running on the same machine. I then started to take advantage of .NET's strong typing for numbers and strings, switched out dictionaries for strongly typed arrays using (obviously) integer lookup, rather than using integers as keys in dictionaries. I also fixed a couple of issues where I was inadvertently using exceptions for flow control, and recalculating strings to generate hashcodes (doh!).
What's surprising is that those changes get me to about 15x faster, and that's really just taking advantage of strong typing and more appropriate data structures that are available in .NET. (JS has arrays, and engines like V8 do optimise for them, even though they're strictly just objects with special properties. Whereas in C, C++, C# arrays are the absolute fastest data structure to access, bar none.) Also worth pointing out that judicious switching from dictionaries to arrays absolutely slashed memory use.
I also implemented the kicker - which happens to be language agnostic - which was to change the moving average calculations to preserve and reuse intermediate results. This got me a 10x performance gain on top of the rest, taking me to about 150x. Like I say, that would probably have been the case in JS as well, so it's not really a fair comparison.
Nevertheless, bear in mind that whilst JavaScript is certainly a lot faster than it used to be, it is not fast, even by the standards of other managed runtime languages. And it's absolutely not fast by the standards of well-written C++ or C compiled down to native code.
I'm not sure how much benefit you'd get from a WebAssembly approach, although possibly something worth looking at.
Don't get me wrong, I've considered similar things, but I'm pretty uncomfortable with the idea for the above reasons. Plus, whatever people might tell you, even the absolute best JavaScript engine is slow by comparison with other languages.
Example: in the last few weeks due to #reasons I've ported a measure calculation engine for market research data from JavaScript to C#. At first, in many ways, it was a like for like conversion, in that I preserved the data model and general mechanism and flow of calculations, albeit that I applied a much stronger separation of concerns. Nevertheless, with no optimisation, it was roughly 2x the performance of the JS version (V8, of course) running on the same machine. I then started to take advantage of .NET's strong typing for numbers and strings, switched out dictionaries for strongly typed arrays using (obviously) integer lookup, rather than using integers as keys in dictionaries. I also fixed a couple of issues where I was inadvertently using exceptions for flow control, and recalculating strings to generate hashcodes (doh!).
What's surprising is that those changes get me to about 15x faster, and that's really just taking advantage of strong typing and more appropriate data structures that are available in .NET. (JS has arrays, and engines like V8 do optimise for them, even though they're strictly just objects with special properties. Whereas in C, C++, C# arrays are the absolute fastest data structure to access, bar none.) Also worth pointing out that judicious switching from dictionaries to arrays absolutely slashed memory use.
I also implemented the kicker - which happens to be language agnostic - which was to change the moving average calculations to preserve and reuse intermediate results. This got me a 10x performance gain on top of the rest, taking me to about 150x. Like I say, that would probably have been the case in JS as well, so it's not really a fair comparison.
Nevertheless, bear in mind that whilst JavaScript is certainly a lot faster than it used to be, it is not fast, even by the standards of other managed runtime languages. And it's absolutely not fast by the standards of well-written C++ or C compiled down to native code.
I'm not sure how much benefit you'd get from a WebAssembly approach, although possibly something worth looking at.
I’m actually super curious to know now the total resource consumption of one video ad versus one hash computation for mining purposes (including bandwidth and battery consumption). The thing with mining is that, since it’s basically a lottery system, you can do as much or as little as you want. With sufficient scale, you could do a few computations and stop when you feel the user has “paid” for access to your page.
Go check it out with this miner implementation I intend using and customizing. [1]
[1] https://coin-hive.com
[1] https://coin-hive.com
Can you/will you run that on a web worker? No way that would work on the foreground thread if you're using that for your game. That obviously depends on how much work your game does but this, for example, would suck with anything else going on in the foreground, especially on a phone: https://arcade.ly/games/asteroids/.
I'll first do whatever's most sensible with current readily available tech and measure. Coin-hive success case [?] (or so I've read), used secondary tab, which contained details about mining data itself.
[?] I think they've removed it from their homepage, most likely because they thought it's not fitting to include it there. :/
[?] I think they've removed it from their homepage, most likely because they thought it's not fitting to include it there. :/
Given their test and API documentation both refer to threads I'm guessing it must be using web workers. I might start offering it to desktop users, along with something like https://www.pollfish.com/, as ways to get rid of adverts on the site (I make sweet f.a. from ads, although I get pretty minimal levels of traffic, tbh - need to spend a lot of time on SEO, which is of course not the fun bit). I'm definitely steering clear of offering it on mobile though.
Feel free to hit me with any data you gather, if you choose to do so, of course.
> Is this not already in the BTC situation, where you burn more in cost of electricity than you gain from mining? Granted, in this case it would be somebody else's electricity, but that feels a bit ethically grey, as well as perhaps environmentally irresponsible.
Probably. But the point is that it's not an alternative to mining for yourself, but an alternative to showing people ads or having people explicitly pay, which is likely to be even less efficient for very small transactions.
If you do it behind the users back, then sure, it might be ethically grey (though how many users care about how much/little electricity the game they play cause their computer to consume?), but the alternative is to clearly show a message saying you do this instead of showing them ads and/or offering them a button to switch to ads instead.
Probably. But the point is that it's not an alternative to mining for yourself, but an alternative to showing people ads or having people explicitly pay, which is likely to be even less efficient for very small transactions.
If you do it behind the users back, then sure, it might be ethically grey (though how many users care about how much/little electricity the game they play cause their computer to consume?), but the alternative is to clearly show a message saying you do this instead of showing them ads and/or offering them a button to switch to ads instead.
I'd suggest finding out, in the real world, if it is ads or mining that gets you the greater income. Then, you optimize for that.
In either case, when you're having them doing the mining, I'd recommend full disclosure AND I'd recommend some explanatory text that goes into the details of what you're doing and why.
Something like, "Click here to learn more about mining." That should do the trick. I'd just put said link below the quick blurb/switch/invite to use mining.
If mining generates more income, then you may wish to offer some sort of in-game bonus for those who enable it. You'll have to figure out some way to ensure they really have it enabled, of course. Giving them a reward may entice more users to choose to do mining.
Another post mentioned that this would be a battery killer. You responded that you'd only enable it for certain devices/browsers. If you want, you could probably enable it (as a choice) for all devices and browsers BUT I'd absolutely ensure that you made it very clear that doing so would mean a rapid depletion of stored energy on battery powered devices.
The reason I'd suggest you do allow it is because many people only use mobile devices and, in some cases, will operate them while they are plugged in. For example, I'm typing this on a tablet that is plugged into an outlet. If mining makes you income, I'd be inclined to want to do mining for you - even while on a mobile device. It'd be even more tempting if there were an in-game reward for it.
Sort of related: That might make an interesting funding option. Free games but mandatory mining on behalf of the creator. Obviously, full disclosure is required if one is concerned with ethics. Inside this idea, they might be able to get rewards directly related to how much they mine on the creator's behalf.
In either case, when you're having them doing the mining, I'd recommend full disclosure AND I'd recommend some explanatory text that goes into the details of what you're doing and why.
Something like, "Click here to learn more about mining." That should do the trick. I'd just put said link below the quick blurb/switch/invite to use mining.
If mining generates more income, then you may wish to offer some sort of in-game bonus for those who enable it. You'll have to figure out some way to ensure they really have it enabled, of course. Giving them a reward may entice more users to choose to do mining.
Another post mentioned that this would be a battery killer. You responded that you'd only enable it for certain devices/browsers. If you want, you could probably enable it (as a choice) for all devices and browsers BUT I'd absolutely ensure that you made it very clear that doing so would mean a rapid depletion of stored energy on battery powered devices.
The reason I'd suggest you do allow it is because many people only use mobile devices and, in some cases, will operate them while they are plugged in. For example, I'm typing this on a tablet that is plugged into an outlet. If mining makes you income, I'd be inclined to want to do mining for you - even while on a mobile device. It'd be even more tempting if there were an in-game reward for it.
Sort of related: That might make an interesting funding option. Free games but mandatory mining on behalf of the creator. Obviously, full disclosure is required if one is concerned with ethics. Inside this idea, they might be able to get rewards directly related to how much they mine on the creator's behalf.
Responding to all your suggestions would take too much time for me right now, but I can tell you that all are very on-point. Thanks.
One interesting trivia though - for few undisclosed games I've checked the data, with following conclusion - they would've made order of magnitude less with mining than with ads. However, if you include currency trend for those periods, they would've made almost the same. :)
I think this might be indication of a possible close-future tipping point in favour of this model. We'll see.
One interesting trivia though - for few undisclosed games I've checked the data, with following conclusion - they would've made order of magnitude less with mining than with ads. However, if you include currency trend for those periods, they would've made almost the same. :)
I think this might be indication of a possible close-future tipping point in favour of this model. We'll see.
>>> I'm doing similar, it's so much fun
Yes, basically this. I think most of us solve the same sort of problems day in and day out with our jobs; it's unbelievably fun to be dropped into a totally new problem space and have that feeling of complete ignorance again.
Yes, basically this. I think most of us solve the same sort of problems day in and day out with our jobs; it's unbelievably fun to be dropped into a totally new problem space and have that feeling of complete ignorance again.
Nice list! These are pretty useful. I found the experience of using protobuf for frontend <-> Go communication to be good, since it forced me to do the work of 'writing my own protocol' but without all the encoding/decoding. There was some issue with naming in protobuf and Go's conventions though.
To add my 2 cents: I think mining Monero is actually very reasonable (you seem to have thought through the impact on your users), and you've nailed the most important thing: making it opt-in.
I'm fine contributing CPU cycles when playing a game, as long as it's in the foreground and I agreed to it.
To add my 2 cents: I think mining Monero is actually very reasonable (you seem to have thought through the impact on your users), and you've nailed the most important thing: making it opt-in.
I'm fine contributing CPU cycles when playing a game, as long as it's in the foreground and I agreed to it.
> I have Node.js serving 200 clients and 10k entities with vision (can run at or from you) at 20 frames per second that keeps under 100MB RAM with no hiccups.
What's in the 100MB?
Doing some napkin math, take 10k hypothetical game objects that each have as state: pos, vel, acc, rot, ang vel, ang acc, name (32 chars), color, health, mana, armor, level, type, energy, kills, timers, inventory (256 bytes)... etc etc, all together comes in at around 512 bytes each. Even if that's double buffered it's still just 10MB.
Curious where all that memory is going!
What's in the 100MB?
Doing some napkin math, take 10k hypothetical game objects that each have as state: pos, vel, acc, rot, ang vel, ang acc, name (32 chars), color, health, mana, armor, level, type, energy, kills, timers, inventory (256 bytes)... etc etc, all together comes in at around 512 bytes each. Even if that's double buffered it's still just 10MB.
Curious where all that memory is going!
To be perfectly honest, I haven't profiled and have been fairly cavalier about it. Seemed good, and most importantly, stable enough - it's not bottleneck. Like you've noticed, it definitely has low-hanging fruits to pick up there, yeah.
game devs... do not understand the web.
well I'm not sure how it could only be 100MB considering that our Angular2/4 Web Application (a big enterprise CRM/ERP) uses 400 MB (no joke).
well I'm not sure how it could only be 100MB considering that our Angular2/4 Web Application (a big enterprise CRM/ERP) uses 400 MB (no joke).
Please tell me what your game is. Even considering abusing my hardware for mining is enough reason for me to avoid anything you create.
You say "Ignore socket.io, do pure binary websocket" and that sort of makes sense to me, along with your advice about not using JSON, but what are you using to do "pure binary websocket"?
Do you have any code samples, please? (Don't need to run or compile, just an example would be handy.)
Also, can you post a link to your game, please? I'd like to give it a try.
Do you have any code samples, please? (Don't need to run or compile, just an example would be handy.)
Also, can you post a link to your game, please? I'd like to give it a try.
There are so many msgpack and protobuff implantations on npm, does anyone have recommendations on which to use / what the tradeoffs are?
https://npmcharts.com/compare/avsc,thrift,msgpack-lite,msgpa...
https://npmcharts.com/compare/avsc,thrift,msgpack-lite,msgpa...
> I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view.
If you don't mind me asking, Why is this the case? I would have assumed world entity deltas is probably how I'd approach things first...
If you don't mind me asking, Why is this the case? I would have assumed world entity deltas is probably how I'd approach things first...
>Write game bots to test your load non-synthetically.
Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.
Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.
In my particular game, for core gameplay, I'm just sending the angle and whether the action button is down or up. So I just slapped together a Node.js script to spawn N clients that, according to some simple logic send that input to server. They don't even interpret what server sends back to them, because I'm testing the server-side with them. Bonus is that you can test your algos in extreme circumstances, like send all clients to particular spot on the map, so you can see how it performs with different distributions of locations.
This was invaluable to almost all multiplayer games I did.
Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).
This was invaluable to almost all multiplayer games I did.
Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).
> Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).
Do what you've gotta do to get paid for your work, but if I found out this was happening it would strongly discourage me from remaining on the website.
Do what you've gotta do to get paid for your work, but if I found out this was happening it would strongly discourage me from remaining on the website.
Thank you, that's great feedback and exactly what I'm after. Nag popup after every few sessions to turn on mining or whitelist the site in adblocker would be better compromise then?
That would work for me.
Honestly this would be a great use for attention grabbing full screen notification (gasp).
"We're cool with adblock, but to continue on this site click here to enable mining, here for paid accounts, or just disable adblock and report inappropriate ads so we can forward that to our syndication."
"We're cool with adblock, but to continue on this site click here to enable mining, here for paid accounts, or just disable adblock and report inappropriate ads so we can forward that to our syndication."
Yeah, actually I think that'd be great place for full disclosure and help user get opportunity to make as informed decision as possible.
"how dare they consume my compute resources to make money?" I mean, what the hell is internet advertising if it's not consuming your compute resources to make money?
The amount of resources required to load a blob of json, some images and do a shitty js callback matroska dance is relatively well understood and accepted as a necessary evil by those that don't/can't block such things. Yes there are bad actors that will churn through your mobile data while your phone is locked, endlessly reloading shitty video ads, but who knows how well miners will be coded or perform.
I'm betting on hearing some horror stories fairly soon as the first gen is rolled out.
I'm betting on hearing some horror stories fairly soon as the first gen is rolled out.
Are you sure about no JSON? My experience is JSON is fairly small, and faster than anything else because it has browser support.
For my use case, replacing JSON with custom binary protocol, some old version being here [1], had savings of about 2 orders of magnitude in bandwidth and at least 1 order of magnitude improvements in server CPU usage. Some significant memory savings too. There's also space for more optimizations to be done, but I consider it viable enough for now.
ArrayBuffer and DataView are fairly fast for my standards.
[1] https://gist.github.com/synthmeat/d23595db732d366f658a1981e9...
ArrayBuffer and DataView are fairly fast for my standards.
[1] https://gist.github.com/synthmeat/d23595db732d366f658a1981e9...
[deleted]
Not OP but yes, absolutely. JSON and MsgPack are schemaless, which means they put the schema, including object keys etc on the wire. On every transfer.
The problem is less the client, you're right with that. The problem is that server bandwidth and CPU scale up really quickly if you try to do 30fps, no matter what method you use to keep state in sync. If you want any significant amount of players in a realtime game, this will be one of your first bottlenecks, probably before any encoding and decoding.
For less realtime-y games, JSON and also socket.io can be perfectly adequate.
Protobuf is just insanely slow to encode/decode client side. So that's out too if you want fast paced games. And for slower games it is not worth the effort.
The problem is less the client, you're right with that. The problem is that server bandwidth and CPU scale up really quickly if you try to do 30fps, no matter what method you use to keep state in sync. If you want any significant amount of players in a realtime game, this will be one of your first bottlenecks, probably before any encoding and decoding.
For less realtime-y games, JSON and also socket.io can be perfectly adequate.
Protobuf is just insanely slow to encode/decode client side. So that's out too if you want fast paced games. And for slower games it is not worth the effort.
It's not for client. It's for server and bandwidth.
Having worked on a commercial networked first person shooter, I can verify that making it smooth and coordinated is a massive pain in the ass. Having a game like World Of Tanks where the server makes all the decisions only works if the game entities are big and don't change much (i.e. tanks). There is a reason why none of the World games have people. Nothing is harder than trying to make a knife fight work over the internet without it looking spastic.
It's also why many MMORPGs rarely try to do collision testing, and have gameplay mechanics where you queue up abilities in advance of expecting to see their effects.
I'm working on a project I hope to make multiplayer eventually, but the issues surrounding it scares me a bit. I wish more companies doing the hard multiplayer work (fps's mostly) would reveal more of their back end and how they do things.
I don't think anyone has really done it well yet but everyone is hiding what they have figured out so far. I have noticed that certain engines tend to have better success at this, for example source engine seems to have drastically improved, allowing games working on this issue like Insurgency to make some noticeable strides towards being better.
This has me remembering UT over dialup lag...
I don't think anyone has really done it well yet but everyone is hiding what they have figured out so far. I have noticed that certain engines tend to have better success at this, for example source engine seems to have drastically improved, allowing games working on this issue like Insurgency to make some noticeable strides towards being better.
This has me remembering UT over dialup lag...
[deleted]
This is a great writeup by VALVE on the subject: https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
Don't not share - which FPS? :)
Yeah, game design and tech to support it are tightly linked, and there's approaches to it from either side.
Yeah, game design and tech to support it are tightly linked, and there's approaches to it from either side.
"spastic" Are there not better words ?
Uncoordinated ? Glitchy ? clumsy ? awkward? blundering?
Just wondering, why are people still using socket.io and not standard websockets, as available in every browser since several years now?
If you don't mind performance, using socket.io gives you stay-alive on the ws connection. If you don't mind writing the 5 lines of code it takes to implement stay-alive for normal websockets then socket.io makes no sense anymore.
I'm personally using it because it gives me:
1. Rooms
2. Disconnects and reconnects,
3. Fallack solutions
for free.
Although not for a game. I trust that its just not made for that kind of performance.
But for a messaging app its actually quite decent.
1. Rooms
2. Disconnects and reconnects,
3. Fallack solutions
for free.
Although not for a game. I trust that its just not made for that kind of performance.
But for a messaging app its actually quite decent.
It provides fallback connection methods if WS connection fails, as it can do if you have, for example, a HTTP/1.0 only proxy between you and the client. For example it was not that long ago that nginx did not support proxying HTTP/1.1. It is getting less and less relevant but at least for me the socket.io code has worked for a long time and there hasn't been a need to rewrite it.
In case you haven't reached that point yet : also evaluate how much network badnwidth is going to cost you if you ever host your servers on aws or gcloud and encounter average success. You may be in for a (bad) surprise.
Now compare that to how much you'll gain from ads. And soon you'll realize one huge issue you're going to have is to maintain your own servers a little bit everywhere in the world.
Now compare that to how much you'll gain from ads. And soon you'll realize one huge issue you're going to have is to maintain your own servers a little bit everywhere in the world.
I've carefully measured and worked toward 1TB/month/VPS at full load. Didn't get there, but now I'm at 2TB (still estimates). With socket.io and JSON, that'd easily blow up to few hundreds TB/month. DigitalOcean for now (but that might change in the future) still doesn't charge on overages, but you do get a slap on the wrist of some kind if you go way overboard, so it's preferable solution to other VPS providers.
It's good that you mention this, since I've heard "horror" stories of people working in this space earning about $10k/mo but spending as much and more on AWS.
It's good that you mention this, since I've heard "horror" stories of people working in this space earning about $10k/mo but spending as much and more on AWS.
Wow, I'm not a game dev pro. But 200 clients + 10K entities could cause that amount of traffic?
Yeah, I've choked one office network on a client project (they went through VPN though) with bot tests. :D
But single-digit TB monthly outbound per VPS was not that surprising to me. Almost triple-digit one in naive implementation, scared the hell out of me first time around. Spent hours double and triple-checking and measuring. :D
But single-digit TB monthly outbound per VPS was not that surprising to me. Almost triple-digit one in naive implementation, scared the hell out of me first time around. Spent hours double and triple-checking and measuring. :D
This is a great write up. I've had similar experiences, and agree, have the server-side game state be the single source of truth.
I basically stuck some visuals on top of the example "counting" snippet on the Elm site (http://elm-lang.org/examples/buttons) and made it multiplayer. First player to count to 50 wins.
I didn't use a tick approach, instead the state of the game was only updated on an event sent from one of the clients. I then broadcast the new state of the game to the other clients.
This is what I made http://retrorace.neillyons.io/ https://github.com/nwjlyons/retrorace. Something is not quite right in the backend code. I think there are problems with waiting for the mutex to unlock. The game feels like it freezes sometimes, but it could also just be a latency issue. The server is in London and I've only tested it with people who are several hundred miles away.
I also had plans to build multiplayer snake but felt like latency would make the game too slow.
I basically stuck some visuals on top of the example "counting" snippet on the Elm site (http://elm-lang.org/examples/buttons) and made it multiplayer. First player to count to 50 wins.
I didn't use a tick approach, instead the state of the game was only updated on an event sent from one of the clients. I then broadcast the new state of the game to the other clients.
This is what I made http://retrorace.neillyons.io/ https://github.com/nwjlyons/retrorace. Something is not quite right in the backend code. I think there are problems with waiting for the mutex to unlock. The game feels like it freezes sometimes, but it could also just be a latency issue. The server is in London and I've only tested it with people who are several hundred miles away.
I also had plans to build multiplayer snake but felt like latency would make the game too slow.
I try to squeeze a little Go side-wise, so I'll be sure to check this out in the future. Thanks.
If you have problems with lag and conflicts resolution of state, take a look at this article, it may be useful.
https://0fps.net/2014/02/17/replication-in-networked-games-l...
On the topic of multiplayer HTML5 games, I've recently started playing with Lance[1] and its pretty good. Its main draw is that it largely does the networking and latency compensation (it has options for interpolation and extrapolation) for you. The only requirement is that you write your game (client and server) in javascript and run the server on node, so that the gameplay logic can be run both in server and client (game is simulated both on client and server, but the clients are periodically synced with the server, using interpolation to make it look smooth).
[1] http://lance.gg
[1] http://lance.gg
If you are writing a game server with web, mobile and desktop clients, is socket.io the way to go?
AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players. Is there a library which abstracts away tcp and udp connections?
And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat? How do you prepare for this? I therefore advise against writing your own network stack if you intend to make the service available to the public. Or am I too cautious?
AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players. Is there a library which abstracts away tcp and udp connections?
And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat? How do you prepare for this? I therefore advise against writing your own network stack if you intend to make the service available to the public. Or am I too cautious?
> If you are writing a game server with web, mobile and desktop clients, is socket.io the way to go?
Socket.io's most discerning feature was fallback to polling. There's also some useful abstractions there, like reconnecting, rooms, as well as some useful ecosystem around it, like socket.io-redis adapter which enables you transparently scale horizontally. If your time is at premium, you might find socket.io work for you, but it's not really hard to write everything better yourself, and better tuned to your use-case.
> AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players.
For fast-action multiplayer, there's currently no sensible offering besides WebRTC's DataChannel, with all it brings on board (I've yet to study it in-depth). I'm having high hopes for QUIC to gain more ground.
> And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat?
From what I've experienced first hand, bots/scripts are most frequent enjoyment-spoiler for other players. I've never ran at scale where I'd be interesting target for actual attacks.
Socket.io's most discerning feature was fallback to polling. There's also some useful abstractions there, like reconnecting, rooms, as well as some useful ecosystem around it, like socket.io-redis adapter which enables you transparently scale horizontally. If your time is at premium, you might find socket.io work for you, but it's not really hard to write everything better yourself, and better tuned to your use-case.
> AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players.
For fast-action multiplayer, there's currently no sensible offering besides WebRTC's DataChannel, with all it brings on board (I've yet to study it in-depth). I'm having high hopes for QUIC to gain more ground.
> And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat?
From what I've experienced first hand, bots/scripts are most frequent enjoyment-spoiler for other players. I've never ran at scale where I'd be interesting target for actual attacks.
As for udp connections, WebRTC looks like the only mature solution (but it's not designed for games and is way more complicated than game devs want it to be). Alternatively, I know about https://github.com/networkprotocol/netcode.io but it's very much in the early stages - afaik you even need a Chrome extension to use it on the client. It's not easy to do udp on the web.
An "UDP" server for browser clients: https://github.com/seemk/WebUDP
I get 60ms RTT for the largest bin of packets with WebUDP many requests goes as high as 140ms (positive skew of the distribution), while ICMP never goes above 29ms RTT. I know UDP can be badly prioritized in some networks but I don't think it can only be that...
This needs more investigations imho, I wonder where this fails so badly; is WebRTC aiming at ~100ms latency?
This needs more investigations imho, I wonder where this fails so badly; is WebRTC aiming at ~100ms latency?
Definitely a bit weird. Where are you located? The echo demo server is hosted in the Netherlands. That's how the round trip histogram looks for me in Estonia: https://i.imgur.com/iBoqwe4.png I'll take a look later if there's any noticeable delays caused by the server.
I left it running for some hours and got alot more 30ms responses:
https://imgur.com/a/SdWtc
https://imgur.com/a/SdWtc
Chrome use ~100% cpu transceiving ~83 packets per second, and I think it sends packets at a slower rate later on, that's why we get so many 30ms responses when we wait some hours.
This is a tcpdump after loading the page:
This is a tcpdump after loading the page:
\time sudo tcpdump -c 1000 -ni eth0 udp and port 9777
1000 packets captured
1000 packets received by filter
0 packets dropped by kernel
0.02user 0.05system 0:12.45elapsed 0%CPU (0avgtext+0avgdata 5472maxresident)k
0inputs+8outputs (0major+1508minor)pagefaults 0swaps
\time sudo ping -f -c 1000 188.166.46.128
PING 188.166.46.128 (188.166.46.128) 56(84) bytes of data.
....
--- 188.166.46.128 ping statistics ---
1000 packets transmitted, 996 received, 0% packet loss, time 14195ms
rtt min/avg/max/mdev = 28.839/38.641/50.433/5.152 ms, pipe 5, ipg/ewma 14.209/41.787 ms
0.02user 0.13system 0:14.31elapsed 1%CPU (0avgtext+0avgdata 2248maxresident)k
0inputs+8outputs (0major+1183minor)pagefaults 0swapshttps://imgur.com/a/9KHYj
this is after start up.
this is after start up.
Cool, thanks. I'll try this out if I ever want to switch to WebRTC.
https://github.com/tinspin/fuse
TLDR: Websockets are overengineered and waste CPU cycles doing nothing but marshalling. HTML is better than canvas because it can be GPU accelerated and you get components and z-order for free.
TLDR: Websockets are overengineered and waste CPU cycles doing nothing but marshalling. HTML is better than canvas because it can be GPU accelerated and you get components and z-order for free.
Any not so expensive hosting options for sockets-based apps? Heroku? Digital ocean? AWS?
Also, what are the limitations by the hosting provider regarding concurrent connections, data transfer, etc.
I am currently working on a poker app using free Heroku dynos but would like to have a budget in mind for when going live.
Also, what are the limitations by the hosting provider regarding concurrent connections, data transfer, etc.
I am currently working on a poker app using free Heroku dynos but would like to have a budget in mind for when going live.
I host https://babbl.xyz on DO with a single $10 server for the socket and the web, and a $10 server for mysql. This can support around 500 active players.
I also did a more action asteroids clone (http://triangle-wars.com), and this would only support I guess 50 players on a $10 server.
The limiting factor seems to be the number of packets / sec that need to be broadcast to all players. If each player is changing state 2/sec, and you have 50 players, then the sever needs to relay 100 packets/sec...
Unlike some other comments in this thread, I didn't find JSON parsing to have any effect on performance.
I also did a more action asteroids clone (http://triangle-wars.com), and this would only support I guess 50 players on a $10 server.
The limiting factor seems to be the number of packets / sec that need to be broadcast to all players. If each player is changing state 2/sec, and you have 50 players, then the sever needs to relay 100 packets/sec...
Unlike some other comments in this thread, I didn't find JSON parsing to have any effect on performance.
On DigitalOcean, I'm planning a following setup to launch:
1. $20 staging at all times, already live [1]
2. $20 static nginx
3. 3x$5 for socket servers on North America / Europe / Asia
4. $5 master server with WS connections to full mesh of socket servers / orchestration / future.
All Debian.
Have no clue what's Heroku like for that, but last time I've checked, it wasn't as good choice as DO was and is for me.
[1] WARNING: might break at any time, by choice or not http://staging.munchies.io
1. $20 staging at all times, already live [1]
2. $20 static nginx
3. 3x$5 for socket servers on North America / Europe / Asia
4. $5 master server with WS connections to full mesh of socket servers / orchestration / future.
All Debian.
Have no clue what's Heroku like for that, but last time I've checked, it wasn't as good choice as DO was and is for me.
[1] WARNING: might break at any time, by choice or not http://staging.munchies.io
Thanks for the info. Good thing about turn-based games is that you send small packets to few players every few seconds, so they're not that intensive in memory/transfer. I hope having a thousand players in a room won't break my piggy bank. Later on I'll venture with chess, checkers and backgammon, all turn-based games.
In this oldie of mine [1][2][3], I used protobuf (with no dictionary), and basically only used real-time information to show other player make moves, in otherwise turn-based/async profile game. It's currently solely on single $20 DO droplet, so I hope no HN effect kicks-in. :)
[1] https://bit.ly/ShowdownFacebook (FB+Flash)
[2] https://bit.ly/ShowdownApple (iOS)
[3] https://bit.ly/ShowdownGoogle (Android)
[1] https://bit.ly/ShowdownFacebook (FB+Flash)
[2] https://bit.ly/ShowdownApple (iOS)
[3] https://bit.ly/ShowdownGoogle (Android)
I tried playing that game but it's almost unplayable on my machine. The input lag seems to be around 200-250ms (so there seems to be no client prediction). Is this how was it supposed to be? Also it's not nice that you didn't mention the demo contains a crypto miner...
I work on it most of this afternoon, including now. Maybe you hit it at wrong time. I use it a lot for load testing, that's mostly why it's live. It did came with a WARNING. :/ Works like a butter for me on EU <-> US path.
> Also it's not nice that you didn't mention the demo contains a crypto miner...
Hopefully it turned on only for Desktop + Chrome and didn't drain your battery? I simply didn't think it'd come as surprise, considering context of the discussion.
> Also it's not nice that you didn't mention the demo contains a crypto miner...
Hopefully it turned on only for Desktop + Chrome and didn't drain your battery? I simply didn't think it'd come as surprise, considering context of the discussion.
For that money you might as well get a beefy dedicated server that's multiple times as powerful and not have to worry about all these small servers.
The bottleneck likely isn't the server here, but geolocation. He has 3 Digital Ocean socket servers in three regions. A single dedicated server will only be in one region. For latency-sensitive applications like realtime games, having servers close to your users is a must.
But can I get 3 dedis in 3 data centers? I'd like to have that lest user experience suffers. I effectively get that with my setup.
Maybe one day when I need it, and certainly evaluating at any point.
Maybe one day when I need it, and certainly evaluating at any point.
If you want to trade off simplicity for lower costs or lower latency, you can also use WebRTC data channels that do p2p communication between the players.
The author mentioned that using a control scheme with small acceleration leads to better synchronizing with the server. Another approach to this is control schemes where the client indicates to the server "I'm planning on being at this location at this future time."
That's then, logically, open to exploitation from user and client-side. Unless deeper logic layers are made to prevent that.
Rabbit-hole, I tell ya. They're fun, though. :)
Rabbit-hole, I tell ya. They're fun, though. :)
I wrote something similar as a very simple demo multi-player tank game/simulator with socket.io and canvas: https://github.com/pdfernhout/TanksInYourBrowser
Do you have a live demo somewhere?
No, sorry -- but it is easy to run it locally if you already have nodejs installed. It's not very fancy -- just shows the basics.
"io.call('up')" broke your server. Sorry :(
Just woke up and it seems to be working now. No harm done, I guess?
Well not really, anyone could type in 'io.call()' and it would add a new player to your server with no controller. 'while(true) { io.call() }' would kill your server in about a minute
- Ignore socket.io, do pure binary websocket
- Do not serialize JSON, or use msgpack or protobuf - write your own protocol.
- Write tests for protocol, you need fast iteration. It's super easy to write tests for codec.
- Quadtrees are kinda lousy compared to r-trees.
- Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view.
- Keep client updates under MTU payload. This is basically impossible with JSON.
- Mutability is the way to go for performance.
- Share code with backend and frontend, you'll need it for client prediction.
- Prefer fixed entity speeds.
- Don't log per-frame in production, journalctl will eat up your memory.
- Write game bots to test your load non-synthetically.
- Test on your VPS, you'll be surprised how underperforming CPUs are in most providers. Neighbours are probably serving some CRUD and are very likely not to be as noisy as you are.
Now, all this might not apply depending on exact gameplay mechanics you have, but it works for me. I have Node.js serving 200 clients and 10k entities with vision (can run at or from you) at 20 frames per second that keeps under 100MB RAM with no hiccups. I think I could use something like Golang for some more advanced physics though.
A question to all reading - I'm considering adding an option to mine Monero in my next game. It'd probably be a flip-switch between ads and mining right there on the home screen, with some intelligent throttle/thread decisions depending on the kind of device game is running on. I'd appreciate any feedback regarding the sentiment on that.