Creating a matchmaker for your multiplayer game(mas-bandwidth.com)
mas-bandwidth.com
Creating a matchmaker for your multiplayer game
https://mas-bandwidth.com/creating-a-matchmaker-for-your-multiplayer-game/
56 comments
This is why smaller AA games still keep this alive - to build community. For instance: Squad, a 80-100 player multiplayer game which primarily depends upon privately hosted servers ran by communities that actually care about the community and game. You'll still run into jerks like any online game, but it's a world of difference to the one-match/single-serving strangers you usually play with in other games.
[deleted]
This is something I really like about Rocket League’s matchmaking. Competitive is random, but Casual play allows you to keep the same lobby. You can even vote to rematch to keep the same teams. I’ve had numerous runs of 3-4 or more matches with the same people. It’s fun to get into a back and forth, especially if you rematch and the other team wins one - then you’ve _gotta_ have a tie breaker!
Yeah, way back when I used to play Counter-Strike, I eventually landed on one server I really liked and kept going to. It would be interesting to see who was on it at different times of the day, and eventually I got to be friendly with most of the regulars that showed up, and they knew me.
Turns out several lived near the server's location, in Texas, and at one point my friends and I just happened to be going there to visit a friend who was stationed at the nearby military base, and so I ended up meeting up with them for lunch. Nice guys.
Turns out several lived near the server's location, in Texas, and at one point my friends and I just happened to be going there to visit a friend who was stationed at the nearby military base, and so I ended up meeting up with them for lunch. Nice guys.
> Turns out several lived near the server's location, in Texas, and at one point my friends
Also ended up in a similar situation multiple times (bunch of randoms found some server we liked, sticked around for matches across weeks, eventually became regulars and eventually figured out we lived nearby). Sometimes we'd bump into each other on other servers too.
After a couple of times of hanging out we've found out why (probably at least) we came across each other, we all default to sorting the server list based on ping (latency), and since we were all geographically close, we tended to end up on the same servers.
Also ended up in a similar situation multiple times (bunch of randoms found some server we liked, sticked around for matches across weeks, eventually became regulars and eventually figured out we lived nearby). Sometimes we'd bump into each other on other servers too.
After a couple of times of hanging out we've found out why (probably at least) we came across each other, we all default to sorting the server list based on ping (latency), and since we were all geographically close, we tended to end up on the same servers.
Texas wasn't that close to me (I was in Illinois), but the server was hosted within a data center, and had pretty low latency for me anyway.
Also it was very consistently up and low latency compared to other servers, so that's why I kept going there at first. Later on I kept showing up because I got to know people on the server.
Also it was very consistently up and low latency compared to other servers, so that's why I kept going there at first. Later on I kept showing up because I got to know people on the server.
Can't you make a group with people you like ? There a feature like that in rocket league.
Getting grouped with the same bad team mates repeatedly would just make me quit the game.
Getting grouped with the same bad team mates repeatedly would just make me quit the game.
You can queue with people you already know, and some games do have a 'stay as a team in the next queue' option, but there's no real opportunity to actually get familiar with anyone, let alone a community, and play with them more in the same way that you get with community servers
Group with the same team until the magic runs out in 3 games then everyone quits lol
can't you queue with people
Glenn Fiedler is the best blogger on game networking. We used your stuff to guide our netcode development ~6 years ago. We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native.
We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage largely due to unity's failure to provide a system to our competitors.
Thanks for helping us, Glenn.
Our matchmaker needs a rewrite, and these posts are nice.
Thanks for helping us, Glenn.
Our matchmaker needs a rewrite, and these posts are nice.
> We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native. We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage
That's awesome! Well done
That's awesome! Well done
[deleted]
[deleted]
What library do you use for WebRTC? The native libwebrtc?
Godot uses libdatachannel. I remember them asking for mbedtls support
Yeah we use libdatachannel. tbh we didnt see much (if any) network improvement using WebRTC over WebSockets, but our testing/analysis is pretty shallow (3 person company - whos got time for that, lets make products)
> our netcode is a serious competitive advantage
What do you sell that isn't already solved by XDP?
What do you sell that isn't already solved by XDP?
Googling XDP tells me it's extreme diesel performance. I'm sure theyre engines are great but we make videogames.
https://en.wikipedia.org/wiki/Express_Data_Path is a feature of recent Linux kernels allowing you to bypass/replace most of the normal Linux networking code, allowing for higher performance. Some network cards will even run your network code directly on their hardware.
(not sure what GP meant, just providing the reference)
(not sure what GP meant, just providing the reference)
I think they were confused by the term “netcode”, which in game developer parlance means “code which implements networked gameplay in a multiplayer game”, rather than “code that implements networking in an OS”.
Yeah, our netcode forward predicts from the server state to the local time to allow the local player to see their own character without network artifacts, and see other players in approximately their true location. It's the 'server authoritative, with client side prediction and rollback' technique. Consult Gaffer On Games' blog posts to learn more.
My understanding of the post:
1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game
2. Wait 1 second on the server, then shuffle the list of clients and group every set of 4 into a match.
3. If no match is found after 10 seconds, add the client to servers in suboptimal datacenters
4. If still no match after another 10 seconds, put them in all the queues and join them to the first game it finds so they can at least play at all, even if they'll have lag no matter where they go
5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. The author links another blog post for further info which is mainly a statistics lecture about averages hiding outliers and doesn't back up this claim that the port number makes a significant difference for latency in any but the most exceptional of situations where you're sent eastward instead of westward around the globe or so) or by collecting statistics to find which server was best for a given location and ISP over the past 30 days (seems over-engineered with extra downsides, like that you will have gaps in the data and that it might have changed)
Did I miss anything? The whole thing seems somewhere between "do the most naïve implication" (1–4) and doubtful (5). It's useful for devs to confirm the intuition they might have for the simplest implementation is correct but other comments saying this is a great blog surprise me when looking at these two posts
The hardest part about matchmaking seems to me finding players well-matched in skill without making them wait long, especially during off-peak hours, or having off-putting loss streaks because they (50% chance for each game) got matched against a slightly stronger or equal team and lost. The article doesn't even mention these aspects exist, let alone offer advice on tackling them
1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game
2. Wait 1 second on the server, then shuffle the list of clients and group every set of 4 into a match.
3. If no match is found after 10 seconds, add the client to servers in suboptimal datacenters
4. If still no match after another 10 seconds, put them in all the queues and join them to the first game it finds so they can at least play at all, even if they'll have lag no matter where they go
5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. The author links another blog post for further info which is mainly a statistics lecture about averages hiding outliers and doesn't back up this claim that the port number makes a significant difference for latency in any but the most exceptional of situations where you're sent eastward instead of westward around the globe or so) or by collecting statistics to find which server was best for a given location and ISP over the past 30 days (seems over-engineered with extra downsides, like that you will have gaps in the data and that it might have changed)
Did I miss anything? The whole thing seems somewhere between "do the most naïve implication" (1–4) and doubtful (5). It's useful for devs to confirm the intuition they might have for the simplest implementation is correct but other comments saying this is a great blog surprise me when looking at these two posts
The hardest part about matchmaking seems to me finding players well-matched in skill without making them wait long, especially during off-peak hours, or having off-putting loss streaks because they (50% chance for each game) got matched against a slightly stronger or equal team and lost. The article doesn't even mention these aspects exist, let alone offer advice on tackling them
> 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo.
No citation, but here's some things to think about.
You can easily have two servers in the same facility, in the same rack, with very different ip addresses. Most of the time you'll get the same general route for all destinations within a /24 (v4) or /48 (v6), but not always. Two /24s at the same facility might get different routes, especially if there are capacity issues on the better route at any point.
From same IP to same IP, anywhere there's a redundant link, which link to use is generally selected with a hash on the 5-tuple {protocol, destination ip, sender ip, destination port, sender port} (sometimes on a smaller tuple). If traffic is unbalanced or if one link is lossy, you can get drastically different experiences with a change in port number.
If you've done traceroutes and seen multiple similar ips at a given hop number, it's probably the second effect; this is pretty common. Typical traceroute sends udp packets to the destination and modulates the sending port number. Sometimes, rarely, you can see drastically different routes within a traceroute. It's easier to do traffic engineering on a /24 level though.
Traffic on redundant links is balanced by hashing flows rather than on a per packet basis so that packets within a flow tend to arrive in order; out of order arrival is very expensive for endpoints, so it's better to ocassionaly have unbalanced usage than to regularly have out of order arrival.
No citation, but here's some things to think about.
You can easily have two servers in the same facility, in the same rack, with very different ip addresses. Most of the time you'll get the same general route for all destinations within a /24 (v4) or /48 (v6), but not always. Two /24s at the same facility might get different routes, especially if there are capacity issues on the better route at any point.
From same IP to same IP, anywhere there's a redundant link, which link to use is generally selected with a hash on the 5-tuple {protocol, destination ip, sender ip, destination port, sender port} (sometimes on a smaller tuple). If traffic is unbalanced or if one link is lossy, you can get drastically different experiences with a change in port number.
If you've done traceroutes and seen multiple similar ips at a given hop number, it's probably the second effect; this is pretty common. Typical traceroute sends udp packets to the destination and modulates the sending port number. Sometimes, rarely, you can see drastically different routes within a traceroute. It's easier to do traffic engineering on a /24 level though.
Traffic on redundant links is balanced by hashing flows rather than on a per packet basis so that packets within a flow tend to arrive in order; out of order arrival is very expensive for endpoints, so it's better to ocassionaly have unbalanced usage than to regularly have out of order arrival.
It's a shame that you missed the key point of the article in #5. Many games today have suboptimal inputs to their matchmaker, because like you, they refuse to believe that the internet can route a significant percentage of players incorrectly.
Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com
Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com
Love the articles but I think you should disclose Network Next is not merely "a" network accelerator. It's the network accelerator that you're the CEO/founder of, which makes this read a bit like an advertisement.
If it can route incorrectly, that seems like a strong signal not to rely on your maps. It doesn't matter if you know what the correct latency is, if the users only experience the incorrect one.
The solution is to pair the matchmaker using maps with a network acceleration product that fixes the bad routes, then the latency map and the actual latency experienced are very close.
The reason you use the maps in the matchmaker is so you can look up the latency from client to datacenter in constant time and you don't have to waste any time on the client pinging ping servers to get the results.
Plus, it costs money to run pings through the network accelerator, and they always (from experience) converge to the latency map anyway.
The reason you use the maps in the matchmaker is so you can look up the latency from client to datacenter in constant time and you don't have to waste any time on the client pinging ping servers to get the results.
Plus, it costs money to run pings through the network accelerator, and they always (from experience) converge to the latency map anyway.
How much time is wasted? 50 milliseconds at most?
And you're about to send 50 packets per second for an hour during the game. Is sending one packet per datacenter really a cost worth mentioning?
And you're about to send 50 packets per second for an hour during the game. Is sending one packet per datacenter really a cost worth mentioning?
[flagged]
[flagged]
[flagged]
Players are coming in and out all the time, you can't "wait 1 second and shuffle the list". 1 second after what?
The algorithm just has a tick rate at 1 second. Assume requests come in continuously, but only do work at some rate, eg. once every 1 second for a matchmaker that is meant to very quickly find games, and maybe once every 10-15 seconds for a higher quality matchmaker that is going to do a lot of work to match players together by skill etc (longer tick times means you'll have more players to consider in each iteration)
I've written game matchmaking before for games and found it far more challenging to do the portions related to matchmaking groups by skill. If you're hyper latency sensitive then perhaps this blog post is really useful? But for the games I worked on we would trade 10ms worse average ping for 10% better skill pairings without question. If you have any advice on improving skill matchmaking I would be quite interested.
> If you have any advice on improving skill matchmaking I would be quite interested.
Samy Duc wrote a really good deep dive into Apex Legends matchmaking that covers skill and team balancing: https://www.ea.com/games/apex-legends/news/matchmaking-2023
Samy Duc wrote a really good deep dive into Apex Legends matchmaking that covers skill and team balancing: https://www.ea.com/games/apex-legends/news/matchmaking-2023
Thanks for the link, I hadn't read this one yet. Very interesting. I feel like as an industry we're still in the stone ages on how we do skill matching systems. A lot of the current and even future system they describe is really not great, and these flaws are definitely not unique to them (the systems I worked on are even more flawed)! Trying to matchmake dozens of players at a time is such a cool, challenging problem.
I really think it's a cursed mission to chase perfectly balanced matches.
In terms of overall experience, the best approach in general is a purely random one. This is how you avoid the experience of getting trapped in a 20 game losing streak, even if you are a really good player.
If you allow the natural balance of good players winning more often and bad players losing more often, you will find things get a lot less messy in between. This also provides your bad players an opportunity to occasionally witness what they could become. When you only play someone 1% stronger than yourself, you probably don't have a great idea of what the upper bound actually feels like. This can become a serious trap for players who are seeking to grow their skills.
In terms of overall experience, the best approach in general is a purely random one. This is how you avoid the experience of getting trapped in a 20 game losing streak, even if you are a really good player.
If you allow the natural balance of good players winning more often and bad players losing more often, you will find things get a lot less messy in between. This also provides your bad players an opportunity to occasionally witness what they could become. When you only play someone 1% stronger than yourself, you probably don't have a great idea of what the upper bound actually feels like. This can become a serious trap for players who are seeking to grow their skills.
A lot of what you're saying depends greatly on the exact game you're talking about and what winning and losing means, for instance if the game has discrete places instead of a bimodal winner/loser.
For a few of the games I worked on, random matchmaking like you describe is a non-starter. If you're a 90th percentile player in one of our games, you effectively never lose to a 70th percentile player or below. Your rating will be so high that we can't give you any rating system points for the win. So the person who won got no reward other than the feeling of winning, and the person who lost played a match they had no hope of winning. It ends up feeling pointless to play as a top player because only 1 or 2 matches in 10 on average have any meaning for you. Needless to say, it also feels worse as a low rated player because you simply lose more often.
For a few of the games I worked on, random matchmaking like you describe is a non-starter. If you're a 90th percentile player in one of our games, you effectively never lose to a 70th percentile player or below. Your rating will be so high that we can't give you any rating system points for the win. So the person who won got no reward other than the feeling of winning, and the person who lost played a match they had no hope of winning. It ends up feeling pointless to play as a top player because only 1 or 2 matches in 10 on average have any meaning for you. Needless to say, it also feels worse as a low rated player because you simply lose more often.
There's different levels of SBMM (skill-based matchmaking) though, most games nowadays have a choice between ranked (where the appeal is the well-matched games and the possibility of increasing your rank) and quick play (where the appeal is the low queue time and more casual play, and the matchmaker basically is making sure that the lobby is filled with players who are very roughly in the same skill range)
While I also have fond memories of pre-SBMM Halo/CoD from the 2000s, they're more focused on pure shooter mechanics rather forcing players to work together to win an objective, so doing well but losing still feels fine. I find SBMM is needed in more objective focused esport games like overwatch, because the game is designed such that it's harder to do well individually (and thus have any fun) if your teammates aren't doing well.
While I also have fond memories of pre-SBMM Halo/CoD from the 2000s, they're more focused on pure shooter mechanics rather forcing players to work together to win an objective, so doing well but losing still feels fine. I find SBMM is needed in more objective focused esport games like overwatch, because the game is designed such that it's harder to do well individually (and thus have any fun) if your teammates aren't doing well.
> it's harder to do well individually (and thus have any fun) if your teammates aren't doing well.
Perhaps this is the actual cursed aspect. In a free-for-all context (i.e. a team size of one), you would never have this kind of a problem. As you increase the mandatory team size to six, you are creating an entirely new universe of effects to compensate for.
In my experience, 99.99%+ of the frustration in Overwatch and League of Legends emerges from dealing with your own teammates, not your opponents.
Perhaps this is the actual cursed aspect. In a free-for-all context (i.e. a team size of one), you would never have this kind of a problem. As you increase the mandatory team size to six, you are creating an entirely new universe of effects to compensate for.
In my experience, 99.99%+ of the frustration in Overwatch and League of Legends emerges from dealing with your own teammates, not your opponents.
I'm confused about the using the greyscale map tiles to estimate ping.
You don't want to have the users ping the servers themselves because those pings could be inaccurate or noisy, so you use historical average data for users in that region instead to get a nice simple number. But... how do you know where the user is? IP Geolocation? Can't that be wrong also?
Isn't it better to have a direct measurement which could be a little wrong than an average of a guess which could be really wrong?
You don't want to have the users ping the servers themselves because those pings could be inaccurate or noisy, so you use historical average data for users in that region instead to get a nice simple number. But... how do you know where the user is? IP Geolocation? Can't that be wrong also?
Isn't it better to have a direct measurement which could be a little wrong than an average of a guess which could be really wrong?
Google has Open-Match[1] for matchmaking where you just have to provide an API that takes a batch of players and returns groups, and it handles the surrounding stuff, also integrates with Agones[2] to automatically provision servers on k8s.
[1] https://open-match.dev/
[2] https://agones.dev/
[1] https://open-match.dev/
[2] https://agones.dev/
Is matchmaking the process of players "always [being] sent to the datacenters with the best chance of having low latency"?
EDIT: No, it seems that it's literally grouping players together in multiplayer games (but doing so in a way where latency between players is minimized.)
EDIT: No, it seems that it's literally grouping players together in multiplayer games (but doing so in a way where latency between players is minimized.)
Yes, but much more too.
The example here is really simplified to focus mostly on the finding datacenters with low latency problem, but it could also include things like matching players of similar skill together, finding a set of players that would make balanced teams, making sure that players who are partied up together play in the same match and so on.
Basically, just like matchmaking in real life. It's the thing that works out groups of players who should play together in a match.
The example here is really simplified to focus mostly on the finding datacenters with low latency problem, but it could also include things like matching players of similar skill together, finding a set of players that would make balanced teams, making sure that players who are partied up together play in the same match and so on.
Basically, just like matchmaking in real life. It's the thing that works out groups of players who should play together in a match.
An option for not shuffling would be nice
The full source code is here: https://github.com/mas-bandwidth/matchmaker
You can turn off shuffling if you want.
You can turn off shuffling if you want.
Great stuff!
This is actually something I hate about multiplayer games with matchmaking nowadays. I made the majority of my childhood friends only because we stayed on the same server and played together for hours on end. I don't think it's a stretch to say that a key reason for why we play multiplayer rather than single player games is to socialize. This has become increasingly more difficult when you just get a new set of people every 10-30 minutes.