League of Legends Chat Service Architecture(engineering.riotgames.com)
engineering.riotgames.com
League of Legends Chat Service Architecture
http://engineering.riotgames.com/news/chat-service-architecture-servers
82 comments
They actually just started it a few months ago:
http://www.riotgames.com/articles/20150817/1697/our-engineer...
http://www.riotgames.com/articles/20150817/1697/our-engineer...
I agree, I really like seeing the EVE development blog. There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up.
They also mention that their users warned them about this battle before hand which was really cool.
They also mention that their users warned them about this battle before hand which was really cool.
> There are a few blogs up about major battles where their servers are just being thrashed and how they were able to stay up. They also mention that their users warned them about this battle before hand which was really cool.
So, I know that I'm expecting too much, but remember that the only way CCP can handle large fights is to simulate them at 1/10th realtime, and get advance notice of said fights so that they can move the simulation for the target system to a single, beefy server ahead of time.
It's a goddamn shame that they don't have the resources available to figure out the -admittedly difficult- problem that is
* Reworking their simulation to be parallelizable across nodes
* Dynamically shifting slices of a battle within a system across physical nodes
So, I know that I'm expecting too much, but remember that the only way CCP can handle large fights is to simulate them at 1/10th realtime, and get advance notice of said fights so that they can move the simulation for the target system to a single, beefy server ahead of time.
It's a goddamn shame that they don't have the resources available to figure out the -admittedly difficult- problem that is
* Reworking their simulation to be parallelizable across nodes
* Dynamically shifting slices of a battle within a system across physical nodes
Here is a related talk from Strange Loop last year:
"Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek https://www.youtube.com/watch?v=_jsMpmWaq7I
"Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek https://www.youtube.com/watch?v=_jsMpmWaq7I
My languages of choice are Python(2, of course) and Erlang. Most of my fellows seem obsessed with multicore programming, which IMO is only useful for latency bound situations such as an OS or game engine. For everything else, multinode wins handily. I'm pleased to see yet-another one of my favorite applications building on the fantastic BEAM VM.
I love Erlang and the concept of multinode programming, but sometimes I wish it could lean back into the multicore/SIMD world. A lot of the time, actors are the right choice; but for things like nodes in a physics engine, you obviously want to take a big pile of them and stream them through a matrix multiplication or something of the sort, if you can.
If there was a language that let me define something in terms of an individual actor-modelled components, but then allowed you to optimize some of them into batched GPGPU passes (and apply strict simulation ticking and so forth in the process), I feel like it would "win" all sorts of domains. Erlang could be that language with a little work.
If there was a language that let me define something in terms of an individual actor-modelled components, but then allowed you to optimize some of them into batched GPGPU passes (and apply strict simulation ticking and so forth in the process), I feel like it would "win" all sorts of domains. Erlang could be that language with a little work.
> Python(2, of course)
Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long been mystified by the almost proud unwillingness of the community to embrace Python 3. It seems like a better language to me. What am I missing?
Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long been mystified by the almost proud unwillingness of the community to embrace Python 3. It seems like a better language to me. What am I missing?
afaik the two main reasons are 1) the expense of porting from 2.* to 3.* and 2) some libraries do not yet support 3 http://python3wos.appspot.com/
Few reasons and there's more but I won't go on forever.
1)The "latest version" of a programming language is not the same as having the latest version of Firefox. Having the latest version can actually be a detriment if you lose libraries as with Python3.
2)It may be unpopular, but I tend to follow the market to a certain degree. Python is my language of choice for home projects and employment. But almost all employers use Python2.x. So I follow them just to keep 1 less hassle out of my life.
3)Losing the long tail of libraries that 2.x has. Many things support Python3, and I'm not against it. I'm just waiting for the day when Python3 is more convenient than Python2 to move. I wasn't one asking for the Py3 changes, and they certainly weren't done for me. So I don't feel like being inconvenienced or doing any porting/heavy lifting in the meantime.
4)My petty reason. I think everyone should just do what's in their best interests. I advertised that "2 of course" because it is a given, but rarely stated. I've met a number of pushy Python3 folks and find them distasteful, which hasn't helped budge me in the slightest. They've probably made me a little edgy on that topic.
5)It's a programming language, so what helps you build something the fastest should be what we use. Today that's still Python2. Some people feel very passionately about Python3 and I hope they spend many hours porting and building the ecosystem for my arrival one day. What seems to be going on instead is they're hoping to convert more devotees that will go out there and do a bunch of free ports.
6)I'm more concerned with moving too early than I am too late when the Python3 ecosystem one day, maybe, eclipses Python2. They're just not that different, I've used both and it's no different from building software with 2. But they have vastly different breadth of libraries in 2's favor. They also screwed up on a few things with 3, it isn't really a flat "upgrade" either.
7)Python3 is just adding more bloat. I generally prefer features are REMOVED from a language until there's nothing left that could reasonably be removed from the core language. Ideally I'd like to see a small version of Python with the basics never touched and features added through addons or a macro system.
Latest version, upgrade, and support-ends-2020 are all political statements in regards to Python3 that mean nothing in reality. Latest version- what can't be backported? Upgrade- more like sidegrade once all things are considered. Support EOL- total smoke and mirrors, code will not stop running post 2020.
They're tossing every feature against the wall that they can to fix that. I'm one that holds feature-soup in disdain though so this is unattractive to me, and I prefer a well thought out "one way to do it" (Zen of Python) approach. Python is well beyond the basics at this point and not a small language, but at least 2.7 is frozen and I can choose to use the backports people provide if I want. I like a smaller core language if possible because I have to read other people's codebases.
1)The "latest version" of a programming language is not the same as having the latest version of Firefox. Having the latest version can actually be a detriment if you lose libraries as with Python3.
2)It may be unpopular, but I tend to follow the market to a certain degree. Python is my language of choice for home projects and employment. But almost all employers use Python2.x. So I follow them just to keep 1 less hassle out of my life.
3)Losing the long tail of libraries that 2.x has. Many things support Python3, and I'm not against it. I'm just waiting for the day when Python3 is more convenient than Python2 to move. I wasn't one asking for the Py3 changes, and they certainly weren't done for me. So I don't feel like being inconvenienced or doing any porting/heavy lifting in the meantime.
4)My petty reason. I think everyone should just do what's in their best interests. I advertised that "2 of course" because it is a given, but rarely stated. I've met a number of pushy Python3 folks and find them distasteful, which hasn't helped budge me in the slightest. They've probably made me a little edgy on that topic.
5)It's a programming language, so what helps you build something the fastest should be what we use. Today that's still Python2. Some people feel very passionately about Python3 and I hope they spend many hours porting and building the ecosystem for my arrival one day. What seems to be going on instead is they're hoping to convert more devotees that will go out there and do a bunch of free ports.
6)I'm more concerned with moving too early than I am too late when the Python3 ecosystem one day, maybe, eclipses Python2. They're just not that different, I've used both and it's no different from building software with 2. But they have vastly different breadth of libraries in 2's favor. They also screwed up on a few things with 3, it isn't really a flat "upgrade" either.
7)Python3 is just adding more bloat. I generally prefer features are REMOVED from a language until there's nothing left that could reasonably be removed from the core language. Ideally I'd like to see a small version of Python with the basics never touched and features added through addons or a macro system.
Latest version, upgrade, and support-ends-2020 are all political statements in regards to Python3 that mean nothing in reality. Latest version- what can't be backported? Upgrade- more like sidegrade once all things are considered. Support EOL- total smoke and mirrors, code will not stop running post 2020.
They're tossing every feature against the wall that they can to fix that. I'm one that holds feature-soup in disdain though so this is unattractive to me, and I prefer a well thought out "one way to do it" (Zen of Python) approach. Python is well beyond the basics at this point and not a small language, but at least 2.7 is frozen and I can choose to use the backports people provide if I want. I like a smaller core language if possible because I have to read other people's codebases.
As an indie game dev I would like to deal with as little of a stack as possible. So what high level services could you use to achieve a similar chat feature without having to roll and maintain all of your own infrastructure?
To me something like Firebase could work for this?
It seems like when game dev companies grow they love to blow out all of their own in house solutions so they can have complete ownership and control. But is the cost of ownership of your stack really worth the maintenance costs? Why not outsource as much of the stack to another company that can take advantage of economies of scale?
To me something like Firebase could work for this?
It seems like when game dev companies grow they love to blow out all of their own in house solutions so they can have complete ownership and control. But is the cost of ownership of your stack really worth the maintenance costs? Why not outsource as much of the stack to another company that can take advantage of economies of scale?
Chat at "indie scale" is fairly easy. 10-20k concurrent users fit into one or two machines with off the shelf software (ejabberd) with minimal expertise. 1k-2k concurrent users could easily be done in almost any stack.
If you're doing more than 20k concurrent users, odds are you can afford to hire someone to scale up or build out a team to do it, or find a 3rd party service and pay them to host your chat.
If you're doing more than 20k concurrent users, odds are you can afford to hire someone to scale up or build out a team to do it, or find a 3rd party service and pay them to host your chat.
Indie don't have 20k CCU.
Openfire is a known server for XMPP:
http://www.igniterealtime.org/projects/openfire/
Openfire is a known server for XMPP:
http://www.igniterealtime.org/projects/openfire/
> Openfire is a known server for XMPP
Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc)
They were talking about indie game studies, not indie chat protocols.
Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc)
They were talking about indie game studies, not indie chat protocols.
You could find someone who would do managed ejabberd.
ejabberd is an XMPP server written in erlang. As soon as I saw they talked about erlang I was pretty confident that ejabberd was the next name they would drop. Very surprised they suggest they rolled their own.
ejabberd is an XMPP server written in erlang. As soon as I saw they talked about erlang I was pretty confident that ejabberd was the next name they would drop. Very surprised they suggest they rolled their own.
Check the blog post's comments. The author mentions that they started with ejabberd and -gradually- rewrote 90% of it. :)
It's nice to see how the backend of my favorite game works. Specially this article, right now I'm developing a chat service for a company and god I hate openfire.
My team also recently built a chat service into a mobile app for a cruise line, and had to have high resiliency to network drops and offline users - while having a mission-critical "guaranteed delivery" requirement. We also chose OpenFire since it was an off-the-shelf XMPP server, and (presumably) was easily extensibile via plugins.
After working with (hacking around) OpenFire for two years, I 100% agree with your statement. The clustering plugin routinely fails (to the point where we have actually investigated not even clustering it anymore), the admin interface routinely displays "wrong" data, and other fun bugs we found along the way.
Does it work for us (with a lot of client XEPs and additional custom plugins)? Yes, it does an acceptable job. Would I choose the same product again, if given the choice? No.
After working with (hacking around) OpenFire for two years, I 100% agree with your statement. The clustering plugin routinely fails (to the point where we have actually investigated not even clustering it anymore), the admin interface routinely displays "wrong" data, and other fun bugs we found along the way.
Does it work for us (with a lot of client XEPs and additional custom plugins)? Yes, it does an acceptable job. Would I choose the same product again, if given the choice? No.
I thought LoL didn't have a chat because it "increases toxicity by 250%"
Officially this is why they don't support voice communication.
Which is also a complete cop-out meant to save $$$ that lacks any basis in fact or reality.
Dota 2 is churning along quite comfortably with integrated voice chat despite having more inter-region mixing. Any time this subject comes up the Dota 2 community nearly unanimously agrees that voice chat in no way encourages "toxicity" and more likely actually prevents it by assigning a human voice to the player as opposed to just disembodied words.
Dota 2 is churning along quite comfortably with integrated voice chat despite having more inter-region mixing. Any time this subject comes up the Dota 2 community nearly unanimously agrees that voice chat in no way encourages "toxicity" and more likely actually prevents it by assigning a human voice to the player as opposed to just disembodied words.
I have ~1k games in Dota 2 and have run across less than a dozen ragers in voice. A quick report+mute and they're taken care of. Usually getting a chat ban in the process.
If anything it makes the game better when people start using voice. I've had games that I've given up on and people started using voice to ask for things like shared unit control for dominated creeps to stack ancients and actually communicate where they're going instead of trying to guess what's going on in four silent teammates' minds.
OTOH, I had one game on EU W where two Russians were talking (nicely) to each other and some British guy comes in and just says "I can't stand Russians. Good bye." and leaves.
And I've had my fair share of LAN cafe players with an open mic blaring music and their friends conversations. Mute and move on.
I really do wish we could mute the mic and not chat and vice-versa.
If anything it makes the game better when people start using voice. I've had games that I've given up on and people started using voice to ask for things like shared unit control for dominated creeps to stack ancients and actually communicate where they're going instead of trying to guess what's going on in four silent teammates' minds.
OTOH, I had one game on EU W where two Russians were talking (nicely) to each other and some British guy comes in and just says "I can't stand Russians. Good bye." and leaves.
And I've had my fair share of LAN cafe players with an open mic blaring music and their friends conversations. Mute and move on.
I really do wish we could mute the mic and not chat and vice-versa.
I can't speak for Dota, but I quit playing league because I got tired of how badly Riot has implemented their ranked system.
There's so much talk about toxicity in league, and very little about what CAUSES the toxicity.
Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth.
For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss.
I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts.
If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game.
Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where.
https://www.youtube.com/watch?v=5UfO7qOHx5M
The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling.
I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me.
I haven't played a game since, although I still watch LCS.
There's so much talk about toxicity in league, and very little about what CAUSES the toxicity.
Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth.
For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss.
I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts.
If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game.
Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where.
https://www.youtube.com/watch?v=5UfO7qOHx5M
The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling.
I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me.
I haven't played a game since, although I still watch LCS.
In Dota 2 if you AFK for more than 5 minutes (no xp) you get an abandon and the game is safe to leave for all players. In ranked your MMR (roughly your Elo) gets updated win or lose if the game goes past first blood. The abandoner gets a loss regardless of outcome.
If you abandon 2 ranked games you get put into the low priority queue and have to play a certain number (at least 5 I think) of all-random games that don't record anything.
If you get reported for communications abuse enough you get a communications ban and can only use the chat wheel and rate-limited pings. Other reports get turned into low-priority queue time (supposedly).
If you dodge a game (even if you don't ready up) you get progressively harsher queue timeouts. It goes something like 4 min, 10 min, 30 min, 1 hour, etc.
There is also rumor of a hidden queue pool of toxic players. Kinda like a shadow ban but it lumps all the ragers and dodgers and AFKers and feeders together to play with each other.
Honestly, LoL never appealed to me. Every champ is about the same with about the same skillset and every game has the same laning. Riot seems to not really be able to make changes and their community management is bad.
If you can get past the turn rate and inability to spam spells in Dota you might enjoy Dota more.
If you abandon 2 ranked games you get put into the low priority queue and have to play a certain number (at least 5 I think) of all-random games that don't record anything.
If you get reported for communications abuse enough you get a communications ban and can only use the chat wheel and rate-limited pings. Other reports get turned into low-priority queue time (supposedly).
If you dodge a game (even if you don't ready up) you get progressively harsher queue timeouts. It goes something like 4 min, 10 min, 30 min, 1 hour, etc.
There is also rumor of a hidden queue pool of toxic players. Kinda like a shadow ban but it lumps all the ragers and dodgers and AFKers and feeders together to play with each other.
Honestly, LoL never appealed to me. Every champ is about the same with about the same skillset and every game has the same laning. Riot seems to not really be able to make changes and their community management is bad.
If you can get past the turn rate and inability to spam spells in Dota you might enjoy Dota more.
See, those things I could get behind because they protect the ranked experience.
The LoL community has this idea that dodging is an important part of climbing BECAUSE of how often you auto-lose games from circumstances outside of your control. And they're right, but only because of how badly Riot protects the ranked experience. I personally hate dodging, but I hate wasting an hour or two due to trolls and AFK's even more.
It kills me to watch riot spend so many resources trying to "fix toxicity" when the real problem is how they give other players so much control over your time in league.
The LoL community has this idea that dodging is an important part of climbing BECAUSE of how often you auto-lose games from circumstances outside of your control. And they're right, but only because of how badly Riot protects the ranked experience. I personally hate dodging, but I hate wasting an hour or two due to trolls and AFK's even more.
It kills me to watch riot spend so many resources trying to "fix toxicity" when the real problem is how they give other players so much control over your time in league.
Yeah, that's crazy. I don't have time for that.
I played 2 games of Dota last night. One 35 min (ranked) and one 43 min game (unranked). It took less than 5 minutes to find each game. I had one game fail to start due to fail to two players failing to load. Reborn has been having some server issues so it's not unheard of. Total time in game was like 1:30.
The ranked game was a complete stomp - 59-14 in 35 min but the losing team stuck it out for the entire game. We could have ended it 10 min earlier but we didn't for whatever reason.
No rage quits or gg ff calls or intentional feeding. I'm not highly ranked (no clue what the equivalent is in LoL; I'm just at or slightly above average) but it's rare to get RQs in ranked at my level.
I played 2 games of Dota last night. One 35 min (ranked) and one 43 min game (unranked). It took less than 5 minutes to find each game. I had one game fail to start due to fail to two players failing to load. Reborn has been having some server issues so it's not unheard of. Total time in game was like 1:30.
The ranked game was a complete stomp - 59-14 in 35 min but the losing team stuck it out for the entire game. We could have ended it 10 min earlier but we didn't for whatever reason.
No rage quits or gg ff calls or intentional feeding. I'm not highly ranked (no clue what the equivalent is in LoL; I'm just at or slightly above average) but it's rare to get RQs in ranked at my level.
As an avid Dota 2 player I can attest this. If people start flaming you, replying on voice chat and saying something positive, like "Sorry, my wife was calling me. But don't worry we can win this", will change the tides immediately.
Yup. The most toxic players in my experience tend to be keyboard warriors anyway.
I'm an avid LoL player and not having voice chat support built in is the largest downside to LoL. Plenty of players are "side-loading" their voice chat through a variety of applications. This is a huge PITA as I need Skype, TeamSpeak, whatever the rest of my team is using.
Players aren't consistently using voice chat since it's not built in. If you are in a match up where one team is on voice chat and the other is not; the team with voice has a huge advantage as text chat is less visible and has more delay.
Players aren't consistently using voice chat since it's not built in. If you are in a match up where one team is on voice chat and the other is not; the team with voice has a huge advantage as text chat is less visible and has more delay.
Or they're just doing what WoW did, WoW if I remember correctly added in-game voice support for raids years ago, but dropped it because of lack of user uptake.
No one was using it because it wasn't as good or easy to use as already existing third party programs, so they dropped it. Riot is probably thinking the same, and can spend their resources elsewhere on something that isn't already done better by someone else.
No one was using it because it wasn't as good or easy to use as already existing third party programs, so they dropped it. Riot is probably thinking the same, and can spend their resources elsewhere on something that isn't already done better by someone else.
WoW already had persisting social structures that lead to those groups setting up voice servers that provided them a lot more intricate controls. People who chose to speak in game had already assembled the infrastructure to do so, why use a worse system? I played Eve Online and it was the same scenario, every single group ran a voice server and often individuals did as well.
Contrast this to LoL where you're dropped into a game randomly with others random players for <60 minutes who you don't know but are forced to communicate with to maximize strategy. Communication is utterly critical yet the best option for doing so is slowly typing a message in a micro intensive game or a nondescript ping. The resulting player behavior is exactly as one would expect.
Contrast this to LoL where you're dropped into a game randomly with others random players for <60 minutes who you don't know but are forced to communicate with to maximize strategy. Communication is utterly critical yet the best option for doing so is slowly typing a message in a micro intensive game or a nondescript ping. The resulting player behavior is exactly as one would expect.
Maybe also they're taking their cues from Xbox Live and all of the raging that comes from a similar aged user-base dropped in with random groups and don't feel like enabling young kids the ability to tell random strangers to go kill themselves...
You'll always have outliers, but nothing stops those same kids from doing so right now with text chat (and probably at a higher rate). Mute functionality is there for a reason.
It does not encourage it.
It also sometimes lead to funny stuff like someone broadcasting sad music if we are losing or epic music if we are winning ahah.
Although the mute system is still a must have.
Although the mute system is still a must have.
Not to mention the fact that most toxic people can only spam the chat wheel commands and cannot actually talk as they are chat banned for X hours.
EDIT:
Are there any articles like this on the architecture of Dota 2 or Steam?
EDIT:
Are there any articles like this on the architecture of Dota 2 or Steam?
And if players start turning to auto-matching voice services (Curse Voice and Razer Connect did this at one stage), they'll go and threaten bans for all.
Look into SteamRE for a good primer on interfacing with Valve chat services, although if you want gory details of server architecture I don't think they've ever written about that in detail.
Maybe, but they just bought a stake in Curse Voice
There's a solution to that (considering that text is acceptable and voice is not for whatever reason) - use voice to text software to convert the user's voice to text and send it over the existing text chat.
It's also why they no longer display user's pings on the loading screen.
I find this trend of social engineering via hiding info to be really annoying/disturbing. LoL doesn't even have a way for a player to find out what their ranking is - the ELO on the back end is hidden for some inane reason, and now I can't even know that I have to play with a certain guy a bit more carefully because he's half a second behind everyone else.
Knowing that I can't trust another player to launch a skill shot attack on target is valuable info - it changes the dynamic of how you'd play with that person.
Knowing that I can't trust another player to launch a skill shot attack on target is valuable info - it changes the dynamic of how you'd play with that person.
> I find this trend of social engineering via hiding info to be really annoying/disturbing.
I totally understand where you're coming from.
However, there exist people who play MOBAS who are easily frustrated and -emotionally- twelve years old. Hiding the rank and ping of all players gives such people two fewer things to get angry or upset about. If they're not angry or upset, they might just either play well and win the game, or -barring that- be a pleasant person to play with. :)
> Knowing that I can't trust another player to launch a skill shot attack on target [because of high ping] is valuable info...
Back in the day, I used to play Descent and Descent 2 [0] over both a 28.8 and -later- a 56k analog modem. My ping times varied from 300ms to 700ms. There were a few weapons in that game that were -effectively- skill shots and tricky to land, but -if you were aware of your ping, and your opponent's likely motion-, landing the shots was totally doable.
In short, just because your teammate is a HPB doesn't mean that he can't cope with that ping. He might always play with that ping, and -thus- be completely accustomed to it.
[0] Both games are 3D, 6DoF shooters.
I totally understand where you're coming from.
However, there exist people who play MOBAS who are easily frustrated and -emotionally- twelve years old. Hiding the rank and ping of all players gives such people two fewer things to get angry or upset about. If they're not angry or upset, they might just either play well and win the game, or -barring that- be a pleasant person to play with. :)
> Knowing that I can't trust another player to launch a skill shot attack on target [because of high ping] is valuable info...
Back in the day, I used to play Descent and Descent 2 [0] over both a 28.8 and -later- a 56k analog modem. My ping times varied from 300ms to 700ms. There were a few weapons in that game that were -effectively- skill shots and tricky to land, but -if you were aware of your ping, and your opponent's likely motion-, landing the shots was totally doable.
In short, just because your teammate is a HPB doesn't mean that he can't cope with that ping. He might always play with that ping, and -thus- be completely accustomed to it.
[0] Both games are 3D, 6DoF shooters.
Here's the problem though - Descent was optimized for those comparatively awful connections. League doesn't specify any minimums, but players report games becoming difficult around 100ms and nigh unplayable >=300ms. (From a casual google)
With LoL being as team-based as it is, knowing if/when my Blitzcrank is going to be able to land his rocket grab lets me know when to launch my ultimate and the rest of the team to join the fight. Having a high ping guy initiate a teamfight is.. unwise.
But quite honestly, I don't believe that hypothesis. If someone is going to get bothered to the point of toxicity by ping, something that is almost completely out of the player's control, they're going to get just as bothered being killed by Malzahar's global ultimate, someone picking the wrong team composition, not getting the lane they want, someone speaking a language other than perfect english, and so on.
It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child. As someone who is not an emotional child (at least I think not): that's disheartening.
I also don't believe Riot is serious about dealing with the problem - it takes a long history of being a serious jerk before the bans start getting trotted out (talk of "correction cards" aside).
And it makes sense why - banned people don't buy champions and skins.
With LoL being as team-based as it is, knowing if/when my Blitzcrank is going to be able to land his rocket grab lets me know when to launch my ultimate and the rest of the team to join the fight. Having a high ping guy initiate a teamfight is.. unwise.
But quite honestly, I don't believe that hypothesis. If someone is going to get bothered to the point of toxicity by ping, something that is almost completely out of the player's control, they're going to get just as bothered being killed by Malzahar's global ultimate, someone picking the wrong team composition, not getting the lane they want, someone speaking a language other than perfect english, and so on.
It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child. As someone who is not an emotional child (at least I think not): that's disheartening.
I also don't believe Riot is serious about dealing with the problem - it takes a long history of being a serious jerk before the bans start getting trotted out (talk of "correction cards" aside).
And it makes sense why - banned people don't buy champions and skins.
> Descent was optimized for those comparatively awful connections.
Descent 3? Yes.
Descent 1 and 2? No. Absolutely not. D1 and D2 were written for play on IPX/SPX LANs. All latency compensation was done manually by the player. It wasn't until the IPX/SPX to TCP/IP bridges known as Khan and Kali came out that folks could play them on the Internet using their shitty analog modems. In fact, a pretty big part of Descent 3's marketing was that it was optimized for play over a analog modem connection to the Internet. [0][1] (Source: I've -sadly- put far more hours into Descent 1 and 2 than probably any other video game. I could fly the first five or ten mines in both installments backwards, with rear-view off and not hit a wall. I also racked up far more in per-minute charges to my dial-up ISP than I ever want to think about.)
Edit: As I was thinking about this, I remembered that both D1 and D2 also offered null-modem (AKA crossover serial) and analog modem options for head-to-head multiplayer. Note that neither of these options transmit data over IP; the analog modem option just acted as a really long null-modem cable. ;) The analog modem option was likely added because once you've written the code to handle null-modem connections, you're at least 90% of the way to handling analog modem connections. But -again-, latency compensation was entirely handled by the player.
> ...players report games becoming difficult around 100ms and nigh unplayable >=300ms...
Here's the thing. If you have -for a little while- a > 300ms ping spike, it's going to disrupt your play. If you always have a > 300ms ping, two things are true:
1) It's the only way you've ever played, so your reaction times are going to tuned to that latency.
2) You're never going to have any basis for comparison, so (and this is key) you're not going to complain about your ping on forums.
> If someone is going to get bothered to the point of toxicity by ping, ... they're going to get just as bothered being killed by Malzahar's global ultimate, [etc.]
I generally agree with this. However, if one can easily remove two barriers to a player's enjoyment and not substantially impact the game's character or mechanics, why shouldn't one do such a thing? The action obviously produces more happiness than it eliminates.
> It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child.
This is hyperbole. You can't seriously believe that the removal of player level and ping information from the match loading screen is "drag[ging] everyone else down to the level of the emotional child".
> I also don't believe Riot is serious about dealing with the problem...
I played LoL for a looong time. I played before the Tribunal was even a thing. I can't speak to how serious Riot is about improving the community, but the difference between the pre-Tribunal community and the community today is night and day. The improvement is vast, and has largely come through the steps that Riot has taken to remind folks that
* This is just a game.
* Your teammates are people, too.
* People who enjoy themselves are more likely to play well. People who play well win more often.
and through steps to identify and prevent chronically toxic players from interacting with non-toxic players. Maybe the isolation machinery doesn't work quickly enough for your liking, but it is there, it does work, and it has had a real, positive effect on the character of the playerbase.
> ...banned people don't buy champions and skins.
Non-toxic players who refuse to play the game because the playerbase is full of toxic players also don't buy champs and skins, and they don't introduce their friends to the game. LoL is bigger than WoW. [2] Riot gains more by permabanning chronically toxic players than they lose in lost potential revenue from said players.
[0] https://www.youtube.com/watch?v=H0aXovYR9iQ
[1] It's more-or-less illegible, but there's a section in the middle of the retail D3 box that mentions how the game is optimized for modem Internet play: http://bubbalou.event3d.com/descent/dvd/d3web.jpg
[2] http://www.forbes.com/sites/insertcoin/2014/01/27/riots-leag...
Descent 3? Yes.
Descent 1 and 2? No. Absolutely not. D1 and D2 were written for play on IPX/SPX LANs. All latency compensation was done manually by the player. It wasn't until the IPX/SPX to TCP/IP bridges known as Khan and Kali came out that folks could play them on the Internet using their shitty analog modems. In fact, a pretty big part of Descent 3's marketing was that it was optimized for play over a analog modem connection to the Internet. [0][1] (Source: I've -sadly- put far more hours into Descent 1 and 2 than probably any other video game. I could fly the first five or ten mines in both installments backwards, with rear-view off and not hit a wall. I also racked up far more in per-minute charges to my dial-up ISP than I ever want to think about.)
Edit: As I was thinking about this, I remembered that both D1 and D2 also offered null-modem (AKA crossover serial) and analog modem options for head-to-head multiplayer. Note that neither of these options transmit data over IP; the analog modem option just acted as a really long null-modem cable. ;) The analog modem option was likely added because once you've written the code to handle null-modem connections, you're at least 90% of the way to handling analog modem connections. But -again-, latency compensation was entirely handled by the player.
> ...players report games becoming difficult around 100ms and nigh unplayable >=300ms...
Here's the thing. If you have -for a little while- a > 300ms ping spike, it's going to disrupt your play. If you always have a > 300ms ping, two things are true:
1) It's the only way you've ever played, so your reaction times are going to tuned to that latency.
2) You're never going to have any basis for comparison, so (and this is key) you're not going to complain about your ping on forums.
> If someone is going to get bothered to the point of toxicity by ping, ... they're going to get just as bothered being killed by Malzahar's global ultimate, [etc.]
I generally agree with this. However, if one can easily remove two barriers to a player's enjoyment and not substantially impact the game's character or mechanics, why shouldn't one do such a thing? The action obviously produces more happiness than it eliminates.
> It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child.
This is hyperbole. You can't seriously believe that the removal of player level and ping information from the match loading screen is "drag[ging] everyone else down to the level of the emotional child".
> I also don't believe Riot is serious about dealing with the problem...
I played LoL for a looong time. I played before the Tribunal was even a thing. I can't speak to how serious Riot is about improving the community, but the difference between the pre-Tribunal community and the community today is night and day. The improvement is vast, and has largely come through the steps that Riot has taken to remind folks that
* This is just a game.
* Your teammates are people, too.
* People who enjoy themselves are more likely to play well. People who play well win more often.
and through steps to identify and prevent chronically toxic players from interacting with non-toxic players. Maybe the isolation machinery doesn't work quickly enough for your liking, but it is there, it does work, and it has had a real, positive effect on the character of the playerbase.
> ...banned people don't buy champions and skins.
Non-toxic players who refuse to play the game because the playerbase is full of toxic players also don't buy champs and skins, and they don't introduce their friends to the game. LoL is bigger than WoW. [2] Riot gains more by permabanning chronically toxic players than they lose in lost potential revenue from said players.
[0] https://www.youtube.com/watch?v=H0aXovYR9iQ
[1] It's more-or-less illegible, but there's a section in the middle of the retail D3 box that mentions how the game is optimized for modem Internet play: http://bubbalou.event3d.com/descent/dvd/d3web.jpg
[2] http://www.forbes.com/sites/insertcoin/2014/01/27/riots-leag...
Heh, fair enough on Descent, I just remember a lot of time spent playing it over modem (and cursing noisy phone lines). Wasn't aware that LAN was the first class citizen there :)
But league, I was more referring to the standpoint of everyone else on the team, not the one guy with high ping. The one guy is living in a world that's offset a quarter to a half of a second behind everyone else - he can't play better than his ping. 350ms is enough time to blink, dodge a skillshot, and so on. In casual games or lower tier ranked that might not matter or even be noticeable, but get up into Diamond/Challenger? Painful.
What if one's enjoyment of the game is based on their performance and competition (i.e. the people who play in the higher levels of ranked?), rather than vague "happiness"?
I've played enough of the game to know that being stuck in a losing game (yes yes, it's just a game, not the point) sucks. It's awful and inescapable (since quitting hurts your team and penalizes you), and a lot of the time, it happens because of factors completely out of your control, whether that be internet connections, that one guy who just disconnected leaving you down a guy, or that one guy who uses phrases ending in "...or I feed". You enter this system to compete with others, and you're denied the one thing you came there to do (and knowing how long matches can take, perhaps the only time you had to do).
Competition as a system necessarily involves unhappiness on the part of one party.. I know few people who enjoy losing.
>You can't seriously believe that..
At the risk of sounding overly snarky, you just said that riot's goal is to maximize happiness. That means removing things used by children to cry about, even if the adults have a legitimate reason for their inclusion. That is the very definition of optimizing to the lowest common denominator.
>It is there, it does work..
And yet the new player experience is still awful. I can say this, because as far as League's system is concerned, I am a new player. There's a reason some high 90s percentage of my games are with friends, rather than matchmaking games that actually raise my standings- dealing with the community sucks. People who leave, people who feed, people who just act like jerks.
If there is improvement, it's completely invisible to me. Perhaps the higher ranks have been cleaned up, but the bottom of the pyramid is still godawful.
But league, I was more referring to the standpoint of everyone else on the team, not the one guy with high ping. The one guy is living in a world that's offset a quarter to a half of a second behind everyone else - he can't play better than his ping. 350ms is enough time to blink, dodge a skillshot, and so on. In casual games or lower tier ranked that might not matter or even be noticeable, but get up into Diamond/Challenger? Painful.
What if one's enjoyment of the game is based on their performance and competition (i.e. the people who play in the higher levels of ranked?), rather than vague "happiness"?
I've played enough of the game to know that being stuck in a losing game (yes yes, it's just a game, not the point) sucks. It's awful and inescapable (since quitting hurts your team and penalizes you), and a lot of the time, it happens because of factors completely out of your control, whether that be internet connections, that one guy who just disconnected leaving you down a guy, or that one guy who uses phrases ending in "...or I feed". You enter this system to compete with others, and you're denied the one thing you came there to do (and knowing how long matches can take, perhaps the only time you had to do).
Competition as a system necessarily involves unhappiness on the part of one party.. I know few people who enjoy losing.
>You can't seriously believe that..
At the risk of sounding overly snarky, you just said that riot's goal is to maximize happiness. That means removing things used by children to cry about, even if the adults have a legitimate reason for their inclusion. That is the very definition of optimizing to the lowest common denominator.
>It is there, it does work..
And yet the new player experience is still awful. I can say this, because as far as League's system is concerned, I am a new player. There's a reason some high 90s percentage of my games are with friends, rather than matchmaking games that actually raise my standings- dealing with the community sucks. People who leave, people who feed, people who just act like jerks.
If there is improvement, it's completely invisible to me. Perhaps the higher ranks have been cleaned up, but the bottom of the pyramid is still godawful.
For the vast majority of the population, pings were just nice to know. For a small minority it was a contributor to toxicity. I think it's just one of those things where a minority of (ab)users was able to ruin it for everyone. Unfortunately I don't feel like this is an isolated case within the League of Legends ecosystem.
For comparison, is there info on how Facebook is using MQTT for chat? EDIT: here's something, but not on backend architecture: https://www.facebook.com/notes/facebook-engineering/building...
All this fancy architecture doesn't change the fact that....I CALLED TOP FIRST!
I'm not a lol player, but I'm amazed to see a similar bug that existed on early versions of Battle.net 15 years ago. However this didn't had many impact since we played top vs bottom with game assigned positions (starcraft).
>I CALLED TOP FIRST!
We'll never know for sure. There have been many times where I'll type in chat: "top", my message will appear, and then somebody else will say "top" immediately after. But, a asking a third person reveals that the second player's message actually arrived first, even though my client shows my message first.
The chat architecture is terrible.
We'll never know for sure. There have been many times where I'll type in chat: "top", my message will appear, and then somebody else will say "top" immediately after. But, a asking a third person reveals that the second player's message actually arrived first, even though my client shows my message first.
The chat architecture is terrible.
Generally speaking users expect instantaneous feedback. It would feel super shitty if every time you hit enter there was lag until a server roundtrip posted the message.
Messages that the server saw earlier could easily be inserted partway up, right? We're talking about ~100ms most of the time; that's not going to cause you to miss messages.
Some chat service I use does this and it's confusing because I don't expect it. I check the last message to see if there's anything new, not necessarily the scrollback.
Basically, it triggers an annoying O(N) operation whenever it happens :)
Basically, it triggers an annoying O(N) operation whenever it happens :)
> It would feel super shitty if every time you hit enter...
I and many people I know don't agree with this. Instant feedback is nice, but not required. Moreover, if we need to choose between (instant feedback) and (knowledge of server receipt of a message and knowledge of message ordering), we'll choose the latter every time. (I'm glaring at you, Google Hangouts.)
Having the chat server lie to you about the state of the world always feels far worse than a delay between the time you submit your message and the time it appears.
I and many people I know don't agree with this. Instant feedback is nice, but not required. Moreover, if we need to choose between (instant feedback) and (knowledge of server receipt of a message and knowledge of message ordering), we'll choose the latter every time. (I'm glaring at you, Google Hangouts.)
Having the chat server lie to you about the state of the world always feels far worse than a delay between the time you submit your message and the time it appears.
Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter.
Even if the server corrected the order messages were received it'd feel shitty to not get your lane because someone has a faster roundtrip to the chat server.
Even if the server corrected the order messages were received it'd feel shitty to not get your lane because someone has a faster roundtrip to the chat server.
1) The likely worst-case trip time to the chat server is 250ms. [0] It's true that that's ~2.5x slower than the typical minimum human reaction time, but it takes a fair bit longer than this to type "top<enter>". [1]
2) The case where the server displays one message ordering to one player and another message ordering to another player is strictly worse than the case where the server takes 250ms longer to display the same message ordering to all players. The first case creates confusion and arguments, as well as feelings of distrust directed towards the chat software. The second case might cause a vague sense of displeasure at the quarter-second delay.
> Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter.
Agreed. Perhaps people could be civil when they play their games. :)
[0] In this case, RTT doesn't matter. Only the time required to get a packet from you to the server matters. I think that we can reasonably assume that the worst case RTT from anyone who's playing a vaguely-latency-sensitive online video game to anywhere else on the Internet is 500ms.
[1] Honestly, it might take half of this time for the pixel issued by a draw command to your graphics card to appear on screen. [2]
[2] http://superuser.com/questions/419070/transatlantic-ping-fas...
2) The case where the server displays one message ordering to one player and another message ordering to another player is strictly worse than the case where the server takes 250ms longer to display the same message ordering to all players. The first case creates confusion and arguments, as well as feelings of distrust directed towards the chat software. The second case might cause a vague sense of displeasure at the quarter-second delay.
> Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter.
Agreed. Perhaps people could be civil when they play their games. :)
[0] In this case, RTT doesn't matter. Only the time required to get a packet from you to the server matters. I think that we can reasonably assume that the worst case RTT from anyone who's playing a vaguely-latency-sensitive online video game to anywhere else on the Internet is 500ms.
[1] Honestly, it might take half of this time for the pixel issued by a draw command to your graphics card to appear on screen. [2]
[2] http://superuser.com/questions/419070/transatlantic-ping-fas...
This was fixed a few patches ago :)
Finally! I haven't played in over a year, but when I did, there was no correlation between the order of messages I saw and the order of messages somebody else saw... made for some rather frustrating arguments.
Although that matchmaking change where you get to say where you want to go and it matches you with a team helped too.
Although that matchmaking change where you get to say where you want to go and it matches you with a team helped too.
Do you know which patch specifically? Because I'm pretty sure I've seen out of order chat delivery in the past few weeks, although I suppose it could have been people being jerks in the pick phase.
Ah this is the reason why I hate that my workplace blocks gaming websites.
I used to work for those sorts of companies.
I fired them because I'm a mature professional who can figure out how best to utilize my time, TYVM.
Find a company that people actually enjoy coming to work and collaborating in. I mean, if you're an engineer, you're already doing the #1 job in the country, why poison the well at a company with a mistrustful IT policy?
Sorry but I have no end of disrespect for that kind of bullshit. If you're at a job and you're jerking around, people will eventually figure it out regardless of stupid content filtering. And in this case, this is a lost learning experience for you, as the article is very interesting.
I fired them because I'm a mature professional who can figure out how best to utilize my time, TYVM.
Find a company that people actually enjoy coming to work and collaborating in. I mean, if you're an engineer, you're already doing the #1 job in the country, why poison the well at a company with a mistrustful IT policy?
Sorry but I have no end of disrespect for that kind of bullshit. If you're at a job and you're jerking around, people will eventually figure it out regardless of stupid content filtering. And in this case, this is a lost learning experience for you, as the article is very interesting.
Look for a new job. Engineers are in high demand, you can find a place that doesn't treat you like a child.
Here's a snapshot that might work for you: https://archive.is/CFboh
An Erlang success story.
Thanks for posting this! I help to produce the blog at Riot and I really hope the tech community finds it useful.
In case anyone's interested, we just posted new content this week - an overview of our RFC process, and the continuation of a Docker&Jenkins tutorial: http://engineering.riotgames.com
In case anyone's interested, we just posted new content this week - an overview of our RFC process, and the continuation of a Docker&Jenkins tutorial: http://engineering.riotgames.com
What happens if a server get partitioned for an extended period of time? In their example, it sounds like Alice and Bob would be isolated from everyone else until connectivity to the cluster is re-established. Is there a mechanism that makes clients connected to an isolated server reconnect to the load balancer?
Assuming state isn't stored in the processes (which it typically isn't in Erlang) and is instead in Riak, I believe it would be Riak's responsibility to resolve the split. Which it is likely capable of doing.
Message ordering is still messed up causing a lot of confusing situations ruining games. "Rito plz."
TL;DR Erlang and distributed Riak using CRDT's (but the article is interesting!)
Why not just use IRC?
I'm a big fan of IRC, but there are two main reasons this argument doesn't work:
* IRC doesn't easily meet their extensibility requirements. Adding a new field to an existing IRC message just isn't doable.
* IRC has no concept of contacts. Their are bots on some servers and some clients will whois a set list of nicks on join to emulate this feature, but having contacts as a real concept is obviously preferable for Riot's use case.
* XMPP is also a standard protocol. If ejabberd didn't meet their needs, it's hard to see any standard ircd meeting them either.
* IRC doesn't easily meet their extensibility requirements. Adding a new field to an existing IRC message just isn't doable.
* IRC has no concept of contacts. Their are bots on some servers and some clients will whois a set list of nicks on join to emulate this feature, but having contacts as a real concept is obviously preferable for Riot's use case.
* XMPP is also a standard protocol. If ejabberd didn't meet their needs, it's hard to see any standard ircd meeting them either.
ejabberd met their needs, they just gradually rewrote 90% of it. :)
I seriously don't understand how IRC has withstood the test of time and gets relegated to the back room. 3d.tv (Twitch before there was twitch) Used IRC in a way I could just go to the IRC channel and connect to it and the people could write IRC programs. Twitch.tv IRC is a little more hidden.
http://help.twitch.tv/customer/portal/articles/1302780-twitc...
http://help.twitch.tv/customer/portal/articles/1302780-twitc...
It only really confuses me because there's not really a lot of value in writing your own "not quite IRC" type chat system. Just think about the development and administration time that went into setting something up that's probably not as good as something you can apt-get install.
I can't agree more.
Are there any great examples of IRC being used for the backend of a games chat system?
How would you envision using IRC in this case?
Would have been great to see RiotGames write their own custom framework to interface between their game and an IRC backend, open source it for other companies to contribute to and prevent further reinventing of the wheel.
Are there any great examples of IRC being used for the backend of a games chat system?
How would you envision using IRC in this case?
Would have been great to see RiotGames write their own custom framework to interface between their game and an IRC backend, open source it for other companies to contribute to and prevent further reinventing of the wheel.
Worms Armageddon and some other Worms games use IRC for lobby chat.
http://worms2d.info/WormNET_(Worms_Armageddon)
http://worms2d.info/WormNET_(Worms_Armageddon)
Pretty sure Eve did/does. Twitch does.
I haven't really used Twitch broadcasting since it was just justin.tv, but back in those days it was very much just IRC. You could just connect to the IRC channel, and in fact if you wanted to capture logs of the chat or do bots that's the way you had to do it. I have no idea what it's like now.
don't really have to deal with scaling when so many players are chat restricted.
I really wish more game studios would do this (Blizzard especially).