Can You See Me Now? A measurement study of Zoom, Webex, and Meet(arxiv.org)
arxiv.org
Can You See Me Now? A measurement study of Zoom, Webex, and Meet
https://arxiv.org/abs/2109.13113
39 comments
> Zoom IMHO has nailed the workflow for clicking an autodial link on my phone, joining the call by video on my laptop and having the participant pin appear on the screen quick enough that I can type it right as the automated voice prompt asks for it on the phone
This sounds inconvenient? For Meet, I click the invite link from my calendar, email, or meeting list for an upcoming meeting list, or from the slack or other message a coworker sent me for the adhoc one, and I'm in, with video and audio going through the one device.
This sounds inconvenient? For Meet, I click the invite link from my calendar, email, or meeting list for an upcoming meeting list, or from the slack or other message a coworker sent me for the adhoc one, and I'm in, with video and audio going through the one device.
You have the same option on Zoom. But then, just as in any other videoconferencing app, your audio is now VoIP - an acronym which stands for "heavily compressed, choppy and muffled"
Dialing in from your phone increases the audio quality for everyone by a factor of approximately ∞
Dialing in from your phone increases the audio quality for everyone by a factor of approximately ∞
Are phone networks better where you're from? The audio compression on the phone network renders the sound quality/clarity inferior to a built in laptop mic in my experience.
I'm in NY currently but have dialed in from several different states in the US and the quality from the phone is always significantly better than VoIP
I just have had a totally different experience then, I have to say. I associate people dialing in from their phones with high levels of background noise, both hissing static and actual background noise, and clipping/lost words. This includes coworkers in Europe and california.
After all, the listener is receiving over VoIP, so you've got phone network issues layered on voip issues.
After all, the listener is receiving over VoIP, so you've got phone network issues layered on voip issues.
Just want to add that I've had this exact experience with people using phone audio with the added variant of them losing cellular signal every few minutes.
Weird, here (france) I made all my family's calls go through VoIP and voice clarity increased a thousandfold, it feels like going from a 1930 radio station to a CD
What’s the preferred service in France for that? Do you mean you set it up yourself with a particular SIP/IAX vendor, or just switched to free.fr or one of those providers?
we all have a VoWifi phone provider and stack
You can also join the Meet in a browser, then under the option memu (...) is “Use a phone for audio”. In that user flow the meeting calls your phone and when you join your devices are linked such that the command-d hot key in the browser will mute/unmute your phone mic.
I've only used Meet once, actually last week, and it took me significantly longer to get on the call because I couldn't start dialing until after clicking "use a phone for audio" whereas with Zoom I call before even joining the call on my computer
Maybe it was user error, but it felt much slower to get everything up and running
Maybe it was user error, but it felt much slower to get everything up and running
Fully agree if you've installed the app on your phone and the drain on your battery is wild.
If you don't have the app, joining via phone is required, finding the number is a few seconds including passcodes and then sometimes passwords which often elude me in terms of their location.
If you don't have the app, joining via phone is required, finding the number is a few seconds including passcodes and then sometimes passwords which often elude me in terms of their location.
Personally, I've been pretty impressed with Jitsi especially given that it's open source. So far it's been easy and the people on the other end have had no problem connecting, despite not having previously downloaded any client software. I had high hopes a ways back for appear.in but they unfortunately lost that domain and I have no idea what the new one is.
Appear.in is now Whereby.com
...and I already forgot the new name
The quality for free plans have tremendously decreased, to the point that I cannot recommend it to family/friends/non-corporate employees.
Huge whereby/appear.in fan, used to run our family wknd catch-ups on there.
Huge whereby/appear.in fan, used to run our family wknd catch-ups on there.
> In case of Webex, all sessions created in the US appear to be relayed via its infrastructure in US-east. This causes the sessions among users in US-west to be subject to artificial detour, inflating their streaming lag.
Former webexer here. That depends on many criteria, sessions can go through multiple locations depending on where the customer is located, the features being used, the type of customer, etc. Since Webex also does transcoding (useful when a 4k wall system and a cell phone are on a same meeting), that can influence where the traffic flows.
Webex, Zoom and Google also all have their own backbones to reduce the number of hops over the public internet.
In this case the researchers appear to be based in New Jersey so they would be more likely to enter the internal network via the eastern POPs.
Former webexer here. That depends on many criteria, sessions can go through multiple locations depending on where the customer is located, the features being used, the type of customer, etc. Since Webex also does transcoding (useful when a 4k wall system and a cell phone are on a same meeting), that can influence where the traffic flows.
Webex, Zoom and Google also all have their own backbones to reduce the number of hops over the public internet.
In this case the researchers appear to be based in New Jersey so they would be more likely to enter the internal network via the eastern POPs.
> Despite its significance, there has not been any systematic study characterizing the user-perceived performance of existing videoconferencing systems other than anecdotal reports.
> We find that the existing videoconferencing systems vary in terms of geographic scope, which in turns determines streaming lag experienced by users. We also observe that streaming rate can change under different conditions (e.g., number of users in a session, mobile device status, etc), which affects user-perceived streaming quality. Beyond these findings, our measurement methodology can enable reproducible benchmark analysis for any types of comparative or longitudinal study on available videoconferencing systems.
Seems like a very useful study at first glance!
> We find that the existing videoconferencing systems vary in terms of geographic scope, which in turns determines streaming lag experienced by users. We also observe that streaming rate can change under different conditions (e.g., number of users in a session, mobile device status, etc), which affects user-perceived streaming quality. Beyond these findings, our measurement methodology can enable reproducible benchmark analysis for any types of comparative or longitudinal study on available videoconferencing systems.
Seems like a very useful study at first glance!
As much as I'd like to save the money, Zoom is evidently still the best option. Hope the others continue to catch up. Neat to see that Meet is almost as good, according to this data - it was hot garbage a few years ago.
I worked at a company that blocked the native clients for every conference system except WebEx. The Zoom and Skype web experience was not good, particularly since all endpoints were loaded with half a dozen Infosec agents and sending all traffic through web proxies on the other coast.
Surprise, the somewhat primitive Webex offered the best experience in that scenario and vendors who wanted to meet were inclined to accommodate us.
Surprise, the somewhat primitive Webex offered the best experience in that scenario and vendors who wanted to meet were inclined to accommodate us.
Webex interface is… not great. Zoom isn’t either and I’m quite surprised it’s as popular as it is given the UX. Surprising these things are as difficult as they are and yet get such adoption.
The webex interface is perfectly fine, but if you are comparing it for the use case of zoom and skype, your target interface is wrong for the tool you're using. Webex is for working on something together. Multiple users using an application to manage some piece of hardware or software for example. Zoom and skype are for talking about something together.
Let's say you get a new server. It needs several teams to set it up - storage to provision/rescan/configure disk, infosec to do domain/antivirus/dlp work, db team to get the database going. They're going to share an rdp session over webex and hand it off to each other to type commands, with everyone observing and catching mistakes. Yes, it's possible to give desktop control to someone on zoom, however that is a secondary afterthought of zoom, while for webex it's the primary function. Right tool for the job - don't transport a piano in a sports car then complain the sports car is.. not great.
Let's say you get a new server. It needs several teams to set it up - storage to provision/rescan/configure disk, infosec to do domain/antivirus/dlp work, db team to get the database going. They're going to share an rdp session over webex and hand it off to each other to type commands, with everyone observing and catching mistakes. Yes, it's possible to give desktop control to someone on zoom, however that is a secondary afterthought of zoom, while for webex it's the primary function. Right tool for the job - don't transport a piano in a sports car then complain the sports car is.. not great.
I am not sure you can answer a subjective assessment with a blanket dismissal - obviously at some level the webex UI can't be completely unusable (lots of people do use it), but apparently some people just don't like it. It has what strikes me as an unpleasant installer (like BlueJeans, it seems to only work if you download an installer and run it for every call, which seems excessive somehow).
YMMV, but one of my customers has to use WebEx and it's terrible. Particularly in large calls with many people talking, it's very slow to switch video to whoever's speaking: it takes a few seconds to even realize the voice has changed, then it flips but shows a spinny loader, and by the time the video finally appears somebody else has started talking and the sequence restarts. By contrast, this is a complete non-issue for Zoom, Meet or Teams.
We use Webex at work (a very large tech company). I use it for it’s intended purposes. My issues:
1. The reaction buttons don’t indicate that anything is actually happening for you when you click them, you only can see others reactions. They’re also quite hidden when they come, make no noise so easy to miss, and don’t tie to the video.
2. The “people joined” ding is very non-intelligent. It used to be when 30+ people were queued for something to start, once it did it just dinged like crazy. Now it seems work has turned that off in a way I cannot turn back on, which means when people are late for a 1:1 they’re staring at me and I have no idea since I now have people blocking. A heuristic that would be aware of # participants, when people are joining, and batching up dings I cannot imagine would be groundbreaking research when this is your product you’re building.
3. The actual controls for video on mobile to switch camera is quite bad.
4. Screen sharing works _ok_, except it has no integration with the resolutions on both sides, Apple’s accessibility-based zoom, I’m given a red dot for a cursor but it doesn’t appear that way on the other side, and cursor tracking is extremely laggy and jumpy on the remote side despite my red cursor that’s giving me the impression of a laser pointer equivalent.
5. When someone is screen sharing, the controls for what to do with the video of others are not very good. Order is non-deterministic (afaik), not easily changed, provides few layouts, and not easily hidden.
6. When screen sharing I’m given a large list of all my windows to choose from. They’re in effectively a random order (not alphabetical, not by z-index on my Mac, etc), and the mini photo of them are usually unintelligible and difficult to pin point the one I want.
These are just some of the bad UX that I encounter daily in work usage. I could probably come up with a list of 10+ more issues without even thinking much about it, and we haven’t even touched the “interface” level issues (eg actual design).
1. The reaction buttons don’t indicate that anything is actually happening for you when you click them, you only can see others reactions. They’re also quite hidden when they come, make no noise so easy to miss, and don’t tie to the video.
2. The “people joined” ding is very non-intelligent. It used to be when 30+ people were queued for something to start, once it did it just dinged like crazy. Now it seems work has turned that off in a way I cannot turn back on, which means when people are late for a 1:1 they’re staring at me and I have no idea since I now have people blocking. A heuristic that would be aware of # participants, when people are joining, and batching up dings I cannot imagine would be groundbreaking research when this is your product you’re building.
3. The actual controls for video on mobile to switch camera is quite bad.
4. Screen sharing works _ok_, except it has no integration with the resolutions on both sides, Apple’s accessibility-based zoom, I’m given a red dot for a cursor but it doesn’t appear that way on the other side, and cursor tracking is extremely laggy and jumpy on the remote side despite my red cursor that’s giving me the impression of a laser pointer equivalent.
5. When someone is screen sharing, the controls for what to do with the video of others are not very good. Order is non-deterministic (afaik), not easily changed, provides few layouts, and not easily hidden.
6. When screen sharing I’m given a large list of all my windows to choose from. They’re in effectively a random order (not alphabetical, not by z-index on my Mac, etc), and the mini photo of them are usually unintelligible and difficult to pin point the one I want.
These are just some of the bad UX that I encounter daily in work usage. I could probably come up with a list of 10+ more issues without even thinking much about it, and we haven’t even touched the “interface” level issues (eg actual design).
I joined an online conference via WebEx earlier this year, made sure my audio/video were muted, and started typing my notes. The presenter stopped mid-slide and called me out because my keyboard audio was interrupting him.
I had to un-mute and then re-mute my audio to continue.
I know I shouldn't have relied on a software switch when I have a perfectly good hardware disable button, but WebEx embarrassed me in front of dozens of my peers, and I will keep that in mind whenever I get to choose what platform a meeting goes on.
I had to un-mute and then re-mute my audio to continue.
I know I shouldn't have relied on a software switch when I have a perfectly good hardware disable button, but WebEx embarrassed me in front of dozens of my peers, and I will keep that in mind whenever I get to choose what platform a meeting goes on.
What, you don't netboot your server to a shared baseimage that runs your infra as code tool of choice?
Having used MSTeams, I've stopped complaining about Zoom. It's stable and straightforward. The on-call UI has its warts, especially on mobile, but otherwise it's fine.
Teams, by contrast, is buggy and inconsistent and weird and has the awful arbitrary rectangle subdividing of the chat window instead of just showing me the other people's full cameras.
Teams, by contrast, is buggy and inconsistent and weird and has the awful arbitrary rectangle subdividing of the chat window instead of just showing me the other people's full cameras.
The company I work for also blocks the native Zoom client, and I know of several other companies that do it as well. If we have a Zoom meeting we have to use the web client. Can anybody give me a reasonable rationale for why this is so common?
> Surprise, the somewhat primitive Webex offered the best experience in that scenario
But it’s not a surprise in this scenario since WebEx is the only one not hobbled, no?
I’ve used all 3 and WebEx is second to Zoom but better than Skype.
But it’s not a surprise in this scenario since WebEx is the only one not hobbled, no?
I’ve used all 3 and WebEx is second to Zoom but better than Skype.
My prayers have been answered. Google Meet devs I beg you, make the app more efficient.
Wait, Google Meet? I thought this was going to be about Jitsi Meet (not joking). Crap. I hadn't heard of Google Meet. So now I still want to know how Jitsi compares.
Ok, because you asked nicely!
When you're hopping from one call to the other, having the ability to quickly drop and join the next one [EDIT: with audio from my phone] is a big win
That, the global Alt+A / Alt+V for the audio / video mute toggle respectively, and the "hold spacebar to talk" hotkeys make it the clear winner from an UX perspective IMHO, even if that X isn't really perfect
At the end of the day, most calls are virtually lag free so I care more about my ability to efficiently interface with the app than I do about a perceived streaming quality [EDIT: except, of course, for those participants who don't join the audio via their phone, as their voices inevitably suffer from some lag and distortion]