Ask HN: How does everyone handle mobile browser testing?
22 comments
For local automated testing of mobile browsers on both iOS and Android you probably want to have a look at https://appium.io/, which uses the Webdriver protocol.
If you want to run the same tests against both mobile and desktop browsers, I recommend https://webdriver.io/, which uses Appium for mobile testing.
If you're interested in a Docker setup as a boilerplate to test against all major browsers (both mobile and desktop) with best practices for e.g. test video recordings, you might also wanna take a look at this repo of mine: https://github.com/blueimp/wdio
If you want to run the same tests against both mobile and desktop browsers, I recommend https://webdriver.io/, which uses Appium for mobile testing.
If you're interested in a Docker setup as a boilerplate to test against all major browsers (both mobile and desktop) with best practices for e.g. test video recordings, you might also wanna take a look at this repo of mine: https://github.com/blueimp/wdio
Appium with Selenium (or XCUITest for iOS) will shorten your life expectancy and grow you some grey hairs, but there is no other better alternative for testing at user level. It is highly unreliable and each new phone OS version will tend to break your automation
As for finding devices there are a few cloud providers like xamarin or perfecto (never really worked with them in production) that gives you remote access to a lot of real devices and software versions
Having said that try to divert more testing towards lower levels- unit tests, testing on simulators, integration tests with a full phone and browser etc.
A good place to also ask is: https://testersio.slack.com
As for finding devices there are a few cloud providers like xamarin or perfecto (never really worked with them in production) that gives you remote access to a lot of real devices and software versions
Having said that try to divert more testing towards lower levels- unit tests, testing on simulators, integration tests with a full phone and browser etc.
A good place to also ask is: https://testersio.slack.com
thank you
The first task to write some sort of test script to have assertions for the core features of your game.
For testing specific UI features, I would recommend looking into Selenium/Appium. Selenium for desktop browser testing, Appium for mobile testing (using Appium you can do mobile specific things like touch actions for instance).
For testing JavaScript, you can take a look at libraries like https://jasmine.github.io/.
Once you have a test script(s) ready, you can point it to cloud based testing services like BrowserStack (I work there).
We provide a large number of actual physical mobile devices to test on (no emulators), For complete list see: https://www.browserstack.com/list-of-browsers-and-platforms/...
Our service is completely free for open source projects !
If you want to go the manual way, you can also do that with BrowserStack, we provide the 'Live' product where you can control and test your website on actual mobile devices straight from your browser.
For testing specific UI features, I would recommend looking into Selenium/Appium. Selenium for desktop browser testing, Appium for mobile testing (using Appium you can do mobile specific things like touch actions for instance).
For testing JavaScript, you can take a look at libraries like https://jasmine.github.io/.
Once you have a test script(s) ready, you can point it to cloud based testing services like BrowserStack (I work there).
We provide a large number of actual physical mobile devices to test on (no emulators), For complete list see: https://www.browserstack.com/list-of-browsers-and-platforms/...
Our service is completely free for open source projects !
If you want to go the manual way, you can also do that with BrowserStack, we provide the 'Live' product where you can control and test your website on actual mobile devices straight from your browser.
Thank you Pulkitsh, late at night now, but I'll have us look at this tomorrow.
Let me add visual diffing constraint to the discussion, otherwise I need the same as OP. percy.io is a cool visual diffing tool I used and liked a lot.
Last time when I used it it supported only chrome/firefox, but both mobile and desktop.
Please let me ask how is visual testing supported in other solutions like: selenium, appium, browserstack, webdriver.io, etc aka what do you use for crossbrowser visual testing?
Last time when I used it it supported only chrome/firefox, but both mobile and desktop.
Please let me ask how is visual testing supported in other solutions like: selenium, appium, browserstack, webdriver.io, etc aka what do you use for crossbrowser visual testing?
They let you test with real hardware. https://www.perfecto.io/ Used them for a long time and everytime amazed by the amount of hardware they have. The whole setup is kind of need.
HeadSpin https://www.headspin.io/ provides a large set of real Android and iOS devices distributed around the globe. Testing on real hardware is always better than testing in an emulator, and testing in a real location provides additional data points that help you eliminate performance issues with your application.
Testing just mobile web browsers is of course supported, but if you ever have the need you can test native game engines and get video and network performance statistics at the same time. Automated testing via Appium is natively supported, or you can bring your own framework.
Testing just mobile web browsers is of course supported, but if you ever have the need you can test native game engines and get video and network performance statistics at the same time. Automated testing via Appium is natively supported, or you can bring your own framework.
We mainly use Browserstack. It's a very powerful tool, especially with their very impressive Chrome dev-tools integration, though it can be a bit laggy at times. Quite unbeatable for the huge array of devices they have available though.
We also have a device table with some common devices that we can use whenever needed. A recent Samsung Galaxy phone (not sure which model), iPhone SE, Kindle Fire and an iPad etc.
We also have a device table with some common devices that we can use whenever needed. A recent Samsung Galaxy phone (not sure which model), iPhone SE, Kindle Fire and an iPad etc.
You mention foreign-language browsers. Interested to know which ones in particular you are dealing with.
From our manual testing the worst seem to be 360, Sogou, and Liebao.
Most devtools on browsers have "responsive tools" - letting you change your screen to mobile or adjusting pixels. I use it sometimes for my web apps.
Responsive design mode helps with a subset of CSS layout issues. It's much faster and easier than hooking up debugging on a live device, so it's invaluable during CSS development, but since it's still the desktop browser with just a few parameters tweaked, it's not going to be adequate for tracking down real-mobile-browser-real-device issues in a game engine.
[deleted]
Judging by the so-called quote-unquote mobile browsing "experience", they don't.
I'm interested in this, too.
Are you manually testing, or programmatically/automated? If it’s the latter then Browserstack (https://www.browserstack) provides mobile devices for testing on mobile browsers in a cloud device farm, there are others too like SauceLabs and AWS Device farm. I’ve had good success with BrowserStack but you should check whether they support the browsers you need, they also provide an easy way to tunnel through to your staging/qa environment if you need to do that. For reporting, especially if it’s not just you and you’re in a team, consider Tesults (https://www.tesults.com), I’m the founder so hit me up with a message and I’ll do what I can to get you setup.
Right now it's been manual stuff. We get reports of users have trouble on bizarre mobile phone combos. Midnight here and just back from the gym, but I'll make sure we check out BrowserStack and Tesults tomorrow and let you know if we hae any questions.
I was curious, so I looked. BrowserStack charges $29/month for live (not-automated) mobile testing, boasting "2000+ Real iOS and Android devices". They also have a "freelance" plan for $19/month for those on a smaller budget.
So I've been working on a mobile game engine for Saito (https://apps.saito.network/arcade) and am running into testing issues.
Right now I'm having to manually run the applications on whatever cellphones and tablets we can find. Is there a better way to do this? Asking as I figure if anyone has solved this problem with an automatic testing suite or something similar it will be someone who reads HN....
Goal is to test mobile browsers to see which ones work and eliminate avoidable failures. Most of the problems we are flagging to date are javascript-implementation issues or things like auto-transaction "features" on foreign-language browsers that strip events from the page and/or stop them from running very quickly.
Any suggestions? Anyone have a startup that tackles this sort of problem?