Ask HN: Browser testing, where are we at?
3 comments
We run farms of puppeteer and headless Firefox.
Not retail, but it's a complex app (that is also available in Electron); things like video plays have trappable events to hook into.
For example, hit play -> time how long until video starts playing -> hit pause at 00:19 (random example time), take a screenshot of the #video-container element DOM, and do a comparison to a known-good screenshot.
It's not overly non-traditional, but still much more than forms and buttons.
Other things include typing into an autocompleting search box, and having middleware capture /autocomplete.json?q=* type of requests and comparing them to see whether they are coming up OK, what queries are coming up, and then dumping the DOM of .autocomplete-search-results-container or similar and iterating them through to turn it into an array again, and ensure sorting order matches.
Not retail, but it's a complex app (that is also available in Electron); things like video plays have trappable events to hook into.
For example, hit play -> time how long until video starts playing -> hit pause at 00:19 (random example time), take a screenshot of the #video-container element DOM, and do a comparison to a known-good screenshot.
It's not overly non-traditional, but still much more than forms and buttons.
Other things include typing into an autocompleting search box, and having middleware capture /autocomplete.json?q=* type of requests and comparing them to see whether they are coming up OK, what queries are coming up, and then dumping the DOM of .autocomplete-search-results-container or similar and iterating them through to turn it into an array again, and ensure sorting order matches.
I feel that nothing significant has changed over the past decade. There newer frameworks that are faster not in order of magnitudes, more stable, have a bit more features or run in the browser itself but at the end you are interacting with elements in your DOM and you are dependent on 20 other cogs and gears that are not part of the browser.
Your best bet would be doing automated smoke GUI testing, lots of API testing and below and above all a layer of manual testing done by the whole team during and after development.
In one team we used high speed cameras, or screen captures on mobiles, to capture images containing debug timestamps and do image comparison of selected regions to reference ones, I think it was using imagemagick. This worked but required constant maintenance and manual interventions.
Your best bet would be doing automated smoke GUI testing, lots of API testing and below and above all a layer of manual testing done by the whole team during and after development.
In one team we used high speed cameras, or screen captures on mobiles, to capture images containing debug timestamps and do image comparison of selected regions to reference ones, I think it was using imagemagick. This worked but required constant maintenance and manual interventions.
They just employ people to do the testing
How are retail websites doing GUI testing (buttons, forms) vs more complex visual cases like Google Maps or websites with non-traditional layouts?
Are there any papers or interesting libs that come to mind?