Show HN: Percy.io, a continuous visual integration tool(percy.io)
percy.io
Show HN: Percy.io, a continuous visual integration tool
https://percy.io/
56 comments
A coworker has has a small tool for accomplishing this:
https://github.com/LesterTheTester/dfrnt
It has been integrated with Travis CI and GitHub pull requests so that non-passing diffs are appended as a comment to the pull request by a bot account.
If there's an intended diff, a pull request can also automatically be submitted against the original pull request which, on merge, updates the "gold master" images and allows the original pull request to pass.
It has been integrated with Travis CI and GitHub pull requests so that non-passing diffs are appended as a comment to the pull request by a bot account.
If there's an intended diff, a pull request can also automatically be submitted against the original pull request which, on merge, updates the "gold master" images and allows the original pull request to pass.
I've used gatling on a few redesigns with success
https://github.com/gabrielrotbart/gatling
I have some tweaks i was going to merge in that gave the diff's as %'s (so you could ignore small differences like fonts being different) as well as animate between the two versions. On my TODO was grabbing the actual element style attributes and then actually just diffing that, so the differences would be more or less spelled out.
Still, cool stuff!
https://github.com/gabrielrotbart/gatling
I have some tweaks i was going to merge in that gave the diff's as %'s (so you could ignore small differences like fonts being different) as well as animate between the two versions. On my TODO was grabbing the actual element style attributes and then actually just diffing that, so the differences would be more or less spelled out.
Still, cool stuff!
If you are interested in how something like this was made, there was a good talk about building something like this at RailsConf this year: http://youtu.be/Q4ttqkIEM7g
It's an awesome talk! Jessica Dillion has actually helped me with Percy a bit. :) She's great, you can see her slides here: http://www.slideshare.net/jessicardillon/implementing-a-visu...
I've been working on VisualReview (https://github.com/xebia/VisualReview), an open source project for visually reviewing a (web-)application as part of a pull-request, very similar to percy.io.
We're releasing our first version somewhere next week which includes a basic comparison GUI and testing api's. PR workflow planned in the release after that. Interesting to see we're not the only ones with this idea.
We're releasing our first version somewhere next week which includes a basic comparison GUI and testing api's. PR workflow planned in the release after that. Interesting to see we're not the only ones with this idea.
This looks really great. The integration with Github and CircleCI makes it a very attractive option for us as it would slot right in to our existing workflows.
I wasn't a fan of giving you github permissions just to sign up for an invite.
I would have agreed with this, but the permissions they ask for are pretty benign and clearly relevant to their mission.
I'm a member of organisations that may not be comfortable with me giving permission by proxy.
I couldn't find a way to sign up and exclude private repos or organizations.
I couldn't find a way to sign up and exclude private repos or organizations.
Fair enough, but I think that's a limitation of Github rather than a problem imposed by Percy.
Percy.io's Invite Request system is tied to github.
How is that not a problem imposed by Percy?
I'd feel more comfortable having a different mechanism to request an invite.
While the permissions don't grant Code Access, it does ask to read Organization and Team membership. I'd expose data for every member of my companies github organization to Percy, JUST for an invite.
How is that not a problem imposed by Percy?
I'd feel more comfortable having a different mechanism to request an invite.
While the permissions don't grant Code Access, it does ask to read Organization and Team membership. I'd expose data for every member of my companies github organization to Percy, JUST for an invite.
Percy is a tool that integrates with Github to set build status on Github pull requests. Of course it needs to tie into Github.
You can sign up for the mailing list instead: https://percy.io/#updates
And yes it's a GitHub limitation, we request some limited readonly access and no code access (we don't need it).
And yes it's a GitHub limitation, we request some limited readonly access and no code access (we don't need it).
That's weird, because I'm a member of an org that Github explicitly requires you approve this access for apps like this.
is this tool any good? anyone used percy and can give a brief review of how they thought it was / what worked well? thank you.
Congrats @fotinakis
Check out also http://shoov.io/ - there are a few blog posts, that explain the concepts and point to examples. Apart of the visual regression testing it also allows monitoring the live sites with your preferred functional testing frameworks (e.g. casperJS, Behat, etc').
Open source of course :)
Check out also http://shoov.io/ - there are a few blog posts, that explain the concepts and point to examples. Apart of the visual regression testing it also allows monitoring the live sites with your preferred functional testing frameworks (e.g. casperJS, Behat, etc').
Open source of course :)
This looks promising. Our project has a plethora of selenium tests, but they're focused on functionality only, not appearance. We can tell there is a table, button, link, form, etc. and click it. On that side we're reliable.
But we really let ourselves down sometimes with regressions to the look and feel of the application.
Are there any competitor tools for this visual testing ?
But we really let ourselves down sometimes with regressions to the look and feel of the application.
Are there any competitor tools for this visual testing ?
We (Harry's) have released a tool called Kontrast: https://github.com/harrystech/kontrast, http://engineering.harrys.com/2014/09/24/kontrast.html
It's written in Ruby but is not really dependent on Rails or anything to run. We use it to compare a list of URLs between the version we are currently testing in CircleCI and our production site. Any differences are highlighted in a nice gallery and we get a message in our Hipchat room.
It's written in Ruby but is not really dependent on Rails or anything to run. We use it to compare a list of URLs between the version we are currently testing in CircleCI and our production site. Any differences are highlighted in a nice gallery and we get a message in our Hipchat room.
You can try https://app.webmate.io. It's a hosted service for layout, regression, and cross-browser testing. Basic features are free. The diff-ing works on the DOM, is visualised in screenshots and identifies the XPath of elements that are found to be changed, removed, or added. Tests are run on browsers/vm that are part of the service, giving you access to 200+ os/browser combinations.
There's VisualReview[0], but only self-hosted.
[0]: https://github.com/xebia/VisualReview
[0]: https://github.com/xebia/VisualReview
ghostinspector.com (one of the products from my company)
check eggplant / testplant (not free though)
Percy.io seems somewhat similar to BBC's wraith[1], albeit easier to integrate.
1.http://bbc-news.github.io/wraith/index.html
1.http://bbc-news.github.io/wraith/index.html
This is neat. Recently started doing Selenium tests with Scrutinizer CI, but we've had issues where the UI is broken and we find out a day later... I'll definitely keep an eye out for you guys
Woohoo! Glad to hear it. :) We have an annoucement-only mailing list if anyone doesn't want to sign up with GitHub: https://percy.io/#updates
Are you an one man army? Congrats! :) This is great. How do you plan to react to potential changes in rendering engines? Would those be shown as test failures as well?
Great question! And yes. :) My plan at the moment for that is, because I know which browser exactly it was rendered in originally, I can put a notice at the top of the page to say something like "We just upgraded our rendering browser! The diffs below might because of that and not because of a change you made." -- and then after that one build, things would be back to normal.
My favorite in this space is Applitools eyes (https://applitools.com/).
Eyes has really advances image processing capabilities, which meant two huge advantages over everything else I looked at:
1. It's not prone to rendering differences. I can set a baseline in one browser/OS combo, run it on other combinations and set it to only consider real regressions as a failure.
2. It can validate the layout of the page even with dynamic content.
Eyes has really advances image processing capabilities, which meant two huge advantages over everything else I looked at:
1. It's not prone to rendering differences. I can set a baseline in one browser/OS combo, run it on other combinations and set it to only consider real regressions as a failure.
2. It can validate the layout of the page even with dynamic content.
Doesn't say what it does. I can deduce from the title and landing page that it has something to do with UIs and CI. From the FAQ it seems it is web only, as it mentions "the page" (assume that is an html page).
Is this an appropriate summary?
"It can help you visually review changes made to a web front end as part of CI/QA"
Would be neat if you could plug any kind of front end screenshot rendering and not just web.
Is this an appropriate summary?
"It can help you visually review changes made to a web front end as part of CI/QA"
Would be neat if you could plug any kind of front end screenshot rendering and not just web.
I signed up. Clicked on the survey, clicked back, got a blank page. Did it again, got a live page. Did it one final time, got a blank page. Tried it again just now, blank page (but there is HTML in the page source).
There's another tool out there that does something similar called Screenbeacon (https://www.screenbeacon.com/).
There's no mention about pricing... any rough idea/estimation?
[deleted]
Devil's advocate - won't this tend to bolster the outdated attitude that the front end must be pixel-perfect?
It depends on the project. But when you're dealing with mobile devices you don't get to test even a fraction of the models available unless you've got enough scale to have a dedicated testing lab.
I'm not sure I consider that to be true. Why is pixel perfection out-dated?
Perhaps because rigid constraints are useless with the number of different devices out today, and layouts should instead be highly adaptive?
No that's wrong. The screenshots are not based on one device looking like another. It's based on one device looking like X all the time.
For example this uses Firefox, if all I change is something in the backend. My front end should look the exact same before the change and after.
For example this uses Firefox, if all I change is something in the backend. My front end should look the exact same before the change and after.
Really interested how this will work with responsive. Does it test a set size each time? Multiple sizes?
Has anyone used visualdiff.com? The services seem pretty similar.
I think they might only do live sites, the thing about Percy is that it can be integrated earlier into your feature/browser tests for all pages, even those behind logins.
[deleted]
Interesting read! Bummer to hear the news though.
I wonder, does it handle responsive UIs?
would pixel by pixel diffs be an overkill?
Has anyone used visualdiff.com?
[deleted]
So.. is this just a lot of wrappers around wraith? Or something else?
Actually it's all home grown, right now it renders the page and screenshots in Firefox 38 ESR and then the actual diff image is produced by imagemagick's compare function, with some special things around it to handle edge cases with width/height differences and some others.
Responsive capabilities will really improve usefulness, and the absence is somewhat of a deal-breaker. On responsive sites developers will have to eyeball each change even when using Percy, so checking the desktop resolution wouldn't be much additional effort relative to checking all other resolutions for each page.
On the biz-dev side, if it's going to be a paid product it might be a good idea to start charging ASAP!