SlimerJS: a replacement for PhantomJS using recent Firefox(slimerjs.org)
slimerjs.org
SlimerJS: a replacement for PhantomJS using recent Firefox
https://slimerjs.org/
34 comments
puppeteer only supports Chorme so you will need to use Slimer if you wish to use Firefox.
Correct for now. However...
Firefox did start investigating supporting the Chrome DevTools Protocol: https://groups.google.com/d/topic/mozilla.dev.platform/4-4A8...
We (Chrome DevTools/Puppeteer team) have had detailed discussions with their DevTools team on this, recommending what subset would enable Puppeteer. I don't know where they are with this, but in the long-term, this solution looks most promising.
Firefox did start investigating supporting the Chrome DevTools Protocol: https://groups.google.com/d/topic/mozilla.dev.platform/4-4A8...
We (Chrome DevTools/Puppeteer team) have had detailed discussions with their DevTools team on this, recommending what subset would enable Puppeteer. I don't know where they are with this, but in the long-term, this solution looks most promising.
Firefox/Mozilla has been doing a great job with cross-browser initiatives like the WebExtension APIs, and dipping their toes into CDP. I am STOKED to hear you guys have been having cross-team conversations on these details!
Shameless fanboy moment: have followed your stuff for years, and I absolutely love the products your team(s) put out (CDP/Puppeteer)!
Shameless fanboy moment: have followed your stuff for years, and I absolutely love the products your team(s) put out (CDP/Puppeteer)!
Aww thanks amigo! Thanks v much. Plz holler my way if there's anything we can do better.
The devtools teams (Chrome, FF, Safari, and Edge) actually talk quite a bit. We collaborate on the console spec [0] and keep each-other in check on feature parity. ;)
[0] https://github.com/whatwg/console
The devtools teams (Chrome, FF, Safari, and Edge) actually talk quite a bit. We collaborate on the console spec [0] and keep each-other in check on feature parity. ;)
[0] https://github.com/whatwg/console
It would be immeasurably useful for Firefox to have support for Chrome DevTools Protocol (or have something with the same functionality).
Anyone from Mozilla know where to follow the work for this?
Anyone from Mozilla know where to follow the work for this?
These are the only public links I've got: https://bugzilla.mozilla.org/show_bug.cgi?id=1391465 https://github.com/devtools-html/rust-cdp
100% honest and non-hostile question - why would you want to use Firefox?
That question could be easily turned around: why would you want to use Chrome?
Personally, if I were to set up automated testing, I'd like to do it with both Chrome and FF, especially given that FF seems to be making a comeback lately. Another reason might be that I've never had anything but problems building Chromium from source, but have successfully built Firefox. So in situations where you want to know with certainty that your binary is derived from the published source code, Firefox wins IMO.
Personally, if I were to set up automated testing, I'd like to do it with both Chrome and FF, especially given that FF seems to be making a comeback lately. Another reason might be that I've never had anything but problems building Chromium from source, but have successfully built Firefox. So in situations where you want to know with certainty that your binary is derived from the published source code, Firefox wins IMO.
> why would you want to use Chrome?
Chrome as a platform has the majority of browser market share and is continuing to be the top contender, I don't think anyone's going to knock them off their high horse anytime soon. Targeting the most common platform for applications deployment is common sense.
> especially given that FF seems to be making a comeback lately
FF is stagnant as far as market share, and has been trending down for at least 2-3 years. Currently it's around 5-6% and has been for awhile. Saying FF is making a comeback is disingenuous to the actual stats.
> Another reason might be that I've never had anything but problems building Chromium from source
Totally understand this one - I've found my roommate's Gentoo laptop in the fridge doing a full Chromium build more than a few times.
All-in-all I totally get where you're coming from, Chrome and FF are my preferred automation/testing platforms for sure =)
Chrome as a platform has the majority of browser market share and is continuing to be the top contender, I don't think anyone's going to knock them off their high horse anytime soon. Targeting the most common platform for applications deployment is common sense.
> especially given that FF seems to be making a comeback lately
FF is stagnant as far as market share, and has been trending down for at least 2-3 years. Currently it's around 5-6% and has been for awhile. Saying FF is making a comeback is disingenuous to the actual stats.
> Another reason might be that I've never had anything but problems building Chromium from source
Totally understand this one - I've found my roommate's Gentoo laptop in the fridge doing a full Chromium build more than a few times.
All-in-all I totally get where you're coming from, Chrome and FF are my preferred automation/testing platforms for sure =)
>Targeting the most common platform for applications deployment is common sense.
If I recall, people used to say something very similar about Internet Explorer. Of course Chrome isn't as awful as IE back then but still...
IMO it makes the most sense to support two independent implementations, especially for browsers.
>Currently it's around 5-6% and has been for awhile. Saying FF is making a comeback is disingenuous to the actual stats.
For desktop FF has about 10% marketshare. 5-6% includes iOS and Android, the later comes with a preinstalled chrome with little incentive to switch...
With the upcoming ESR 60 release, I also expect (and heard) a lot of companies will switch to Firefox, atleast in the german region (where marketshare increased from Dec to Jan). Some Linux distros might switch over from the previous ESR release and a lot of users could potentially reevaluate.
>I've found my roommate's Gentoo laptop in the fridge doing a full Chromium build more than a few times.
Chromium is not Chrome, it's Chrome with all the Google Binaries ripped out. Chrome itself is, to my knowledge, not open source, only Chromium is.
If I recall, people used to say something very similar about Internet Explorer. Of course Chrome isn't as awful as IE back then but still...
IMO it makes the most sense to support two independent implementations, especially for browsers.
>Currently it's around 5-6% and has been for awhile. Saying FF is making a comeback is disingenuous to the actual stats.
For desktop FF has about 10% marketshare. 5-6% includes iOS and Android, the later comes with a preinstalled chrome with little incentive to switch...
With the upcoming ESR 60 release, I also expect (and heard) a lot of companies will switch to Firefox, atleast in the german region (where marketshare increased from Dec to Jan). Some Linux distros might switch over from the previous ESR release and a lot of users could potentially reevaluate.
>I've found my roommate's Gentoo laptop in the fridge doing a full Chromium build more than a few times.
Chromium is not Chrome, it's Chrome with all the Google Binaries ripped out. Chrome itself is, to my knowledge, not open source, only Chromium is.
Linux has ~1% market share, though, and probably alreay has a pretty large proportion of firefox users. I doubt linux will affect anything much.
Linux has 2.3% marketshare (counting ChromeOS and not accounting for users blocking scripts, which is also disproportionally large on linux)
It may be disproportionately large, but it's still small enough to not have useful effect on such statistics.
Do you know this wouldn't have an effect or do you guess it wouldn't?
In the end, support chrome is simply supporting another Internet-Explorer-era. Maybe not as IE but it would be very similar.
If you or your company only develop for chrome you're not better than any of the websites that proudly state they will only work under IE6. End of story.
In the end, support chrome is simply supporting another Internet-Explorer-era. Maybe not as IE but it would be very similar.
If you or your company only develop for chrome you're not better than any of the websites that proudly state they will only work under IE6. End of story.
> In the end, support chrome is simply supporting another Internet-Explorer-era. Maybe not as IE but it would be very similar.
IMO - it is foolish to put Chrome and IE in the same box.
> If you or your company only develop for chrome you're not better than any of the websites that proudly state they will only work under IE6. End of story.
I don't think anyone is going to make an argument to only support Chrome, there's clearly other browser tech out there that we as devs must support. I will however say that you should prioritize supporting the most popular browsing platforms (browser, screen, device, OS, etc) when building applications and/or during testing.
Since you're touting Linux utilization - I will say that I better have a helluva testing budget (time and money) if I'm going to even touch on the long-tail platform combos (anything < 5%). The only reason my apps are heavily tested in Linux is because they're developed on a Debian Jessie box =D
IMO - it is foolish to put Chrome and IE in the same box.
> If you or your company only develop for chrome you're not better than any of the websites that proudly state they will only work under IE6. End of story.
I don't think anyone is going to make an argument to only support Chrome, there's clearly other browser tech out there that we as devs must support. I will however say that you should prioritize supporting the most popular browsing platforms (browser, screen, device, OS, etc) when building applications and/or during testing.
Since you're touting Linux utilization - I will say that I better have a helluva testing budget (time and money) if I'm going to even touch on the long-tail platform combos (anything < 5%). The only reason my apps are heavily tested in Linux is because they're developed on a Debian Jessie box =D
Glad to see there's an alternative to headless Chrome that's not tied to Google in any way.
Without being too cynical, is the fact that Mozilla receives most of their funding from Google not relevant when considering Google's influence?
Not funding. Revenue.
What influence are you talking about? Can you be more specific?
What influence are you talking about? Can you be more specific?
Is the amount of money Mozilla is able to extract from Google a statement about Google’s influence or Mozilla’s?
i think it depends if mozilla can still operate if googles funding were to dry up
Does this support FreeBSD? The docs say it's supports anything that can run Firefox so I would think yes. I can't check cause I'm in a car for the next 6 hours. But if this supports FreeBSD I would definitely use it over puppeteer because puppeteer does not support FreeBSD and it doesn't really look like they care to add support [1].
[1]: https://github.com/GoogleChrome/puppeteer/issues/1049
[1]: https://github.com/GoogleChrome/puppeteer/issues/1049
How does SlimerJS compare to Selenium which is truly cross-browser and the cornerstone of our internal web automation tests?
Afaict the difference is that slimerjs uses Firefox Headless while Selenium spins up the entire browser.
For Web Testing spinning up the browser is okay, for some other tasks using just a headless version can be good enough.
For Web Testing spinning up the browser is okay, for some other tasks using just a headless version can be good enough.
Selenium can also use headless if you use the MOZ_HEADLESS environment variable.
so I have an upcoming side project that will heavily leverage scraping/screenshotting and I plan to use either headless chrome or slimerjs. It will be a Rails app, queueing up lots of sidekiq jobs, that I (presumably) will have calling a headless browser instance, probably running in a container. Ideally I'll run this either on Heroku or even better, just a DO droplet running dokku.
I found browserless.io and the supporting github project over the weekend, looks promising.
Who else is doing this? Is the architecture I just touched on the right way to go about doing this? Or should I have each sidekiq job spin up its own browser container for the scraping job?
Also, chrome vs firefox won't really matter in this instance, but reliability/performance/memory will...any recommendations?
I found browserless.io and the supporting github project over the weekend, looks promising.
Who else is doing this? Is the architecture I just touched on the right way to go about doing this? Or should I have each sidekiq job spin up its own browser container for the scraping job?
Also, chrome vs firefox won't really matter in this instance, but reliability/performance/memory will...any recommendations?
I used slimer because scraper detection didn't pick it up as much. Beware that slimer (when I used in 2015) still require xvfb.
Chrome headless should be fastest. And if you're printing to pdf phantom and chrome had the better ability to do that..
I had each sidekiq job spinning up its own process -- and needed to do some X display avoidance.
Chrome headless should be fastest. And if you're printing to pdf phantom and chrome had the better ability to do that..
I had each sidekiq job spinning up its own process -- and needed to do some X display avoidance.
Nice, thanks for the reply. Looking at their site, it looks like they now support a true headless mode:
https://docs.slimerjs.org/master/release-notes-1.0.html
"With Firefox 56 and more, SlimerJS can be trully headless by adding the –headless option on the command line"
So that could be an interesting thing to play with. I've thought about just having my sidekiq jobs spin up a docker container with the browser in it and then just limiting the amount of jobs at once for resource constraints, and now seeing your comment I think I might just give it a shot.
https://docs.slimerjs.org/master/release-notes-1.0.html
"With Firefox 56 and more, SlimerJS can be trully headless by adding the –headless option on the command line"
So that could be an interesting thing to play with. I've thought about just having my sidekiq jobs spin up a docker container with the browser in it and then just limiting the amount of jobs at once for resource constraints, and now seeing your comment I think I might just give it a shot.
I built https://github.com/danielwestendorf/pdf-bot-pro a few days ago, a scalable 1-click heroku deploy rewrite of pdf-bot. It's fully functional, except I built the background queuing on Faktory, and there is not working Faktory add-on available for heroku yet (and thus, no real promotion of the project yet). You could set up a faktory server elsewhere and define the URL as an ENV var and it would work today.
Nice thanks for the link. So if I'm understanding correctly, your app essentially runs as a standalone api, receiving pdf requests, queueing them, processing them, then hitting a specific webhook url on success for other systems to then query the pdf tool to get the resulting pdf.
This is approximately how I was thinking of settings thing up as well, so thanks for this!
This is approximately how I was thinking of settings thing up as well, so thanks for this!
Yup, exactly.
Nice, thanks for sharing! I built a simple scraper and I need to add background processing etc:
https://github.com/RealEstateWebTools/property_web_scraper
Will dig into your code to see what I can learn.
https://github.com/RealEstateWebTools/property_web_scraper
Will dig into your code to see what I can learn.
If you decide to open source your project, please share the link here. Thanks
The "simple example" on the front page uses promises which silence all exceptions by default. Not a good idea. Futhermore, with synchronous operations the code would be much simpler.
One of the most active browser automation libs out there and it's just a pure joy to use =)