A list of headless web browsers(github.com)
github.com
A list of headless web browsers
https://github.com/dhamaniasad/HeadlessBrowsers
56 comments
It's a strange omission given that CEF is free and seemingly in rude health while Awesomium, number 1 on the list (if only by dint of its name), looks dead, even though you have to pay for it.
I'm using it for off-screen rendering. It's pretty straightforward, once past the inevitable initial hump. The code is sensible (if a little smart pointer-happy), and you're provided with a good range of hooks. Documentation is OK. Nothing you can't figure out if you've got a debugger...
(I used Visual Studio with the prebuilt Spotify builds. This addin really helps: https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f...)
I'm using it for off-screen rendering. It's pretty straightforward, once past the inevitable initial hump. The code is sensible (if a little smart pointer-happy), and you're provided with a good range of hooks. Documentation is OK. Nothing you can't figure out if you've got a debugger...
(I used Visual Studio with the prebuilt Spotify builds. This addin really helps: https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f...)
I made a PR [1] based on your endorsement. EDIT: it's merged. Yay, we did it HN!
[1] https://github.com/dhamaniasad/HeadlessBrowsers/pull/16
[1] https://github.com/dhamaniasad/HeadlessBrowsers/pull/16
You can add "open source contributions" to your CV now ;)
I'm going to add "thought leader".
I'm going to add "thought leader".
Especially surprising since CEF's Wikipedia page lists 73 projects that use it (many of the well known and widely used, like Spotify, Second Life, Steam Client, Adobe Creative Cloud, Dreamweaver CC, Edge Animate and Edge Reflow).
I first looked at Awesomium because it had a Unity3D interface [1], but that's gone stale (Awesomium doesn't support 64 bits, much has changed in Unity since then, not to mention Chrome itself), and some developers on the Unity discussion groups were suggesting CEF as an alternative. [2] [3]
I'm interested in integrating the latest CEF with the latest Unity3D on Mac and Windows desktop, as well as for consoles.
There are (non-Unity-specific, i.e. maybe too complex and modern) C# wrappers for CEF [4], which looks pretty elaborate, but I think it would be tighter to integrate the CEF C++ code directly as a native Unity extension, without thunking through too much Windows-oriented C# glue that might not be compatible with Unity's older version of C#.
On iOS I'm simply using WkWebKit, and WebKit on Chrome, which both support off-screen rendering.
Here's some interesting stuff comparing Awesomium and CEF and other stuff from Stratoscope. [5] It's from a 446 day old thread about CefSharp [6] so it might predate the Awesomiumocalypse. The thread is interesting reading! Really makes me wonder why Awesomium is on the list at all.
It looks like Coherent UI is a commercially supported alternative to Awesomium, but I haven't looked into it yet. Their employee said good things about CEF and offered some fair criticism:
"Hi, Stoyan from Coherent Labs here. My impressions of CEF are very good. They manage to keep it very close to the chromium HEAD, which is great. The only technical downside of CEF is the imperfect GPU acceleration. In many cases however the architecture of chromium itself is a hindrance. Their processes hog memory like crazy and introduce perceivable lag. We had to write a complete new product to solve the issue for games." [7]
[1] http://labs.awesomium.com/unity3d-integration-tutorial-part-...
[2] http://answers.unity3d.com/questions/945207/unity3d-v5-aweso...
[3] http://gamedev.stackexchange.com/questions/111846/chromium-e...
[4] https://github.com/cefsharp/CefSharp
[5] https://news.ycombinator.com/item?id=10089645
[6] https://news.ycombinator.com/item?id=10087013
[7] https://news.ycombinator.com/item?id=10096616
I first looked at Awesomium because it had a Unity3D interface [1], but that's gone stale (Awesomium doesn't support 64 bits, much has changed in Unity since then, not to mention Chrome itself), and some developers on the Unity discussion groups were suggesting CEF as an alternative. [2] [3]
I'm interested in integrating the latest CEF with the latest Unity3D on Mac and Windows desktop, as well as for consoles.
There are (non-Unity-specific, i.e. maybe too complex and modern) C# wrappers for CEF [4], which looks pretty elaborate, but I think it would be tighter to integrate the CEF C++ code directly as a native Unity extension, without thunking through too much Windows-oriented C# glue that might not be compatible with Unity's older version of C#.
On iOS I'm simply using WkWebKit, and WebKit on Chrome, which both support off-screen rendering.
Here's some interesting stuff comparing Awesomium and CEF and other stuff from Stratoscope. [5] It's from a 446 day old thread about CefSharp [6] so it might predate the Awesomiumocalypse. The thread is interesting reading! Really makes me wonder why Awesomium is on the list at all.
It looks like Coherent UI is a commercially supported alternative to Awesomium, but I haven't looked into it yet. Their employee said good things about CEF and offered some fair criticism:
"Hi, Stoyan from Coherent Labs here. My impressions of CEF are very good. They manage to keep it very close to the chromium HEAD, which is great. The only technical downside of CEF is the imperfect GPU acceleration. In many cases however the architecture of chromium itself is a hindrance. Their processes hog memory like crazy and introduce perceivable lag. We had to write a complete new product to solve the issue for games." [7]
[1] http://labs.awesomium.com/unity3d-integration-tutorial-part-...
[2] http://answers.unity3d.com/questions/945207/unity3d-v5-aweso...
[3] http://gamedev.stackexchange.com/questions/111846/chromium-e...
[4] https://github.com/cefsharp/CefSharp
[5] https://news.ycombinator.com/item?id=10089645
[6] https://news.ycombinator.com/item?id=10087013
[7] https://news.ycombinator.com/item?id=10096616
It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb
Xdummy is another headless X display that has a more modern, more complete stack.
https://xpra.org/trac/wiki/Xdummy
https://xpra.org/trac/wiki/Xdummy
TIL. I did not know Xdummy. I have just followed your link. It is trivial to launch. It avoids installation of vnc, has a better compatibility and is already present everywhere.
This is the way I am going.
I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line.
In the end unless you are severely resource constrained,it is a very good option.
I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line.
In the end unless you are severely resource constrained,it is a very good option.
I've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).
Could you say more about what you mean by "clobbering each other" here?
I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT.
I've always assumed it had something to do with Xvfb/browser interaction.
I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT.
I've always assumed it had something to do with Xvfb/browser interaction.
When you run "firefox", it will check if the current user already has an instance running; even if it's on a different display.
To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance").
Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T".
As an example, the Firefox command I'm using in one of my scripts is currently:
I use a variant of the script at http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation.
NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)
To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance").
Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T".
As an example, the Firefox command I'm using in one of my scripts is currently:
timeout 60 firefox -safe-mode \
-profile "$FF_DIR" \
-no-remote \
-new-instance \
"$URL"
echo "$DISPLAY"
export DISPLAY=:99
I use a variant of the script at http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation.
NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)
Firefox shares sessions (cookies, etc.) between all running instances by default.
Just to throw in another option, there's selenium grid docker containers: https://github.com/SeleniumHQ/docker-selenium. The containers are a bit heavy IMHO (I suppose because they're aimed more at 'testing' instead of scraping), but I think there are a few Alpine Linux versions floating around (and easy enough to set up yourself).
Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).
Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).
+1 you can run things in real browsers.
fwiw Travis CI supports it.
You can also go with a regular browser tied on a vncserver display (pros: you have real captures/video of the tests, cons: the solution gets a bit heavier than when using xvfb)
It's also fairly easy to take a capture of an xvfb or an xdummy display. Dunno about video though.
I really wish we had a good set of scriptable browser engines (i.e. everything but the graphics part) for all browsers. That would make testing so much easier.
Even simple tasks like taking a screenshot of a given web page involves PhantomJS, which seems really heavy-handed.
Thanks for compiling this list but the issue I'm having with this is the lack of a rating system based on usage, Github stars, downloads, repo activity, project creation, corporate support, etc. to get a measure of how reliable those tools are.
Even simple tasks like taking a screenshot of a given web page involves PhantomJS, which seems really heavy-handed.
Thanks for compiling this list but the issue I'm having with this is the lack of a rating system based on usage, Github stars, downloads, repo activity, project creation, corporate support, etc. to get a measure of how reliable those tools are.
The problem is that to take a screenshot of how the page actually looks, you need almost everything from the browser. Rendering of HTML/CSS/images/videos, Javascript execution, networking, cookies... you end up with 90% of the browser, and then removing the last 10% is not worth it.
If all you want to do is take a screenshot, I've had success with http://wkhtmltopdf.org/ (which also outputs to image formats.) You can't script it though.
Yeah, some are quite old and discontinued. I currently use Nightmare which is electron based and has a nice api. Tried a couple others but every time something went wrong.
The problem is that the "graphics part" is often the part that knows how to deal with fonts, which you sort of need for layout.
Here's a thing I made to script WebDriver from the terminal or shell scripts:
https://github.com/mbrock/wd
https://github.com/mbrock/wd
Most of the tools listed use PhantomJS or QT core browser as the base. The problem is that PhantomJS doesn't support many new API-s or the rendering is different.
There is also headless chrome browser in development, not sure if there are any prebuilt binaries out there?
There is also headless chrome browser in development, not sure if there are any prebuilt binaries out there?
I've been using chromium-headless for certain automated tasks, and I've been getting a lot better results than from PhantomJS in terms of correctly rendered resources, resource consumption and execution speed. If your business depends on rendering engines and you're using PhantomJS, chromium-headless is probably something you want instead.
I build from chromium src, and my code is more or less an extension with its own BUILD.gn with a deps on //headless. Once you've built the initial tree (took a night for me), builds will be really fast.
Most of the interaction with the browser is by the chrome debugging protocol, as with "normal" chromium. Generally speaking, you want to target that, instead of the internal APIs.
There are some tweaks you need to be aware of. Chromium needs a homedir, so you want to configure that. There's state stored between sessions, so if you want process invocations stateless, consider spending some time hacking or run the process in a fresh namespace/chroot/docker/jail/whatever each time. I had to write my own protocol handler and URLRequestJobs because I render pages from archives on disk and that was the easiest way to get that started at the time.
I don't know if vanilla chromium supports it, but I'm pretty sure I've seen a --headless flag somewhere... If not, headless_shell is pretty easy to build following the official build instructions[1]. I'm running on RHEL/Fedora/CentOS. There's also unofficial images on docker hub if you want to go that route.
[1]: https://chromium.googlesource.com/chromium/src/+/lkgr/headle...
I build from chromium src, and my code is more or less an extension with its own BUILD.gn with a deps on //headless. Once you've built the initial tree (took a night for me), builds will be really fast.
Most of the interaction with the browser is by the chrome debugging protocol, as with "normal" chromium. Generally speaking, you want to target that, instead of the internal APIs.
There are some tweaks you need to be aware of. Chromium needs a homedir, so you want to configure that. There's state stored between sessions, so if you want process invocations stateless, consider spending some time hacking or run the process in a fresh namespace/chroot/docker/jail/whatever each time. I had to write my own protocol handler and URLRequestJobs because I render pages from archives on disk and that was the easiest way to get that started at the time.
I don't know if vanilla chromium supports it, but I'm pretty sure I've seen a --headless flag somewhere... If not, headless_shell is pretty easy to build following the official build instructions[1]. I'm running on RHEL/Fedora/CentOS. There's also unofficial images on docker hub if you want to go that route.
[1]: https://chromium.googlesource.com/chromium/src/+/lkgr/headle...
Here's the bug to follow if you want to follow progress (the anchor comment in the url is for when screenshot capturing landed)
https://bugs.chromium.org/p/chromium/issues/detail?id=546953...
https://bugs.chromium.org/p/chromium/issues/detail?id=546953...
What about startup time? phantomjs is the only headless browser I have tested which starts up relatively fast.
Can't test now, but around a few tens of milliseconds for about:blank
Also selenium grid docker containers. Unfortunately their Python documentation isn't the best (though there's some excellent third-party created Python documentation available). To save you the head-scratching I went through, it's basically just:
- Start a 'selenium hub'
- Start up as many firefox/chrome 'node' containers and link them to the hub.
- Point webdriver at the hub port and url.
I will eventually get around to building some Alpine containers for this, one day. I find the official ones a bit 'heavy' given all it really invoves in installing openJDK and running a jar file / browser driver.
- Start a 'selenium hub'
- Start up as many firefox/chrome 'node' containers and link them to the hub.
- Point webdriver at the hub port and url.
I will eventually get around to building some Alpine containers for this, one day. I find the official ones a bit 'heavy' given all it really invoves in installing openJDK and running a jar file / browser driver.
what new apis should phantomjs support?
download support[1] is the only new api in development. it's not tagged for the 2.5 release[2].
[1] https://github.com/ariya/phantomjs/pull/14225 [2] https://github.com/ariya/phantomjs/issues/14458
download support[1] is the only new api in development. it's not tagged for the 2.5 release[2].
[1] https://github.com/ariya/phantomjs/pull/14225 [2] https://github.com/ariya/phantomjs/issues/14458
FileReader support is not fully implemented (addEventListener is not present). Just ran into this last week.
There has been an open issue for this for three years[1].
[1] https://github.com/ariya/phantomjs/issues/11733
There has been an open issue for this for three years[1].
[1] https://github.com/ariya/phantomjs/issues/11733
Going to ask same question, about headless Chrome, any recent information on that, anyone?
There is a Google Group for it: https://groups.google.com/a/chromium.org/forum/#!forum/headl...
I check every now and again. I'm super interested in headless PDF generation (using Nigtmare now)
I check every now and again. I'm super interested in headless PDF generation (using Nigtmare now)
it is issue 546953[1] in the chromium issue tracker.
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=546953
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=546953
Anybody else seeing something similar to this? https://i.imgur.com/Dz1FLGQ.png. It's as if GitHub was embedded in an iframe on this repo's page.
Ditto. Probably someone messed up some piece of dynamically loaded server-side rendered stuff in the repository info bar. Since not everyone seems to have it it's probably not on full-roll-out, but just a small part of the user base.
Yup, I'm seeing the same thing. Weird.
Chrome will officially support headless mode: http://crbug.com/546953
The problem I found with this was mostly that there are bugs that only show in one browser (only FF, only Chrome, only Edge, etc) so these tools get their own share of bugs.
At the moment I use Ghostinspector (a SaaS) which works okay. Their UI is a bit clunky, but recording stuff via Chrome extension and modifying it later works okay. Also they let me run the tests with Firefox later. So I develop with Chrome and run tests with Firefox.
At the moment I use Ghostinspector (a SaaS) which works okay. Their UI is a bit clunky, but recording stuff via Chrome extension and modifying it later works okay. Also they let me run the tests with Firefox later. So I develop with Chrome and run tests with Firefox.
i've tried some of these for a project ( i had to cough automate download of files from a site made in old asp.net and with a lot of new js on it), only awesomium was able to do the job, but, man, it was overwelming difficult! i had to inject fiddler as proxy to catch exactly what was happening, awesomium by itself just can't do it, we are very far from a real scriptable headless browser.
Unfortunately, Awesomium seems to have been in "lights on, nobody home" mode for over a year now, and uses an outdated version of Chrome. [1] The nail in the coffin is a predictably unfulfilled promise (see Yannou's rule #1 and #2) followed by radio silence: [2]
"The product is very well alive. Version 1.7.6 will be released in a few days with many new features and an updated Unity plugin as well. Meanwhile, we are also working on our next major version (v2.0)." Perikles C. Stephanidis, Jun 15
The other big problem is that it's not free. So nobody is going to pick it up and run with it.
[1] http://answers.awesomium.com/questions/6880/does-the-project...
[2] http://answers.awesomium.com/questions/7255/is-awesomium-dea...
"The product is very well alive. Version 1.7.6 will be released in a few days with many new features and an updated Unity plugin as well. Meanwhile, we are also working on our next major version (v2.0)." Perikles C. Stephanidis, Jun 15
The other big problem is that it's not free. So nobody is going to pick it up and run with it.
[1] http://answers.awesomium.com/questions/6880/does-the-project...
[2] http://answers.awesomium.com/questions/7255/is-awesomium-dea...
What about Electron? Multiple people, including myself, have used it to make a web browser
fwiw, nightmare is just a thin layer above electron's browser.
I'm currently using Nightmare for a little automation. It's fantastic. A few small gotchas, but so easy to set up.
Missing: lynx, links (both have headless mode), curl, wget, nscurl.
Pretty sure being able to render JavaScript is needed to be on the list. So not most of what you listed.
It's missing the browser engine and version that each of them use. Even latest PhantomJS only uses Chromium 40. Most important metric IMO.
Would be nice to include if is it maintained or their last update. There are a lot of them that aren't updated in years.
I know it's somewhat a vanity metric, but this list should have GitHub stars listed for each project. Would save us some work :)
Unless someone creates a bot that will automate the process of updating the number of stars, it's a cat and mouse game.
Maybe something like this would be an easy start: http://githubbadges.com/
number of contributors, last commit date too.
I haven't used it myself, but I'm considering it, so I'd appreciate hearing other people's thoughts about it.
[1] https://bitbucket.org/chromiumembedded/cef
[2] https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Ap...
[3] https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage...