Readability.js(github.com)

164 pointsby stefankuehnelvor 2 Jahren23 comments

https://github.com/mozilla/readability

23 comments

simonwvor 2 Jahren

I like using Readability.js as a demo for my shot-scraper CLI utility: https://shot-scraper.datasette.io/en/stable/javascript.html#...

Running this in a terminal (after installing shot-scraper):

    shot-scraper javascript \
      https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"

Outputs this:

    {
        "title": "Scraping web pages from the command line with shot-scraper",
        "byline": null,
        "dir": null,
        "lang": "en-gb",
        "content": "... long string of HTML ...",
        "length": 7104,
        "excerpt": "I\u2019ve added a powerful new capability to my shot-scraper command line browser automation tool: you can now use it to load a web page in a headless browser, execute JavaScript \u2026",
        "siteName": null,
        "publishedTime": null
    }

stefankuehnelvor 2 Jahren

Wow, that's really cool! Thanks for sharing. I knew about "shot-scraper" before, but I didn't know you could do something so cool with it.

nozzlegearvor 2 Jahren

This is really cool, thanks for sharing!

zerojamesvor 2 Jahren

I have used and love readability.js. I used it in an application that lets you run various NLP analyses over a web page (surprisals, reading time, word counts, etc.). For that, I needed only the main page content. readability.js retrieves main page content well, consistently.

The Alan Turing Institute maintains a Python wrapper around readability.js, too: https://github.com/alan-turing-institute/ReadabiliPy.

crabmusketvor 2 Jahren

I had not heard the term "surprisal" before. It's a delightful word. https://en.wikipedia.org/wiki/Information_content

tuananhvor 2 Jahren

I love this feature so much, I built a website to let me share clean text URL with others.

The first one use js lib but it's kinda limited. the second one use Go, compiled to wasm. Both deployed to Cloudflare workers

- https://github.com/tuananh/reader

- https://github.com/tuananh/reader2

johnchristophervor 2 Jahren

> minScore (number, default 20): the minimum cumulated 'score' used to determine if the document is readerable;

Ahaaa, so that's why sometimes I get the reader icon and sometimes not (especially on mobile).

harrisonjacksonvor 2 Jahren

This is pretty good for stuffing web scrapings into an LLM context.

Can also use `JSDOM ---> element.textContent` depending on your needs. Useful for snagging all the text content or a specific element's.

flirvor 2 Jahren

I wrote a browser plugin that does this. Could never figure out how to deal with paged articles though.

sabrvor 2 Jahren

Readability is awesome! I used it to build Smort.io [1] to easily read articles & ArXiv papers.

[1] https://smort.io

bluish29vor 2 Jahren

That would be a good chance to mention postlight reader [1] which is a browser extension (both Firefox and chrome versions available) that uses Readability to give you a better reading experience. I recommend it

[1] https://reader.postlight.com/

iansinnottvor 2 Jahren

Excellent library. This is what i'm using to build a full text index of every website I visit [1]. It doesn't work perfectly, but well enough.

[1]: https://github.com/iansinnott/full-text-tabs-forever

r-k-jovor 2 Jahren

Clipper.js is built on top of Mozilla's Readability library, Turndown to convert HTML to Markdown https://github.com/philschmid/clipper.js

oulipovor 2 Jahren

Is there a way to use it as a copy-paste in the console, or link to paste in the URL bar, in order to convert a poorly formatted webpage to a nicer one?

Zhylvor 2 Jahren

Is there a way of using this in conjunction with, say, curl? I'd love to be able to grab clean web pages for offline use and printing.

simonwvor 2 Jahren

See my comment here! https://news.ycombinator.com/item?id=39504105

input_shvor 2 Jahren

If you run it inside a container, it's fairly simple: https://github.com/phpdocker-io/readability-js-server

zyangvor 2 Jahren

This is the type of work ML should excel at.

noman-landvor 2 Jahren

Has anyone been able to get this to work on Cloudflare Workers?

tuananhvor 2 Jahren

I did this few years back https://github.com/tuananh/reader

also 2nd version use golang and copmile to wasm https://github.com/tuananh/reader2

iansinnottvor 2 Jahren

I seem to remember it can be run with JSDom rather than a real DOM. I've never tried with CF workers specifically though.

shiomiruvor 2 Jahren

See also the C port here: https://github.com/eafer/rdrview/

It works well with text-mode browsers like w3m.

nicman23vor 2 Jahren

huh that is probably good for ereaders