RSS-proxy: create an RSS/ATOM or JSON feed of almost any website(github.com)
github.com
RSS-proxy: create an RSS/ATOM or JSON feed of almost any website
https://github.com/damoeb/rss-proxy
25 comments
Ah, I like FreshRSS, based on what I can see from it. It's written in PHP and free.
Thanks! RSSHub is another option we could add to the mix https://github.com/DIYgod/RSSHub.
Also, all projects mentioned on this thread can be tracked and easily compared on LibHunt https://www.libhunt.com/posts/500676-rss-proxy-create-an-rss...
Also, all projects mentioned on this thread can be tracked and easily compared on LibHunt https://www.libhunt.com/posts/500676-rss-proxy-create-an-rss...
Netnewawire looks great! Any similar recommendations for Android ?
To elaborate on this, RSS-Bridge maintains custom extraction rules for many sites [1], but RSS-proxy attempts to do that for any site using some pretty nifty logic [2]. I tried it on a few pages and it seems to do its job accurately, if the HTML is good enough.
Semantic HTML5 can be so helpful, especially in regards to tools like this one, or crawlers, or generally accessibility-wise. Unfortunately, its syntax is not enforced by browsers (for good reasons), so many developers never leverage it - and we end up with many custom-built sites falling back to <div>s everywhere, JS click handlers on non-interactive elements etc. I really wish modern framework docs and guides would more strongly hint to accessibility concerns.
[1] https://github.com/RSS-Bridge/rss-bridge/tree/master/bridges [2] https://github.com/damoeb/rss-proxy/blob/master/packages/cor...
Semantic HTML5 can be so helpful, especially in regards to tools like this one, or crawlers, or generally accessibility-wise. Unfortunately, its syntax is not enforced by browsers (for good reasons), so many developers never leverage it - and we end up with many custom-built sites falling back to <div>s everywhere, JS click handlers on non-interactive elements etc. I really wish modern framework docs and guides would more strongly hint to accessibility concerns.
[1] https://github.com/RSS-Bridge/rss-bridge/tree/master/bridges [2] https://github.com/damoeb/rss-proxy/blob/master/packages/cor...
Seconding NetNewsWire. A recent version added iCloud sync so your feeds and unread statuses sync across the desktop and mobile version. Somehow it's free.
Monitoring Public Facebook pages? A very painful process. Facebook requires a login to view a public pages -- and most public offices like governments post content exclusively on Facebook so as things stand, you have to have an account on Facebook to access government communication, which is plain sad.
I do not remember any good tool to access Facebook pages. If you get one (rss-hub), it breaks in a week after Facebook changes their API endpoints.
If you have one, let me know.
If you have one, let me know.
Good work there. Imagine if we had a universal feed generator that could generate a feed out of everything. No more manual checking labor.
RSS saved me the "right click on a bookmark folder -> open all" every hour or so. Does RSS increase my productivity or decrease it? On one hand, I'm not obsessively checking a bunch of websites. On the other hand, I have a firehose of content directed at my face with hundreds of feeds.
Next step is local analytics to sort articles based on local-only rules/analytics/goals.
RSS Hub also attempts to be exactly that: https://docs.rsshub.app/en/
Interesting idea, but the fact so many of the feeds are broken probably indicates how difficult this is to maintain. Ideally something that needs to be done via algos and machine learning rather than custom parsing for each site (not unlike what search engines already do internally).
Great idea, but it seems it does not work for basic static sites like those generated by SvelteKit (http://svekyll.com generated this site: https://webiphany.com). That's too bad.
Author here, thanks for those tough nuts :) I am working on a successor of this POC called rich-rss [0], which contains a full rewrite of rss-proxy in a strongly typed language.
[0] https://github.com/damoeb/rich-rss
[0] https://github.com/damoeb/rich-rss
Well that site is a bunch of divs. Afaik the tool leverages semantic html (section, article etc).
This is great. Many scrapers require constantly monitoring which is a bit heavy for the scraping process. I can see this tool making it easy to monitor for changes fairly quickly. Great work!
A replacement for Yahoo Pipes!
In the early Android days, I wrote a few apps that relied on web site scraping to provide their data. Even back then, these websites evaded scraping. Yahoo Pipes to the rescue. You'd set it up to scrape their site and provide a well-formed RSS feed of the data. Since these sites saw Yahoo connecting, they typically didn't block it, presumably because they thought it was search crawling.
In the early Android days, I wrote a few apps that relied on web site scraping to provide their data. Even back then, these websites evaded scraping. Yahoo Pipes to the rescue. You'd set it up to scrape their site and provide a well-formed RSS feed of the data. Since these sites saw Yahoo connecting, they typically didn't block it, presumably because they thought it was search crawling.
It doesn't work well everywhere and there isn't a way to customize its selection of divs e.g. see this thread itself https://rssproxy-v1.migor.org/?url=https:%2F%2Fnews.ycombina...
Throwing https://heartfeedrss.wordpress.com/usage/ into the ring as its been my go-to rss reader since LinkedIn's Pulse was decommissioned.
I really dig the grid-based approach for both increased content density and easy cross-feed navigation.
I really dig the grid-based approach for both increased content density and easy cross-feed navigation.
A good way to find new pages on a site is to use the site map feed in xml or text format.
Then look for open graph tags in the page for titles descriptions, canonical urls and images.
Most modern websites have all of those these days and they are automatically updates.
Making an RSS feed reader is thus easier and less prone to errors from analyzing the markup of pages.
Neat product though.
Then look for open graph tags in the page for titles descriptions, canonical urls and images.
Most modern websites have all of those these days and they are automatically updates.
Making an RSS feed reader is thus easier and less prone to errors from analyzing the markup of pages.
Neat product though.
This is quite useful, especially the self host option.
What type of websites don't work?
What type of websites don't work?
RSS scrapers I want: something that consolidates twitter threads like threadreaderapp into articles.
"... almost any website."
What are the domain names of some (dynamic) websites where this will not work.
What are the domain names of some (dynamic) websites where this will not work.
Only guessing, but https://oeis.org/ ?
Great tool, something I didn't know I need in my life. Thanks.
If any of you are looking into RSS, check out FreshRSS[2] (an RSS aggregator). A mobile client I especially like is NetNewsWire[3] (for Apple products only, but is imo amazing)
[0] https://hnrss.github.io/
[1] https://github.com/RSS-Bridge/rss-bridge
[2] https://www.freshrss.org/
[3] https://netnewswire.com/