HackerTrans
TopNewTrendsCommentsPastAskShowJobs

tyropita

no profile record

Submissions

[untitled]

2 points·by tyropita·3 yıl önce·0 comments

comments

tyropita
·3 yıl önce·discuss
+1. Also according to some swedish friends, aftonbladet is not a very high quality news service.
tyropita
·3 yıl önce·discuss
Makes you wonder if they use LLMs and a chat interface to generate text-based role-playing games, renamed as “simulations”.
tyropita
·3 yıl önce·discuss
Documentation looks really neat and in-depth, always appreciated. Looks like you’re missing a .gitignore file. Folders like __pycache__ don’t need to be checked in.
tyropita
·4 yıl önce·discuss
It was indeed very unexpected. The first time you would try out a cli tool you’d expect just calling its name to return help info and maybe an error.
tyropita
·4 yıl önce·discuss
After a certain scale I think you can let clients do double-work and let the most common crawl data, among different clients, win.

And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.

There certainly are a lot of elegant ways to reduce spam for this particular problem imo.
tyropita
·4 yıl önce·discuss
Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home.

How are the batches of URLs to be crawled generated/discovered and posted at your API?

How do you deal with duplicate crawls?