HackerTrans
TopNewTrendsCommentsPastAskShowJobs

tyropita

no profile record

Submissions

[untitled]

2 points·by tyropita·3년 전·0 comments

comments

tyropita
·3년 전·discuss
+1. Also according to some swedish friends, aftonbladet is not a very high quality news service.
tyropita
·3년 전·discuss
Makes you wonder if they use LLMs and a chat interface to generate text-based role-playing games, renamed as “simulations”.
tyropita
·3년 전·discuss
Documentation looks really neat and in-depth, always appreciated. Looks like you’re missing a .gitignore file. Folders like __pycache__ don’t need to be checked in.
tyropita
·4년 전·discuss
It was indeed very unexpected. The first time you would try out a cli tool you’d expect just calling its name to return help info and maybe an error.
tyropita
·4년 전·discuss
After a certain scale I think you can let clients do double-work and let the most common crawl data, among different clients, win.

And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.

There certainly are a lot of elegant ways to reduce spam for this particular problem imo.
tyropita
·4년 전·discuss
Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home.

How are the batches of URLs to be crawled generated/discovered and posted at your API?

How do you deal with duplicate crawls?