Ask HN: What Would It Take to Make an Open Source Google?
11 comments
I posted this a few years back, problem has since become only more difficult: https://news.ycombinator.com/item?id=11626999
Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in:
* a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically
* Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage
* Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.
Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in:
* a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically
* Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage
* Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.
What about having a variety of different possible ranking functions? With people being able to select one they prefer or defaulting to a random selection/aggregation of them? Just an idea.
People would still game the most popular (default probabbly) system and it wouldn't take long for folks to give up on the search aspect as it would be dorked up pretty regularly.
Scrapping seems to be a fairly solved problem, there are open source scrapers that are competitive with google, and just download an entire site into an archive, ssl headers could be included to prove the authenticity of the archive in a distributed environment.
Indexing I have no idea how this could be done in a distributed way, as far as I know it basically cant. Distributed p2p apps use a central server or will send the search query to everyone on the entire network until some one gives them a result. Which isn't efficient enough to work on googles scale and the privacy issues.
My best guess is to break a query into tokens and then have a dht of indexes for each token and then have the peer request all the needed indexes compile them together or something? Anyway even if there is a way to distribute an index efficiently, I cant think of a way to generate the indexes in a secure way, at least without some sort of human oversight, which might be acceptable.
Distributed ranking runs into all the problems sdrinf pointed out. A better solution is to have lost of ranking factors enough that its easier to just make a good website than figure out the system and how to game it. Then let the end user weight the ranking factors to suite them, this should give better results than google and make SEO harder to fake.
Indexing I have no idea how this could be done in a distributed way, as far as I know it basically cant. Distributed p2p apps use a central server or will send the search query to everyone on the entire network until some one gives them a result. Which isn't efficient enough to work on googles scale and the privacy issues.
My best guess is to break a query into tokens and then have a dht of indexes for each token and then have the peer request all the needed indexes compile them together or something? Anyway even if there is a way to distribute an index efficiently, I cant think of a way to generate the indexes in a secure way, at least without some sort of human oversight, which might be acceptable.
Distributed ranking runs into all the problems sdrinf pointed out. A better solution is to have lost of ranking factors enough that its easier to just make a good website than figure out the system and how to game it. Then let the end user weight the ranking factors to suite them, this should give better results than google and make SEO harder to fake.
Which part(s) of Google? Search? Docs? Drive? Analytics? There are at least a fee open-source alternatives already under development, such as Nextcloud (Docs/Drive) and Piwik (Analytics).
In order for open-source projects to be successful we need to contribute resources (time and money) and encourage coordination (I.e. contribute to existing efforts rather than bikeshedding or starting from scratch.)
In order for open-source projects to be successful we need to contribute resources (time and money) and encourage coordination (I.e. contribute to existing efforts rather than bikeshedding or starting from scratch.)
While the other pieces would be nice, I am primarily concerned about search. That product seems like the toughest nut to crack.
"Common Crawl" and Elastic Search? Elastic Search for ipfs. Adding a few petabytes with a lot of bandwidth to ipfs, maybe modifying the warc/wat headers to support ipfs address pinning? Some extra web browser code to sanitize, token harvest, pipe webpages that users visit back into the indexing/pinning locally with metadata for times visited and other counts, with a ui that allows user to pick and choose what gets indexed. Some extras around front page click through thrashing because of poor results, incentivizing quality with upvotes/downvotes/flags, charging for search, and paying for it with resource (bandwidth/storage) commitments.
The idea is intriguing. Perhaps it'd be possible to structure web crawling and indexing as a decentralized operation - drawing inspiration for crypto based storage networks.
People would get paid to run crawlers / store indexes of the data.
Gonna ponder this
People would get paid to run crawlers / store indexes of the data.
Gonna ponder this
How is blockchain technology relevant at making a search engine. It sounds like you're looking for a way to use the tech instead of purely trying to fix the problem.
I'm definitely just throwing that out there, as a solution for decentralized monetization and incentivizing resource sharing. So far, blockchain/crypto tech is the only sustainable solution I've seen to distributed computing where no one entity owns the execution of computation or storage of data. Definitely open to alternative ideas though, this is just a thought experiment.
There is https://github.com/commonsearch
Also, it is a lot of money only for handling the storage costs.
Also, it is a lot of money only for handling the storage costs.
Could the execution and runtime of said search engine be powered by blockchain/cryptocurrency tech? There are several existing technologies like Sia that decentralize file storage. Or are other technologies better suited?
What are other challenges of operating a search engine? Regulatory compliance? Ethical scrubbing of results?
Would this solution be tenable or does search lend itself too much to a centralized entity like Google?