My 15 Years Striving for Search Engine Independence. AMA(blog.mojeek.com)
blog.mojeek.com
My 15 Years Striving for Search Engine Independence. AMA
https://blog.mojeek.com/2021/03/to-track-or-not-to-track.html
19 comments
Hi, Marc from Mojeek here.
A while ago we had an add URL page but it was better at catching low quality pages than pages we wanted to index. We are considering ways to enable it again without that issue.
I don't really have any favourites, except Mojeek of course. But I've always used multiple search engines and rarely perform a search on just one.
We have provided our results to multiple metasearch engines over the years and currently provide them to a number of people for various uses. It's not our main focus right now, but could well become as time goes on.
We're at a very early stage with our new ad platform and we have a select number of testers helping us to improve it. So far, we're very happy with the interest that's been shown in it. Our current financials are good and I've tried to only move onto the next stages when I felt it was achievable, so right now we're stable.
A while ago we had an add URL page but it was better at catching low quality pages than pages we wanted to index. We are considering ways to enable it again without that issue.
I don't really have any favourites, except Mojeek of course. But I've always used multiple search engines and rarely perform a search on just one.
We have provided our results to multiple metasearch engines over the years and currently provide them to a number of people for various uses. It's not our main focus right now, but could well become as time goes on.
We're at a very early stage with our new ad platform and we have a select number of testers helping us to improve it. So far, we're very happy with the interest that's been shown in it. Our current financials are good and I've tried to only move onto the next stages when I felt it was achievable, so right now we're stable.
Thanks for answering.
> We have provided our results to multiple metasearch engines over the years and currently provide them to a number of people for various uses.
Are you able to name any?
Also: what (if any) are some "starting points" for your crawl? I know that Right Dao uses Wikipedia dumps as one of its sources, for instance.
> We have provided our results to multiple metasearch engines over the years and currently provide them to a number of people for various uses.
Are you able to name any?
Also: what (if any) are some "starting points" for your crawl? I know that Right Dao uses Wikipedia dumps as one of its sources, for instance.
https://www.etools.ch/ is one.
We originally seeded it with URLs from DMOZ. Now we have our own page weight that in combination with modified dates decides what is crawled and when.
We originally seeded it with URLs from DMOZ. Now we have our own page weight that in combination with modified dates decides what is crawled and when.
Interesting. As you scale from 100 to 200 to 300 servers, how does that translate into differences in the product? More pages indexed? More frequent indexing? Faster response times?
Hello, Mojeek back end dev here, I mostly work with Marc on the engine and backend services.
More pages indexed, yes, as there's a finite/specific amount stored on each server. Also for redundancy to back up sections of the index in case of any hardware failure on any one disk/server, you can fall back to the other server containing that slice of the index.
It also raises new problems as the server count grows. One way of doing things is for one server to ask all the other servers to return their 1000 best results for a query. Over time, that can amount to a considerable amount of bandwidth and sorting for that single server to do. We'd recently made a change to proxy requests more along a per-rack level and let them sort the results and send back. This improved our median search time dramatically and is in good timing for our next batch of 100 servers.
We noticed a visible improvement in response times via a 3rd party that uses our API for results. https://www.etools.ch/searchInfo.do
More pages indexed, yes, as there's a finite/specific amount stored on each server. Also for redundancy to back up sections of the index in case of any hardware failure on any one disk/server, you can fall back to the other server containing that slice of the index.
It also raises new problems as the server count grows. One way of doing things is for one server to ask all the other servers to return their 1000 best results for a query. Over time, that can amount to a considerable amount of bandwidth and sorting for that single server to do. We'd recently made a change to proxy requests more along a per-rack level and let them sort the results and send back. This improved our median search time dramatically and is in good timing for our next batch of 100 servers.
We noticed a visible improvement in response times via a 3rd party that uses our API for results. https://www.etools.ch/searchInfo.do
Marc Smith is the founder and lead developer of Mojeek. Today Mojeek celebrates 15 years since publicly announcing a no tracking policy for their independent search engine; built through a long struggle, mainly by Marc, who saw early the need for search privacy. This story has never been told; until now. You can read it in the post above.
We are very happy to have reached this milestone of 15 years as a search engine without tracking. We invite you to put questions to the Mojeek team, it's a nice break from developing the Mojeek service, web crawler and indexing infrastructure.
We are very happy to have reached this milestone of 15 years as a search engine without tracking. We invite you to put questions to the Mojeek team, it's a nice break from developing the Mojeek service, web crawler and indexing infrastructure.
Quite impressive. There are now companies coming out with the option of being paid search engines, most recently Neeva. What's your opinion on this approach?
We think it’s an interesting development and something we considered in early 2020 before I formally started. Marc also considered it over the years. We might look at it in the future but for now, we think that’s not the route for us, nor the peple using Mojeek. There’s something a bit elitist and anti-democractic about paywalling access to information. Neeva have a lot of VC money to test the idea (but are closed for now) so let’s see how they get on. Infinity search are offering it now.
> There’s something a bit elitist and anti-democractic about paywalling access to information.
The premise of a paid search engine is such that information still remains free on the web, it is the access to the convenience of a (superior/premium) search product that you are paying for. This is different to what you are claiming?
I'd also go as far to say that paid search is the only model that removes perverse incentives that ad-driven business models for search engines have historically produced. We all know that both Larry and Sergey agreed that advertising in search engines is a bad thing in their seminal 1997 paper, yet we also know how that ended ( no ads -> contextual ads -> privacy intrusive ads -> ?). The pressure from the true customer filling the pockets (advertisers), makes this path hard to avoid given enough time.
A paid search product would face pressure from a very different angle - improve quality of the search product itself or risk losing a customer. This is IMO a much better position to be in from a product standpoint as at least the incentives of both the company and the user/customer are aligned. Business wise, probably less lucrative because of "search has to be free" intertia.
The premise of a paid search engine is such that information still remains free on the web, it is the access to the convenience of a (superior/premium) search product that you are paying for. This is different to what you are claiming?
I'd also go as far to say that paid search is the only model that removes perverse incentives that ad-driven business models for search engines have historically produced. We all know that both Larry and Sergey agreed that advertising in search engines is a bad thing in their seminal 1997 paper, yet we also know how that ended ( no ads -> contextual ads -> privacy intrusive ads -> ?). The pressure from the true customer filling the pockets (advertisers), makes this path hard to avoid given enough time.
A paid search product would face pressure from a very different angle - improve quality of the search product itself or risk losing a customer. This is IMO a much better position to be in from a product standpoint as at least the incentives of both the company and the user/customer are aligned. Business wise, probably less lucrative because of "search has to be free" intertia.
These are good points. Paid search is something we have considered and may turn too; but others are testing that with a lot more $ from VCs. Our take on ads has always been different [0]:
Unlike the Google founders who started off not wanting to do ads, but then relented under financial pressures, we have never been anti-ads. Ads convey information, they can even entertain. They can be a useful economic signal for a buyer and a way for upstarts and smaller brands to get attention and compete.
[0] https://www.mojeek.com/support/ads/
Unlike the Google founders who started off not wanting to do ads, but then relented under financial pressures, we have never been anti-ads. Ads convey information, they can even entertain. They can be a useful economic signal for a buyer and a way for upstarts and smaller brands to get attention and compete.
[0] https://www.mojeek.com/support/ads/
What do you see as the biggest challenge at getting Mojeek integrated into existing platforms, like browsers or mobile devices, that typically rely on less privacy-preserving options, namely Google?
The App Store duopoly is likely to be the biggest challenge. Followed closely by the large Chrome/Safari market share. The agreement between Google and Apple worth $10bn in cash, and more in P/E to Apple, and distribution for Google that comes from it is another well recognised moat.
You mention rewriting Mojeek three times. Do you expect a fourth iteration?
No definitely not. This version has scaled and works well on nearly 4 billion pages. It works efficiently, is fast for users and was a breakthrough from the previous iteration. Hereon it will be incremental improvements.
Could you help me understand two fundamental issues? (1) Why would I use Mojeek over Google and (2) how do you make money?
Key constraint: you cannot answer "because privacy" for #1, unless you can explain what that means.
Fundamentally, I don't understand why privacy matters.
Google finds stuff that I want , when I want it. How it does it doesn't concern me, as long as it can. It seems (based on using say Duck vs Google) that Google wins by a landslide in quality of search results.
Can you deliver better results with less tracking, and how would I know those are in fact better results?
And of course, if I am to trust all my search to you tool, how do I know you'd be around in 10 years?
Thanks!
Key constraint: you cannot answer "because privacy" for #1, unless you can explain what that means.
Fundamentally, I don't understand why privacy matters.
Google finds stuff that I want , when I want it. How it does it doesn't concern me, as long as it can. It seems (based on using say Duck vs Google) that Google wins by a landslide in quality of search results.
Can you deliver better results with less tracking, and how would I know those are in fact better results?
And of course, if I am to trust all my search to you tool, how do I know you'd be around in 10 years?
Thanks!
1) Different results. Google may have a near monopoly on search but their algorithm is not the only, nor perfect, lens.
2) Contextual Ads
Search is best done with more than search tool. Google+Duck+Mojeek if you like.
https://blog.mojeek.com/2020/12/popping-filter-bubbles-in-fi...
2) Contextual Ads
Search is best done with more than search tool. Google+Duck+Mojeek if you like.
https://blog.mojeek.com/2020/12/popping-filter-bubbles-in-fi...
What kind of innovations do you plan to differentiate yourself from other search options: Google, Bing, Duck ... there are a lot of options. Why you?
There appear to be a lot of options, but as mentioned in the article, most of them are syndication partners of others; and mostly of Bing.
Google and Bing rank and present in very similar ways. The has has been little radical innovation in search in many years. AI improves auto-suggest and all sorts of things but those things often reinforce pathways. Oveall though we have to accept their major lens on the world. To break out of such straitjackets we will have innovations that free up users from those.
Why us now?
1) We don't track AND we don't pass any data to Google or Microsoft, nor anyone else for that matter.
2) You will get very different alternative results from Mojeek. Sometimes these can better and often not so good. But why use just one search engine? I use eight regularly, easily set-up in FF.
Why us later? Watch this space.
Google and Bing rank and present in very similar ways. The has has been little radical innovation in search in many years. AI improves auto-suggest and all sorts of things but those things often reinforce pathways. Oveall though we have to accept their major lens on the world. To break out of such straitjackets we will have innovations that free up users from those.
Why us now?
1) We don't track AND we don't pass any data to Google or Microsoft, nor anyone else for that matter.
2) You will get very different alternative results from Mojeek. Sometimes these can better and often not so good. But why use just one search engine? I use eight regularly, easily set-up in FF.
Why us later? Watch this space.
- What are some of your other favorite search engines and why?
- Bigger engines like Google, Bing, Yandex, and even Gigablast are providers for other engines. Does Mojeek plan to become a provider for any other engines/has it done so already?
- You've described how you are making/will make money (contextual ads). How well does it seem to be working? How's Mojeek's current and future financial stability?