Documentation looks really neat and in-depth, always appreciated.
Looks like you’re missing a .gitignore file. Folders like __pycache__ don’t need to be checked in.
Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home.
How are the batches of URLs to be crawled generated/discovered and posted at your API?