I was wrong to be upset this whole time that the rewrite would hurt Zig. This is one of those rare occasions when I’m glad I was wrong. Interesting insights.
Hister has a built in crawler with standard HTTP lib and browser based backends, you can feed your link collection to it. Also, Hister supports importing your existing browser history automatically using either of the mentioned backends.
Hister stores only the text content of HTML/pdf pages. 1000 documents require around 80-100MB of storage and there is still plenty of room to optimize for storage space.
I'm using it for 6-7 months and my index size is below 1GB with almost 10k pages.
Also, a downside of the proxy approach: it does not handle properly JS based websites and cannot identify dynamic content changes. Our extension periodically checks if the browser tabs' content has been changed and automatically updates the index when change detected.
It is possible. The automatic website indexing can be turned off in the extension and manual indexing can be triggered via the command line tool, the extension popoup or by hotkeys.
> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?
There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.
> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.
Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.
I agree, we should add more extractors [1]. Can you recommend extractors you missed?
Absolutely, this is a great example where Hister can shine.
I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.
In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.
Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers.
Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/
Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
I'm still working on a self-hosted search service called Hister with the goal to reduce dependence on online search engines.
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. It provides offline result previews, a flexible web (and terminal) search interface & query language to explore saved content with ease or quickly fall back to traditional search engines.
I've been using it for a few months and as my local index is growing I can avoid opening google/duckduckgo/kagi - and even websites listed in results - more and more frequently.
The initial reception is overwhelmingly positive with already more than 30 contributors and hundreds of contributions - perhaps you can find it useful as well. (Or at least have some constructive criticism =])
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. It provides a flexible web (and terminal) search interface with offline result previews & detailed query language to explore collected content or quickly fall back to traditional search engines.
It can provide a privacy-respecting search experience for serving "recall" type searches where users retrieve previously visited content, but falls short in "discovery" type searches (yet).
As a search/metasearch developer, I can heavily recommend Uruky. It is by far the best third party search alternative in terms of how they approach privacy and transparency. Keep up the good work Bruno! <3