Building a custom code search index in Go(boyter.org)
boyter.org
Building a custom code search index in Go
https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/
19 comments
It’s actually sitting on my hdd. Just need to finish it off. I got sidetracked.
See also https://codesearch.debian.net/ - https://github.com/Debian/dcs for a similar project that may fit your needs better. I've not compared them both, but I use dcs frequently
The blog posts about it are great too. This one https://michael.stapelberg.ch/posts/2019-09-29-dcs-positiona... in particular.
Assuming this is different from: https://github.com/boyter/searchcode-server which is in Java.
Yes. Very different.
Congratulations to the author. It seems like they have excelled at a very fast pace from using third party solutions to building their own in a short time. I look forward to their progress and seeing where this goes, to maybe becoming an excellent open source copilot alternative.
I don't know about a fast pace, but I did have fun with it!
Great read, but a basic search yields zero results:
https://searchcode.com/?q=kong.New+lang%3Ago
Same search on GitHub:
https://github.com/search?type=code&q=kong.New
Edit: there also doesn't seem to be any ranking at all, such as exact word matches being boosted
https://searchcode.com/?q=kong.New+lang%3Ago
Same search on GitHub:
https://github.com/search?type=code&q=kong.New
Edit: there also doesn't seem to be any ranking at all, such as exact word matches being boosted
Probably because I don’t prioritise GitHub anymore. Their own search is great, but it might get picked u[ eventually.
There is ranking, first by a pre rank popularity of the repository and secondly by tf/idf of the trigrams. It’s weighted towards longer matches in the display as well.
But let me know where it’s not doing what you expect and I’ll fix it.
There is ranking, first by a pre rank popularity of the repository and secondly by tf/idf of the trigrams. It’s weighted towards longer matches in the display as well.
But let me know where it’s not doing what you expect and I’ll fix it.
> But let me know where it’s not doing what you expect and I’ll fix it.
I would expect > 0 results for the above search
Same search on SourceGraph: https://sourcegraph.com/search?q=context:global+kong.New&pat...
I would expect > 0 results for the above search
Same search on SourceGraph: https://sourcegraph.com/search?q=context:global+kong.New&pat...
Ah I see a vanity search! The ones that always cause issues :)
Its just down to it not being in the index, I shall ensure I add it just for you based on this.
Done. That repository https://github.com/alecthomas/kong will get picked up when I kick off the indexing again (sometime next week once all the activity dies down)
Its just down to it not being in the index, I shall ensure I add it just for you based on this.
Done. That repository https://github.com/alecthomas/kong will get picked up when I kick off the indexing again (sometime next week once all the activity dies down)
What a patronising response to a report of 500+ results missing from your index :) How will you improve your product if you can't accept feedback?
Edit: typo
Edit: typo
Hah!
I actually checked, it was aware of your repository and has been tracking it for a while. Just it didn't get indexed due to me de-prioritizing github. If I can scrounge some more RAM back I will try to expand the index yet again.
In any case I manually updated your github star score, so it should pop to the top when I do get around to starting things again. Probably after re:invent.
I actually checked, it was aware of your repository and has been tracking it for a while. Just it didn't get indexed due to me de-prioritizing github. If I can scrounge some more RAM back I will try to expand the index yet again.
In any case I manually updated your github star score, so it should pop to the top when I do get around to starting things again. Probably after re:invent.
Is fitting the index in ram really that important? Obviously it is fast, but if you can get away with storing it on a fast disk like an nvme gen4 then why not?
Extremely important, search indexes are cache optimized. Live update indexes even more so.
Also, off-topic but as you know, I recently tried out your scc tool and am eagerly awaiting its support for Elixir templates (.eex, .heex)! You said it was a day from done a while back and would go out in the next release. What’s the release schedule like?
https://github.com/boyter/scc