lunaticd

vor 2 Jahrendiscuss

3. The project started under a Harvard affiliated Github org during the course of PhDs. These same people later joined Google where it continued to be developed and over time adopted more and more in place of TensorFlow.

lunaticd

vor 2 Jahrendiscuss

i did it does a ton of allocations, which is why i made a simple maxheap implementation and then sort the final result

lunaticd

vor 2 Jahrendiscuss

please make it even faster!

lunaticd

vor 2 Jahrendiscuss

it doesn't seem to have better support for things like xor and count_ones. I believe the main use case is comparisons.

lunaticd

vor 2 Jahrendiscuss

julia> @code_typed hamming_distance(Int8(33), Int8(125)) CodeInfo( 1 ─ %1 = Base.xor_int(x1, x2)::Int8 │ %2 = Base.ctpop_int(%1)::Int8 │ %3 = Base.sext_int(Int64, %2)::Int64 │ nothing::Nothing └── return %3 ) => Int64

julia> @code_llvm hamming_distance(Int8(33), Int8(125)) ; Function Signature: hamming_distance(Int8, Int8) ; @ /Users/lunaticd/code/tiny-binary-rag/rag.jl:13 within `hamming_distance` define i64 @julia_hamming_distance_16366(i8 signext %"x1::Int8", i8 signext %"x2::Int8") #0 { top: ; @ /Users/lunaticd/code/tiny-binary-rag/rag.jl:14 within `hamming_distance` ; ┌ @ int.jl:373 within `xor` %0 = xor i8 %"x2::Int8", %"x1::Int8" ; └ ; ┌ @ int.jl:415 within `count_ones` %1 = call i8 @llvm.ctpop.i8(i8 %0) ; │┌ @ int.jl:549 within `rem` %2 = zext i8 %1 to i64 ; └└ ret i64 %2 }

it lowers to the machine instruction now.

I also tried 8 Int64s vs 64 Int8s and it doesn't seem to make a difference when doing the search.

EDIT: apologize for the formatting

lunaticd

vor 2 Jahrendiscuss

exact in this case means that all the vectors are compared against the query vector. Where as other search methods such as HNSW are approximate searches.

lunaticd

vor 2 Jahrendiscuss

author here. I thought there might be a machine instruction for this but wasn't sure, I also didn't know Julia had a count_ones that counted the 1s.

Thanks! With this the timings are even faster. I'll update the post.

Submissions

Optimizing RAG to Search PubMed Articles in Milliseconds Instead of Minutes

Approximate Binary Vector Search for RAG in Julia

Exact binary vector search for RAG in 100 lines of Julia

Generative AI vs. AWS Textract – A Case for GPT Wrappers

LLMs can dream of electric sheep, but can they add?

Take1 – An Exploration into LLM Writing

Counting NanoGPT FLOPs in PyTorch

Show HN: DataGPTd – Making data analysis as easy as asking a question

comments