mkcg

6 months agodiscuss

@see: https://news.ycombinator.com/item?id=46766004

4 years agodiscuss

I don't know about CUDA, however I've made an AVX2 based encoder a few months ago : https://github.com/phoboslab/qoi/pull/143

I intend to create an AVX2 based decoder but I had absolutely no time to work on side projects in the past three months.

You might also want to take a look at this streaming encoder if you want to encode large files with a tiny memory footprint : https://github.com/MKCG/php-qoi/blob/main/src/FFI/lib/qoi.c

mkcg

5 years agodiscuss

goto statements are also very useful to parse, compress and uncompress data in the critical loop.

mkcg

5 years agodiscuss

You could also prefix it with a varint : in this case it would not require any extra byte compared to the NUL terminated string when the length of the string is less than 128 chars.

mkcg

5 years agodiscuss

Thanks for the feedback, my issue is that I allocate only a few MB to my indexing thread so I'm looking for a more efficient implementation to avoid having to produce then merge too many segments from disk.

I'm currently considering using compressed pointers on some part of the tree to reduce the memory footprint as much as I can. Let's see how it goes...

mkcg

5 years agodiscuss

Hi,

do you have any metric regarding the memory usage of your ART implementation ?

I tried to implement one for the database I'm currently working on, however I feel that I am using way too much memory.

Basically, with my current implementation a dictionary containing about distinct 2857086 words would require 341MB.

mkcg

6 years agodiscuss

What is your stack ?

I made a simple Proof-of-concept of a PHP backend based on Redisearch compatible with the Instantsearch.js library.

You can take a look at it here : https://github.com/MKCG/algolia-on-redisearch/tree/master/ap...

With the Covid crisis we were no longer able to pay 3K per month for a basic search engine and now we are using a solution based on this on production (> 30M search/month).

mkcg

6 years agodiscuss

You missed the "streaming" part.

Being able to allocate only a few KB to ingest any JSON is a killer feature.

There is a valid reason to reinvent the wheel : in my case I had to do something similar in C99 for a SaaS federated search engine to have the lowest memory footprint possible.

mkcg

6 years agodiscuss

I've been working on a lite PHP library acting as an agnostic query engine to aggregate content from different (files, API, SQL, document oriented, ...) kind of datastore and ease the usage of some aggregations : terms, facets, ranges, min, max, average, ...

https://packagist.org/packages/mkcg/php-query-model

And also working on a search engine (in C).

mkcg

6 years agodiscuss

http://travaux.ovh.net/?do=details&id=43213& ?

Submissions

Show HN: Single header branchless ASCII folding filter

Show HN: Nua.ge Terraform Provider

PHP – simple document-oriented model built from multiple datasources

comments