HackerTrans
TopNewTrendsCommentsPastAskShowJobs

bugboy73

no profile record

comments

bugboy73
·3 anni fa·discuss
All major cloud providers (AWS, Azure and GCP) provide instance-types that support AVX512-VBMI2. Iguana targets large amounts of that are decompressed often and speed is of the essence. Those are typical server scenarios, so the AVX512-VBMI2 requirement isn't really an issue in common-case scenarios.

DISCLAIMER: I work for Sneller
bugboy73
·3 anni fa·discuss
DISCLAIMER: I'm one of the Sneller developers

Compression is required to make more efficient use of the available bandwidth and RAM. Although AWS provides up to 100Gb/s networking, we improve the net throughput using compression. Also we cache data on the computing nodes, so storing it in compressed form allows us to store more data on the node.

Our query engine is often limited by memory bandwidth when using uncompressed data. So it needs a lot of data and it needs it fast. Because we cache data in compressed form, decompression is always needed, so it should be as fast as possible. That's the primary reason why we developed Iguana, because a faster decompression results in a faster query engine.
bugboy73
·4 anni fa·discuss
If you check out the Kubernetes folder in the repo, then you find the Kubernetes setup to run in a distributed environment (that is also highly available).