sagia · HackerTrans

vor 2 Jahrendiscuss

I had similiar issue, but for aggreagations. Use case was to "compress" large datasets into smaller aggregations for insertion into a costly db. At first we used duckdb but memory became an issue there and we also bumped into a couple of issues with how duckdb handles arrays. We then moved this workload to clickhouse local, which was faster and had more fine tuning options to our liking. in this case was limiting ram usage with i.e. max_bytes_before_external_group_by

sagia

vor 2 Jahrendiscuss

    Location: Israel
    Remote: Yes
    Willing to relocate: Yes
    Technologies: go-lang, java, kotlin, javascript (ts if i must), linux, bash, k8s, helm, clickhouse, druid
    Other-notable-skills: People management, tech project management, AWS->GCP
    Résumé/CV: Available on request
    Email: asagia [at] icloud [dot] com

Hi, My name is sagi. In my current role I am a software engineering manager, in which I (hands-on) lead a very small team (5-6) of full stack engineers. Coming from ad-tech, I have managed to tackle multiple challenges in a variety of technologies and teams - be it Android SDKs to mid-scale (~2-10k qps) backends. Also, worked on multiple other products spanning from clickhouse analytical systems to full stack applications in whatever-gets-the-job-done fashion. I have 12 years (7 years in the current company, after 2 exits) of software engineering experience, 4 of them as a lead. Looking forward hearing about opportunities that might require someone who doesn't see any barriers between data-eng/backend/frontend/mobile/ops.

sagia

vor 2 Jahrendiscuss

I wonder if a similar (but maybe more bloated) implementation using interfaces (and probably generics too?) will work in this case.

comments