OCaml for Kafka stream processing, analytics and telemetry data(github.com)

4 pointsby tingfirst10 mesi fa3 comments

OCaml for Kafka stream processing, analytics and telemetry data

https://github.com/mfreeman451/proton-ocaml-driver

4 comments

For OCaml users interested in data streaming processing (similar to Flink or Spark), but looking for a faster and more efficient option, check out this OCaml plugin Timeplus Proton. Concise, safe, highly performant and fun

-> Streaming Queries - Process large datasets with constant memory usage

-> Async Inserts - High-throughput data ingestion with automatic batching

-> Compression - LZ4 and ZSTD support for reduced network overhead

-> TLS Security - Secure connections with certificate validation

-> Connection Pooling - Efficient resource management for high-concurrency applications

-> Rich Data Types - Full support for #ClickHouse types including Arrays, Maps, Enums, DateTime64

-> Idiomatic OCaml - Functional API leveraging OCaml's strengths

rainmaker1110 mesi fa

Interesting. How does the performance look like?

gangtao10 mesi fa

Micheal, the author of the project did some performance test and he might annouce it later, but one of his first test show Async Insert with Rate: 3422488 rows/second, but I beleive after optimization and tuning, it could be much faster.

mfreeman45110 mesi fa