11x Faster Ingest of 1B Rows Using a Controversial DB Architecture(docs.synnaxlabs.com)
docs.synnaxlabs.com
11x Faster Ingest of 1B Rows Using a Controversial DB Architecture
https://docs.synnaxlabs.com/guides/comparison/performance/one-billion-rows
2 comments
[deleted]
We're engineers working on a time-series database engine specifically designed for high speed sensor data. We come from backgrounds in aerospace engineering, and were surprised by the lack of database engines built specifically for very high speed data i.e. megahertz vibrational readings.
We wanted to take a much simpler approach than solutions like Timescale or InfluxDB by building a system that was more along the lines of "object storage for sensor data". We don't support complex query languages like SQL or Flux, and instead we're focused on providing great integrations for scientific programming languages like Python.
Since we're tailored towards small numbers of very high speed time-series, we made the somewhat unintuitive decision to keep data for each sensor in its own file (these sensors are called channels). The result was a pretty dramatic increase in read and write speeds for measurement applications like Aerospace, Automotive, and Fusion/Fission research.