I'm Emiliano, an aerospace engineer and co-founder of Synnax. Our team spent years in industry writing Excel based control sequences and the runtime environments that execute them. After founding Synnax, we spent two years trying to avoid building a new language (Python, then Lua), before giving in.
This is the story of why we eventually decided to take the leap. The next set of posts will be a deep dive into the language design.
We're engineers working on a time-series database engine specifically designed for high speed sensor data. We come from backgrounds in aerospace engineering, and were surprised by the lack of database engines built specifically for very high speed data i.e. megahertz vibrational readings.
We wanted to take a much simpler approach than solutions like Timescale or InfluxDB by building a system that was more along the lines of "object storage for sensor data". We don't support complex query languages like SQL or Flux, and instead we're focused on providing great integrations for scientific programming languages like Python.
Since we're tailored towards small numbers of very high speed time-series, we made the somewhat unintuitive decision to keep data for each sensor in its own file (these sensors are called channels). The result was a pretty dramatic increase in read and write speeds for measurement applications like Aerospace, Automotive, and Fusion/Fission research.
Yes! I'm a fan of Asimov and the foundation series. Synnax is the home planet of Gaal Dornik, who studies Psychohistory, which is (loosely) related to time. I mostly thought the name sounded cool, and then came up with this explanation later :)
> I don’t think saying you’re 5x InfluxDB on writes is going to persuade too many people.
I definitely agree with this. Our early prototype of Synnax actually sat on top of a combined Redis/S3/SQL stack and focused on those high level features. We found that it was challenging to deploy, manage, and synchronize data across these services, especially when you're running everything on prem.
We've come to believe that a re-architecture of the underlying infrastructure can actually unlock the high level workflows. For example, to compare a real-time telemetry stream with a historical data set you'd need to query across tools like Kafka and Influx at the same time. For an experienced software engineer this isn't too hard of a task, but they don't tend to be the people who understand the physics/mechanics of the hardware. We want it to be possible for say, a Turbo machinery expert, to translate a Python script they wrote for post-processing a CSV into something Synnax compatible without a huge amount of work.
In short, we're working on finding a way for subject matter experts in hardware to implement the anomaly detection mechanisms they already have in their head, but don't have the software expertise to implement.
> The thing I wanted was one solution for something that was always two: a properties/metadata database, and a separate time series database.
What do you think about TimeScale for this sort of use case? Haven't run it in production myself, but having your time-series data in the same place as a SQL table seems pretty nice.
> We found that visualization layers tended to reach down just far enough into the data intake world that it was really hard to sell just another tsdb.
This is a good point. We think that focusing exclusively on the DB is probably not the right approach. Most of our focus nowadays is on building out higher level workflows on top of the new database.
Hey, we're aware and a little embarrassed! We use some nested CSS selectors that aren't compatible with some browsers and haven't gotten around to fixing it yet, sorry!
The core time series engine, called cesium (https://github.com/synnaxlabs/synnax/tree/main/cesium), is written completely from scratch. It's kind of designed as a "time-series S3", where we store blobs of samples in a columnar fashion and index by time. We can actually get 5x the write performance of something like InfluxDB in certain cases.
For other meta-data, such as channels, users, permissions, etc. we rely on CockroachDB's Pebble, which is a RocksDB compatible KV store implemented in pure go.
It's been a really interesting problem to tackle - we've seen so many different ways that companies try to tackle this problem, and while theres one or two companies that have made really fantastic internal tools, most are ... lacking, to say the least.
Our plan so far has been to try to interface with the bigger companies through the drivers we make for their hardware. We haven't reached out about partnerships yet, but that is a really good idea.
Thank you for the offer - will definitely reach out if and when we need more help on the implementation side.
This is the story of why we eventually decided to take the leap. The next set of posts will be a deep dive into the language design.