Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.
We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.
Note: the linked position is focused on the data integration aspect of Materialize but we're also looking for wider database roles so feel free to apply if the general area of Materialize seems interesting to you!
What we’re looking for:
- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).
Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.
We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.
What we’re looking for:
- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).
Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.
We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.
What we’re looking for:
- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).
> However, while checksums can be used under the “Crash Consistency Model” to solve consistency through power loss, PAR showed that checksums are not sufficient to be able to distinguish between a torn write at the end of the (uncommitted) WAL caused by power loss, and a torn write in the middle of the (committed) WAL caused by bitrot.
The PAR paper states that "although Crash preserves safety, it suffers from severe unavailability". I assume that when TigerBeetle loads state from RAM into a CPU cache/register it operates under the NoDetection consistency model or the Crash consistency model if ECC RAM automatically resets the CPU on read errors. At the same time it doesn't suffer from severe unavailability so what gives?
The answer is probably that ECC RAM is just reliable enough that the NoDetection/Crash models are fine in practice.
I can believe that off-the-shelf checksum and redundancy options offered by filesystems like ext4 and ZFS or systems like RAID don't hit the required error probabilities but why does the argument stop there? Couldn't a distributed database generate error correcting data on every write in the application layer so that the probability becomes low enough such that NoDetection/Crash become a non-issue for storage, just like RAM? Is there some other fundamental difference between reading and write data from RAM versus a disk?
> Sure, we’re not yet injecting storage faults, but then formal proofs for protocols like Raft and Paxos assume that disks are perfect, and depend on this for correctness? After all, you can always run your database over RAID, right? Right?
> If your distributed database was designed before 2018, you probably couldn’t have done much. The research didn’t exist.
I'm trying to understand this part but something seems off. It seems to imply that those proofs do not apply to the real world because disks are not perfect, but neither is RAM. The hardware for both RAM and disks has an inherent error rate which can be brought down to arbitrary levels by using error correction codes (e.g ECC RAM).
I'm assuming that the TigerBeetle correctness proofs are predicated on perfect RAM even though in reality there is a small probability of errors. This tells me that there is an error rate which they consider negligible. If that's the case what is the difference between:
* TigerBeetle's storage approach
* Paxos or Raft with enough error correction on disk writes that the probability of errors equals that of ECC RAM which is considered negligible
I've probably made a logical error in my reasoning but I can't see it. Can someone enlighten me?
I must be missing something. The paper describes the algorithm of the CRDT and mentions that "timestamps ’t need to be globally unique and totally ordered".
Then it mentions multiple times that Lamport clocks/timestamp can be used as the timestamp in their system but as far as I know these only give a partial order of events. How is this reconciled in their system?
Yes, and this problem is exacerbated by the raspberrypi's microUSB power input. We've seen countless number of cases where a Pi exhibiting SD card corruption was also experiencing undervoltage. If you're using raspberrypis in production it's paramount to have a good microUSB cable and power supply
I'd be interested to hear about the failure modes of I2C that you've observed. Unfortunately I'm not familiar with CAN or SPI. Do these alternatives provide error correction as part of the protocol? Do they offer superior reliability in some other way?
balena founder here. balenaOS does use the hardware watchdog available on the raspberry pi to detect CPU lock ups and automatically reset the board. On top of that we're also running software watchdogs that check the health of key system components and restart them if they become unhealthy.
It's true that SD cards are known for getting corrupt. balenaOS separates the partitions of the device and keeps the userspace in a readonly one while keeping mutable OS state in a separate partition. We are very conscious of writing as infrequently as possible to the SD card for this reason. The partition that accepts the most writes is the one holding the user container, which will get written to during an update, and also in case the user container stores any data on the device.
I'm aware that SD cards will internally swap blocks and don't really care about partition boundaries but assuming you're using an SD card with a well designed firmware it shouldn't lose a block during wear leveling.
That said, the SD card problem is one reasons we designed balenaFin :)
balena founder here. balenaOS comes with all the infrastructure needed for robust host OS updates. We expose this functionality to our users via a button in the web dashboard. We don't yet have an automated, rolling upgrade style mechanism.
The main consideration for a feature like this is that sometimes containers have dependencies to interfaces exposed by the operating system which are not always stable. This is especially true for IoT usecases because containers will typically interface with some device connected to the system.
Tangential to this, we're working on an extended support release schedule (a la firefox) for balenaOS. I could see us building an automated OS update mechanism on top of that. We'll definitely think about it, thanks a lot for your feedback :)
Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.
We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.
Note: the linked position is focused on the data integration aspect of Materialize but we're also looking for wider database roles so feel free to apply if the general area of Materialize seems interesting to you!
What we’re looking for:
- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).
- Strong CS fundamentals (B.S./M.S. equivalent).
- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.
Compensation: $200K–$225K + equity.
Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.