As one of the contributors, I'm always happy to see interest and people using it.
Keeper is a really interesting challenge and we're really open to any kind of feedback and thoughts.
If you tried it out and have some feedback for it, I encourage you to create an issue (https://github.com/ClickHouse/ClickHouse), ask on our Slack, ping me directly on Slack... (just don't call me on my phone)
And don't forget that it's completely open-source like ClickHouse so contributors are more than welcome.
Also, we apply many different faults in our Jepsen tests which are run 3 times a day and we never had a problem with leader election. I know this doesn't confirm that there is no bug in it but it's pretty reassuring I would say.
I didn't think about the vDSO, and that it's used for `clock_gettime`, so thank you for the information and correction!
It's really hard for me to get a feel of what is okay to use, what is the standard way to do something without actually implementing everything and seeing how it behaves, so information like this help me a lot.
> hey probably just got lucky in that the drift always exceeded the time that elapsed between calls on other cpus
And this is also true. :D
But I would say we were lucky to notice this problem at all because it really doesn't happen on that many CPUs, at least on the ones we tried out.
Sorry if something was explained awkwardly or incorrectly, I'm trying to get a hang of writing posts like this, technical posts that are interesting and fun to read.
> If they were using unsigned arithmetic then they would've gotten the expected value.
And we did, but we did not expect to get the input values we got, which are explained by your last paragraph: "time stamp counter drift across cores." This was the real problem, and something new we learned.
> use clock_gettime+CLOCK_MONOTONIC
I forgot to include this because I was focused on the solutions we ended up using, but after some research this is 100% correct! We were afraid that too many syscalls during the query execution can produce huge performance hits so we decided to look for a different solution.
There is some basic knowledge on how to achieve the replication, but, in our case, a lot of it was improvised and we're not sure what is the best approach.
We tried to minimize the overhead (keep as little information as possible) and that was more or less the only guidance we used.
Well, consistency and availability are the problem here. While enforcing one of the property, you lose on the other. Not sure how would you achieve those opposing properties without introducing huge amount of complexity.
In this case, replication consists of communication between 2 instances, and when one them fails you need to pick consistency or the availability.
Paxos and Raft introduce consensus using multiple instances communicating with each other to mitigate those problems. Implementing correctly communication with 2 instances is hard enough, implementing it correctly for n instances is for sure harder.
In Memgraph, we actually tried to implement Raft and we did manage to some degree, but the effort was too much, especially when you look at the results we got (protocol that didn't work for many edge cases).
Keeper is a really interesting challenge and we're really open to any kind of feedback and thoughts.
If you tried it out and have some feedback for it, I encourage you to create an issue (https://github.com/ClickHouse/ClickHouse), ask on our Slack, ping me directly on Slack... (just don't call me on my phone)
And don't forget that it's completely open-source like ClickHouse so contributors are more than welcome.