Open sourcing Dicer: Databricks's auto-sharder(databricks.com)
databricks.com
Open sourcing Dicer: Databricks's auto-sharder
https://www.databricks.com/blog/open-sourcing-dicer-databricks-auto-sharder
14 comments
The Dicer Slicelet supports such a pattern, you can poll the assignment directly on the Slicelet:
https://github.com/databricks/dicer/blob/master/dicer/extern...
(btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/extern...)
https://github.com/databricks/dicer/blob/master/dicer/extern...
(btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/extern...)
[deleted]
Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?
Right - this is not about sharding data/datasets. This is for sharding in-memory state that a service might have. The problem of building services at low cost, high scale, low latency and high throughput is common in many environments including our services at Databricks, and Dicer helps with that.
Does anyone else have something similar?
What are some use cases that you found are useful?
What are some use cases that you found are useful?
Sounds related to Google Slicer: https://research.google/pubs/slicer-auto-sharding-for-datace...
These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies.
I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc
I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc
Sharded in-memory caching turns out to be rather useful at scale :)
Some of the key examples highlighted on our blog are Unity Catalog, which is essentially the metadata layer for Databricks, our Query Orchestration Engine, and our distributed remote cache. See the blog post for more!
Some of the key examples highlighted on our blog are Unity Catalog, which is essentially the metadata layer for Databricks, our Query Orchestration Engine, and our distributed remote cache. See the blog post for more!
For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.
[0] https://aws.amazon.com/builders-library/reliability-and-cons...
[1] https://brooker.co.za/blog/2021/05/24/metastable.html