What’s new with our in-house load balancing service(dropbox.tech)
dropbox.tech
What’s new with our in-house load balancing service
https://dropbox.tech/infrastructure/robinhood-in-house-load-balancing-service
54 コメント
> Why even consider ZK in 2024?
I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.
I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.
Great question
Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.
Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.
We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2....
Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.
Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.
Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis....
I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.
I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.
[deleted]
Very cool. Low-state approaches are always more interesting.
I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?
[stub for offtopicness]
Bees gather pollen. In the fall, as temperatures get lower, they still want to keep gathering pollen, but sometimes they run out of warmth and energy to fly all the way back to the nest. Thus you will find bees sleeping on flowers through the night. When morning comes and the sun warms them, they'll pick up and fly off to the nest to drop off the last night's load of pollen.
[flagged]
"load balancing" is a cold way to describe a layoff...
"fully automated market-driven load balancing" more like.
"load balancing" is apt description of the role of HR
More like container orchestration and bin packing, amirite? Faulty container? Terminate and launch a new container. Cattle, not pets.
He does → https://www.linkedin.com/in/yi-shu-tai-13160154/
According to his LinkedIn profile: 7 years 6 months into the job.
Although, it looks like he transfered to a new team a year ago to do ML Ops.
According to his LinkedIn profile: 7 years 6 months into the job.
Although, it looks like he transfered to a new team a year ago to do ML Ops.
Dropbox just laid off 20% of the company yesterday. In many cases, people don't rush to update their LinkedIn so he may or may not still be there.
[deleted]
[flagged]
Round Robin [1] is a popular algorithm for load balancing.
That’s the only reason they named the software “Robinhood”.
[1] https://en.wikipedia.org/wiki/Round-robin_scheduling
That’s the only reason they named the software “Robinhood”.
[1] https://en.wikipedia.org/wiki/Round-robin_scheduling
Why specifically "Robinhood", though? Robinhood is a fairly well known brokerage.
Why not Robin Hood or some other unique or generic name?
Why not Robin Hood or some other unique or generic name?
Many things are named after Robin Hood. It's not as though the brokerage invented the term or has any right of exclusive use. You'd have to be quite dim-witted to muddle up a brokerage and a load balancer.
Many things are named after Robin Hood, but I only know of two things specifically calling themselves Robinhood and one of them I only learned from this thread.
In fairness the load balancer presumably came first, but the point remains.
In fairness the load balancer presumably came first, but the point remains.
I think Robinhood the brokerage was _fairly_ obscure til 2020 or so, no? This appears to have gone into production in 2020, so is likely older than that.
Robinhood is a large filesystem policy engine and maintenance tool since 2004, too. Soo Robinhood Markets are already late to the game. :)
I don't think there are many people who will confuse an in-house load balancing service used at dropbox with a brokerage even if both are called Robinhood. Similarly, I don't think that people will confuse said brokerage with an old character of English folklore (and no I don't think it's the space that's important).
Context matters. There simply isn't a risk for meaningful confusion here. So what's the issue exactly?
Context matters. There simply isn't a risk for meaningful confusion here. So what's the issue exactly?
>(and no I don't think it's the space that's important).
That space or the lack thereof is important.
To date I only know of two Robinhoods, the brokerage and the load balancer, and I only learned about the second with this thread.
That space or the lack thereof is important.
To date I only know of two Robinhoods, the brokerage and the load balancer, and I only learned about the second with this thread.
> I don't think that people will confuse said brokerage with an old character of English folklore
I don't know about that. Why do you think people would not link a financial company to the folklore character famous for redistributing money? That's kind of disingenuous claim to me.
When I first heard about it, I actually assumed it was a reverse Robin Hood by taking money from the poor to give back to the rich type of scheme.
I don't know about that. Why do you think people would not link a financial company to the folklore character famous for redistributing money? That's kind of disingenuous claim to me.
When I first heard about it, I actually assumed it was a reverse Robin Hood by taking money from the poor to give back to the rich type of scheme.
You cannot avoid all names that were or will be ever used by anyone. Collisions will happen. And Robin Hood is not particularly obscure…
Probably because it takes from the rich and gives to the poor.
Weird that a gambling app would take thename of a popular character from legend!
Or maybe not, try actually naming something sometime and you'll see. It's not easy to come up with something that fits all the requirements AND has never been used before.
And it's not like anyone is ever going to get confused and accidentally try and use Dropbox's internal load balancer to buy options on a stock or something. Or try and make a gambling app the star of their next hit movie for that matter.
Or maybe not, try actually naming something sometime and you'll see. It's not easy to come up with something that fits all the requirements AND has never been used before.
And it's not like anyone is ever going to get confused and accidentally try and use Dropbox's internal load balancer to buy options on a stock or something. Or try and make a gambling app the star of their next hit movie for that matter.
A gambling app that --like all-- takes from the poor and gives to the rich. Sarcasm, or bitter misantropy, I suppose.
because it is an internal tool? why does that cause an issue?
[flagged]
There's a tool called Robinhood which started its life around 2004 [0], In turn, Robinhood Markets are founded in 2013.
Imagine knowing that Robinhood exists as one of the most popular (or the only) policy engines for large (EB sized) filesystems for but still naming your new company Robinhood.
I think Robinhood Markets should C&D immediately and find themselves a new name.
[0]: https://github.com/cea-hpc/robinhood
Imagine knowing that Robinhood exists as one of the most popular (or the only) policy engines for large (EB sized) filesystems for but still naming your new company Robinhood.
I think Robinhood Markets should C&D immediately and find themselves a new name.
[0]: https://github.com/cea-hpc/robinhood
The Robinhood being discussed in this post is not a product, it's internal technology. The name clashing doesn't really matter.
There are near-infinite things they could name their software, but instead they copied an existing name of another software project.
Quite insufferable if you ask me.
Quite insufferable if you ask me.
Software developers who love their job generally are into fantasy worlds, RPG games, astronomy and whatnot. If we nitpick every name choice, then life will have no fun.
I have computers named after stars and galaxies. Some codenames of the applications I have developed myself has stupid connections to characters because of what they (the software) do are similar to characters themselves. I named a module "Scrat" because it was a state machine which changed thousands of states per second according to input it received from a file I/O module called Manfred (which carries things on his back or with its trunk).
I have my names clash with others, and others' names clash with mine. Who cares, it's fun and makes things enjoyable for everyone, plus it allows one to inspire themselves by setting a vision for their software.
Or, maybe we should start naming our software with UUID strings. That won't clash in a long time. :D
I have computers named after stars and galaxies. Some codenames of the applications I have developed myself has stupid connections to characters because of what they (the software) do are similar to characters themselves. I named a module "Scrat" because it was a state machine which changed thousands of states per second according to input it received from a file I/O module called Manfred (which carries things on his back or with its trunk).
I have my names clash with others, and others' names clash with mine. Who cares, it's fun and makes things enjoyable for everyone, plus it allows one to inspire themselves by setting a vision for their software.
Or, maybe we should start naming our software with UUID strings. That won't clash in a long time. :D
Why does every single thing need a unique name? Even trademarked names for businesses and products are scoped to business sectors - its why apple the computer company who took the name of apple the record label was just fine (until apple invented itunes and had to buy the record label).
Heck even code does scoped naming... the compiler can't really handle punctuation that is slightly wrong and even it allows me to reuse variable names in the same program.
I don't know why we'd need a univerally unique name for each piece of software ever created. The existing software called robinhood is unrelated to this load balancer, so even if I could download and use it, I doubt I'd have much of a problem distinguishing the two.
Heck even code does scoped naming... the compiler can't really handle punctuation that is slightly wrong and even it allows me to reuse variable names in the same program.
I don't know why we'd need a univerally unique name for each piece of software ever created. The existing software called robinhood is unrelated to this load balancer, so even if I could download and use it, I doubt I'd have much of a problem distinguishing the two.
imo its your mindset that is quite insufferable, not the companies decisions of naming internal tooling.
[deleted]
[deleted]
biddendidden(1)
[flagged]
Totally. They should just continue to carry a workforce they don't need, risking the viability of the business and the other 80% of the jobs.
[flagged]
Why use both?
Why even consider ZK in 2024? Is it just popular in java shops?
Maybe I am misunderstanding?