Someone like Amit doesn't need money; by the time you get as highly placed at Google as he was you will be wealthy. I believe that Uber has interesting engineering challenges but likewise have doubts about their profitability.
SRE here. If you can handle your work nobody cares how much you are in the office. At all. However, unless you are insanely organized you won't be able to do this every single week--most successful googlers have several projects going on at once and there will be some times when you need to work hard on more than one.
Can you go into more detail regarding the prohibitive consistency issues? How do you maintain consistency in steady-state (ie. not during migrations?) Also, how do you make the call as to whether to bring your site down vs. attempting a live migration?
It would be interesting if someone from Github could discuss why they chose to do this migration by taking the whole site offline and doing the migration all at once. Did anyone investigate if this could be done without taking the site offline?
Consider that if you collect device fingerprints, you can detect users who live together, because they are likely to share devices. If somebody bad then gets access to that data, they could do creepy things to your users.
This is generally known as "device fingerprinting"--there are many ways to do it but they all involve probing for unique properties of a client via JS / Flash (listing installed fonts, drawing invisible characters and measuring via JS, etc.), then hashing them together to generate a unique ID for that user.
Some people think this practice violates users' privacy, and I'm one of them. This technology can be used to uniquely identify a user across multiple logins on the same site, or even multiple sites. It's quite widespread.
This paper[0] is mostly a survey of prominent DF providers and sites using this technology, and it's also a good primer on device fingerprinting techniques.
I think what will actually happen to requests in flight is:
- partial data received by old HAProxy is lost as old HAProxy exits
- new HAProxy comes online, binds to port, receives fd
- iptables rule removed. new HAProxy starts receiving new requests
- in-flight requests from the old HAProxy are timed out by the kernel (TCP RST) as nothing is there to read request data from the old fd or send response data.
So I think this is actually "worse" in some sense than the other retry behavior since it's not recovered inside the same TCP session but instead forces the client to open a new TCP session.
I feel I must point out that this relies on your clients to retry packets that iptables drops, so at the least they'll have a slower experience than they otherwise would. I believe that this will also break requests that are in flight when the reload happens, if they have sent partial data.
Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.
This is generally correct. In particular, when the HAProxy process is stopped/restarted there is a brief period during which the port is not bound by either process. (If the new process isn't able to get the socket when it boots it will sleep ~XXms, then try to bind/listen in a loop until it gets it or a retry threshold is hit.) During this time the kernel will reject incoming connections to the HAProxy port, so you are in danger of dropping incoming requests on the ground.
This release contains a neat feature: you can now bind HAProxy to a specific FD opened by its parent process. This means that you can babysit your HAProxy processes underneath a parent process that opens ports and get hitless HAProxy restarts, which I've long desired.
This is a terrible question so I'm not going to answer it, but if you are interested in reading about the corrupt history of Myrtle Beach (and the greater Grand Strand area) I strongly recommend Will Murdock's Banana Republic: A Year in the Heart of Myrtle Beach. http://www.amazon.com/Banana-Republic-Heart-Myrtle-Beach/dp/...
Funny to see this post pop up in a thread about how dysfunctional SF city government is. Myrtle Beach is effectively governed by an oligarchy of property development families.