500 miles (2002)(web.mit.edu)
web.mit.edu
500 miles (2002)
http://web.mit.edu/jemorris/humor/500-miles
34 comments
I had an actual high severity issue on-call page over this kind of thing once.
When I first started with Amazon, I was a dev on the team owning the UI component that all warehouse Picking associates used. Had to run on mobile handscanners with IE5.5. The UI was dead simple, pretty darn fast and that speed matters for worker efficiency. Taking 3 seconds between an item scan and the next pick being displayed was unacceptably slow.
One night a few weeks after Christmas, I got an autocut alarm at around 10pm. It said "PickUI page loads 90th percentile > 5 seconds" or something to that effect. I logged in and sure enough, the latency was spiking up and down like crazy for the entire European stack. All our dependencies were rock solid, metrics showed stable performance. Network was fine, no issues. Rendering was normal. But the round trip latency from the users' perspective (sent via Ajax post after a page load) was showing some round trips were taking 15 seconds!
After 30 minutes of scratching my head, I happened to accidentally switch my graph from "P90 latency" to "number of data points". There were very, very few. It turned out that since the Christmas season was just over, all the EU wearhouses were only running day shifts that week. (That doesn't happen very often anymore, I hear). In the entire European region, there was one picking associate working. And she was picking in some area with poor WiFi signal. Around 20% of the time, her page loads were slow because her WiFi sucked. I drew out a rough map of the locations she picked from and the latency- they was some kind of dead zone. But she was the entirety of my metrics for Europe, so that meant my 90th percentile was terrible.
I modified the alarm to have a minimum number of data points threshold and went to bed.
When I first started with Amazon, I was a dev on the team owning the UI component that all warehouse Picking associates used. Had to run on mobile handscanners with IE5.5. The UI was dead simple, pretty darn fast and that speed matters for worker efficiency. Taking 3 seconds between an item scan and the next pick being displayed was unacceptably slow.
One night a few weeks after Christmas, I got an autocut alarm at around 10pm. It said "PickUI page loads 90th percentile > 5 seconds" or something to that effect. I logged in and sure enough, the latency was spiking up and down like crazy for the entire European stack. All our dependencies were rock solid, metrics showed stable performance. Network was fine, no issues. Rendering was normal. But the round trip latency from the users' perspective (sent via Ajax post after a page load) was showing some round trips were taking 15 seconds!
After 30 minutes of scratching my head, I happened to accidentally switch my graph from "P90 latency" to "number of data points". There were very, very few. It turned out that since the Christmas season was just over, all the EU wearhouses were only running day shifts that week. (That doesn't happen very often anymore, I hear). In the entire European region, there was one picking associate working. And she was picking in some area with poor WiFi signal. Around 20% of the time, her page loads were slow because her WiFi sucked. I drew out a rough map of the locations she picked from and the latency- they was some kind of dead zone. But she was the entirety of my metrics for Europe, so that meant my 90th percentile was terrible.
I modified the alarm to have a minimum number of data points threshold and went to bed.
Took me a couple times to read through this to realize that this was asking about the intermediate trip time between the server and the client rather than the total round-trip time that it took to process requests, including processing time.
Very creative question, requires thinking outside the box. I don't think I would be able to answer it correctly.
Very creative question, requires thinking outside the box. I don't think I would be able to answer it correctly.
As measured it was actually talking about total round-trip time including server processing, but it was measured from within the JS on the client, from start-of-AJAX-request to finish, and so it included both server/client hops in addition to processing time. That's part of the red herring: since you expect the bottleneck to be server processing time, you look there first.
A decent partial answer to "How to prevent this?" is to look at additional metrics - graphing server-only latency would show the expected correlation with server load and an inverse correlation with total RTT, and would narrow the search space significantly.
A decent partial answer to "How to prevent this?" is to look at additional metrics - graphing server-only latency would show the expected correlation with server load and an inverse correlation with total RTT, and would narrow the search space significantly.
> The actual answer (I could tell this was drawn from a real experience) was "China"
“China” would have been my first answer to that question.
After spending so much time in a project that deals with massive amounts of HTTP requests from China, I realized that the majority of web developers are making too many assumptions. Several times I have found myself patching pieces of code because the connection was suddenly dropping in the middle of the operation, so I had to add retries and read incomplete streams of bytes to recover at least pieces of the actual data from our Chinese netizens.
In retrospective, I have learned a lot about TCP in the last year with this project.
“China” would have been my first answer to that question.
After spending so much time in a project that deals with massive amounts of HTTP requests from China, I realized that the majority of web developers are making too many assumptions. Several times I have found myself patching pieces of code because the connection was suddenly dropping in the middle of the operation, so I had to add retries and read incomplete streams of bytes to recover at least pieces of the actual data from our Chinese netizens.
In retrospective, I have learned a lot about TCP in the last year with this project.
It's extra fun when you're debugging strange connection resets that occur on a seemingly-random subset of A/B tests, and then find out that it's because a substring of the digits in the experiment ID happens to match the numeric date of the Tiananmen Square massacre, which is censored by the Great Firewall.
I just tried https://www.microsoft.com/19890604 from inside China (Wuhan, to be specific) on a naked (non-back-doored) connection. I got Microsoft's 404, not any of the GFW behaviour I've come to know and loathe.
I think I'll be calling bullshit on this.
I think I'll be calling bullshit on this.
And another data point.
https://www.baidu.com/s?wd=19890604&rsv_spt=1&rsv_iqid=0xfd2...
No blocking there and it's a Chinese search engine!
https://www.baidu.com/s?wd=19890604&rsv_spt=1&rsv_iqid=0xfd2...
No blocking there and it's a Chinese search engine!
Read vermilingua's Wikipedia link on Illegal Numbers. This was in 2012, which lines up with the reference cited on Wikipedia for the GFW blocking the numbers 89, 6, and 4. They reversed course after getting called out for this being brain dead.
To clarify, I actually queried http://www.microsoft.com/19890604 and had it redirected to https://... by Microsoft's server. So the URL query was in plaintext.
Just, you know, so the undies come unbunched before people comment.
Just, you know, so the undies come unbunched before people comment.
Several months ago, I was doing ping geolocation on a bunch of VPN servers. To see if they're actually located where claimed. I looked at several AirVPN servers, claimed to be in Hong Kong. Based on data from hundreds of probes, they did in fact seem to be in Hong Kong.
However, minimum(rtt) was ~300 msec for probes in several mainland locations, some <100 km from Hong Kong. Why might that have been?
However, minimum(rtt) was ~300 msec for probes in several mainland locations, some <100 km from Hong Kong. Why might that have been?
What kind of position were you applying for that they asked you this very interesting question?
Frontend software engineer. Latency management is included in front-end engineer's job (because most of the time, it's Javascript's fault), so debugging something like this would not fall outside of the job description. My interviewer was also a frontend software engineer. Both of us ended up doing more general-SWE/TL jobs later on, though (by coincidence, I ended up on the same team as her a year and a half later).
I doubt that they would ask the same question now - for one, most Chinese queries are served out of datacenters in Southeast Asia now (at least when I left; if I believe the Project Dragonfly news I hear about on HN they may be served from inside China), so there's no trans-Pacific cable hop. The Chinese population has also gotten a lot more access to broadband & high-bandwidth mobile connections in the decade since this interview.
I doubt that they would ask the same question now - for one, most Chinese queries are served out of datacenters in Southeast Asia now (at least when I left; if I believe the Project Dragonfly news I hear about on HN they may be served from inside China), so there's no trans-Pacific cable hop. The Chinese population has also gotten a lot more access to broadband & high-bandwidth mobile connections in the decade since this interview.
Somewhat related, I just found out that OSX ships with units version 1.0.
The prompt gives "586 units, 56 prefixes" instead of the post's "1311 units, 63 prefixes".
So if you want to try this you can `brew install gnu-units` and run gunits instead:
"3070 units, 109 prefixes, 109 nonlinear units"
It's pretty annoying that OSX ships with such outdated utilities [1]
[1] http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/
The prompt gives "586 units, 56 prefixes" instead of the post's "1311 units, 63 prefixes".
So if you want to try this you can `brew install gnu-units` and run gunits instead:
"3070 units, 109 prefixes, 109 nonlinear units"
It's pretty annoying that OSX ships with such outdated utilities [1]
[1] http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/
In this particular case, you could also just use the equivalent units of c-msec.
I love this story and find it refreshing each time I read it. That said, has anyone ever looked into if it's real or apocryphal? I'd be curious to know.
I don't know whether the original story was -ahem- embellished a bit or not, but I do know that after forwarding this to a sysadmin friend of mine running a host of Sun boxes at the Norwegian University of Science & Technology, he got somewhat obsessed with replicating this bug - turned out on his hardware du jour the range was some 1100km (700 miles.)
Well, Trey Harris at least claims that it really happened ( but he may have taken liberty with some details): https://www.ibiblio.org/harris/500milemail-faq.html
[deleted]
I believe the problem here can be used as a proof of proximity in a blockchain. Imagine 2 specially designed devices that can receive a string, perform a public key encryption on it, and transmit it with very low turnaround time.
A bitcoin block hash is found, and 2 participants begin a back and forth hashing session in which billions of round trips are performed...every minute or so, a transaction containing the latest result is submitted to the network, and eventually one is locked in.
An interested party could then verify that the transaction could not have occurred unless the 2 keys involved were within a certain physical distance from one another.
Not sure what it could be used for yet, but it's something that feels like it could be important for some purpose.
A bitcoin block hash is found, and 2 participants begin a back and forth hashing session in which billions of round trips are performed...every minute or so, a transaction containing the latest result is submitted to the network, and eventually one is locked in.
An interested party could then verify that the transaction could not have occurred unless the 2 keys involved were within a certain physical distance from one another.
Not sure what it could be used for yet, but it's something that feels like it could be important for some purpose.
That would certainly demonstrate that the keys were brought geographically near each other for some amount of time, which is interesting, but of what use would it be? It can't usefully test where I am, just my key. I can rent a cloud machine anywhere in the world for 15 minutes and put my key there temporarily.
If you had a way of projecting a pattern onto a surface you are taking a picture of, you could receive a hash, project that out onto whatever surface, take a picture, and send back the signed hash of the picture you just took, thus proving that a particular picture was taken in a particular place.
If that picture is of you, you can thus prove that you were in a particular place at a particular time.
If that picture is of you, you can thus prove that you were in a particular place at a particular time.
photographic proofs are not trustworthy... even with projections.
https://youtu.be/p5_tpq5ejFQ
https://youtu.be/p5_tpq5ejFQ
Only works if there's no router delays in the middle. The vast majority of consumer Internet connections fail this - I've already got an 8ms ping just to get to the first Comcast node, from my home wi-fi router. It's near instantaneous within Comcast's network, but then loses a couple milliseconds handing off to their fiber backbone provider.
He could use timeout as a proxy for distance because all the e-mail recipients were within the same university, with a direct-switched network. Once it got on the wire, the only latency was from speed-of-light.
He could use timeout as a proxy for distance because all the e-mail recipients were within the same university, with a direct-switched network. Once it got on the wire, the only latency was from speed-of-light.
You could use this to prove that N devices are spread fairly geographically evenly near the line between two devices you know the location of, if the transport latency was fairly constant.
If you cross this line over multiple countries, you now have a system where you can prove that the devices are spread across some number of the different countries, which probably has some useful properties, like making nation-state attacks slightly more difficult.
If you cross this line over multiple countries, you now have a system where you can prove that the devices are spread across some number of the different countries, which probably has some useful properties, like making nation-state attacks slightly more difficult.
I don't think you can prove they have a minimal distance between them, just maximal. Thus, any data stream may have always came from a single country, you can only impose an upper limit.
Sure, you can prove a minimal distance. Think of this problem as a metal chain. Each link of the chain enforces a maximal distance on the nodes it adjoins. The chain is bound on either end to some fixed point.
In an non-constrained case, sure, there's no-minimal link distance. You just have a pile of chain.
However, if you make the chain taut between the two points, say by decreasing the number of nodes in the chain or the maximal distance of links, the minimal distance between nodes then approaches the maximal distance between nodes. When you have a chain constrained to the straight line between the two points, the maximal distance between nodes is equal to the minimal distance between nodes.
In an non-constrained case, sure, there's no-minimal link distance. You just have a pile of chain.
However, if you make the chain taut between the two points, say by decreasing the number of nodes in the chain or the maximal distance of links, the minimal distance between nodes then approaches the maximal distance between nodes. When you have a chain constrained to the straight line between the two points, the maximal distance between nodes is equal to the minimal distance between nodes.
[deleted]
I was today years old when I discovered the units program.
I read this every time it gets posted. Just too good a story.
I’ve seen this several times before through the years, but somehow missed the ending.
> “I'm looking for work. If you need a SAGE Level IV with 10 years Perl, tool development, training, and architecture experience, please email me at [email protected]. I'm willing to relocate for the right opportunity.”
I wonder how things turned out for Trey...
> “I'm looking for work. If you need a SAGE Level IV with 10 years Perl, tool development, training, and architecture experience, please email me at [email protected]. I'm willing to relocate for the right opportunity.”
I wonder how things turned out for Trey...
From Trey's FAQ on the story, linked below [0]:
22. The signature says you're looking for work. Are you still?
Nope, but thanks for asking!
Looks like he is doing well https://www.linkedin.com/in/treyharris/
The actual answer (I could tell this was drawn from a real experience) was "China". The valleys in server load corresponded to EST nights, which happens to correspond to the workday in China. During these time periods, there are fewer users online, but the vast majority of them are located in Asia, where a response to them needs to get through the Great Firewall of China and cross a trans-Pacific cable. Meanwhile, the U.S. west coast is just going to bed and the U.S. east coast & Europe are sleeping, so all of the low-latency users drop out of the population sample. It was a nifty application of both Simpson's Paradox, correlation-is-not-causation, and speed-of-light limits.
Extra points if you can think of how to avoid strange debugging situations like this in the future.