Wikimedia narrows down the app sendin 90M requests to a pic of flower(phabricator.wikimedia.org)
phabricator.wikimedia.org
Wikimedia narrows down the app sendin 90M requests to a pic of flower
https://phabricator.wikimedia.org/T273741#6815828
104 comments
> Resorting to Twitter for support is increasingly common.
Always maintain some out of band support system. If that's email, so be it.
If there's business owners watching this, please do not make Twitter (or any other social media) your primary point of contact for support or abuse. I stay far away from Twitter such that I don't have an account and can't even see a single tweet without jumping through some hoops. I've learned via posting here that I'm not alone in this and that this trend will likely grow as time goes on.
Always maintain some out of band support system. If that's email, so be it.
If there's business owners watching this, please do not make Twitter (or any other social media) your primary point of contact for support or abuse. I stay far away from Twitter such that I don't have an account and can't even see a single tweet without jumping through some hoops. I've learned via posting here that I'm not alone in this and that this trend will likely grow as time goes on.
On one side, StackOverflow has been a blessing.
On the other, it made "Copy Paste Programming" go to eleven. (There was even a C# example the other day that famously broke in a big project but I can't find it)
Maybe it would be a case of Stack Overflow linting examples to remove stuff like builtin urls and such.
I've seen "developers" complaining that example code with a very explicit >replace this part for your use case< complaining that the example didn't work. I guess making some things harder would just be an overall gain.
On the other, it made "Copy Paste Programming" go to eleven. (There was even a C# example the other day that famously broke in a big project but I can't find it)
Maybe it would be a case of Stack Overflow linting examples to remove stuff like builtin urls and such.
I've seen "developers" complaining that example code with a very explicit >replace this part for your use case< complaining that the example didn't work. I guess making some things harder would just be an overall gain.
I recall once, there's at least a prototype visual studio plugin which would copy code examples straight from the internet.
I would love a bet it list of common use case templates which can just pop up when I use visual studio.
Or maybe a sort of snippet box to drag and drop in my code. For example reading and writing a file in C sharp isn't something I exactly know off the top of my head.
I would love a bet it list of common use case templates which can just pop up when I use visual studio.
Or maybe a sort of snippet box to drag and drop in my code. For example reading and writing a file in C sharp isn't something I exactly know off the top of my head.
You're probably thinking about some app not being able to start when some other specific app is running because both were using a GUID copied from SO for implementing single-instance apps.
Possibly the SO question in question: https://stackoverflow.com/a/522874/615306
Possibly the SO question in question: https://stackoverflow.com/a/522874/615306
Yeah, it was Docker for Windows and Razer Synapse 3 who couldn't work together because they both copied the same SO code: https://twitter.com/Foone/status/1229641258370355200 :)
I cannot find the source, but the folklore says a Linux kernel developer wrote a USB tutorial with his USB VID/PID, years later, he found he became the manufacturer of all sorts of gadgets he never heard about.
It's a good example but I think it's not exactly that. Maybe it was a GUID generation code that would generate the same one for every instance?
> I know it has something to do with some mythical thing called a mutex, rarely can I find someone that bothers to stop and explain what one of these are.
Oh my
> I know it has something to do with some mythical thing called a mutex, rarely can I find someone that bothers to stop and explain what one of these are.
Oh my
In this particular example, this is not an ignorant bad question. It's a self-QA by an experienced developer.
Some time we'll get round to writing this up but there's a small customer of Cloudflare that gets a very high HTTP requests per second rate. It's a simple service (bit like a "what's my IP address" but not that) and it turns out that a quite popular hardware device hard-coded requests to this service and doesn't appear to cache the results and so it asks over and over and over again for the same information.
We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long...
Yesterday: over a billion HTTP requests...
We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long...
Yesterday: over a billion HTTP requests...
This is similar to how Qualcomm's DNS servers got knocked off the air. An OEM shipped an update which would query a development TURN server we were running - once per connection, over millions of devices. It was a crazy day.
Are you not tempted to just block the requests from these devices, and let the manufacturer take the loss? I imagine serving all those requests is costing real money.
It's not a TOS violation. It did cause us some ops pain at one point (they were getting hit with > 50,000rps concentrated in certain locations). But one of the reasons Cloudflare can operate our service is we have 3.2 million customers who are doing all sorts of stuff. We get so much stronger from that great variety of traffic.
Right. I'm sure you can explain more but I've read due to peering agreements and the like having a lot of one-directional traffic can be a good thing.
Could you be more explicit on the nature of the service?
I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?
I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?
One day we'll write this up. It is not time or connectivity.
> To narrow down the app, we decided to observe connections to the image from clients (phones) to our servers. We did this by opening the popular apps one-by-one and noting down the time. After doing this for all the apps, we then ran this query in Hive: SELECT * FROM wmf.webrequest WHERE year=2021 AND month=2 AND day=9 AND parse_media_file_url(uri_path).base_name='/wikipedia/commons/1/16/AsterNovi-belgii-flower-1mb.jpg' AND webrequest_source='upload' AND uri_host = 'upload.wikimedia.org' AND user_agent='-' AND ip=<IP>;
> We then found the specific app that was making the request by matching the time when it was opened and the time image was requested from our servers, restricting the results to the User-Agent '-' and from the IP we tested.
Unless I missed something, running mitmproxy/Charles etc. in front of the phone would have been way easier than querying the entirety of Wikimedia server logs and trying to match IP & timing windows.
> We then found the specific app that was making the request by matching the time when it was opened and the time image was requested from our servers, restricting the results to the User-Agent '-' and from the IP we tested.
Unless I missed something, running mitmproxy/Charles etc. in front of the phone would have been way easier than querying the entirety of Wikimedia server logs and trying to match IP & timing windows.
"way easier" depends on what you are familiar with.
If you are familiar with "querying the entirety of wikimedia server logs" and do it all the time (the word "entirety" makes it seem like a big deal, but they clearly have tools meant for this that they use all the time)... and have never learned to use "mitmproxy/Charles etc" before....
It sounds like the "querying the entirety of the server logs" for this task probably took them tens of minutes at most. It would probably take me at least an hour or two to learn how to use "mitmproxy/Charles etc".
So "way easier"?
If you have to do this sort thing all the time, it might be useful to install and learn how to use "mitmproxy/Charles etc", why not? Certainly worth considering. But if the tools you have are working for you...
I mean, what they did seems like it worked to get them the answer and was pretty efficient, using the toolset they use all the time for dealing with wikimedia ops... Seems like some good detective work to me. I get the desire to point out other tools that would be well-suited for this kind of task, but why the need to point it out as if they did something wrong or not "way easier"? Sounds like what they did was pretty easy for them, and they didn't need to learn new tools to make it "way easier".
I enjoyed hearing about how they tracked this down, and found it useful. Pointing out how they didn't use the "right" tools just makes it less likely people will be willing to share their processes.
If you are familiar with "querying the entirety of wikimedia server logs" and do it all the time (the word "entirety" makes it seem like a big deal, but they clearly have tools meant for this that they use all the time)... and have never learned to use "mitmproxy/Charles etc" before....
It sounds like the "querying the entirety of the server logs" for this task probably took them tens of minutes at most. It would probably take me at least an hour or two to learn how to use "mitmproxy/Charles etc".
So "way easier"?
If you have to do this sort thing all the time, it might be useful to install and learn how to use "mitmproxy/Charles etc", why not? Certainly worth considering. But if the tools you have are working for you...
I mean, what they did seems like it worked to get them the answer and was pretty efficient, using the toolset they use all the time for dealing with wikimedia ops... Seems like some good detective work to me. I get the desire to point out other tools that would be well-suited for this kind of task, but why the need to point it out as if they did something wrong or not "way easier"? Sounds like what they did was pretty easy for them, and they didn't need to learn new tools to make it "way easier".
I enjoyed hearing about how they tracked this down, and found it useful. Pointing out how they didn't use the "right" tools just makes it less likely people will be willing to share their processes.
You added way too many assumptions to a short non-accusatory comment.
I mean, the query doesn't look that complicated, and is something they'd obviously be already familiar with how to use - not that mitmproxy etc is particularly hard, but its another thing.
Judging by the last point they seemed to want to doublecheck findings and confirm before throwing accusations around.
> To further confirm this finding and to ensure that we had the correct app, we decided to log DNS queries from a phone by setting up a local resolver to capture DNS traffic. After pointing the phone towards it and launching the app, we noticed that it was indeed the one looking up upload.wikimedia.org on startup.
> To further confirm this finding and to ensure that we had the correct app, we decided to log DNS queries from a phone by setting up a local resolver to capture DNS traffic. After pointing the phone towards it and launching the app, we noticed that it was indeed the one looking up upload.wikimedia.org on startup.
Certificate pinning has made this a pain in the arse.
You can only pin your own certificate, not someone else’s. In this case you probably don’t even need SSL proxying to pin down the culprit, as I dare say not many apps connect to wikimedia on startup. You do need SSL proxying to be sure though.
The app may not load at all with mitmproxy if it has pinned its server cert though.
No, you can selectively decrypt HTTPS requests for only some domains, and act as passthrough for others.
Nope. Starting from Android 10, unless an app has explicitly allowed user certificates (and no-one reasonably does, it's all behind a <debug-overrides> flag), you will not be able to MITM it. You may inject your certificates as much as you want. The only option is to have a device on which you have root access, which can push system certificates with adb. This pretty much only means the android emulator these days.
I don’t use Android so I wasn’t aware of that. But that’s a completely separate concern from cert pinning which does not hinder decrypting third party connections at all.
Edit: after looking into this a bit, this is pretty nuts. How do enterprises inject certificates now?
Edit: after looking into this a bit, this is pretty nuts. How do enterprises inject certificates now?
re enterprise injection:
They don't. It's been made increasingly clear that allowing certs roots to infect unrelated apps is a Bad Thing. MDM profiles etc presumably allow internal certs to be deployed, but those are hopefully limited as countries, let alone companies, have attempted to use those mechanisms to spy on millions of people.
They don't. It's been made increasingly clear that allowing certs roots to infect unrelated apps is a Bad Thing. MDM profiles etc presumably allow internal certs to be deployed, but those are hopefully limited as countries, let alone companies, have attempted to use those mechanisms to spy on millions of people.
So it's still possible on rooted devices? Seems good enough.
If they have the logs in Elasticsearch or something similar it wouldn't be all that hard.
It looks like it wasn't a big deal to perform the lookup, so I guess it comes down to what the engineer was familiar wit.
It does say later in the post that they used a local proxy to confirm their findings, too. Maybe they wanted to check from both sides, just to be sure.
It does say later in the post that they used a local proxy to confirm their findings, too. Maybe they wanted to check from both sides, just to be sure.
[deleted]
They didn't have 'the phone'. They only had traffic logs and had to find the app that matched a certain traffic pattern. And they did have 'something like ElasticSearch', it's all in the link, and even in this reply.
I'm confused, it says
> We did this by opening the popular apps one-by-one and noting down the time
> We did this by opening the popular apps one-by-one and noting down the time
It sounds like they did have the phone. Otherwise, Wikipedia is doing some sketchy stuff if they can figure out what apps are on my phone just from my IP address (IMO unlikely)
The most impressive thing about this is that the offending app doesn't even display the image, it was some copypasta code the app developers apparently didn't even understand.
Welcome to the Internet. Here we welcome amateurs and it shows. Or not, like in this case.
Don’t know why you would go with didn’t understand rather than didn’t care. I thought the prevailing theory was that it was used for measuring internet speeds.
the hard part about this, if it were not wikimedia but some individual person's server, is that the traditional method of using something like an apache rewrite rule to replace the jpg/png with goatse wouldn't work, because the image isn't even being displayed.
Even worse: You might be seen as an easy way to Goatse people. Just give the yahoos the URL of an image you're rewriting to be the infamous gape. The traffic would only grow once you came to be known for Goatse as a Service.
> Goatse as a Service (YC S21)
Implemented via several dozen geographically distributed 1U servers with 10 or 100GbE network interfaces, and anycast DNS. You could do it pretty much the same as anycast DNS recursive resolvers, but all sharing the same hostname, IP and TLS keys.
Help us solve the problem of hotlinking.
Related: https://news.ycombinator.com/item?id=26072025, 20% of requests for Wikimedia Commons are for one image of a flower, 2 days ago
> 12. By this time, we had isolated the app and were convinced that this is the one that is fetching the image on startup. We could not find the image anywhere in the app, confirming our theory that it fetches the image but does not display it.
The analysis stops right when things start to become interesting. I was hoping there'd be a decompiled code snippet to see what the app in question is actually doing with the image, since it's not displaying it.
The analysis stops right when things start to become interesting. I was hoping there'd be a decompiled code snippet to see what the app in question is actually doing with the image, since it's not displaying it.
If one were to decompile and check what the app is doing, I'd guess it would still be a good idea to not say so on a public forum. Especially, in cases where there is so much public attention. :)
I agree with what you're saying about not going public immediately if anything malicious had been discovered, but then they probably wouldn't have written:
> We will thus hold back the banning of the url for now [...]
If on the other hand it's really just some benign leftover example code downloading the image and not doing anything with it later as has been suggested and is indeed the most likely, there'd be no harm in confirming that's the case.
They went to great lengths with their investigation, and this would be the obvious final step to wrap it up. Posting a couple of the relevant .smali lines wouldn't have to reveal the name of the app in question (which at this point can be identified by anyone sufficiently motivated anyway).
> We will thus hold back the banning of the url for now [...]
If on the other hand it's really just some benign leftover example code downloading the image and not doing anything with it later as has been suggested and is indeed the most likely, there'd be no harm in confirming that's the case.
They went to great lengths with their investigation, and this would be the obvious final step to wrap it up. Posting a couple of the relevant .smali lines wouldn't have to reveal the name of the app in question (which at this point can be identified by anyone sufficiently motivated anyway).
Evidently the download has been specially crafted and surreptitiously emplaced to globally disseminate a steganographically embedded key that decrypts tailored malware aimed at disrupting the [REDACTED] nuclear weapons programme and for which the app is a weaponised delivery sabot distributed and marketed as part of the same covert operation.
What I'm trying to say is, the image is a plant
What I'm trying to say is, the image is a plant
4cao(1)
> To recap, we were aware of the following at this stage:
> it is a popular chat/social media mobile app used in India
> it sets the User-Agent and Referer to '-'
> it fetches the image from Wikimedia Commons but does not display it
And then they identified which app it was, but it is not revealed
> it is a popular chat/social media mobile app used in India
> it sets the User-Agent and Referer to '-'
> it fetches the image from Wikimedia Commons but does not display it
And then they identified which app it was, but it is not revealed
Any idea why they don't name the app?
Wikimedia is simply trying to be polite and not to publicly shame the company.
In addition to that, one of the culprits here is a widespread sample code that was carelessly copied to a popular app. Shaming does penalize the other culprit but not that one.
Surely the code sample is not to blame here? Or do you truly think the author of the sample is also deserving of being called a "culprit"?
Just to be clear, I don't think every author using this image for their sample code is to blame. I'm specifically looking for someone using the public Wikimedia CDN for speed tests [1] and I think that someone is probably the sample code author.
[1] https://news.ycombinator.com/item?id=26073450 has located the actual app and intended purpose, for your information.
[1] https://news.ycombinator.com/item?id=26073450 has located the actual app and intended purpose, for your information.
It could provoke an angry mob, which doesn't help anybody.
Given that it doesn't display it, and that the image u.r.i. is frequently used in example code rather than something with example.com, it was almost certainly an innocent mistake of copying example code.
> it fetches the image from Wikimedia Commons but does not display it
If I were to guess, they use the picture as a connectivity/speed test. They probably figured Wikipedia has unlimited free bandwidth, so they didn't care.
If I were to guess, they use the picture as a connectivity/speed test. They probably figured Wikipedia has unlimited free bandwidth, so they didn't care.
The fact the picture has "1mb.jpg" in the file name suggests it is 1 megabyte and therefore a good candidate for use as a speed test.
However the image is in fact 160 kilobytes, so I suspect whatever speedtest is being done is getting the wrong results...
However the image is in fact 160 kilobytes, so I suspect whatever speedtest is being done is getting the wrong results...
Could be worse. Imagine if the original "1mb.jpg" was correctly linked, boom, 10x more traffic...
This seems interesting from a legal point of view as well:
Is an app downloading, but never displaying, creative commons content infringing on copyright (by not showing correct attribution and violating the CC terms)?
Besides copyright, could this be considered theft of service?
Is an app downloading, but never displaying, creative commons content infringing on copyright (by not showing correct attribution and violating the CC terms)?
Besides copyright, could this be considered theft of service?
> This seems interesting from a legal point of view as well
Is it? If you make a resource freely available to people online, and people access said resource, what's the legal ramification there? It would appear there is no malicious intent which would be necessary to make the case for abuse, and theft of service would be a stretch given that Wikimedia doesn't charge for their service.
Is it? If you make a resource freely available to people online, and people access said resource, what's the legal ramification there? It would appear there is no malicious intent which would be necessary to make the case for abuse, and theft of service would be a stretch given that Wikimedia doesn't charge for their service.
I think OP is referring to the fact that your device is internally making a copy of the image during the download process, yet the creative commons license requires that copies of an image have attribution. The terms of the license are therefore likely not being met.
Very unlikely anyone will care...
Very unlikely anyone will care...
It's a violation of the terms of use
>Disrupting the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
>Disrupting the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
Though it's interesting to think of the possible legal ramifications, I doubt there will be a court case. The "damages" looks like about ten terabytes of bandwidth, and lawyer fees would surpass that in days .
NTP domains have had a history of similar problems, and they seem to be resolved by apologizing, fixing the problem, and sometimes a donation.
NTP domains have had a history of similar problems, and they seem to be resolved by apologizing, fixing the problem, and sometimes a donation.
I don't think this is theft. Nothing was stolen. At most Wikipedia suffered some damage in the form of slightly increased bandwidth costs. Also, theft is involuntary. The Wikipedia server has the power to simply refuse the connection.
It's difficult to steal something offered for free. You could try to charge them with attempted DDOS or something like that but since wikimedia did not suffer any actual degradation of service. I think that at most you can go for something like "causing harmful traffic through negligence" but you'd need to prove the traffic was actually harmful.
In any case let's not get carried away. 90 million requests for a 70KB file is only 5.8 TB. Wikimedia mentions in their about pages that they are hosted on bare metal servers in various places around the world. Just going on the bandwidth charges of the first provider in the list, that'd be about $30 USD per month if they have the "bulk" pricing or $300 USD per month if they use the list pricing. I don't think that is worth going to court over for the Wikimedia foundation.
In any case let's not get carried away. 90 million requests for a 70KB file is only 5.8 TB. Wikimedia mentions in their about pages that they are hosted on bare metal servers in various places around the world. Just going on the bandwidth charges of the first provider in the list, that'd be about $30 USD per month if they have the "bulk" pricing or $300 USD per month if they use the list pricing. I don't think that is worth going to court over for the Wikimedia foundation.
> it is a popular chat/social media mobile app used in India
Indians of Hacker News, what are the likely candidates?
Indians of Hacker News, what are the likely candidates?
Considering that this seems to have been code accidentally left in while copy pasting from a tutorial, it's very hard to say, without doing the exact same investigation the Wikimedia team did.
There are a lot of apps that have launched in India around that time frame with huge numbers of users thanks to nationalistic rhetoric. They are terrible apps, but they are made in India terrible apps, and that apparently is enough to get a large following in India of late.
There are a lot of apps that have launched in India around that time frame with huge numbers of users thanks to nationalistic rhetoric. They are terrible apps, but they are made in India terrible apps, and that apparently is enough to get a large following in India of late.
To be fair, a nation has to make lots of terrible apps first, before they can make mediocre apps, and then good apps. The logic of wanting homegrown apps could still be correct, even though there is a painful period of terrible apps.
Now, what the U.S. excuse is for its terrible apps, I'm not sure...
Now, what the U.S. excuse is for its terrible apps, I'm not sure...
> They are terrible apps, but they are made in India terrible apps, and that apparently is enough to get a large following in India of late.
That's awesome!
Every app has to start somewhere. I mean, even ISRO started by transporting rocket parts on bikes, and now has a satellite around Mars.
https://www.indiatimes.com/technology/science-and-future/fro...
That's awesome!
Every app has to start somewhere. I mean, even ISRO started by transporting rocket parts on bikes, and now has a satellite around Mars.
https://www.indiatimes.com/technology/science-and-future/fro...
ShareChat may be? Or some video apps like TikTok
Moj? That was the seemingly most viable TikTok alternative that cropped up.
Yeah, guess that the Wikimedia guys are not adepts of the "name and shame" method. Although it would be entirely justified in this case, and stop people trying to guess it and implicating the wrong app...
It's pretty lame they didn't name the app, it's clear some programmer(s) somewhere needs the wake up call.
[deleted](1)
This reminds me of a Stack Overflow answer that became popular but instead of using 'example.com', they used some other random, but valid, URL that suddenly created a huge spike in traffic for the unsuspecting web page.
I'm now curious, how much traffic does example.com receive? Does it use Anycast? Does IETF publish statistics? Searched and found the answer here, no known statistics, but it's backed by a CDN.
* Ask HN: What does traffic to example.com look like?
https://news.ycombinator.com/item?id=8057442
* Ask HN: What does traffic to example.com look like?
https://news.ycombinator.com/item?id=8057442
This is going to be so much fun when the owner of foo.bar brings their domain online :D
I'm pretty sure that this pic was also linked to in some of the examples in the Coding Together course from Apple and Stanford a few years back.
It makes sense that's still included deep down in some copy/paste app stuff.
It makes sense that's still included deep down in some copy/paste app stuff.
[deleted]
Dupe
More discussion https://news.ycombinator.com/item?id=26072025
More discussion https://news.ycombinator.com/item?id=26072025
it's not a dupe. This is the follow on.
I wonder if the chat app will stop working. I'm betting it is just using this image retrieval as a test to see if it's online.
Insert a delay before the image is returned by the server, watch who starts complaining.
frankly, I would have redirected the image to their own homepage.
If only for the outrageous user agent.
If only for the outrageous user agent.
Guys can you stop the "hello from Hacker news" comments? its spam.
There was ”Hello to Hacker News” from one of the moderators, to which someone replied ”Hello from Hacker News”. No that much of a spam IMO. And it was from one person, not much all of ”guys” in HN
Never link to issues on bug tracker. This always, without fail, happens. It makes more work for people who are already stressed about trying to solve a problem.
Just don't link to bug trackers, ever.
Just don't link to bug trackers, ever.
Nah, if it's a historically or culturally significant issue, I'm definitely linking to it so people can read about it. Just add a warning not to post meaningless garbage in the ticket's thread, if you're that concerned about it.
TL;DR The thread doesn't name the specific app, but it's Indian, likely a social Android app, probably Say Namaste or Mutton TV. The right app was found by installing apps and checking the connection logs.
Wasn't say namaste. I suggested that within the thread and contacted them, but ssingh has said it's not them.
Looks like a case that the developers carelessly copied and pasted some "sample code" into the app...
> [...] it seems that there is no good way to get in touch with them through email (I sent an email to all publicly available channels, only to get back an autoresponder that assumes I'm an user of the app and asking for my phone number). I eventually resorted to DM their CEO on twitter.
Resorting to Twitter for support is increasingly common. The importance of having an "abuse@" email (and possibly some social media bots to DM all sysops when a mail arrives)...