In my ex-job we were developing e-commerce system, which was super-old|big|messy 0-test PHP trash. After two years of actively working on it, I still couldn't form a clear picture about the details of its subsystems in my head.
One day there is a call from a client, saying that he is missing many of his orders. The whole company is on its feet and we are searching for what went wrong. We are examining the server logs just to find out that someone is making thousands of requests to our admin section and tries to linearly increment order IDs in the URL. Definitely some kind of attack.. Our servers are managed by different company so we are opening a ticket to blacklist that IP. Quick search told me that the requests are coming from the AWS servers, and the IP leads me to an issue on GitHub for some nginx "bad bots" blocking plugin, saying that this thing is called Maui bot and we are not the first one experiencing it. Nice. Anyway, this thing is still deleting our data and we can't even turn off the servers because of SLAs and how the system was architected. So we are trying to find out how is it even possible, that unauthorized request can delete our data. We are examining our auth module, but everything looks right. If you are not logged in and visit the order detail (for example) you are correctly redirected to login screen. So how? We are reading the documentation of the web framework that the application is using. There is it. $this->redirect('login');. According to the documentation it was missing return before that statement. So without the return, everything after that point was still executed. And "everything" in our case, was the action from the URL. No one ever noticed, because there were no tests, and when you tried it in the browser, you were "correctly" presented with the login screen. Unfortunately, with side-effect..
Guy that wrote that line did it 5-6 years before this incident, and was out of the company for many years even before I joined. I don't blame him..
Fix. Push. Deploy.
No more deleted orders.
POST MORTEM:
The Maui bot went straight to the disallowed /admin paths in robots.txt and tried to increment numbers (IDs) in paths.
I remember, that because Maui bot actions were (to the system) indistinguishable from the normal user actions, someone had to manually fix the orders in the database just by using server logs and comparing them somehow.
Sorry for my English, and yeah, (obviously) don't use robots.txt as security measure of any kind...
In my ex-job we were developing e-commerce system, which was super-old|big|messy 0-test PHP trash. After two years of actively working on it, I still couldn't form a clear picture about the details of its subsystems in my head.
One day there is a call from a client, saying that he is missing many of his orders. The whole company is on its feet and we are searching for what went wrong. We are examining the server logs just to find out that someone is making thousands of requests to our admin section and tries to linearly increment order IDs in the URL. Definitely some kind of attack.. Our servers are managed by different company so we are opening a ticket to blacklist that IP. Quick search told me that the requests are coming from the AWS servers, and the IP leads me to an issue on GitHub for some nginx "bad bots" blocking plugin, saying that this thing is called Maui bot and we are not the first one experiencing it. Nice. Anyway, this thing is still deleting our data and we can't even turn off the servers because of SLAs and how the system was architected. So we are trying to find out how is it even possible, that unauthorized request can delete our data. We are examining our auth module, but everything looks right. If you are not logged in and visit the order detail (for example) you are correctly redirected to login screen. So how? We are reading the documentation of the web framework that the application is using. There is it. $this->redirect('login');. According to the documentation it was missing return before that statement. So without the return, everything after that point was still executed. And "everything" in our case, was the action from the URL. No one ever noticed, because there were no tests, and when you tried it in the browser, you were "correctly" presented with the login screen. Unfortunately, with side-effect.. Guy that wrote that line did it 5-6 years before this incident, and was out of the company for many years even before I joined. I don't blame him..
Fix. Push. Deploy. No more deleted orders.
POST MORTEM:
The Maui bot went straight to the disallowed /admin paths in robots.txt and tried to increment numbers (IDs) in paths.
I remember, that because Maui bot actions were (to the system) indistinguishable from the normal user actions, someone had to manually fix the orders in the database just by using server logs and comparing them somehow.
Sorry for my English, and yeah, (obviously) don't use robots.txt as security measure of any kind...