Most open-source L7 DDoS mitigation and bot-protection approaches rely on challenges (e.g., CAPTCHA or JavaScript proof-of-work) or static rules based on the User-Agent, Referer, or client geolocation. These techniques are increasingly ineffective, as they are easily bypassed by modern open-source impersonation libraries and paid cloud proxy networks.
We explore a different approach: classifying HTTP client requests in near real time using ClickHouse as the primary analytics backend.
We collect access logs directly from Tempesta FW (https://github.com/tempesta-tech/tempesta), a high-performance open-source hybrid of an HTTP reverse proxy and a firewall. Tempesta FW implements zero-copy per-CPU log shipping into ClickHouse, so the dataset growth rate is limited only by ClickHouse bulk ingestion performance - which is very high.
* periodically executes analytic queries to detect spikes in traffic (requests or bytes per second), response delays, surges in HTTP error codes, and other anomalies;
* upon detecting a spike, classifies the clients and validates the current model;
* if the model is validated, automatically blocks malicious clients by IP, TLS fingerprints, or HTTP fingerprints.
To simplify and accelerate classification — whether automatic or manual — we introduced a new TLS fingerprinting method.
WebShield is a small and simple daemon, yet it is effective against multi-thousand-IP botnets.
It's useful to store a web server access logs in an analytics database, e.g. to fight against bot attacks. We store structured access logs in Clickhouse, which is already good, but compression and data ordering from the post may improve performance even more - we'll try this.
The thing is that a web server, especially under DDoS, may produce much more records than Clickhouse can ingest. But there is good news: for Nginx, if you build a fast pipeline to feed access logs to Clickhouse, you can increase performance, I'd say up to x2, thanks to faster access logging.
This particular project, WebShield, is simple and it didn't take too long to develop. Basically, with this project we're trying to figure out what can be built having fingerprints and traffic characteristics in an analytic database. It's seems easy to make PoCs with these features.
For now, if this tool can stop some dummy bots, we'll be happy. We definitely need more development and more sophisticated algorithms to fight against some paid scrapping proxies.
It's more or less simple to classify DDoS bots because they have clear impact - the system performance degrades. For some bots we also can introduce the target, for the bots and the protection system, e.g. the booked slots for a visa appointments. For some scrappers this is harder.
Another opportunity is to dynamically generate classification features and verify resulting models, build web page transition graphs and so on.
This is a good point about possible blocking of ~50% of the Internet. For DDoS we _mitigate_ an attack, not _block_ it, so probably for bots we should do the same - just rate limit them instead of full blocking.
Technically, we can implement verification of client side certificates, but, yes, the main problem of adoption on the client side.
In our approach we do our best to not to affect user experience. E.g. consider an example of a company website with a blog. The company does it's best to engage more audience to their blog, products whatever. I guess quite a part of the audience will be lost due to requirement of authentication on website, which they see first time.
However, for returning, and especially regular, clients I think that is a really simple and good solution.
We explore a different approach: classifying HTTP client requests in near real time using ClickHouse as the primary analytics backend.
We collect access logs directly from Tempesta FW (https://github.com/tempesta-tech/tempesta), a high-performance open-source hybrid of an HTTP reverse proxy and a firewall. Tempesta FW implements zero-copy per-CPU log shipping into ClickHouse, so the dataset growth rate is limited only by ClickHouse bulk ingestion performance - which is very high.
WebShield (https://github.com/tempesta-tech/webshield/), a small open-source Python daemon:
* periodically executes analytic queries to detect spikes in traffic (requests or bytes per second), response delays, surges in HTTP error codes, and other anomalies;
* upon detecting a spike, classifies the clients and validates the current model;
* if the model is validated, automatically blocks malicious clients by IP, TLS fingerprints, or HTTP fingerprints.
To simplify and accelerate classification — whether automatic or manual — we introduced a new TLS fingerprinting method.
WebShield is a small and simple daemon, yet it is effective against multi-thousand-IP botnets.