Performance optimizations for storing web server access logs in ClickHouse(clickhouse.com)

2 pointsby krizhanovskyvor 9 Monaten1 comments

Performance optimizations for storing web server access logs in ClickHouse

https://clickhouse.com/blog/log-compression-170x

1 comments

It's useful to store a web server access logs in an analytics database, e.g. to fight against bot attacks. We store structured access logs in Clickhouse, which is already good, but compression and data ordering from the post may improve performance even more - we'll try this.

The thing is that a web server, especially under DDoS, may produce much more records than Clickhouse can ingest. But there is good news: for Nginx, if you build a fast pipeline to feed access logs to Clickhouse, you can increase performance, I'd say up to x2, thanks to faster access logging.