avogar · HackerTrans

clickhouse-local is a really good tool for analyzing local data, as it supports a wide range of data formats and it's really fast. It can save you a lot of time, because there is no need to install a batch of packages to use it or setup a database server. Just download a binary/install one package and enjoy. It has developed rapidly in recent years and continues to improve. https://clickhouse.com/docs/en/operations/utilities/clickhou...

avogar

4 anni fadiscuss

SPyQL looks very promising, great work!

I can't help but mention clickhouse-local tool: https://clickhouse.com/docs/en/operations/utilities/clickhou...

clickhouse-local is a single binary that enables you to perform fast data processing using SQL - effectively database features without a database. This tool supports the full breadth of ClickHouse functions, many popular file formats and recently added automatic schema inference. You can query not only local files, but also remote files (from S3/HDFS/static files accessed by URL). Moreover, clickhouse-local tool has interactive mode where you can create tables, play with data and do almost everything that you can do wih ordinary database. And let's not forget, this tool is written in C++, so it's incredibly fast.

Disclaimer: Work at ClickHouse

avogar

4 anni fadiscuss

Using ClickHouse you can also process local files in one line using clickhouse-local command tool. And it will look a lot easier:

clickhouse local -q "SELECT passenger_count, COUNT(*), AVG(total_amount) FROM file(taxi.csv, 'CSVWithNames') GROUP BY passenger_count"

And ClickHouse supports a lot of different file formats both for import and export (you can see all of them here https://clickhouse.com/docs/en/interfaces/formats/).

There is an example of using clickhouse-local with taxi dataset mentioned in the post: https://colab.research.google.com/drive/1tiOUCjTnwUIFRxovpRX...

comments