pradeepchhetri

9 mesi fadiscuss

Very cool project indeed!

I tried to check the kind of flights they flew in the world's dangerous airport (Lukla, Nepal)[0] and found they use ATR-72 series.

[0] https://adsb.exposed/?dataset=Planes&zoom=12&lat=27.7136&lng...

pradeepchhetri

anno scorsodiscuss

> Debian package is of poor quality

Can you elaborate more please? I would love if you can say what all can be improved to make debian package up to standards.

pradeepchhetri

anno scorsodiscuss

clickhouse-local is similar to duckdb, you don't need a clickhouse-server running in order to use clickhouse-local. You just need to download the clickhouse binary and start using it.

  clickhouse local
  ClickHouse local version 25.4.1.1143 (official build).

  :)

There are few benefits of using clickhouse-local since ClickHouse can just do lot more than DuckDB. One such example is handling compressed files. ClickHouse can handle compressed files with formats ranging from zstd, lz4, snappy, gz, xz, bz2, zip, tar, 7zip.

  clickhouse local --query "SELECT count() FROM file('top-1m-2018-01-10.csv.zip :: *.csv')"
  1000000

Also clickhouse-local is much more efficient in handling big csv files[0]

[0]: https://www.vantage.sh/blog/clickhouse-local-vs-duckdb

pradeepchhetri

anno scorsodiscuss

I prefer to use clickhouse-local for all my CSV needs as I don't need to learn a new language (or cli flags) and can just leverage SQL.

    clickhouse local --file medias.csv --query "SELECT edito, count() AS count from table group by all order by count FORMAT PrettyCompact"

   ┌─edito──────┬─count─┐
   │ agence     │     1 │
   │ agrégateur │    10 │
   │ plateforme │    14 │
   │ individu   │    30 │
   │ media      │   423 │
   └────────────┴───────┘

With clickhouse-local, I can do lot more as I can leverage full power of clickhouse.

pradeepchhetri

anno scorsodiscuss

glad you are using CH :)

pradeepchhetri

anno scorsodiscuss

I am looking forward to learn about such upcoming features in the community call https://clickhouse.com/company/events/v25-1-community-releas...

pradeepchhetri

anno scorsodiscuss

ClickHouse has a solid Iceberg integration. It has an Iceberg table function[0] and Iceberg table engine[1] for interacting with Iceberg data stored in s3, gcs, azure, hadoop etc.

[0] https://clickhouse.com/docs/en/sql-reference/table-functions...

[1] https://clickhouse.com/docs/en/engines/table-engines/integra...

pradeepchhetri

2 anni fadiscuss

To my knowledge, both Postgres and MySQL has their own strengths and weaknesses. Example: mvcc implementation, data replication, connection pooling and difficulty of upgrades were the major weaknesses of Postgres which are much improved over time. Similarly mysql query optimizer is consider lesser developed than that of Postgres's.

Overall I think Postgres adoption and integrations and thus community is much more wider than MySQL which gives it major advantage over MySQL. Also looking at the number of database-as-a-service companies of Postgres vs those of MySQL we can immediately acknowledges that Postgres is much widely adopted.

pradeepchhetri

2 anni fadiscuss

> If I had to only pick two databases to deal with, I’d be quite happy with just Postgres and ClickHouse - the former for OLTP, the latter for OLAP.

As the author mentioned, I completely agree with this statement. In fact, many companies like Cloudflare are built with exactly this approach and it has scaled them pretty well without the need of any third database.

> Another reason I suggest checking out ClickHouse is that it is a joy to operate - deployment, scaling, backups and so on are well documented - even down to setting the right CPU governor is covered.

Another point mentioned by author which is worth highlighting is the ease of deployment. Most distributed databases aren't so easy to run at scale, ClickHouse is much much easier and it has become even more easier with efficient storage-compute separation.

pradeepchhetri

2 anni fadiscuss

Streaming queries is coming in ClickHouse too: https://github.com/ClickHouse/ClickHouse/pull/63312

pradeepchhetri

2 anni fadiscuss

There was a great talk[0] recently from Laravel core team member about ClickHouse, Probably you will enjoy watching it.

[0] https://www.youtube.com/watch?v=_jjvaFWWKqg

pradeepchhetri

2 anni fadiscuss

You should look at ClickHouse which has good PHP client

[0] https://github.com/smi2/phpClickHouse

pradeepchhetri

2 anni fadiscuss

Since analytics data is generally write-heavy, I would recommend to use ClickHouse. You can use async-insert[0] feature of ClickHouse, thus you don't need to worry about batching events on your side. If you are looking for an embedded solution, you can use chDB which is built on top of ClickHouse.

[0] https://clickhouse.com/blog/asynchronous-data-inserts-in-cli...

pradeepchhetri

2 anni fadiscuss

You can leverage ClickHouse to process your music data. ClickHouse supports both TSV[0] and XML[1] data formats.

[0] https://clickhouse.com/docs/en/interfaces/formats#tabseparat...

[1] https://clickhouse.com/docs/en/interfaces/formats#xml

pradeepchhetri

2 anni fadiscuss

This syntax looks a lot like PRQL. ClickHouse supports writing queries in PRQL dialect. Moreover, ClickHouse also supports Kusto dialect too.

https://clickhouse.com/docs/en/guides/developer/alternative-...

pradeepchhetri

2 anni fadiscuss

I would recommend using ClickHouse which provides very efficient compression and thus reduces the size of data drastically.

Apart from that, it provides various other feature:

- Dynamic datatype [0] which are very useful for semi-structured fields which generally logs contains very often.

- You can configure column's & table's TTL [1] which provides efficient way to configure retention.

At my previous job (Cloudflare), we migrated from Elasticsearch to ClickHouse and saved nearly 10x reduction in data size and got 5x perf improvement. You can read more about it [2] and watch the recording here [3]

Recently, ClickHouse engineers published a wondering detailed blog about their logging pipeline [4]

[0] https://clickhouse.com/docs/en/sql-reference/data-types/dyna...

[1] https://clickhouse.com/docs/en/engines/table-engines/mergetr...

[2] https://blog.cloudflare.com/log-analytics-using-clickhouse

[3] https://vimeo.com/730379928

[4] https://clickhouse.com/blog/building-a-logging-platform-with...

pradeepchhetri

2 anni fadiscuss

I would recommend to look at ClickHouse[0] which is Apache-License and is completely open-source. You can self-host it and it is one of the fastest growing OLAP database. It has good integrations with BI tools and it can be used for wide varieties of use cases. It is widely adopted[1]. I would recommend you to give it a try and validate it for your use-case.

I work for ClickHouse and available for any queries or help you need.

[0]: https://clickhouse.com/ [1]: https://clickhouse.com/docs/en/about-us/adopters

pradeepchhetri

2 anni fadiscuss

If you are interested in reading internals of using Parquet with ClickHouse, do read following articles:

- https://clickhouse.com/blog/apache-parquet-clickhouse-local-...

- https://clickhouse.com/blog/apache-parquet-clickhouse-local-...

pradeepchhetri

2 anni fadiscuss

Reading through this blog, to me it seems Parquet is lot like ClickHouse native data format.

Best part of ClickHouse native data format is I can use the same ClickHouse queries and can run in local or remote server/cluster and let ClickHouse to decide the available resources in the most performant way.

ClickHouse has a native and the fastest integration with Parquet so i can:

- Query local/s3 parquet data from command line using clickhouse-local.

- Query large amount of local/s3 data programmatically by offloading it to clickhouse server/cluster which can do processing in distributed fashion.

pradeepchhetri

2 anni fadiscuss

> Postgres is still way ahead of ClickHouse in terms of operational simplicity

Having served as both ClickHouse and Postgres SRE, I don't agree with this statement.

- Minimal downtime major version upgrades in PostgreSQL is very challenging.

- glibc version upgrade breaks postgres indices. This basically prevents from upgrading linux OS.

And there are other things which makes postgres operationally difficult.

Any database with primary-replica architecture is operationally difficult IMO.

pradeepchhetri

Submissions

Kim Kardashian is welcome to join my Research Team on 3I/ATLAS

The Scientific Revolution of Interstellar Objects

ClickHouse's JSON datatype gets better

A Postgres Mystery: The SIGTERMs do nothing!

Ursa: ClickHouse Research Fork

One Billion JSON Challenge

comments