Yes, MySQL-DuckDB columned read only node will continuously get data from transactional workload by binlog.
Then people will not need to maintain tools like kafka/debezium to sync between two node.
I’m quite certain that if DuckDB had been open-sourced and reached stability around 2020, TiDB would have definitely chosen DuckDB instead of ClickHouse.
Actually, that’s not the case. I also support PostgreSQL products in my professional work. However, specifically regarding this issue—as I mentioned in my article—it is simply easier to integrate DuckDB by leveraging MySQL's binlog and its pluggable storage engine architecture.
Here is the professional English translation of your analysis, optimized for a technical audience or a blog post:
Why I Believe MySQL is More Suited than PostgreSQL for DuckDB Integration
Currently, there are three mainstream solutions in the ecosystem: pg_duckdb, pg_mooncake, and pg_lake. However, they face several critical hurdles. First, PostgreSQL's logical replication is not mature enough—falling far behind the robustness of its physical replication—making it difficult to reliably connect a PG primary node to a DuckDB read-only replica via logical streams.
Furthermore, PostgreSQL lacks a truly mature pluggable storage engine architecture. While it provides the Table Access Method as an interface, it does not offer standardized support for primary-replica replication or Crash Recovery at the interface level. This makes it challenging to guarantee data consistency in many production scenarios.
MySQL, however, solves these issues elegantly:
Native Pluggable Architecture: MySQL was born with a pluggable storage engine design. Historically, MySQL pivoted from MyISAM to InnoDB as the default engine specifically to leverage InnoDB's row-level MVCC. While previous columnar attempts like InfoBright existed, they didn't reach mass adoption. Adding DuckDB as a native columnar engine in MySQL is a natural progression. It eliminates the need for "workaround" architectures seen in PostgreSQL, where data must first be written to a row-store before being converted into a columnar format.
The Power of the Binlog Ecosystem: MySQL’s "dual-log" mechanism (Binlog and Redo Log) is a double-edged sword; while it impacts raw write performance, the Binlog provides unparalleled support for the broader data ecosystem. By providing a clean stream of data changes, it facilitates seamless replication to downstream systems. This is precisely why OLAP solutions like ClickHouse, StarRocks, and SelectDB have flourished within the MySQL ecosystem.
Seamless HTAP Integration: When using DuckDB as a MySQL storage engine, the Binlog ecosystem remains fully compatible and intact. This allows the system to function as a data warehouse node that can still "egress" its own Binlog. In an HTAP (Hybrid Transactional/Analytical Processing) scenario, a primary MySQL node using InnoDB can stream Binlog directly to a downstream MySQL node using the DuckDB engine, achieving a perfectly compatible and fluid data pipeline.
On this page, we introduce how to implement a read-only Columnar Store (DuckDB) node leveraging the MySQL binlog mechanism. https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/du... In this implementation, we have performed extensive optimizations for binlog batch transmission, write operations, and more.
I think the most interesting part is PolarFS taking full advantage of the emerging techniques like RDMA, NVMe, and SPDK. And the Parallel raft consensus algorithm
when you deal with multi threads, not only epoll will cause some problems, but also global variable, memory, etc. global variable solved by mutex, but epoll solved by avoid using it to epoll_wait fds in multi threads.
For a system engineer, it's hard to draw visual csv file by java script.
markdown is the most simple and convenient tool, it's a good idea to combine markdown with visualization.
I think the different between redis and webserver like nginx is that all the operations in redis is almost the same, it is about less than 1ms. However the request to nginx fall in a widely range, some request need 10ms, while some request need 10s. Since nginx need do some file operations.
So the single model work well for redis, but it doesn't work well for nginx, since if there is a request in nginx that is blocking for about 10s, people can't tolerate this situation.