Snowflake has skyrocketed in popularity over the past 5 years and firmly planted itself at the center of many companies' data stacks. Snowflake came into existence in 2012 with a unique architecture, described in their seminal white paper as "the elastic data warehouse". Rather than have compute and storage coupled on the same machine like their competitors did, they proposed a new design that took advantage of the near-infinite resources available in cloud computing platforms like Amazon Web Services (AWS). In this post, we'll dive into the three layers of Snowflake's data warehouse architecture: cloud services, compute and storage.
Outside of being open source, how does ClickHouse differ from Snowflake/BigQuery? In what scenarios would I choose ClickHouse over those existing solutions?
Completely agree a DS shouldn't be responsible for all those things you mentioned (all the infrastructure, networking, security, etc.). In today's age, "data pipelines" can look like plugging into tools & frameworks that abstract away a lot of this complexity. For example - fivetran for extraction & dbt for data modelling. So once those engineering/devops heavy things you listed are abstracted away, I think it's fair for a DS to be responsible for building the data transformation logic in the pipeline. Of course, not all companies will have such tooling available, in which case it's not fair to place those expectations on a DS.