Thank you for mentioning Data Diff! Founder of Datafold here.
We built Data Diff to solve a variety of problems that we encountered as data engineers: (A) Testing SQL code changes by diffing the output of production/dev versions of SQL query. (B) Validating that data is consistent when replicating data between databases.
Data Diff has two algorithms implemented for diffing in the same database and across databases.
The former is based on JOIN, and the latter utilizes checksumming with binary search, which has minimal network IO and database workload overhead.
As a data engineer, validating your SQL code as you develop pipelines is hard. It requires running the code to materialize a "development" dataset, then auditing the resulting dataset to ensure your code produces the correct output. That tedious process usually involves writing lots of ad-hoc SQL scripts.
We built an integration with a popular framework for orchestrating SQL pipelines – dbt – that enables developers quickly profile their development datasets and diff them against production to audit their work as they code. It's free for individual developers. (We make money by selling CI/CD validation to teams)
Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code impacts data produced throughout the entire data pipeline/DAG. Datafold is used by data teams at Patreon, Thumbtack, Substack, Angellist, among others, and raised $22M from YC, NEA & Amplify Partners. Our founding story and Launch HN: [https://news.ycombinator.com/item?id=24071955]
Roles:
Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code impacts data produced throughout the entire data pipeline/DAG. Datafold is used by data teams at Patreon, Thumbtack, Substack, Angellist, among others, and raised $22M from YC, NEA & Amplify Partners. Our founding story and Launch HN: [https://news.ycombinator.com/item?id=24071955]
Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source code impacts data produced throughout the entire data pipeline/DAG. Datafold is used by data teams at Patreon, Thumbtack, Substack, Angellist, among others, and raised $22M from YC, NEA & Amplify Partners. Our founding story and Launch HN: [https://news.ycombinator.com/item?id=24071955]
Data Diff has two algorithms implemented for diffing in the same database and across databases. The former is based on JOIN, and the latter utilizes checksumming with binary search, which has minimal network IO and database workload overhead.