Show HN: Census – Export for data warehouses
7 comments
Brad from Census (https://www.getcensus.com) here - I wanted to add some technical detail for the HN crowd.
On its face, a Census workflow is a simple "program" that our users author in a point-and-click manner – read some data from System A and broadcast it to Systems B, C, and D. But to "compile" that program, Census has to determine:
There's a lot of depth to this (and we haven't "solved" it by any means) - happy to answer questions here or at [email protected] if you have them!
On its face, a Census workflow is a simple "program" that our users author in a point-and-click manner – read some data from System A and broadcast it to Systems B, C, and D. But to "compile" that program, Census has to determine:
- Which APIs to use (bulk, streaming, etc)
- The schema of each destination (which can change out from underneath us at run time!)
- The semantics of reads and writes from each system (atomicity, isolation, how roll back if a write fails)
- How to map data with high-fidelity across strongly-, weakly-, and dynamically-typed data stores
There's a lot of depth to this (and we haven't "solved" it by any means) - happy to answer questions here or at [email protected] if you have them!
Excited to follow your progress! I view this problem as the one of the biggest gaps in today's "Cloud Data Ecosystem". Tools like Stitch and Fivetran make it super easy to extract data from source systems; next-gen cloud data platforms like Snowflake make storing, transforming, and querying that data a breeze (especially with the help of tools like dbt and dataform); and there are a ton of powerful and easy to use BI tools for visualizing and digesting that data. But the minute you need to send that data to other systems, it's back to painful, failure-prone, and mind-numbing scripts.
So nice to hear! That's precisely how we felt :-)
Nice, congrats on launching! I'm excited to try it out.
Curious - what if there's a transformation that happens with data from the data warehouse but can't be performed with SQL (such as a Python script)? Is there a way to send that data back into the integrations that you support? Or would be it best to push back into the data warehouse and use Census from there?
Aka, can you only transform with Census using SQL? Or other languages as well?
Curious - what if there's a transformation that happens with data from the data warehouse but can't be performed with SQL (such as a Python script)? Is there a way to send that data back into the integrations that you support? Or would be it best to push back into the data warehouse and use Census from there?
Aka, can you only transform with Census using SQL? Or other languages as well?
Thanks! We've been talking about that on the team. I kind of want to expose any table automatically as a pandas dataframe so that you could transform it with python on the fly and we'd take care of the execution (and then sync the result back to the apps). Do you have something specific in mind?
When can we create an account with email/password instead of Google?
Very soon! Send me an email at [email protected] and I'll get you going asap!
We’ve built a product that connects directly to your data warehouse and syncs into apps like Salesforce, Customer.io and even Google Sheets. In fact, your business teams won’t even need to rely on engineering to manage all these pipelines.
The tech stack for analyzing customer data in 2020 looks pretty great. You can load almost any data into an auto-scaling data warehouse (Snowflake, BigQuery) with easy point & click tools like Fivetran. You can build SQL models with dbt and create visual reports in Metabase. But you can’t easily push insights back into the marketing/sales/support apps.
You can’t solve this with direct app integrations or “event routers” like Zapier and you definitely shouldn’t over-engineer a solution with Spark/Kafka/Airflow. We designed Census to make your data warehouse a single source of truth for modeling and transformation before publishing data back into your SaaS tools quickly and reliably.
We’re proud of what we’ve built so far and there’s a lot more work to deliver on our dream of saving us all from generating & uploading yet another CSV file so we can spend more time actually building our products (or reading HN). You can check it out at https://www.getcensus.com.
Since this is HN, we’d love to hear everyone’s war stories on building internal ETL solutions!