hey bserial, I'm part of the team working on Dagster.
While there are many things we're working on, there are 3 goals that got me excited about working on this system:
1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.
2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.
3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.
Thumbtack | Software Engineer, SRE, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Have you considered just applying through the jobs pages of the larger tech companies? Apple, Google, Facebook, LinkedIn are all on the peninsula and more accessible than startups in SF.
We're also doing this at Thumbtack. We run all of our Spark jobs in job-scoped Cloud Dataproc clusters. We wrote a custom Airflow operator which launches a cluster, schedules a job on that cluster, and shuts down the cluster upon job completion. Since Google can bring up Spark clusters in < 90s and bills minutely, this works really well for us, simplifying our infrastructure and eliminating resource contention issues.
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
I'm constantly surprised by how much work Amazon expects its customers to do themselves. The work that Segment has done here should be a service provided by AWS directly, continuously updating cost data in a Redshift database without any customer work required.
We just migrated our data infrastructure to GCP. One of the big motivators was experiences like this with AWS. We've got near-realtime GCP cost dashboards in BigQuery, and the only meaningful work on our end to make that happen was writing the SQL queries.
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Thumbtack | San Francisco, CA | Full-Time | Onsite
Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.
We are a friendly, ambitious team of 80 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.
We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.
We are a friendly, ambitious team of ~70 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.
We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.
We are a friendly, ambitious team of ~60 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.
We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
While there are many things we're working on, there are 3 goals that got me excited about working on this system:
1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.
2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.
3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.
Hope that makes sense!