hey bserial, I'm part of the team working on Dagster.
While there are many things we're working on, there are 3 goals that got me excited about working on this system:
1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.
2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.
3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.
Thumbtack | Software Engineer, SRE, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
- too many moving parts, especially given the track record of EMR
- might not even be available when your requirement is to keep the data in the same AWS region as the DynamoDB table, as only five regions support Data Pipeline
The best approach I've seen so far is to use DynamoDB Streams and an AWS Lambda function to create incremental backups in an versioned S3-bucket. dynamodb-replicator (https://github.com/mapbox/dynamodb-replicator) implements that together with some scripts to do management tasks like back filling an S3 bucket with data which is already in DynamoDB or joining incremental backups into a single file.
It's still pretty unpolished and definitely needs some love, but I think it's the right approach.
Have you considered just applying through the jobs pages of the larger tech companies? Apple, Google, Facebook, LinkedIn are all on the peninsula and more accessible than startups in SF.
We're also doing this at Thumbtack. We run all of our Spark jobs in job-scoped Cloud Dataproc clusters. We wrote a custom Airflow operator which launches a cluster, schedules a job on that cluster, and shuts down the cluster upon job completion. Since Google can bring up Spark clusters in < 90s and bills minutely, this works really well for us, simplifying our infrastructure and eliminating resource contention issues.
Awesome stuff, glad to see folks leveraging the possibilities! Perhaps as a follow-up you could write a guest blog on how this works for you! Feel free to ping me offline.
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
I'm constantly surprised by how much work Amazon expects its customers to do themselves. The work that Segment has done here should be a service provided by AWS directly, continuously updating cost data in a Redshift database without any customer work required.
We just migrated our data infrastructure to GCP. One of the big motivators was experiences like this with AWS. We've got near-realtime GCP cost dashboards in BigQuery, and the only meaningful work on our end to make that happen was writing the SQL queries.
Agreed. I don't know why Amazon (and Azure) make this so hard. I've done something pretty similar to what Segment did (except it supports normalizing stats from Azure and AWS), and 90% of the work is stuff you don't feel like you should have to be doing.
- The team is reviewing what to build for the next quarter.
- You have a long list of revenue-generating features
- At the end of the list there's one more item that'll help your customers spend LESS money on your product.
- You can only build so many things and you know that you, your department, and Amazon the company will get pats on the back in $$$ form if you focus on the revenue generating features
- Sure, it'd be really good to help longterm customers understand their costs better, but your biggest ones have the resources to build that infrastructure themselves anyways.
I think you're absolutely right, but what's hilarious to me is that they have private pricing for many of their services. For the traffic we're already doing, we just asked and they knocked our cloudfront bill down nearly 75%. We didn't have to change our usage at all. Granted, we serve a lot of traffic.
Second this. We just moved our sizable infrastructure to GCP for more or less the exact same reason. GCP makes all of this a breeze -- I don't understand why other providers can't do the same.
This way of looking at things has never been particularly convincing to me... Of all the work that's done, at any organisation, almost none of it can truly answer those questions affirmatively.
"That lightbulb is broken!"
"I was going to replace it, but I couldn't find a customer to pay for it, and nobody has threatened to change to a brighter competitor"
Have you ever founded a startup? Did you spend your time working on features users will pay for, or an awesome billing dashboard (assuming billing is not your core product of course?) If so, how successful were you?
It's about focus. Only so many hours in a day, so many dollars, so many developers. If AWS got a nicer billing daashboard but Lambda was delayed by 6 months, how would it have changed the place AWS has in the market?
Yes you could take this too far and never change a light bulb. But I would answer "I will be more productive if I have a nice light bulb, so it is worth the 5 minutes to change it"
Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE
Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.
We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.
We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!
While there are many things we're working on, there are 3 goals that got me excited about working on this system:
1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.
2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.
3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.
Hope that makes sense!