I'm not a data engineer but was tasked with building an ETL pipeline for a large company. It's all just Step Functions, looping through file streams in a Lambda, transforming, then putting the data into Snowflake for the analytics team to view. My pipeline processes billions of rows from many different sources. Each pipeline runs daily on a cron job (maybe that's the key differentiator, we don't need live streaming data, it's a lot of point of sale data).
Whenever I hear actual data engineers talk about pipelines there are always a million different tools and complicated sounding processes. What's am I missing?
Hi, data engineer here (of sorts). I'll take a crack.
> It's all just Step Functions, looping through file streams in a Lambda, transforming, then putting the data into Snowflake for the analytics team to view.
So, you're right. Data going from A -> B is the most important part of the job. Like, the pipeline you've got is like the 60-80% critical bit. Get data from systems A through Y and put it into system Z. Without it, there's no point talking about anything else.
But what happens when you need to rollback all of the data from system F for the last week? What if you need to rollback system C for the last 2 days as well? How do you keep track of whether data from systems G, H and I is still "good" quality? How do you handle frequently changing source systems, which could do any one of drop, rename, add columns on a daily basis? Do you ignore those columns forever? Or do you siphon that data off to a special place for manually review? Do you process all previously seen data every single night? If you only process the newest subset of data, what do you do if you've noticed a bug in the pipeline code? Process everything again and spend a whole bunch of money every time we fix buggy pipeline code? Or do a "migration" on already processed data? How do you track which subsets were processed with which version of your pipeline? How did that run get triggered -- was it a glue crawler scheduled cron job, or did someone trigger it manually? Does any of this actually matter? Does anyone even look at the dashboards any more?
Getting the pipeline running is the first bit. With non-volatile/slow-changing source data/systems, you're fine with just the pipeline most of the time. Depending on the volume of the data, you could probably get away with just rerunning the pipeline over all previously seen data when there's been a bugfix and just eat the cost. You don't need anything more complicated than that.
But if the source data/systems are volatile [0], then keeping it working, keeping it accurate and keeping it cheap comes next. And the complexity of that depends on how complex X -> Y is.
A simple "back of a napkin" way to determine complexity is to sit down and ask some questions about the 6 V's: Volume; Velocity; Variety; Veracity; Value; Variability [1]. It sounds like [2] you have high volume, some velocity (nightly), some variety, low veracity, high value and low variability data. There's a lot of data coming from multiple source systems, and it's important data. But it doesn't sound like the changes that much and is all probably all in the same or similar formats [3] and good quality. So... a simple pipeline triggered nightly works in that case.
When things get more complicated, that's when the system moving data from A -> B gets more complicated.
---
Having said all that, there has been a lot of resume driven development in the industry. $LAST_JOB i worked at the previous team thought it would be a great idea to roll their own job orchestrator product [4]. from scratch. in django. not a library that django called, not some django API endpoints that forwarded requests to some Airflow backend. nope. a job orchestrator written from scratch in django.
Because "bare metal" apparently.
They were also obsessed with this idea of reproducibility, but what they actually wanted was idempotency... but that's a story for another reply.
---
[0]: even if each upstream is slow to change, the cumulative effect could be there is at least one upstream change breaking something every day, so the cumulative effect is that the upstreams together are volatile even if individually they are not
I'm not a data engineer but was tasked with building an ETL pipeline for a large company. It's all just Step Functions, looping through file streams in a Lambda, transforming, then putting the data into Snowflake for the analytics team to view. My pipeline processes billions of rows from many different sources. Each pipeline runs daily on a cron job (maybe that's the key differentiator, we don't need live streaming data, it's a lot of point of sale data).
Whenever I hear actual data engineers talk about pipelines there are always a million different tools and complicated sounding processes. What's am I missing?