> Data engineering can be unsatisfying if you thrive on writing reliable systems
This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.
Unless you get to work at the place that values your test driven perspective. Either because they are doing it or desperately want to (where want = they are already investing heavily in it)
We build pipelines in staging schemas, and we only promote them to prod once they pass tests. This is pretty analogous to SWE (it's where we got the idea) and is a pretty crucial part of us meeting the requirement where we don't put inaccurate data in front of customers. It's--of course--only as good as the tests, but that's every reliable system.
This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.