This almost feels like a case of Betteridge's law.
Every single job I've had in the software space in the last 6 years (and I've had a bunch) has made pretty extensive use of SQL. Obviously this is a sample size of one, but me being good with SQL wasn't especially impressive to any of my coworkers at any of my jobs.
Ironically a lot of my data science peers only use SQL to extract data, then they like to bring it into excel or python to analyze. I’m pretty convinced that you should use SQL right up until you can’t. If you need to do text analysis or compute complex statistical measures you should just use SQL for extraction (to max effect) and then use the appropriate tools. Unfortunately a lot of my peers only use SQL for about 10% of the analysis that could be done in SQL which makes peer review and team work extremely difficult. I really wish data science had just been a sub category of software engineer, where programming skills were required full stop. In data science sometimes people can have terrible programming skills but good math skills which in my opinion is like an airplane with a great engine but no wings.
My first engineering job was doing ColdFusion in 2012. Even at the time ColdFusion was considered dated, and it was extremely slow.
When I learned SQL at that job, and when I learned you can squeeze a lot of computation out of SQL, it was huge for me. I started doing as much pre-processing with SQL as I could, just because it felt like it was giving me a lot of free performance upgrades.
Yeah I have yet to really open the hood and learn more about query plans (which enable the computation ability you mention). I’ve been running into OOM errors all week and luckily the system gives me advice on where to find the cause. I’d like to learn more about how to determine what parameters to set (like partitioned vs broadcast joins) because even after all this time using SQL it still feels like an alien technology to me especially when my SQL is syntactically correct but causing worker explosions left and right lol. There’s no better feeling than refactoring the code and seeing a job that used to time out run and you see how much data is being processed efficiently like that.
Honestly even very crappily written SQL will probably still be faster than ColdFusion, particularly if you're using stored procedures. I'm not going to claim I'm the world expert of anything, and the SQL I write was never hyper-optimized, but ColdFusion was just ridiculously slow.
Every single job I've had in the software space in the last 6 years (and I've had a bunch) has made pretty extensive use of SQL. Obviously this is a sample size of one, but me being good with SQL wasn't especially impressive to any of my coworkers at any of my jobs.