I've been curious: what does it take to find a job in software that isn't just gluing things together? Where do you have to go to write interesting algorithms at work? In our golden age of open-source, does that stuff only ever happen at FAANG and research labs?
Those jobs exist, but are few and far between, since cool ideas tend to be packaged up into reusable libraries, and don't need to be redeveloped, while everyone else uses these reusable libraries differently, so you have a constant need for "sanitation engineers". Tensorflow or OpenCV or UnrealEngine get written once, then everyone uses them to build whatever they're building. You write some cool glue around these, that's the interesting code, then spend most of your time wiring everything together into a larger package that actually does something useful.
Once you've wired everything together, now you need to track version, build artifacts, manage release processes, test them and qualify them, etc, so you do a lot of QA and DevOps work.
QA, DevOps and sanitation engineering is so product specific that it can't be packaged up, it's a craftsmanship position, and that's why all of us spend most of our time doing this kind of work.
Yeah, that's what I figured. It's a bit depressing when you think about it. I guess for most of us that itch just has to be scratched by pleasure-programming.
I don't know how long you've been doing this, but I've been continuously at it for almost 30 years now, and no matter where, there are challenging problems to solve, particularly outside the "web app with a backend" family of products. Doing optical recognition of malformed chicken patties as they whiz by at 60mph on a conveyor belt sounds mundane, but it's a lot more fun than writing REST API's to a SQL backend.
In your career, you will go through new and excited, then sort of bored, then you will find a niche that's both interesting to you, and that you're good at - you become an expert in something. At that point, you do as much of that as you can, and the sanitation engineering doesn't seem so bad when its directed at something interesting to you, especially when you work with colleagues who are better than you at something and challenge you professionally.
I look at this the other way around: if you’re only writing CRUD applications and relatively simple user interfaces for them, you tend to end up doing join-the-dots programming, but for almost any other field I’ve ever worked in, things get more interesting.
For example, we have a client at the moment who makes a type of device with a lot of user-configurable behaviour. An embedded web server allows access to its UI from a browser, and we were originally brought in to build that web UI for them. On the face of it, this is a substantial but straightforward SPA development, just one where the back end happens to talk to APIs that communicate with various physical components in the device rather than a traditional database.
However, it turns out that the way users view that device and how they want it to behave is very different to the way you have to program the various physical components to make useful things happen. That means even in this superficially simple project, we have some interesting algorithms in the front-end to present application-specific visualisations of the current state of the system, and we have a much more involved algorithm sitting behind that UI that converts the user’s tidy, intuitive view of the world into the very untidy and often counter-intuitive data formats required internally to program the hardware components.
To make this comment slightly more topical, I’ll also mention that the behind-the-scenes algorithm is essentially a form of compiler, taking a serialised version of the user’s world as input and running through a pipeline that systematically converts the data into the required internal formats. The first generation was written in Python, and has proven to be reasonably successful, but we are always a bit nervous about maintaining it just because of the number of edge cases and interactions inherent in the world we’re operating in. For the second generation, we made a big decision to go with Haskell instead, and for this sort of work, there were very welcome benefits including greater expressive power when writing data-transforming code and a strong type system that prevents mistakes like accidentally forwarding data from one pipeline step to the next without applying an appropriate transformation.
I agree with Beltiras’s point in the GP comment, and I probably wouldn’t choose Haskell to implement the kinds of software mentioned in that comment for much the same reasons. However, it definitely has real value in the sort of situation I described above, where we have both integrations with other systems but also substantial data crunching to do.
Everywhere! 3rd gen factory robotics, and manipulator control. Movie cg renderers and asset transformation pipelines (heck, get busy and finish inkscape!) Ehr systems. ERP systems. Risk management systems. Decision support systems. Trading systems. Field service support systems. Scheduling, packing, routing systems.
Now everything has an ui, and data innies and outies, but gosh! That's just connectors to the diamond!
