This has been my experience too - not in ML (i have zero experience there) but in software architecture more generally. The idea of locality of change, of encapsulation, of interfaces as a way to reduce the amount of stuff you need to know in order to change the system. It is a pretty effective strategy for keeping the "cost to change" a software product low.
There have been downfalls for me though - interestingly never the obvious problem. To me the obvious problem is performance, there's usually an overhead in a plugin system of some sort but it hasn't been an issue for me yet, maybe i'm just lucky.
An annoying real issue has been adding in dependencies between plugins, that has always introduced horrible issues later on. I think based on my current life experiences i'd even choose to duplicate functionality over introducing dependencies now - despite the fact that idea is basically anethema to conventional wisdom.
>The idea of locality of change, of encapsulation, of interfaces as a way to reduce the amount of stuff you need to know in order to change the system. It is a pretty effective strategy for keeping the "cost to change" a software product low.
Yes, not only keeping the cost of change low, but unlocking potential in a way that seems prescient but is just common sense. I like to think about this in terms of "unit of thought", "protocols, interfaces, specs", "impedance matching" to maximize power transfer, unknown future consumers.
Doing that adaptation upstream prevents people writing adapters downstream. It may be adding a "REST" API so that consumers you may ignore can work with that. It may be using Cloudevents spec for your events to make it easier for people to work with that. It may be making sure the output of your system is a Docker image to make it easier to use that elsewhere for a user, whether that may be a human or something else.
Systems that produce known/standardized/understood building blocks unlock a lot of potential and avoid downstream work.
There have been downfalls for me though - interestingly never the obvious problem. To me the obvious problem is performance, there's usually an overhead in a plugin system of some sort but it hasn't been an issue for me yet, maybe i'm just lucky.
An annoying real issue has been adding in dependencies between plugins, that has always introduced horrible issues later on. I think based on my current life experiences i'd even choose to duplicate functionality over introducing dependencies now - despite the fact that idea is basically anethema to conventional wisdom.