Are there any drawbacks of using a plugin centric approach? Typically there is loss of expressiveness in code, loss of performance or disconnect between core and plugin development.
> Are there any drawbacks of using a plugin centric approach?
Here are some:
1. You're perhaps more subject to Hyrum's law. If plugin devs can see it, they will use it. The general observation here is that it's harder to control the visible interfaces and implicit dependencies you export than the dependencies and interfaces you rely on. As one example, semantic versioning doesn't cater for this at all. Plus, most of the practice knowledge in software is on managing relied on dependencies.
2. Dog follows tail. It can happen that a plugin becomes so successful the overall system evolution slows down. The core system can upgrade, but adoption/deployment can be constrained when a particularly valuable plugin doesn't move up to the latest and the customer base sees more value in the plugin than its core platform. This can compound poorly over time, and in extreme cases the desirable plugin can become its own platform/system (something I think business savvy tech leaders are increasingly aware, and wary, of).
3. Operational complexity. It can be harder to run and maintain a plugin based system than a closed one. 2 is a consideration here, but so are other concerns, such as security and resource isolation. Strategies vary, but who pays this cost on a relative basis is one of the more (and perhaps the most) interesting aspect of working on or using plugin systems. As one example of this, think about allocating responsibility for bugs.
4. R&D complexity. It may take more time to design and build a plugin system than a closed one. Incrementally evolving to a plugin system can be difficult if you didn't start there to begin to with. So you usually need a clear opening motivation to delay reward (or avoid over engineering) to invest in a system design where functionality can be extended by non-core developers.
Not that I could see at this point. It is not a panacea, but for us, it was a way to contain scope at different levels.
For exaple, we have the platform and it has icons on the sidebar for Notebook, Object Storage, etc.
Every single one of these is a separate application and a separate repository. These applications are independent in how they deal with business logic, so there's no loss of expressiveness. They just must present certain "receptors" or interface if they want to be plugged into the system. The "interface" is a big word, and someone can produce a valid minimal plugin (that does nothing except be loaded) in two minutes.
This allows us to contain details of a plugin to the plugin itself, and not having it leak to other parts of the product. If we want to activate/de-activate the plugin, it takes less than 10 seconds manually.
Now, sometimes a plugin depends on another plugin. But they make their requests to that plugin, and fall-back to something else in case that plugin is unavailable.
The amount of engineering time this has saved us is delightful. I think of all the code we did not have to write and it makes me smile.
That's for containment and encapsulation at the application level. But we also follow that mode at the functionality level, too. For example, model detection and tracking is done by plugins.
We like to do things and have an abstraction so that we can churn out functionality for similar things "industrially", without thinking too much, but also so we could remove things easily without breaking the rest. Making code not just easy to add, but easy to remove is important. When we did that, we were able to remove a lot of code, too.
It is a spectrum, and we started by using it to contain at the "app" level.
That’s a rosy picture, but I’d point out that atrocities such as Eclipse Rich Client platform and associated OSGi specs are where this plugin concept can lead and it has its own problems! Complexity and discoverability are two!
It's not a panacea, but as a guiding thought it has served us well. We don't go all in on things "just because", and we develop what we need as opposed to what we imagine we need, and stop at an abstraction level that gets the job done.
you need to have someone who is very comfortable saying "no" to be in charge of maintaining the interface. otherwise, and especially if the plugin devs have access to the core code, they will say stuff like "hey, I see you have a very convenient function in the core, can you expose it for my plugin?". once you open the door to this, your encapsulation suffers death by a thousand cuts. you can end up with a de facto monolithic codebase that also has a complicated plugin interface that doesn't really encapsulate anything.
> you need to have someone who is very comfortable saying "no" to be in charge of maintaining the interface.
Of which the end result will be that the desired functionality will be somehow hacked within the plugin or will not be available at all.
Problem with such plugin based architecture is that it relies on a well designed interface. Person which designs the interface needs to have very good idea of how that interface will be used in the future, which is difficult / often impossible thing to do.
When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.
> your encapsulation suffers death by a thousand cuts. you can end up with a de facto monolithic codebase that also has a complicated plugin interface that doesn't really encapsulate anything.
Yes, worst outcome of all. In reality, plugin based architecture is no silver bullet. It can be very counter productive, especially when you're figuring out what you actually want to build, as you build it.
> When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.
one thing I will add is that every new feature does not have to be a plugin just because you have a plugin interface. "implement it directly in the core" is a perfectly valid fourth choice. some things just aren't suited to a plugin implementation.
>one thing I will add is that every new feature does not have to be a plugin just because you have a plugin interface. "implement it directly in the core" is a perfectly valid fourth choice. some things just aren't suited to a plugin implementation.
Yes. Quoting my answer to your reply's parent:
"""
It depends on the scope of the functionality. For example, right now, authentication and token generation are in the core, but it's okay right now because authentication spans across the whole product.
We eventually will extract it out, so we could use it as a component in another product, but for now, it's not inappropriate to leave it in the core.
"""
>Of which the end result will be that the desired functionality will be somehow hacked within the plugin or will not be available at all.
It depends on the scope of the functionality. For example, right now, authentication and token generation are in the core, but it's okay right now because authentication spans across the whole product.
We eventually will extract it out, so we could use it as a component in another product, but for now, it's not inappropriate to leave it in the core.
>When business requirements change, you then have the difficult dilemma - just insist on "no", introduce a minimal hack, redesign interfaces to support the use case in a clean way (possibly big task) etc.
Some days are easier than others.
>Yes, worst outcome of all. In reality, plugin based architecture is no silver bullet. It can be very counter productive, especially when you're figuring out what you actually want to build, as you build it.
That's why I talked with the scope and abstraction level. We tend to make the few and loose assumptions that get us a lot of leg work done automatically. We won't make further assumptions just for 1% advantage. And if we do, there's a fallback. For example, we say that a plugin has a certain structure and expects say an icon file. If it's not there, it's not there, the plugin is loaded but just not displayed. We issue a warning, in case it was by mistake, but the application does not break.
Very few and loose "specs" that one can go through quickly and easily without looking at a checklist or something.
Again, it's not a panacea. The underlying assumption in what I wrote is that neither I nor the reader believe in silver bullets. It's not a dichotomy. The question of course is not whether a plugin architecture solves all problems and makes bacon or solves nothing, and I may have been unclear in my message. My point was that it's one of the most useful things we have done because it reduced the amount of work we had to do. We still wake up and build product.
The core pretty much does nothing, except load the plugins and a few functions that we will extract into their components. This is what we did for the other parts.
One reason we did this was because we built custom, turn-key, ML products for large enterprise. Complete applications, from data acquisition and model training to "the JavaScript", admin interface, user management, etc.
Now... these large enterprise clients were in a sector. We could hardly sell the product to other similar clients because we couldn't just pick and choose which component or features to put on a skeleton.
It took us a lot of time, because these projects were both "software engineering" and "machine learning". In other words, we were toast. The worst of both worlds, as we were doing complete applications that even allowed their people to train models themselves.
It took a toll on morale. At some point working on eight different projects with different code bases and subsets of the team. We were fed up with this. We wanted to do things differently. We wanted to be able to get the time it took to ship the project the closest possible to the time it took to train models, which we historically did rapidly. It was all the rest that took time.
Total time = time to define problem + time to get data + time to produce models + time to write application + a big hairy epsilon
We wanted to bring "Total time" to its irreducible form. We didn't want to keep writing different applications for clients. We knew how to do it, but we did it enough times for several clients to notice patterns we wanted to extract into components. We also were losing time with the ML project lifecycle (experiment tracking, model management, collaboration, etc). We didn't want to ask the question "Which model is deployed again? What data produced that model? I tried your notebook on my machine, it doesn't work! DS: Hey, Jugurtha... Can you deploy my model? Jugurtha: I'm busy right now. I'll do it as soon as possible".
So we started building our ML platform[0] to remove as much overhead as possible, while being flexible. For example, one of our design goals is that everything one can do on the web app, they should be able to do with an API call.
