I'm surprised that things like this don't happen more often. It seems like uploading malicious binaries to npm or other registries would be fairly easy.
Then again, convincing people to play around with a model might be easier than getting them to use your library in an application.
I've seen an alarming number of Colab notebooks that casually ask for access to your entire Google Drive and then download a few thousand Python packages (and who knows what else) and execute code.
maybe it is more like - one million transactions in good faith and then an actor or group does a Bad Faith action.. is the tolerance for 'perfect' in the way ? How can ordinary use be enabled while correctly blocking out malicious use, without crazy and ugly intrusive security engineering ?
put another way, maybe the cost of a few bad faith actors is much higher than anticipated among people doing skilled work? How is that handled in a human way?
I can guess that de-platforming based on whatever is much easier for the server side. This dynamic is how we got to the personal computer "revolution" thirty+ years ago
You'd hope, but a lot of models are still distributed as .pt files, which are Pickle — which means they can contain and execute arbitrary code on load.
Some popular applications (Automatic1111 etc.) take measures against that, but the real fix is to ignore anything that isn't safetensors.
There are action models based on LLMs like Gorilla LLM or autoGPT that take a natural language input and convert that to API calls that could do all sorts of things from sending emails to performing stock trades.
To my knowledge it's still the harnesses that perform those actions, not baked into the weights (other than perhaps special tokens denoting that an action should be taken)
Then again, convincing people to play around with a model might be easier than getting them to use your library in an application.