Recently there has been a trend in calling models with weights and code available "open source" even if the training data is not available. For safe deployment in health care and other safety critical fields, transparency on the training data and process are vital too, which means developing clear terminology for models full transparency! Even this article title suffers from this ambiguity.
I can't fathom why they didn't just require the models to make available the training data itself. Sure you might need to fork some cash so they can ship you hard drives but surely being audited by someone anyone is better than none.
Training data for a medical diagnosis model would likely include enough info to de-anonymize the info for some participants (age, sex, zip code, descriptions). I'm not sure what the answer should be but I'm uncomfortable with the medical training data being provided freely to the world.
There is still a large difference between an AI being able to cite it's claims, and having the original training data and exact code and process in which to convert the training data into the weights used in inference.
If you cannot re-create the weights and model used for inference, a release's value is somewhat limited vs releases where the inference model can be re-created. (It's kind of like the limited value of scientific papers where the results cannot be reproduced due to a lack of detail)
What would you do with the training data if you had it? I see absolutely no reason why the training data is needed to evaluate a model, or how any kind of guarantees could be made about the model if you did have the training data. With the weights and code it's perfectly possible to interrogate and evaluate it.
I suspect a lot of people asking for training data are mainly looking to complain about some aspect of it (bias, copyright, etc etc) instead of actually thinking they can somehow use it to devine how the model will perform.
One can never practically evaluate it on all possible inputs/prompts, so an understanding of the training data distribution is important to generate the right test queries and create guardrails for desired use cases.
How does open source improve safety if we simply don't have the analytical tools to intuitively reason about LLMs?
You can't use this to prove that the model will always behave correctly (or desirably). At best, you can build test-suites to empirically check that it kinda-sorta appears to be doing the right thing most of the time. Which you can just as easily do with a black-box model.
It's not that I'm against openness. I just don't see how you can posit that it gets us close enough to safety.
Full openness in healthcare also comes with a cost: no training or fine-tuning on patient records or real world interactions which really really ought to not be fully open.
But your fine with a human that might spend 5 minutes looking at your chart after a heavy day of drinking/pills/10,000 other things to do?
The problem with 'Doctor AI' isn't that it's going to make mistakes, we already have doctors doing that and killing 100s of thousand per year. It's that we'll only have 1 doctor AI everywhere and there won't be a thing as a second opinion because "Computer Don't Argue"
It seems like all of their criticisms can be easily applied to essentially any technology used or medical company relied on.
For example:
>In the rush to deploy off-the-shelf proprietary LLMs, however, health-care institutions and other organizations risk ceding the control of medicine to opaque corporate interests. Medical care could rapidly become dependent on LLMs that are difficult to evaluate, and that can be modified or even taken offline without notice should the service be deemed no longer profitable
Even:
>LLMs often generate ... convincing outputs that are false
is already a problem the medical community has to address with existing tests.
Or:
>Another problem specific to proprietary LLMs is that companies’ dependency on profits creates an inherent conflict of interest that could inject instability into the provision of medical care.
Seemingly applies to essentially the entirety of medical supplies and medications.
Because if you are trying to assess the safety of some process and there is a 0.5% chance that it will give you deadly results, its not good.
Moreover, if its not deterministic, how can you assess if its safe? Sure you can run many many iterations, but how do you know when its safe enough? LLMs encourage freeform entry, which means the testing space is fucking massive.
Does writing with a different syntactic style give different outcomes? Does spelling mistakes lead to increase morbidity? thats a test plan I don't want to have to run (unless you are paying me megabucks.)
Humans are not deterministic as you point out, which is why you need to control for as many unknowns as possible when testing in a health setting.
If the training data for this is completely published, we'll have a huge head-start in terms of building a deterministic LLM once someone figures out how to do that.
As mentioned in another comment, the problem is that Open Source does not necessarily apply to all aspects of models. Open code allows everyone access to the "source" of an application. It does not mean the information that the code stores, when used, is also open to viewing.
In models, the training data (dataset) is frequently "closed", where it is not open to viewing. That's just the default behavior of publishing models. You don't need the dataset to use the model. The weights or tensors may be "open" in that we can see them, but they are fairly "not worth viewing" if we don't know the nature of the relationships between the tensors.
If we were able to figure out relationships between the tensors, and the dataset was not made open, then there might be a debate on whether or not certain use of that extracted or "transfer" knowledge is allowed.
For a "model" to be fully "open", it must publish the data it was trained on, the code it used to train itself, and its tensors or weights must not be encrypted or disallow establishing relationships in the weights.
It would seem to me, that the data is the more important part of the equation, and the health care providers, being the holders of this data, and also needing to find new revenue streams, want to profit from this.
With federated learning and homomorphic encryption, can we satisfy both parties?
I agree that open source (or source available) is better, in particular the weights and code, the training data is immaterial. But I think a lot of this is pretty naive. The "best" model is the best and it's unlikely to come from some idealistic consortium. And the data they have is virtually irrelevant, as every company that thinks they have a great trove of data finds out. My recommendation would be to use whatever the leading source available models is (one of the big llamas?) and focus on the guardrails needed to make it a helper for medicine. Reinventing the wheel is a bad idea.
To deploy generative AI in healthcare someone has to pay for the salaries of a lot of people to do the work. That means there needs to be a business model.
I am not sure who will take an AI through regulatory procedures if it is open source and there is no way to make money from it.
Open source is a useful tool for research yes. More of it would be nice.
But I don’t understand how or why anyone is going to go through all the hurdles of deploying technology if all of it is open source.
Maybe an open source enthusiast can explain to me how that is supposed to work?
Redhat, Element, Prusa, Adafruit, Sidero Labs, Arduino... plenty of companies that open source everything or almost everything and have have staying power.
Many consumers, myself included, will -only- pay for technology if it is open source. In fact if something is proprietary I feel I am being cheated anyway and I might as well pirate it until I find something open to support.
Many of us are willing to pay for time and labor and to support development for our personal projects and businesses so long as we have the power to change that relationship later and keep the tech if third party company later goes evil or goes under.
If I do not have the source code, I do not own it. If I cannot own it, then why pay for it.
Also everything becomes open source eventually. Companies can choose to accelerate this and earn community goodwill that might make them money selling open source turn-key services, or be replaced by that same community eventually doing it all themselves.
No one pays for a license fee for the Linux kernel, but they pay their choice of cloud provider to host it. Choice. That is what I will pay for.
I looked through a bit of the companies here and I don’t see any companies that have to retain a quality system and staff to stand behind their products permanently. These models seem to work better when you can just put stuff out there and occasionally pop in to help?
This is not a model that can work with regulated medical products. There is a very significant cost to maintaining static artifacts and I don’t see how you can defensively do that if anyone can access the artifacts?
How many hospitals download the linux kernel and manage their own servers?
I would assume very few. They likely lack the expertise, and pay third party companies to manage the servers, patches, and updates for them... even when the software is open source.
If installing these things required passing FDA trials before you could even do the first install, it’s hard to imagine anyone self funding the regulatory trial and then open sourcing the result.
These businesses you mention are different because you can start getting paid as soon as you build some expertise in how to operationalize the systems.
Only certain classes of healthcare products require regulatory approval. For example, you could likely build and distribute an open source AI tool for summarizing patient charts, and the FDA probably wouldn't object (this is not legal advice).
