I think a deeper question to ask is:

Can AI create at all?

By our own human definitions of creation, does anything spit-out by LLM, ML, AI, have any merit as a created -thing-?

Can the sum of what is learned by a model become more, and if so, can that create something? Anything?