Yes, that's called 'ensembling'. There is a lot of work being done on this kind of solution. One way in which it could work is that you can use multiple models that have been fine tuned for various problems and then use the answer that returns the highest confidence.

You can also have adverserial generation where models given different expertises and attitude can go back and forth criticizing each others work

Sounds like the 'dead internet' is just around the corner!

Ask the LLM to perform a complex task by splitting it into sub tasks to be performed by other LLM instances, then integrate the results...

Is this something like langchain is working towards?

AutoGPT, BabyAGI