You don’t know which expert you’ll need for each layer, so you either keep them all loaded in memory or stream them from disk