Yep while this manifest v3 ugly thing is putting me on the brink of jumping ship (I compromise by having two browsers), as for your concern I found that Chrome is going to allow blacklisting extensions for sites, so now I can turn them off for the few sites that I really worry to grant extension read access.
If you are fine with two browsers, maybe you could instead look into separate Firefox profiles with different sets of extensions. I have added "-p" to my Firefox shortcut so it always starts with the profile picker thing.
chrome://flags, "Extensions Menu Access Control" flag. https://developer.chrome.com/blog/new-extensions-menu-testin...