If you write your operating system to use the paravirtual devices and run inside a hypervisor then you don’t need to worry about drivers for all the physical devices you might find.

By the time you have attracted enough users that those few percent of performance you are losing on the paravirtual drivers is an issue, the extra code for some performance critical native drivers will be a small part of your OS.