The core should trap an illegal instruction on unrecognized custom instruction opcode, and the handler could call an emulator. Just like if one issued a floating point instruction on a core without the F extension. Of course that means whomever writes that trap handler and emulator needs to actually keep up, and custom opcodes are unlikely to be unique across all RISCV implementations. So it’s theoretically possible but you know that saying about theory and practice…