You can prefix a prompt for Whisper with a small text section containing desired vocab, and it will likely improve accuracy for that specific domain.

Whisper source is very readable, check out https://github.com/openai/whisper/blob/main/whisper/decoding...