SSH blocked by firewall for any connections except incoming from VPN IP address. VPN server is mine and located on separate server.

This has proven to be working solution for years. Any monstrous "security" constructions with keeping it open or partially open will backfire once one attack vector will be discovered.