I'm not aware of a way to do that easily using VNC.
Also, is there a good API for programming VNC? So I can inspect frames and act upon them?