You aim for 3ms, 10ms can be described as annoying. 50ms is really bad.

In reality you either move to analog processing when you reach that much delay or design your creative workflow around it (learn not to expect immediate feedback) so you move towards offline processing.