if you have time booked on your calendar, no matter what it is, it's safe to assume anything else put on top is going to be declined, because you already have something scheduled.

You can add a regular meeting for yourself too, instead of focus time, also set your calendar not to share meetings' details with other folks, so they will only see 'busy'