"Can we talk about this later? I need to get <task> done by <time>." isn't rude (at least by my barbarian standards), and it would be very rude to ignore such a request.
It doesn't seem rude to my standards, either. I'd be interested to hear others' experiences using it, though.
My experience has been that, unless delivered in an exceptionally conciliatory manner (exaggeratedly, actually), this kind of request gets interpreted as rude by a lot of people. For me, at least, I think it comes from many people assuming they have a "quick question" or that their "small" request isn't difficult to fulfill.
When someone walks up to me and opens with "quick question" the hairs on my neck stand up. I had a job where that became an office trend, everyone would start with "quick question".
I started responding to "quick question" with "long answer". That and I also hung a sign on my chair with my official office hours. Neither actually worked, and people thought I was rude, but it did make me feel better to try to push back. ;)
Long term my takeaway is that while too many meetings drain the life out of people, there's also a layer of communication that needs to happen, and when too much interrupting is going on, it's a sign that the right kind of meetings aren't taking place. Now instead of getting annoyed, I start to track the "quick question" categories, and create new channels for the commonalities, ideally Slack or email rather than a meeting, but in person when it gets big enough.
This is really why I prefer remote. I mean besides wearing pajamas all day. If I need to be scarce to get some work done, I can make myself completely scarce. Nobody can route their problem around my barriers by just showing up at my desk.
But isn't the mental energy required to come up with this answer (and judge its rude/nonrude status) already a distraction? Once you've left the bubble, there is a cost to coming back into the zone.
In my experience it's difficult to set the bubble boundaries. Some days I'm working on small tasks and wouldn't mind being interrupted, other days I need uninterrupted concentration. It's hard to communicate that with colleagues. Maybe there should be a general rule — mornings in the zone, then after lunch handle all tasks that require sync-up + training + selling.
