Then such a crawler should mark itself with all purpose tags and thus be blocked in this scenario.

Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.