A lot of dependecies are pulled from the (recommended) asciidoc-dblatex package, which depends, indirectly, on x11 packages. Don't install the recommended packages (whose concept is anyway in conflict with keeping a system lean) and see what happens.
How to not install recommended packages? (there wasn't a choice, and anyway not installing recommended packages often implies - rightly or wrongly - security risks).
Regarding the apt "recommended" concept, it's a generic concept of optional functionality; it's orthogonal to security. If one wants to make a generic association with security, less recommended packages -> less functionality -> smaller attack surface.