For some applications this matters, but for a typical web server or web architecture, which is what go is optimized for, rarely if ever would a couple MB of binary size make a difference.
Especially when tossed in a container for most workloads nowadays. You can have a full go binary statically linked and copied into a FROM SCRATCH docker image and it just works, and is miniscule.