That is false; the license doesn't speak about any such right.
Rather, the copyright holder of the AGPLed work is exercising their right to dictate the following:
[Y]our modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.
Under the AGPL, the visitors to the program need not request the source; it must be prominently available for them to obtain in a self-serve manner. No mention is made that they have any kind of right. They are not parties to the license at all. A copyright license can only grant rights to licensees.
By the way, "all users interacting" could be interpreted to literally mean all users, including users who visit the login page, but do not have an account (and are not authorized to have one due to not belonging to the organization that runs the service).
> if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network [...] an opportunity to receive the Corresponding Source of your version [...]
From the Merriam-Webster dictionary:
> interact, verb: to act upon one another
When distributing a binary from a compiler to a user, the user does not act upon the compiler. So the user is not interacting with the compiler, so you don't need to distribute your modified compiler's code any more than you would with the GPL.
That's not what is in question. Of course if you someone has reason to comply with section 13 of the license, then users who download the code are having the code distributed to them.
The disputed claim is that users who interact with a remote program are, by that interaction, being distributed the program.
The GNU AGPL itself explicitly disagrees with this; check its definition of "propagate" and "convey".
Depending on the license of the compiler, yes, the binary is related to the compiler in some way. In practice it literally contains code from the compiler.
This is not how I see it at all. The ones using it are the users, and if they sit across the network you are distributing it in a very tangible sense.
The idea that the admin running a service is the one doing the "use" and not the users of your service is debatable to say the least.