As someone who left a job in academic administration recently and who is also faculty at another university, this is not true. Administrators can invite and often can disinvite or make other decisions about talks. This kind of day-to-day running of the university has been largely ceded to administrators.