Free speech is (sadly) already restricted at universities.
Make however a distinction between the free speech of individuals, and that of institutions (such as the church) operating on university premises. There is a place for indiscriminately allowing the former, but not the latter.