I've always wondered what actually breaks if 1 is prime or conversely what defining 1 as not prime gives us. Got just far enough into my math degree before switching to CompSci to stay of of universities the rest of my life to want to know.
The biggest problem is that you lose unique prime factorization. With prime factorization, I get a unique representation of every positive integer. Let's consider a way to write positive integers in "base prime", similar to base 10 or base 2. I'll start counting from 1 and write numbers as a tuple of prime factors. Similar to base 10, "base prime" has an infinite set of 0s that we're leaving out for brevity (e.g. 19 = 0000019), although it's on the right side instead of the left.
The i th position in every tuple is the power of the i th prime in the factorization of that number. So 10 = (1, 0, 1) = 2^1 * 3^0 * 5^1. 84 would be (2, 1, 0, 1) = 2^2 * 3^1 * 5^0 * 7^1. If we have unique factorization, there is exactly one way to write every positive integer like this, and there are many insights we can gain from this factorization. If 1 is prime, then we can write 6 = 1^257 * 2^1 * 3^1, or any other power of 1 we like. We just gain nothing from it.
There are often many equivalent ways to define any mathematical object, and I'm sure there are plenty of ways to define a prime number other than "its only factors are itself and 1". These other definitions are likely to obviously exclude 1. One obvious one is the set of basis coordinates in this "unique factorization" space that I just laid out here. And we're never really excluding or making a special case for 1, because 1's factorization is simply the absence of any powers -- empty set, all 0s, whatever you want to call it.
Keep in mind that "unique factorization" turns out to be very interesting in all sorts of other mathematical objects: rings, polynomials, symmetries, vector spaces, etc. They often have their own notion of "prime" or "primitive" objects and the correspondence with integer-primes is much cleaner if we don't consider 1 prime.
Some examples are in these comments, e.g. the Fundamental Theorem of Arithmetic. The Sieve of Eratosthenes is an amusing outcome, where 1 is the only prime if you take it literally.
But also mentioned elsewhere in the thread: if we declared 1 to be a prime, then many (I daresay "most") of our theorems would have to change "prime number" to "prime number greater than one".
