I never understood how or why patenting math is a thing.

It is patenting the application of math.

How do you differentiate between math and the "application of math"?

You can't patent an S-box transform. You can patent using a specific S-box transform to encrypt / decrypt data. That's the justification at least.

More like you can patent math when it is applied via a computer. So you would not be violating the patent by evaluating the S-box with pen and paper.

It makes no particular sense, but the theory behind math patents being out of bounds is that math is a fact. Software patents, including crypto patents, do not "feel" like facts to most judges, even if they technically are (see: Curry-Howard Correspondence).