A refusal to acknowledge such integral parts of these systems as semantics renders that line of argument wholly irrelevant. No good faith discussion cannot be conducted without participants who already understand the meaning behind symbols.
It doesn't matter at all what "mathematics" you use to "analyze" music because the purpose of these modes of communication goes beyond the arbitrary sounds chosen -- what does matter is what is being represented by those sounds.
Statistics is for the analysis of systems in equilibrium. Language by definition is a non-equilibrium system.