There was a stronger anti-tax argument for other things (stamp act, especially) but the tea issue was mostly a bunch of smugglers and smuggler-connected folks who were pissed off that the British had not just made any gray-area work they were doing explicitly illegal, but were also undercutting them such that they couldn't compete on price, even by breaking the law.

For ordinary tea drinkers the whole tea thing actually made tea cheaper. It's one of the weirder things to get attention as a reason-for-revolt, really. I suppose the "tea party" protest is just odd and interesting enough that we can't resist lumping that in with far more legitimate grievances.