Well, there's two issues I would have with your statement.
The first is that "pointing out" is not where the story ends; like parent said, there is a small minority of people who use this "average man"(an amalgamation of features that doesn't actually exist in reality) to shame specific individuals(e.g. calling out novel authors for being "privileged" and thus shouldn't write characters with a different skin colour). This usage opens you up to an obvious reciprocal attack you would probably object to - one could point out that the average black person is much more likely to be arrested(https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-...) and use this to call specific black people "dangerous". In both cases, you are prejudging based on a stereotype.
The second objection is that once you start to delve into the "privileges", the label just doesn't seem to fit the actual things being described. Actually searching for "white male privileges" leads me to articles that are so easy to fight that I feel like I'd be addressing a strawman[0][1][2], so if possible, could you provide a list? My general contention is that the lists of "privileges" end up being a mix of: significant outcome benefits which have controversial correlation-not-causation interpretations(e.g. is blacks being arrested more "white privilege" or simply the amalgamated effect of high poverty, low education and latent racism?); things you would not really describe as "privileges" but simply different choices(e.g. men having longer careers due to spending less family time); things that are incredibly petty("I can if I wish arrange to be in the company of people of my race most of the time." from [0])
The first is that "pointing out" is not where the story ends; like parent said, there is a small minority of people who use this "average man"(an amalgamation of features that doesn't actually exist in reality) to shame specific individuals(e.g. calling out novel authors for being "privileged" and thus shouldn't write characters with a different skin colour). This usage opens you up to an obvious reciprocal attack you would probably object to - one could point out that the average black person is much more likely to be arrested(https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-...) and use this to call specific black people "dangerous". In both cases, you are prejudging based on a stereotype.
The second objection is that once you start to delve into the "privileges", the label just doesn't seem to fit the actual things being described. Actually searching for "white male privileges" leads me to articles that are so easy to fight that I feel like I'd be addressing a strawman[0][1][2], so if possible, could you provide a list? My general contention is that the lists of "privileges" end up being a mix of: significant outcome benefits which have controversial correlation-not-causation interpretations(e.g. is blacks being arrested more "white privilege" or simply the amalgamated effect of high poverty, low education and latent racism?); things you would not really describe as "privileges" but simply different choices(e.g. men having longer careers due to spending less family time); things that are incredibly petty("I can if I wish arrange to be in the company of people of my race most of the time." from [0])
[0] - http://www.winnipeg.ca/clerks/boards/citizenequity/pdfs/whit...
[1] - http://everydayfeminism.com/2015/11/lessons-white-privilege-...
[2] - http://www.care2.com/causes/what-exactly-is-white-privilege-...