It's not a retort to that -- It's a retort to the person asking for their money back.

It is saying "what you are upset about does not matter. Don't bother someone else with it", which is both an insult and correct.