It still massively affects you (assuming you are in the US), because of the 4th Amendment. That's the one that prohibits unreasonable search and seizure.
A key factor in whether a search or seizure is unreasonable is whether or not you have a reasonable expectation of privacy. The standard for what is a reasonable expectation of privacy is what society considers to be reasonable, not what you individually consider to be reasonable.
So even though you are not on Facebook, all those people who ARE on it and are sharing publicly the most intimate details of their lives are moving the 4th Amendment boundary between searches and seizures that require a warrant and those that do not.
Good luck with that. Even the credit reporting companies get confused and can't figure out who I am accurately. And I actively help them since I like having credit cards and the ability to get mortgage loans.
Facebook has built a platform that's much more accurate and comprehensive than anything experian has, because experian isn't in the business of selling your political beliefs and sexual orientation and a hundred other demographics to marketers.
You can file a FOIA request with Facebook to see what they have on you. Other people have, and have gotten their dossiers. Have you had a phone number or mailing address in the past 10 years? If so, they've gotcha covered.