Yes, I have the exact same feeling. I like seeing all the places I've been, but I hate that google also sees all the places I've been.
I've recently installed Owntracks on my phone and have started logging all my location updates, so soon I may be able to just drop the google location tracking.
I just checked, they list location data back to 2009.
https://www.google.com/maps/timeline