You still have to cross at intersections and driveways. I've been hit multiple times (at low speed) by motor vehicles crossing the sidewalk at a driveway. I don't ride on sidewalks anymore.
This sounds like a speed issue. If you’re doing 20 mph on the sidewalk then yes you weren’t in view when the driver started backing out of their driveway.
I don’t fully think I’m a bike I get right of way. If I see a car reversing ahead I move into the street then prepare to full stop but proceed with caution and watch the cars acceleration to see if they see me. If I get the hint they don’t. I stop. It’s scratch on the bumper for them and life for me I am more than glad to yield.