Aesthetics is a lot of it, but also the ability to build things like sidewalks or other public infrastructure. If people can build up to the property line, you’re going to have a hard time making many neighborhoods accessible or healthy without having to move structures.

I don’t think I’ve ever seen a neighborhood that made any particular public accessible use of the area between the sidewalk and the buildings. There are occasional utility easements, but those don’t need much space. Some neighborhoods have no fences and let pedestrians walk between the houses. Other than that, setbacks seem to be mostly for aesthetics and front yards.

It’s definitely both but it comes up all of the time where I live because the white-flight suburbs didn’t have that and either don’t have sidewalks at all or have these tiny sidewalks which aren’t contiguous and any attempt to fill the gaps or widen them to be comfortable for wheelchairs or strollers has problems with built structures.

Also for light and shadow management. This is most pronounced in Manhattan where tall vertical buildings right at the property line would create deep dark canyons. This is still true (to a much lesser extent) in suburbia.