At some point, autonomous vehicles will be carrying cargo instead of passengers. When that happens, fatalities per mile driven will no longer be a valid comparison between manned and unmanned vehicles, as catastrophic accidents will have fewer people involved.
I do think it'll always be fair to consider total lives lost per usage metric, regardless of if people were in the vehicle or not. The lives of drivers & passengers have equal value to the lives of pedestrians.
