We currently have about 800 times the population as we did during the time of hunter gatherers, so we can lose quite a large portion of our population while still greatly exceeding the previous levels. It could be that we are seeing the end game of logistic growth. A decline in population would mean that resources would become cheaper, which in turn could stimulate population growth again.
A population that declines through birth rate attrition gets old. The average age in hunter gather communities was about 15 years old. In the next 10-20 years, the average age in a number of countries is going to approach 60.
You can't just think about raw numbers, you have to think about demography.
