You have not only the impact of mining but also the problem of reverse logistics for correct disposal and recycling of those toxic batteries that don’t even last long. Batteries contaminate soil and water in a way that’s much harder to control if it start piling up everywhere. And today’s world can’t even solve the disposal of plastic.
With intelligent charging practices (which often come at the expense of stored energy) and temperature control, modern lithium cells asymptote to ~20% capacity loss and stay there for, well at least since the model S came out, still counting. Or that was the case a couple years ago when I looked into it. The difference in the battery life in a leaf vs a tesla is qualitative not quantitative. The motivation to make phone batteries last longer wasn't there at first until it provided negative press, now intelligent charging is fairly common in phones. Thermal management is harder.
Lithium batteries are a great source of lithium, and bigger of them, like laptop batteries, largely get recycled even now, AFAICT. Lithium + iron chemistries in particular avoid seriously toxic components.
Both lithium and plastics are far less nasty than, say, ash from a coal-burning plant, with its sulfur, mercury, and radioactive stuff. Retiring these is a higher priority thing, IMO, than improving lithium mining cleanliness (though an improvement is always welcome).
