Here here, I agree. In the most recent versions we have seen speed increases. The fact that polars exist shows that there is a tone of low hanging fruit.
There is also dask to increase parallelism and performance which I’ve used on some massive datasets 200GB+
Isn't the low hanging fruit that polars picked: "how about lazy evaluation to allow the query to be optimised?" ... which is mostly anathema to the design of pandas?
Pandas (the API) is also getting better at big data. I'm an advisor at a company, Ponder, that will take your Pandas code and execute it on "big data".
There is also dask to increase parallelism and performance which I’ve used on some massive datasets 200GB+