I strongly suspect that per-student spending is heavily biased towards special education programs at the not highest-achievement end of the spectrum.
I support the necessity of those programs, but the idea that money is flowing upwards on the achievement curve is probably contradicted by the data, given the intense needs of special education programming.
I think there may be some truth to that, I think some gifted programs pay the teachers a bit extra.
But, often, it may just be the appearance of extra things; if the gifted class does more 'enrichment' activities, it's often because they've got more time left after doing the required curriculum. Or, in my district, the gifted program had busing (which costs money) to get enough kids together to run one class per grade.
If the issue is money though; maybe it could be addressed through demonstrably worse conditions in the gifted program. Maybe a cap of 35 students per class in gifted, and 25 in mainstream. Then there's a tradeoff. My district had bussing as a negative characteristic for the gifted program as well (although my neighborhood school was the bus destination for four out of six years, so it wasn't bad for me)