The biggest reason people think this kind of thing is scary? They're remembering back to highschool when they had to march through math while being taught by people who barely understood it. They know they never would have done it without having been forced, and that idea kind of leaks backwards to early education.

There's a mythical "self-driven and exploratory" math education out there, but the cold reality is while essentially anyone would be able to supervise a journey through colors and counting only very few have even seen what's good about math. (Moreover, what small fraction of them would be willing to teach teenagers?)