>why a whole new project.el instead of adopting or at least compatibility with projectile

Probably because projectile is not FSF compatible and getting all of the copyright assignments would be way more expensive.