The genetic evidence points into the other direction:

> There is considerably less Neanderthal ancestry on the X-chromosome compared to the autosomal chromosomes, which similarly suggests that admixture with modern humans was primarily the result of mating between modern human females and Neanderthal males.

https://en.wikipedia.org/wiki/Neanderthal_genetics#Parentage