Ultimately, this algorithm tries to find an single directional vector in a 512 dimensional space that approximates what it means to change skin color.

Expecting that this works all the time (or expecting that all points in this 512-dim space result in a beautiful person) is probably a bit too much to ask. :-)