Another thing that is rarely mentioned, is that it’s very common to use color filters in B&W photography to tune the contrast.

That means the luminance values can be completely different from what they actually were, and skew the color choices. On the other hand, this might be something that a good ML algorithm can detect and compensate for.