Yes it was python but that has nothing to do with it. Same would happen in go, rust, R, or matlab.
Correct answers: 1.0, 0.0, 1.0
Confidence from algo: 1.0, 0.0, n/a
Confidence on the wire: 1.0, 0.0, 0.0
Score after bug: 66%
Score as it ought to be scored: 100%
It was enough to make several algorithms which were very selective in the data they would attempt to analyze (think jpg vs png images) went from "kinda crap" in the rankings to "really good"
Well, only in python is that N/A value also a float. In protobuf, go or Java for that matter, that data model must somehow be changed to communicate the difference.
If you had use 3 float values in Go or Java you would have had the same problem.
Correct answers: 1.0, 0.0, 1.0
Confidence from algo: 1.0, 0.0, n/a
Confidence on the wire: 1.0, 0.0, 0.0
Score after bug: 66%
Score as it ought to be scored: 100%
It was enough to make several algorithms which were very selective in the data they would attempt to analyze (think jpg vs png images) went from "kinda crap" in the rankings to "really good"