Show the training set, and PROVE that the tasks and answers aren't in there. I don't understand why this is not a default first step for proving that this is creating new knowledge.
Well that's harder than maybe solving well-known open problems (whose soln's are presumably not in training set lol) but it seems that their examples are not clearly breaking sota, especially on matmul
No, I'm saying they have a massive database of solutions (the training set) and don't even bother proving that their solution isn't in there. I'm not claiming something, they are failing to provide some necessary information here
If they wanna do research where they claim they did something novel, without showing that they didn't just "look it up" in their massive training set, then yes, they should share what is and what isn't contained within.
