I really want to stress: it's excellent science, and that's why they hold their code tightly. You can say 'no, a true scientist publishes everything' but---says you.
As someone in the field let me assure you: everything, of course, is more complicated than you make it out to be. I understand the absolutist position. But in a world of finite and ever-shrinking resources (grants, positions, etc.) it's fair to try to push your advantage. If funding were plentiful, adopting standards of publish-every-line-or-it-doesn't-count would be fair. People would have plenty of time and resources to get that done. As it stands there are basically no incentives to behave that way and being strapped for human resources puts the issue at the bottom of the list compared to actually getting results.
I'm not an absolutist, and don't want to come off as one. I'm just not in lattice QCD :)
What degree of data sharing is considered normal there? Across experimental physics it varies a lot: astronomers are often required by the funding agencies to make the data public, whereas particle physics experiments have traditionally shared very little (although pressure from funding agencies has started to change this too).
Given the ways you described this collaboration, my questions are:
- As an experimental physicist, when will I be able to believe them? Do we wait around for someone else to cook up a batch of similar secret sauce to confirm the result? Will they release their gauge configurations after some embargo period? Or should we believe them just because they are top-notch? I've seen top-notch groups like this fall before, so it seems quite reasonable if experiments aren't citing them now.
- Should funding agencies be attaching more importance to openness in science? From what you describe (and sorry if I'm misinterpreting you) there is very little incentive to share things that would make their results far more useful. Of course nothing is simple, but I've seen collaborations reverse their stance on open data overnight in response to a bit of pressure from the people writing the pay checks.
> Do we wait around for someone else to cook up a batch of similar secret sauce to confirm the result?
It took you folks 20 years to redo the experiment. Independent lattice calculations have already been underway for some time; I would expect (but I won't promise, not working on the topic myself and not having any particular insider information) results on the year-or-two timescale.
> Will they release their gauge configurations after some embargo period?
BMW probably will not do this. In their recent Nature paper they do say that upon request they'll give you a CPU code BUT when they provide a nerfed CPU code that produces the same numbers, rather than their performant production code. ... annoying.
> Or should we believe them just because they are top-notch?
Well, maybe? Why do you believe the theory initiative's determination of the vacuum polarization or the hadronic light-by-light? Some how it's more sensible to back out those things by fitting experimental data than by doing a direct QCD calculation? There's no free parameters in a QCD calculation, but fitting... well, give me a fifth and I can wiggle the elephant's trunk.
> I've seen top-notch groups like this fall before, so it seems quite reasonable if experiments aren't citing them now.
I think it's wrong not to hedge the experimental results and it's wrong not to cite them, but I understand why experimentalists wouldn't take their result as final either.
As someone in the field let me assure you: everything, of course, is more complicated than you make it out to be. I understand the absolutist position. But in a world of finite and ever-shrinking resources (grants, positions, etc.) it's fair to try to push your advantage. If funding were plentiful, adopting standards of publish-every-line-or-it-doesn't-count would be fair. People would have plenty of time and resources to get that done. As it stands there are basically no incentives to behave that way and being strapped for human resources puts the issue at the bottom of the list compared to actually getting results.