> The amount of rubbish I see published with editorial "standards" is enormous (in my own field and others).
Every human institution is flawed; that doesn't mean the alternative institution, or no institution at all, is better.
What I'm really wondering is, how can you keep up efficiently? And how do you have more objective standards? The system you describe seems very prone to popularity and political contests, and the ubiquitous Internet mob actions. Critiques by others aren't really useful signals unless you critique the critiques carefully - and who has time?
I'm not saying you have no answer, I'm just trying to understand how it works.
> Critiques by others aren't really useful signals unless you critique the critiques carefully - and who has time?
Well, I take the time and about a quarter of my most impactful papers have been such critiques. How do we encourage it? Well, ICLR (or was it NeurIPS?) a few years ran a replication challenge where if you could replicate a paper you got co-authorship. Not sure how much I love that strategy, but I am sure there are ways to create a sane "economy" around it.
As for whether we are better or worse off with the current state in my own field: I do not know. We end up in some sort of social science-esque argument where we simply can not prove the experiment either way as it can only be run once and have to argue on very shaky grounds (it also does not help that the field is exploding unlikely pretty much no other field ever has, which comes with its own issues). I think I am keeping up and I think that I personally have a decently objective view, but I can not prove that to you. What I can say is that there is not a single scientist around me that is not acutely aware of the problems with the previous and current systems. But given how "bottom up" we are without big beasts like Elsevier around that would have a deep financial interest to enforce the status quo, I believe we will arrive at solutions and faster than we otherwise would. There will be pain, yes, but see for example TACL, ACL Rolling Review, ICLR, etc. These are all initiatives that have been fielded by the community and I would argue two have already been great successes and one is struggling, but, could succeed.
> The amount of rubbish I see published with editorial "standards" is enormous (in my own field and others).
Every human institution is flawed; that doesn't mean the alternative institution, or no institution at all, is better.
What I'm really wondering is, how can you keep up efficiently? And how do you have more objective standards? The system you describe seems very prone to popularity and political contests, and the ubiquitous Internet mob actions. Critiques by others aren't really useful signals unless you critique the critiques carefully - and who has time?
I'm not saying you have no answer, I'm just trying to understand how it works.