I developed and maintain a large and very widely used open source agent-based modeling toolkit. It's designed to be very highly efficient: that's its calling card. But it's old: I released its first version around 2003 and have been updating it ever since.
Recently I was made aware by colleagues of a publication by authors of a new agent-based modeling toolkit in a different, hipper programming language. They compared their system to others, including mine, and made kind of a big checklist of who's better in what, and no surprise, theirs came out on top. But digging deeper, it quickly became clear that they didn't understand how to run my software correctly; and in many other places they bent over backwards to cherry-pick, and made a lot of bold and completely wrong claims. Correcting the record would place their software far below mine.
Mind you, I'm VERY happy to see newer toolkits which are better than mine -- I wrote this thing over 20 years ago after all, and have since moved on. But several colleagues demanded I do so. After a lot of back-and-forth however, it became clear that the journal's editor was too embarrassed and didn't want to require a retraction or revision. And the authors kept coming up with excuses for their errors. So the journal quietly dropped the complaint.
A while back I wrote a piece of (academic) software. A couple of years ago I was asked to review a paper prior to publication, and it was about a piece of software that did the same-ish thing as mine, where they had benchmarked against a set of older software, including mine, and of course they found that theirs was the best. However, their testing methodology was fundamentally flawed, not least because there is no "true" answer that the software's output can be compared to. So they had used a different process to produce a "truth", then trained their software (machine learning, of course) to produce results that match this (very flawed) "truth", and then of course their software was the best because it was the one that produced results closest to the "truth", whereas the other software might have been closer to the actual truth.
I recommended that the journal not publish the paper, and gave them a good list of improvements to give to the authors that should be made before re-submitting. The journal agreed with me, and rejected the paper.
A couple of months later, I saw it had been published unchanged in a different journal. It wasn't even a lower-quality journal, if I recall the impact factor was actually higher than the original one.
If it makes you feel any better, the problem you’re describing is as old as peer review. The authors of a paper only have to get accepted once, and they have a lot more incentive to do so than you do to reject their work as an editor or reviewer.
This is one of the reasons you should never accept a single publication at face value. But this isn’t a bug — it’s part of the algorithm. It’s just that most muggles don’t know how science actually works. Once you read enough papers in an area, you have a good sense of what’s in the norm of the distribution of knowledge, and if some flashy new result comes over the transom, you might be curious, but you’re not going to accept it without a lot more evidence.
This situation is different, because it’s a case where an extremely popular bit of accepted wisdom is both wrong, and the system itself appears to be unwilling to acknowledge the error.
It seems that the failure of the scientific process is 'profit'.
Schools should be using these kinds of examples in order to teach critical thinking. Unfortunately the other side of the lesson is how easy it is to push an agenda when you've got a little bit of private backing.
Many people do not know that Impact Factor is gameable. Unethical publications have gamed it. Therefore a higher IF may or may not indicate higher prominence. Use Scimago journal rankings for non-gameable scores.
Science and Nature are mol-bio journals that publish the occasional physics paper with a title you'd expect on the front page of The Weekly World News.
I was an undergraduate at the University of Maryland when you were a graduate student there in the mid nineties. A lot of what you had to say shaped the way I think about computer science. Thank you.
This reminds me of my former college who asked me to check some code from a study, which I did not know it was published, and told him I hope he did not write it since it likely produced the wrong results. They claimed some process was too complicated to do because it was post O(2^n) in complexity, decided to do some major simplification of the problem, and took that as the truth in their answer. End result was the original algorithm was just quadratic, not worse, given the data set was easily doable in minutes at best (and not days as claimed) and the end result did not support their conclusions one tiny bit.
Our conclusion was to never trust psychology majors with computer code. And like with any other expertise field they should have shown their idea and/or code to some CS majors at the very least before publishing.
When I was a grad student I contacted a journal to tell them my PI had falsified their data. The journal never responded. I also contacted my university's legal department. They invited me in for an hour, said they would talk to me again soon, and never spoke to me or responded to my calls again after that. This was in a Top-10-in-the-USA CS program. I have close to zero trust in academia. This is why we have a "reproducibility crisis".
PSA for any grad student in this situation: get a lawyer, ASAP, to protect your own career.
Universities care about money and reputation. Individuals at universities care about their careers.
With exceptions of some saintly individual faculty members, a university is like a big for-profit corporation, only with less accountability.
Faculty bring in money, are strongly linked to reputation (scandal news articles may even say the university name in headlines rather than the person's name), and faculty are hard to get rid of.
Students are completely disposable, there will always be undamaged replacements standing by, and turnover means that soon hardly anyone at the university will even have heard of the student or internal scandal.
Unless you're really lucky, the university's position will be to suppress the messenger.
But if you go in with a lawyer, the lawyer may help your whistleblowing to be taken more seriously, and may also help you negotiate a deal to save your career. (For example of help, you need the university's/department's help in switching advisors gracefully, with funding, even as the uni/dept is trying to minimize the number of people who know about the scandal.)
I found mistakes in the spreadsheet backing up 2 published articles (corporate governance). The (tenured Ivy) professor responded by paying me (after I’d graduated) to write a comprehensive working paper that relied on a fixed spreadsheet and rebutted the articles.
I had a similar experience where a competitor released an academic paper rife with mistakes and misunderstandings of how my software worked. Instead of reaching out and trying to understand how their system was different than mine they used their incorrect data to draw their conclusions. I became rather disillusioned with academic papers as a result of how they were able to get away with publishing verifiably wrong data.
Unfortunately, academia is subject to the same sorts of social things that anything else is. I regularly see people still bring up a hoax article sent to a journal in 1996 as a reason to dismiss the entire field that one journal publishes in.
Personally, I would agree with you. That's how these things are supposed to work. In practice, people are still people.
I think the publish or perish academic culture makes it extremely susceptible to glossing over things like this - especially for statistical analysis. Sharing data, algorithms, code and methods for scientific publications will help. For papers above a certain citation count, which makes them seem "significant", I'm hoping google scholar can provide an annotation of whether the paper is reproducible and to what degree. While it won't avoid situations like what the author is talking about, it may force journal editors to take rebuttals and revisions more seriously.
From the perspective of the academic community, there will be lower incentive to publish incorrect results if data and code is shared.
I take the occasion to say that I helped making/rewriting a comparison between various agent-based modelling software at https://github.com/JuliaDynamics/ABMFrameworksComparison, not sure if this correctly represents all of them fairly enough, but if anyone wants to chime in to improve the code of any of the frameworks involved, I would be really happy to accept any improvement
SeanLuke, I tried to fix an issue about Mason I opened when I was looking into this a while back two years ago and tried to notify people about that (https://github.com/JuliaDynamics/ABMFrameworksComparison/iss...) with https://github.com/JuliaDynamics/ABMFrameworksComparison/pul..., hopefully the methodology is correct, I know very little about Java...In general, I don't think there is any very good comparison on performance in this field unfortunately at the moment, though if someone is interested in trying to make a correct one, I will be happy to contribute
Is this the kind of thing that retractions are typically issued for, or would it simply be your responsibility to submit a new paper correcting the record? I don't know how these things work. Thanks.
My answer: while 99% of the AI community was busy working on Weak AI, that is, developing systems that could perform tasks that humans can do notionally because of our Big Brains, a tiny fraction of people promoted Hard AI, that is, AI as a philosophical recreation of Lt. Commander Data.
Hard AI has long had a well-deserved jet black reputation as a flakey field filled with armchair philosophers, hucksters, impressarios, and Loebner followers who don't understand the Turing Test. It eventually got so bad that the entire field decided to rebrand itself as "Artificial General Intelligence". But it's the same duck.
The only difference is the same hucksters are trying to sell the notion that LLMs are or will become AGI through some sort of magic trick or with just one more input.
US News rankings are garbage based in no small part on opinion surveys and famously manipulated year over year.
Though I strongly disagree with their choice of conferences, probably the best regarded ranking of computer science schools is CSRankings.org (https://csrankings.org/)
If you think this is dense, try Italy some time. Huge numbers of highly distinct dialects, because until the mid-1800s Italians spoke huge numbers of entirely different languages, complete with their own full literature traditions. During unification the country settled on Florence's language (the language of Dante) as the "official" language: but everyone still proudly speaks their own language. To my knowledge, Italy is regarded as the densest diverse dialect region in Europe.
How different? What Americans call arugula the British call rocket. Because the British word is derived from the French roquette, which is from ruchetta, a word in italian dialects along the French border. But Americans got their word from aruculu in the southern Calabrese dialect, a result of immigration. The Italian word is rucola, from the Latin eruca.
Americans think "Capeesh" is an Italian word because they heard it in The Godfather. But it's not: it's Sicilian, as is much of the film.
Capisce, pronounced with a distinct 'eh' at the end ('capeesh-eh'), would be standard Italian for 'he/she understands' or 'you (polite) understand'.
But 'capeesh' tends to be used differently in American mob films, meaning either 'Got it, pal?' or 'Yeah, I got it' ("Capeesh? Capeesh."). Those would be different in standard Italian: capisci ('capeesh-ee'), and capisco ('cap-is-coh'), respectively. It's that final example that makes it obvious that the mobsters aren't speaking standard Italian - there is no 'sh' sound in capisco, so eliding the final vowel wouldn't get you to 'capeesh', but to something more like 'cap-isk'.
However, the corresponding forms in Sicilian are capisci and capisciu. Eliding the final vowel yields the observed 'capeesh' in both cases.
It makes perfect sense that the mobsters would be speaking Sicilian rather than standard Italian. Italian immigrants in the US were overwhelmingly from Italy's south, which is generally poorer than the north. (The Mafia, in particular, is an organization with its roots in western Sicily.) Most of these immigrants came before the advent of standardized/centralized schooling in Italy, and so were never taught modern standard Italian. Instead, they spoke their native Romance languages, generally dialects of Sicilian and Neapolitan.
Even today, most Italian-Americans will be able to tell you which 'dialect' their grandparents spoke.
[Capisce is] borrowed from the spoken Sicilian and Neapolitan equivalents of Italian capisci, the second-person singular present indicative form of capire (“to understand”).
No, it is not. When they say "capeesh" in the movie, they're trying to say "do you understand?" (second person singular). In Italian, that would be "capisci" (pronounced "ca-pee-shee").
Additionally, capisce ("does he understand?") in Italian, is pronounced "ca-pee-sheh".
Meh. If this was the wrong guy to threaten, then he would have sued to overturn their design patents. Instead he just told them where they could stick it. That's done all the time.
I get the feeling that, no matter how slow Linus goes, this is going to lead to a split. If Linus eventually pushes through Rust, the old guard will fork to a C-only version, and that won't be good.
Seems highly unlikely. Note that Hellwig is the only major remaining independent Linux kernel developer. All the rest have salaries paid by the Linux Foundation, Red Hat, Google, et cetera. They are highly unlikely to take an action that threatens their salary.
And Hellwig works as a contractor, he's not a volunteer in the same way that Con Kolivas was. Hellwig isn't truly independent either.
Recently I was made aware by colleagues of a publication by authors of a new agent-based modeling toolkit in a different, hipper programming language. They compared their system to others, including mine, and made kind of a big checklist of who's better in what, and no surprise, theirs came out on top. But digging deeper, it quickly became clear that they didn't understand how to run my software correctly; and in many other places they bent over backwards to cherry-pick, and made a lot of bold and completely wrong claims. Correcting the record would place their software far below mine.
Mind you, I'm VERY happy to see newer toolkits which are better than mine -- I wrote this thing over 20 years ago after all, and have since moved on. But several colleagues demanded I do so. After a lot of back-and-forth however, it became clear that the journal's editor was too embarrassed and didn't want to require a retraction or revision. And the authors kept coming up with excuses for their errors. So the journal quietly dropped the complaint.
I'm afraid that this is very common.
reply