My wife is working on a PhD in psychology (she intends to do clinical work) and after hearing about the mathematical "training" they receive in order to do research, I was astonished at the lack of rigour. One personal anecdote stands out. Myself being a professional statistician, she would sometimes come to me for help with her assignments. On one assignment, her professor had misunderstood one particular statistic and had given then a problem that wasn't well defined. I helped her craft a polite email to her professor explaining the ambiguity and asking for a resolution. The professor got extremely defensive and did not answer the question, and none of her colleagues realized that the question was incorrect either. And don't get me started on the conclusions they draw from the smallest of trials... /rant Sorry.
P-hacking can be done by analyzing the data in different ways until a significant effect is found, and then failing to report all of the analyses done. There is no universal way to analyze data (especially what collected data to include); there are choices that are made and this leads to a “garden of forking paths” where if you look around the garden enough noise alone will give you ‘signicance’. Preregistration makes this impossible by declaring your analysis before collecting data.
As a psychology undergrad, it's hard to be surprised at the lack of rigor when you realized a large part of the faculty does not understand what science is and therefore does not feel the need to believe in it. I'm still astonished that the entire subfield of psychodynamics is allowed to persist based on "case studies" ie phenomenology ie not science.
As a scientist should, I reserve the right to change my mind should the data or my understanding of it. I freely admit that I currently know little, but I'm still taken aback by the shortsightedness of some of my professors with regards to their... Epistemological relativism.
Fair enough, though as a computer scientist my statistics training was also awful. I actually tried several different stats professors' courses before I found one where I managed a barely passing grade. They all taught terribly and preferred solving meaningless integrals to teaching statistical techniques, applications, and intuition. /rant as well
