Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> One thing that troubles me in the paper is that the researchers appear to have gone looking for precursor patterns in an ad hoc way, with no physical theory in mind, just trying different binning techniques and delays until they got a signal.

I believe in particle physics this is known as the “look-elsewhere effect”. Basically if look long and hard enough for a pattern, you will eventual find it if your parameter space is large enough: https://en.m.wikipedia.org/wiki/Look-elsewhere_effect



When I was taught statistics, this was regularly brought up as a big no-no in science. However, I read a guide to practical statistics that had a gem about predicting the stock market. If we discovered surges in the market correlated to newspaper sales, we wouldn't discard this as look-elsewhere. In fact, we'd follow newspaper sales very closely.

Predicting earthquakes has a big upside for humanity. If there is even a small correlation -- even if we don't yet understand it -- we can benefit from it.


It can be useful, it's just you need to keep monitoring it: if the effect seems to shrink or disappear on future tests, it's probably spurious.


In GWAS studies there's a nice visualization for dealing with this called Manhattan Plots. https://en.wikipedia.org/wiki/Manhattan_plot

Basically if you test a lot of hypotheses plot all the p-values and look to see if there is something that is a true outlier or whether it is expected given so many tests.


Most of particle physics is people searching for jobs that give meaning to their life. This often biases them against finding results against meaning.


Not to mention that they are themselves made of the particles they study, which ineradicably biases their outlook.


Never be made out particles. That's true.


This is related to the fact that in a higher-dimensional space the hamming distance between two points is compact, compared to 2d/3d intuition. This problem is related to the way that e.g. embedding text into an AI for comparison purposes often produces surprisingly closely related vector distances for relatively unrelated strings.


The obligatory XKCD: https://xkcd.com/882/


Can we transfer this to the financial markets?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: