> Using data from the Panel Study of Income Dynamics reveals that the relationship between predictors typically used to predict mortgage performance and the subsequent arrival of mortgage default trigger events has exhibited significant intertemporal heterogeneity.
A pretty roundabout way of saying that the future was being predicted by extrapolating the past, and that the actual future was unlike the past.
> Interestingly, diversifying the macroeconomic conditions from which the training data are sampled did not generally improve the out-of-time performance of the machine learning methods that we considered.
Mildly interesting indeed, but not quite surprising given that the macroeconomic conditions form a pretty large space. All sets of conditions for which data is available form only a tiny subset of all possible conditions, so increasing the training data from (say) 0.1% to 0.2% of possible inputs is not going to move the needle much.
In this instance in particular (mortgage default prediction), generating better predictions will produce shifts in the very conditions observed. The possibilities for emergent behaviour in a system as complex as the mortgage market are very large.
To rephrase... because the models predicting model failure (often) use the same assumptions/knowledge as the original models, so they both fail at the same time once assumptions become invalidated.
"intertemporal heterogeneity in the relationship between variables that are frequently used to predict [...] imply that model instability is a significant source of risk for [folks] that rely heavily on predictive statistical models and machine learning algorithms [...]"
No shit, Sherlock! aka "overfitting"
I don't mean to deride the actual research that was done, but I would hope that people understand this general property better before jumping on the bandwagon to use statistical models -- without needing the benefit of hindsight.
A pretty roundabout way of saying that the future was being predicted by extrapolating the past, and that the actual future was unlike the past.
> Interestingly, diversifying the macroeconomic conditions from which the training data are sampled did not generally improve the out-of-time performance of the machine learning methods that we considered.
Mildly interesting indeed, but not quite surprising given that the macroeconomic conditions form a pretty large space. All sets of conditions for which data is available form only a tiny subset of all possible conditions, so increasing the training data from (say) 0.1% to 0.2% of possible inputs is not going to move the needle much.
In this instance in particular (mortgage default prediction), generating better predictions will produce shifts in the very conditions observed. The possibilities for emergent behaviour in a system as complex as the mortgage market are very large.