That reminds me of the findings of Google’s paper on EfficientT5 (https://arxiv.org/abs/2109.10686). They refer to it as “DeepNarrow”.