Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.