Actually, seems like they did try the suggestion out, basically by training a mo...

		huevosabio on Oct 2, 2023 \| parent \| context \| favorite \| on: Efficient streaming language models with attention... Actually, seems like they did try the suggestion out, basically by training a model with a dedicated sink token with all zeros. The verdict seems to be that you still end up with other initial tokens being used as sinks, so it is better to have a dedicated sink token.