My take on this is that each narrow field goes through an S curve with a very exciting exponential beginning followed by a long tail of slow and boring progress. Then there is a new field with a new S curve.
The current hotness is experimental and theoretical investigations of large language models. This is just maths, and the papers coming out recently have been amazing.
I’ve read papers showing things like that neurons pack information into nearly-orthogonal spaces with beautiful geometric symmetries.
Just yesterday there was a paper showing that the middle layers of a deep language model can be interchanged and still work!
The current hotness is experimental and theoretical investigations of large language models. This is just maths, and the papers coming out recently have been amazing.
I’ve read papers showing things like that neurons pack information into nearly-orthogonal spaces with beautiful geometric symmetries.
Just yesterday there was a paper showing that the middle layers of a deep language model can be interchanged and still work!