I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834
This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a generalized convolution, not the standard one.
The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.
I argue that autodiff can be replaced with what I call Hopf coherence which happens within the single layer as opposed to across the whole graph.
Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.
I'm working on a next gen Hopf algebra based machine learning framework. The beautil part is that it ties nicely to PL aspect as well.
Look at the diagram. Do you see the path going through the middle? And the paths at the top and bottom (they are generalized convolutions)? Well a Hopf algebra "learns" by updating it's internal state in order to enforce an invariance between the middle path and the top and bottom paths.
Allow me to restate it, it's an algebra that "learns". Reading about Hopf algebras is tripper than dropping acid.
This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a generalized convolution, not the standard one.
The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.
I argue that autodiff can be replaced with what I call Hopf coherence which happens within the single layer as opposed to across the whole graph.
Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.
I'm working on a next gen Hopf algebra based machine learning framework. The beautil part is that it ties nicely to PL aspect as well.
Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW