I’ve been working on a physics project that unexpectedly led to a new transformer architecture.
*TickBlock* replaces QKᵀ with a learnable banded positional operator (“tensor mode”).
On Tiny Shakespeare, it matches GPT-2-small-level performance with only 0.64M parameters (≈0.5% of the size). It trains in ~12 minutes on a Mac laptop — no kernel optimizations yet.
The design is motivated by my research in fundamental physics, which reinterprets relativity without tensors. That work (Project Belgrade) models reality as a sequence of discrete publishing events and photons as standing sheets of information:
[Structure and Mechanics of Reality: Project Belgrade](https://doi.org/10.5281/zenodo.17191747)
Right now the results are small-scale, but the efficiency gains suggest a bigger question:
- With kernel optimization, parameter sweeps, and compression, how far can this scale?
- Could we see GPT-4-class assistants running fully on laptops? GPT-3.5-class reasoning on phones?
- If that became practical, what would be the most compelling on-device applications?
Curious to hear how the community thinks about the tradeoffs and opportunities.
There are many apps. Voice Dream isn't up to date with voice quality (which was amazing 10 years ago when it started, but now even Apple gives you voices of similar quality for free), but is up to date with prices.
It is free at the moment. They clearly specify in their terms that it won't be completely free in the future. The price of their same product if accessed from the web is $100/month for the amount (barely) sufficient for book reading. It may be smarter to skip until they reveal their official pricing for the mobile app.
*TickBlock* replaces QKᵀ with a learnable banded positional operator (“tensor mode”). On Tiny Shakespeare, it matches GPT-2-small-level performance with only 0.64M parameters (≈0.5% of the size). It trains in ~12 minutes on a Mac laptop — no kernel optimizations yet.
Repo: https://github.com/projectbelgrade/tickblock
The design is motivated by my research in fundamental physics, which reinterprets relativity without tensors. That work (Project Belgrade) models reality as a sequence of discrete publishing events and photons as standing sheets of information: [Structure and Mechanics of Reality: Project Belgrade](https://doi.org/10.5281/zenodo.17191747)
Right now the results are small-scale, but the efficiency gains suggest a bigger question:
- With kernel optimization, parameter sweeps, and compression, how far can this scale? - Could we see GPT-4-class assistants running fully on laptops? GPT-3.5-class reasoning on phones? - If that became practical, what would be the most compelling on-device applications?
Curious to hear how the community thinks about the tradeoffs and opportunities.