TickBlock: GPT-2 performance at 0.5% size, trained on a Mac, Physics-inspired(github.com)

1 pointsby ivan_icin10 เดือนที่ผ่านมา1 comments

TickBlock: GPT-2 performance at 0.5% size, trained on a Mac, Physics-inspired

https://github.com/projectbelgrade/tickblock

2 comments

ivan_icin10 เดือนที่ผ่านมา

I’ve been working on a physics project that unexpectedly led to a new transformer architecture.

*TickBlock* replaces QKᵀ with a learnable banded positional operator (“tensor mode”). On Tiny Shakespeare, it matches GPT-2-small-level performance with only 0.64M parameters (≈0.5% of the size). It trains in ~12 minutes on a Mac laptop — no kernel optimizations yet.

Repo: https://github.com/projectbelgrade/tickblock

The design is motivated by my research in fundamental physics, which reinterprets relativity without tensors. That work (Project Belgrade) models reality as a sequence of discrete publishing events and photons as standing sheets of information: [Structure and Mechanics of Reality: Project Belgrade](https://doi.org/10.5281/zenodo.17191747)

Right now the results are small-scale, but the efficiency gains suggest a bigger question:

- With kernel optimization, parameter sweeps, and compression, how far can this scale? - Could we see GPT-4-class assistants running fully on laptops? GPT-3.5-class reasoning on phones? - If that became practical, what would be the most compelling on-device applications?

Curious to hear how the community thinks about the tradeoffs and opportunities.

[deleted]10 เดือนที่ผ่านมา