Research & Experiments
Technical trinkets, paper implementations, and experiments.
#research#ml
This section is dedicated to technical "trinkets" and paper implementations.
Implementations
- Attention Residual Connections
- summary from kimis attention residual notes, convert qwen 3.5 from standard residual connections to blocked attention residuals
Experiments
- mnesh: training bugs and lessons learnt
- post-mortem on mnesh, a hierarchical rnn that predicts your next terminal command from shell history — the bugs, the actual training logs, and the one lesson that kept biting
- Parameter Golf
- notes and logs from the parameter-golf challenge: squeezing bpb out of a tiny transformer under 16mb, where the bottleneck turned out to be architecture, not compute