q2 quantization is kinda cool

i spent the last few days studying some inference topics, the baseten inference engineering book has been a massive help but most of my studying has been on reading antirez's d24 inference engine which is a DeepSeek 4 Flash local inference engine for Metal, insanely cool and well written project, i love it, i fucking love it, well written, well documented, beautiful c

immediately you open the ds4.c file and scroll a bit you're hit with these numbers

static const uint64_t iq2xxs_grid[256] = {
    0x0808080808080808, 0x080808080808082b, 0x0808080808081919, 0x0808080808082b08,
    0x0808080808082b2b, 0x0808080808190819, 0x0808080808191908, 0x08080808082b0808,
    0x08080808082b082b, 0x08080808082b2b08, 0x08080808082b2b2b, 0x0808080819080819,
    ...

this can be found here -- https://github.com/antirez/ds4/blob/920f9872ec98602e899c1f0bfec9f2f0f6103017/ds4.c#L236 and the project ican be found here -- https://github.com/antirez/ds4/tree/main

my stupid ass first thought this was some mmap thing but after some investigation i found out this was a q2 quantization technique - the codebook trick they call it and before you know it i was spiralling down quantization techniques but the q2 trick antirez used stuck with me because it's pretty cool and i wanna try to explain it to you because i spent all night reading on this and i think i'm the best person to do this job

simple linear quantization is cool, it actually is really cool, impressive results, if a model is shared in half precision fp16 you can q4 the guy and that makes the model 4x smaller, cool, very cool, insanely cool, it's just that linear quantization can't work for q2 basically because we're reaching a very coarse zone, q4 means 4 bits for every weight whichhhh is still pretty good, but q2 is just 2 bits, it's too stiff, using linear quantization at q2 you're almost guaranteed to make your model stupid. so a new technique is used, called the codebook trick, and from my understanding this trick has sacrificed decoding the exact weights back for the overall shape of the weights, which will still work in practice

don't take my word on this but understanding this reminded me of my brief study of the Johnson–Lindenstrauss (JL) Lemma which basically explicitly implies that precision can be sacrificed for relationship — in our case shape. pls don't take my word on this

side note: the JL lemma is from the 80s (i did a quick revise for this paper) and was originally about geometry and dimensionality reduction, the fact that it keeps showing up in ML is one of those things that makes you appreciate how deeply math connects everything

how does this codebook trick work?

trying to convert the storage a billion fp16 numbers will take to be 8x smaller is a crazy feat and this codebook trick kinda does it. it's more of pattern finding here, we're not targeting precise values, we're trying to retain structure and local shapes.

assuming you have a weight of 32 numbers

float weights[32] = { 
    0.01, 0.02, 0.01, 0.00, 1.5, 1.7, 1.6, 1.8,
    -0.01,-0.02,-0.01, 0.00,-1.4,-1.5,-1.6,-1.7,
    0.2 , 0.25, 0.3 , 0.28, 0.22, 0.27, 0.26, 0.24,
    2.1 , 2.0 , 2.2 , 2.3 , 0.1 , 0.0 , 0.1 , 0.0
};

in linear quantization we'd try to compress each value individually

but q2 gets insanely hard because now every weight only has 4 possible values — 2 bits = 4 combinations and that is terrible for precision

so instead of preserving individual values, codebook quantization preserves local structure. the trick is grouping weights together into small clusters

32 weights -> 4 clusters -> 8 weights per cluster

cluster 0 becomes

[
    0.01, 0.02, 0.01, 0.00,
    1.5, 1.7, 1.6, 1.8
]

notice something interesting here, the actual values are different, but the shape is obvious

tiny tiny tiny tiny
BIG  BIG  BIG  BIG

and apparently transformers tend to repeat these kinds of patterns everywhere. so instead of storing the 8 weights individually, we try finding the closest "known pattern"

this is where the codebook comes in — those numbers i mentioned earlier that can be found here

at first glance these numbers look completely insane

0x0808080808080808
0x080808080808082b

but they're actually much simpler than they look, these are hexadecimal numbers, and each pair of hex digits represents 1 byte

08 08 08 08 08 08 08 08

is literally just:

[8, 8, 8, 8, 8, 8, 8, 8]

and

08 08 08 08 08 08 08 2b

becomes:

[8, 8, 8, 8, 8, 8, 8, 43]

because 0x2b = 43

suddenly the codebook starts looking less like black magic and more like actual patterns. the first pattern:

[8, 8, 8, 8, 8, 8, 8, 8]

basically represents a flat structure, everything has roughly similar magnitude. while the second one:

[8, 8, 8, 8, 8, 8, 8, 43]

represents something closer to:

small small small small
small small small BIG

and apparently transformer weights repeat these kinds of local structures constantly. that's why this works at all.

the really interesting thing is that these aren't "real weights", they're more like compressed shape templates. it's almost like this starts feeling less like "number compression" and more like signal compression or image compression

note: this is conceptually really close to how vector quantization works in image codecs like VQ-VAE. you're not storing the image, you're storing an index into a learned dictionary of patches. quantization for LLM weights is basically doing the same thing, just in weight space instead of pixel space

at this understanding i thought i'd visualize the IQ2_XXS codebook values and gain some insight from that and yh some insight was really gained. the small magnitudes are darker while the larger are brighter

from bottom to top and left to right it's obvious that we're going from simple structures to complex patterns. i also kinda noticed that this looks somehow like an image kernel but i'll let that slide lol

codex wrote me the script btw

i only visualized for this codebook family, would probably wanna see some others later

even after finding the closest pattern, we're still missing two really important things: the sign of each weight, and the actual magnitude scale of the cluster. because the codebook only stores rough magnitudes and local structure, a pattern like:

[8, 8, 8, 8, 43, 43, 43, 43]

doesn't tell us whether the real weights were positive or negative. it also doesn't tell us whether the cluster originally looked like:

[0.01, 0.02, 0.01, 1.5]

or:

[10, 12, 11, 150]

same shape. completely different scale. so quantization stores two extra pieces of information alongside the codebook index: a pattern index, a sign mask, and a scale.

sign mask

the sign mask is actually pretty clever. since each cluster has 8 weights, we store 8 bits — 1 bit per weight:

00111100

each bit represents the sign of a reconstructed weight, where 0 = positive and 1 = negative. so if our codebook pattern was:

[8, 8, 8, 8, 43, 43, 43, 43]

and our sign mask was 00111100, we'd reconstruct something closer to:

[+8, +8, -8, -8, -43, -43, +43, +43]

the actual values still aren't correct yet, but now the local structure and sign information are preserved.

note: storing 8 sign bits per cluster of 8 weights is essentially free overhead. you're already using most of your bit budget on the codebook index and scale. the sign mask is elegant because it costs almost nothing but recovers a massive amount of structure

scale

this part is actually very similar to normal linear quantization. we first look at the cluster magnitudes and figure out roughly how "large" this cluster is. for example this cluster:

[0.01, 0.02, 0.01, 0.00, 1.5, 1.7, 1.6, 1.8]

has a largest magnitude around 1.8, while the largest value inside the matched codebook pattern is 43. so we compute something roughly like:

scale = 1.8 / 43  ≈  0.041

this scale basically tells the decoder: "these template values are too large, shrink them down by 0.041x."

decoding at inference

decoding becomes surprisingly straightforward. during inference the model streams quantized weights from memory and reconstructs approximate fp16/fp32 weights on the fly. for every cluster, it reads the codebook index, looks up the pattern, applies the sign mask, multiplies everything by the scale, and feeds the reconstructed weights into the matrix multiplication.

so our pattern after scale multiplication:

[8, 8, 8, 8, 43, 43, 43, 43]  ×  0.041

becomes:

[0.328, 0.328, 0.328, 0.328, 1.763, 1.763, 1.763, 1.763]

which vaguely resembles the original cluster shape — tiny tiny tiny tiny, BIG BIG BIG BIG. and that's really the important thing here.

the actual bit budget

so let's just sanity check what we're actually storing per cluster of 8 weights, because this is the part that genuinely surprised me when i sat down and counted

- codebook index : 8 bits  (picks 1 of 256 patterns)
- sign mask      : 8 bits  (1 bit per weight)
- scale          : 16 bits (fp16 or a compressed float)

that's 32 bits total for 8 weights. 32 bits / 8 weights = 4 bits per weight, which is still q4 technically, but the kind of information stored is completely different from vanilla q4. instead of 4 bits trying to encode one number's value, you've got 32 bits encoding a shared template, the sign of each weight, and the cluster's magnitude in one shot.

the tradeoff is real though. when you decode a weight you're getting a weight that's shaped like the original and signed correctly, but the fine-grained numerical precision is gone. whether that matters depends entirely on the layer, the model, and how sensitive that particular weight cluster is to noise.

why it works

the model doesn't necessarily need exact weights - it needs enough of the statistical structure preserved for the transformer to behave similarly. at ultra-low-bit quantization we're no longer preserving exact numbers. we're preserving repeating local patterns in the learned weight space, which is honestly what makes this entire thing feel more like compression engineering than traditional numerical approximation.

and that was most of it, q2 quantization in a nutshell and what i could remember from a night's study session thank you for reading my simple explanation