Quantization
undoes alignment.
Shrink an instruction-tuned LLM from 16-bit to 3-bit and its perplexity barely flinches. Its fairness does not survive. Between 6 and 21% of items it had answered perfectly develop brand-new stereotypes, and standard quality metrics never see it coming. This piece walks through how, and why.
Based on Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels (Rath & Maliakkal, 2026). 3 models × 5 precision levels × 12,148 BBQ items × 5 seeds = 911,100 inference records.
What quantization actually does
Before the bias story: quantization maps each weight onto a small grid of representable values. The trap is that one large outlier per group sets the spacing for everyone, so the small weights get crushed together. Drag the dial and watch them collapse.
At this precision you can count the horizontal levels. Every weight must snap to one of them, and the small weights collide onto the same few lines.
Dropping precision is not a gentle blur. Each bit you remove halves the number of representable values: 8-bit gives 256 levels, 4-bit gives 16, 3-bit gives just 8. Every weight in the group has to round to one of them.
And the spacing is not the model’s to choose. A single large outlier stretches the group’s range, so the step size grows and the many small weights all snap onto the same handful of lines. Those small weights are where the model’s learned restraint lives, the ability to recognise an unanswerable question and abstain. Coarsen them and that restraint is the first thing to go.
The test: questions with no answer
We use the ambiguous half of BBQ, where the context genuinely does not say who. The correct answer is always “unknown,” so any demographic pick is, by construction, stereotyping. Click around an item to see how the two metrics are scored.
BBQ’s ambiguous condition only. 12,148 items across these five categories. The context is always insufficient, so the right answer is always “unknown.” Click an answer as if you were the model.
The dose-response
Five precision levels, not two. That is what lets you see a dose-response instead of a single before/after snapshot. Stereotyping rises monotonically as bits fall, and the models pull apart sharply.
Stereotype Reliance Score against precision, one line per model. The overall mean across all 911,100 records moves 0.175 → 0.242 (37.8% relative). illustrative
First, the climb is monotonic for every model: more aggressive quantization means more stereotyping, a clean dose-response. Second, the models are not equal. Drag to Q3 and Mistral pushes past the random-guessing line, meaning it is now actively selecting stereotypes more often than chance, while Qwen stays well below at every level.
The emergence of new biases
This is the result everything else builds to. Take only the items the model got perfectly right at full precision, then drop the bits and watch clean items light up as newly biased. This is not amplification of an existing tendency. It is new behaviour.
Mistral-7B-Instruct-v0.3 · 312 cells stand in for the 8,642 items this model got perfectly right at full precision (SRS = 0 across all 5 seeds). A lit cell is an item that has since developed a new stereotype. from paper
These are not items the model was “a bit biased” on and got worse. At full precision it never once picked the stereotype, across five seeds. The bias is genuinely new, manufactured by the compression itself as quantization noise erodes the alignment that was suppressing the pretraining prior.
The baseline is sharply bimodal: 71–92% of items sit at SRS = 0 and 8–27% at SRS = 1, with almost nothing in between. Counter-intuitively, the already-biased items actually improve under Q3 (0.958 → 0.706): compression scrambles every learned pattern. But far more clean items break than broken ones heal, so the net direction is worse.
The evaluation gap
Here is why nobody catches it. The metric teams actually check, perplexity, stays nearly flat while the fairness damage races ahead. Same model, same dial, two completely different stories.
Mistral-7B-Instruct-v0.3. The quality metric a practitioner actually checks (perplexity) against the fairness damage they never see (new biases). Both on one scale. from paper
At Q3, perplexity moved +10.20% (a number most teams would wave through), yet 17.7% of previously clean items have developed new stereotypes. The fairness signal is moving 1.7× faster than the quality signal. The paper frames Mistral at 3-bit as a 173× gap between the two signals. Aggregate perplexity gives false assurance that the compressed model is behaviourally equivalent to the original. It is not.
The model stops saying ‘I don’t know’
The mechanism behind the bias is a loss of epistemic humility. As precision drops, the model’s willingness to abstain falls in lockstep with its rising stereotype reliance. It is not becoming more biased; it is losing the ability to withhold judgement.
Unknown Selection Rate: how often the model is willing to say “I cannot tell.” The overall mean falls 0.764 → 0.631 (17.4%) as precision drops. illustrative
Since BF16, USR has moved −0.14. That mass did not split evenly: stereotype +0.16, anti-stereotype −0.02. The abandoned “unknown” mass flows overwhelmingly toward the stereotype, not the other way. That asymmetry is the whole point of tracking both metrics.
The two meters move in opposite directions in lockstep. As compression eats the model’s ability to recognise that a question is unanswerable, it stops abstaining and falls back on the strongest thing it has left: the statistical prior from pretraining. For demographic questions, that prior is the stereotype. The model is not acquiring new beliefs. It is losing the nuance to say “I don’t know,” which is exactly what instruction tuning had taught it.
The average hides the damage
Population-level effect sizes look negligible. But that average is diluted by the items that never change. Restrict to the susceptible ones and the effect leaps from negligible to medium-large. Toggle between the two views.
Cohen’s h (BF16 vs Q3), averaged over model × category. Toggle to the items that already had a weak tendency at full precision (per-item SRS ≥ 0.2). from paper
The subset is simply the items whose per-item SRS was already ≥ 0.2 at full precision: the model slipped toward the stereotype on at least one of its five seeds before any compression. Those are the items a quantization nudge tips over. Two examples, where each dot is one seed.
Across all items the effect is only 0.179, technically significant but negligible in size. That is the headline most aggregate evaluations would report, and it is diluted by the huge majority of items that never change. Restrict to the items that were actually susceptible and the effect jumps to 0.742, medium-to-large, with every single comparison significant. The damage is real and concentrated; the average just hides it.
A logistic regression across all 911,100 responses confirms the trend: bit-width predicts stereotype-consistent answers with β = -0.0205 (< 0.0001), controlling for category. Each bit you strip away systematically tilts the model toward the stereotype.
What this means
Aggregate quality metrics give false assurance that a compressed model is equivalent to the original. Evaluating fairness under compression needs its own protocol.
Compression is never just compression. Quantization does not add a thin, uniform layer of noise across a model’s behaviour. It selectively dissolves the epistemic calibration that instruction tuning installed, and the model reverts to pretraining-era priors. For demographic questions, those priors are stereotypes.
The uncomfortable part is the timing. The fairness damage arrives before the quality damage is visible. By the time perplexity tells you something is wrong, 6 to 21% of your previously clean items have already broken. Any pipeline that gates on aggregate quality alone is shipping that breakage with a clean bill of health.
The same evidence points to a fix. USR declines smoothly and the per-item SRS distribution is bimodal, which suggests the parameters responsible for abstention are a small, identifiable set. A mixed-precision scheme that keeps those groups at higher precision while aggressively compressing the rest could buy most of the memory win without the 17.4% collapse in epistemic humility. Failing that, a short post-quantization fine-tune on ambiguous prompts may restore the calibration compression strips away.
Found this useful?