Measuring impact of KV quantization on Qwen3.8

I’ve been looking for numbers to quantify the impact of KV cache quantization on Qwen3.8 27B. I’ve come across a lot of details for Qwen3.6 and other models, but not so much on 3.8.

The general consensus is that KV quantized to Q8_0 is quite acceptable, and that quantizing further is problematic, more so on K than on V.

I am summarizing my findings first for brevity, and I will expand on assumptions, methodology and data later.

Summary of results

• For KV quantization, Q8 is fairly close to F16, and distinctly better than Q4.

• Dropping from Q4 to Q3 weight quantization (0.07 vs 0.19 mean KLD) is worse than aggressively quantizing cache KV at same weight quantization (0.09 mean KLD at Q4 weights, Q4 KV)

Assumptions

• Assuming Q8_K_XL weight quantization model with f16 dtype KV cache is the source of truth. According to Unsloth blog post for the model, this should be close to ~0.001 mean KLD to the unquantized model, which makes be believe this is not a big assumption. This model size is practically the biggest I can run on my machine.

Methodology

• Using llama-perplexity (commit 465e49b9c) to compute probability/surprise distributions
• Baseline is from Unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf
• Corpus is prose/narrative heavy, conversation, ~450K words
• Context is 20K tokens

Limitations

• The corpus is prose / narrative heavy, results on STEM/code corpus might differ significantly
• Small context window (20K tokens), only 10K have their surprise distribution evaluated, same tokens for all run variations
• No qualitative assessment
• Not testing against a benchmark of known answers
• I am no expert, likely unknown unknowns :)

Discussion

Generally, for KV quantization, F16 and Q8 are fairly close. Q4 departs meaningfully from those two, see the diagonal in the table below.

Q4_K_XL mean KLD by KV quantization

Q4_K_XL same top-1 (%) KV quantization

Quantizing K and V separately at different dtypes is not as obvious. The error bars are large enough that we shouldn’t read too much into adjacent cells. Frankly, after looking at the data, I am less worried of running K at Q8 and V at Q4.

On Q3 weight quantization model, the probability distribution is already so distinct from the base model, that KV quantization is a much less dominant factor, than in the Q4 weight model.

That is to say, if running Q4 weight quant is an option, deciding on KV quantization for Q3 is not a good use of time.

Q3_K_XL mean KLD by KV quantization

Q3_K_XL same top-1 (%) KV quantization

I don’t have results for higher bit per weight quantizations, so it’s hard to know if this holds at other quants. If it does, one is better off running the least quantized weight model with an aggressive KV quantization than the other way around.

Results

Q4_K_XL vs Q8_K_XL (17.56 GB weights):

Q3_K_XL vs Q8_K_XL (13.15 GB weights):

What next

• Measure Q5 and Q6 weight quantization and compare weight vs KV cache quantization impact
• Vary the corpus
• Larger context window