Explain quantization for inference: INT8/INT4, GPTQ/AWQ, what breaks, and how you validate it.

TL;DR: Quantization converts weights/activations from FP16 into lower-precision integers (INT8, INT4), cutting memory and accelerating memory-bound inference. Plain rounding breaks down when a handful of large-magnitude outliers inflate the scale; GPTQ and AWQ address this (error-compensated rounding, guarding salient weights). Weight-only quantization is safe and widespread; activation quantization is tougher. Always confirm on your eval set, not perplexity alone.

How to approach it

Say why you quantize (fitting memory, plus speedups on bandwidth-bound decode), then the core difficulty (dynamic range and outliers), then the methods and the validation step, since "did quality actually hold" is the real question. Candidates who pass point to outliers as the failure mode and to a task eval, not perplexity, as the proof.

A strong answer

Why. Weights at FP16 are 2 bytes each; INT8 halves that, INT4 quarters it. Beyond fit, LLM decode is memory-bandwidth-bound (you stream weights per token), so reading fewer bytes per weight directly speeds generation. That is why a quantized 70B can serve on one GPU and run faster.

The hard part: dynamic range. Quantization maps a float range to a small integer grid via a scale. If a tensor has a few huge-magnitude outliers, the scale stretches to cover them and the many normal values collapse into a couple of buckets, destroying precision. Naive round-to-nearest at INT4 wrecks accuracy for exactly this reason.

Methods.

• Weight-only PTQ (GPTQ, AWQ). GPTQ quantizes weights greedily while compensating for the error introduced, using second-order (Hessian) information; AWQ identifies and protects the salient weight channels that matter most for output. Both are post-training (no retraining), need only a small calibration set, and hold quality well at INT4 for many models.
• Activation quantization (INT8 weights+activations, SmoothQuant) is harder because activation outliers are large and input-dependent; SmoothQuant shifts difficulty from activations into weights to make it tractable.
• FP8 (on newer hardware) keeps floating-point dynamic range at 8 bits and is increasingly used for both training and inference.
• QLoRA is the training-side cousin: fine-tune adapters on top of a 4-bit frozen base.
• Sub-4-bit PTQ exists, so do not claim QAT is the only route below INT4. QuIP and QuIP# push post-training weight quantization to two bits with incoherence processing and lattice codebooks, and AQLM uses additive quantization for the same range. Whether you should ship any of it is a separate question from whether it is possible: expect real quality loss, verify on your own tasks, and check that fast kernels exist for the format before you count the memory saving.

Validation. Perplexity alone is a weak proxy; run your task eval set (accuracy, exact-match, judge scores) and compare to the FP16 baseline. Quantization error is non-uniform across capabilities, so a model can hold perplexity yet regress on reasoning or code. Keep sensitive components (sometimes the embedding/output layers, or attention) at higher precision if needed. The defensible position: weight-only INT4 (AWQ/GPTQ) is a safe default for serving, validated on real tasks; push to activation or INT4-everything only with measurement.

Key takeaways

• The failure mode is outliers stretching the scale, not "fewer bits" in the abstract; GPTQ/AWQ exist to fix it.
• INT8 is near-lossless; INT4 weight-only is the workhorse serving default; below INT4 needs either QAT or a method built for it, plus proof on your tasks.
• Activation quantization is strictly harder than weight-only because activation outliers are input-dependent.
• Validate on a task eval against the FP16 baseline, not perplexity alone, because errors hit capabilities unevenly.

What interviewers probe next

• "Why does INT4 help latency, not just memory?" Decode is bandwidth-bound; fewer bytes per weight means less memory traffic per token.
• "Per-tensor vs per-channel/group scales?" Finer-grained scales (per-channel, group-wise) handle varying ranges far better than one global scale, at a small overhead.
• "QAT vs PTQ?" Quantization-aware training recovers more accuracy at very low bits but costs a training run; PTQ (GPTQ/AWQ) is cheaper and usually enough at INT8/INT4.
• "How low can you go?" INT8 is nearly lossless for most models; INT4 is usually fine with good methods; below that quality degrades fast without QAT.

Common mistakes

• "Just use fewer bits" with no mention of outliers or dynamic range, the thing that actually breaks.
• Validating on perplexity only and missing task-specific regressions.
• Treating activation quantization as easy as weight quantization.
• Calibrating on unrepresentative data, so the scales are wrong for production inputs.