quantization

Fine-tuning a 65B model once demanded a node of A100s. QLoRA compresses it onto one card with three specific tricks. The signal is knowing what gets quantized, what stays trainable, and why quality barely shifts. Here is the answer.

At long context it is the KV cache, not the weights, that saturates the GPU. What matters is naming the levers (quantization, token eviction, head sharing, low-rank) and the quality cost of each.

Both compress an LLM to 4-bit weights, yet they choose what to protect in very different ways. What matters is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each demands.

GGUF powers most local LLM runs, and its quant names look cryptic. What matters is decoding what Q4_K_M means and why mixed-precision k-quants outperform naive uniform quantization.

FP8 underpins modern H100-class training and serving. What matters is knowing the two FP8 variants, why a floating format beats INT8 on dynamic range, and where scaling still counts.

At a billion vectors the index choice is a memory budget question before it's a recall question. Flat search is off the table, HNSW may not fit, and PQ swaps recall for RAM. Here is the call a staff engineer makes on the whiteboard.

float32 is far more precision than ranking needs. int8 is close to free, binary is 32x smaller, and the difference between a working binary index and a recall disaster is one word: rescoring.

On-device AI gives up raw capability for privacy, offline use, and latency, under tight memory and battery limits. The signal is the small-quantized-model stack plus a hybrid router that sends the hard queries to the cloud without leaking private context.

The trap is leaping straight to 'add more GPUs.' The signal is profiling first, then reaching for the cheap, no-retrain levers in the correct sequence. Here is the diagnose-then-optimize playbook for p99.

Interviewers here want concrete figures, not 'grab a bigger GPU.' Weight memory stays constant, the KV cache scales with traffic, and the order you pull levers in settles the outcome. This walks through the napkin math and the serving stack.

Quantization is the opening move for shrinking and accelerating models, and the interviewer expects more than 'use fewer bits.' What they grade is whether you know what each precision level gains you, why outliers wreck naive quantization, and how you demonstrate quality survived. This lays that out.

Distillation gives you a small fast model that retains most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

On-device AI is a genuine product surface (privacy, offline, latency), and it imposes hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.

Past FP16/BF16, FP8 sits on the next rung of the precision ladder, and the newest tensor cores compute on it natively. What counts is knowing the two formats, what FP8 really buys, and why dynamic range demands careful scaling.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. What counts is spotting it as the binding constraint and naming the levers that shrink it without wrecking quality.

Pruning strips redundant weights to make a model smaller and sometimes faster. The signal is the structured-versus-unstructured divide and why only structured pruning dependably cuts latency on commodity hardware.

Naive 4-bit quantization can wreck quality, and the reflex to give up and serve fp16 leaves a large speedup unclaimed. The accuracy is usually recoverable. Here is the ladder of fixes.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.