Quantization — How LLMs Fit on Your Laptop Without Losing Their Mind

A 7B parameter model in full precision (FP32) needs about 28 GB of RAM just to load.

Most people don't have 28 GB of GPU memory lying around. Yet somehow, people are running 7B models on MacBooks and gaming laptops. Some are even running 13B models.

How? Quantization.

---

What Is Quantization?

Quantization is the process of storing and computing a model's weights at lower precision so the model takes less memory and runs faster.

The simplest version: instead of storing a weight as a 32-bit decimal like 0.7319284, you store it as an 8-bit integer like 94.

0.7319284  →  94
(FP32)        (INT8)

Memory saved: 4 bytes → 1 byte per weight. 4× smaller.

You're not throwing away the model. You're compressing it — the way JPEG compresses a photo. Some precision is lost, but the result is still useful.

---

Why Does This Actually Make Things Faster?

Two reasons most explanations skip:

Memory bandwidth wins. Moving 1 byte is 4× faster than moving 4 bytes. On a GPU or CPU, the bottleneck during inference is usually not the math — it's how fast data moves from memory to the compute units. Smaller numbers = faster data movement.

Integer math is faster than decimal math. CPUs and GPUs have dedicated integer units that run faster and use less power than floating-point units. INT8 operations are significantly faster than FP32 on modern hardware.

Combined: quantized models are faster, cheaper, and more power-efficient — without changing the architecture at all.

---

The Core Mechanism: Scale and Zero-Point

You can't just round 0.7319284 to 94 and hope for the best. You need a way to go back. That's what scale and zero-point are for.

Scale — how much real value each integer step represents. The size of one step.

Zero-point — which integer represents the real value 0.0. Needed when the weight range is asymmetric.

Quantization formula (real → integer):

quantized_int = round(real_value / scale) + zero_point

Dequantization formula (integer → real):

real_value = (quantized_int - zero_point) × scale

---

Worked Example — Symmetric Range

Say a weight is 0.7319284, INT8 range is 0–255, scale is 0.007787:

q = round(0.7319284 / 0.007787)
  = round(93.9963...)
  = 94

To read it back:

real ≈ 94 × 0.007787 ≈ 0.7319

Tiny rounding error. Acceptable.

---

Worked Example — Full Range with Zero-Point

Suppose model weights range from -1.0 to +1.0, mapped to INT8 (0–255):

Step 1 — compute scale:
  full range = 1.0 - (-1.0) = 2.0
  scale = 2.0 / 255 ≈ 0.00784

Step 2 — zero-point (maps 0.0 to middle of INT8):
  zero_point = 128

Step 3 — quantize weight = 0.5:
  q = round(0.5 / 0.00784) + 128
    = round(63.7) + 128
    = 64 + 128
    = 192

Step 4 — dequantize back:
  (192 - 128) × 0.00784 ≈ 0.502  ✓ (tiny gap)

The gap from 0.5 to 0.502 is the quantization error. It's real, but small. Models are surprisingly robust to it.

---

Asymmetric Ranges Need Careful Zero-Points

When the weight range isn't symmetric — say -3.0 to +1.0 — the zero-point shifts:

scale = (1.0 - (-3.0)) / 255 = 4.0 / 255 ≈ 0.01569

zero_point = round((0 - (-3.0)) / 4.0 × 255)
           = round(3/4 × 255)
           = round(191.25)
           = 191

Quantize x = -1.2:
  q = round(-1.2 / 0.01569) + 191 = round(-76.5) + 191 = 209

The zero-point formula for asymmetric ranges:

zero_point = round((0 - min) / (max - min) × 255)

This is what makes asymmetric quantization tricky to implement but more accurate — it uses the full integer range even when weights aren't centered around zero.

---

Per-Tensor vs Per-Channel vs Per-Batch

One scale and zero-point for the whole tensor isn't always the best idea. Here's how the three approaches differ:

Per-tensor quantization One scale + one zero-point for the entire weight matrix. Simple and fast. But if one row has weights ranging 0.001–0.002 and another ranges 5.0–100.0, the same scale crushes the small values into indistinguishable integers.

Per-channel quantization Each row (channel) gets its own scale and zero-point. A tiny row gets a fine-grained scale. A large row gets its own scale. Much better accuracy — this is what most modern quantization tools use for weights.

Tensor:
Channel 0: [0.2, -0.4, 0.7, 0.1, 0.5, -0.2]  → scale_0, zp_0
Channel 1: [-1.1, 0.3, 0.8, 0.4, -0.6, 0.2]  → scale_1, zp_1

Each channel gets its own ruler. Better accuracy, slightly more metadata to store.

Per-batch quantization Used for activations (the intermediate numbers flowing between layers during inference), not weights. One scale is computed dynamically across the current batch. Precision adapts on-the-fly to whatever inputs the model is currently processing. Common in INT8 activation quantization and TensorRT.

per-channel  →  each row/channel gets its own scale
per-tensor   →  entire layer shares one scale
per-batch    →  all items in a batch share one scale (dynamic, for activations)

---

PTQ vs QAT

Two ways to quantize a model — and they're very different in cost and quality:

Post-Training Quantization (PTQ) Take an already-trained model, quantize its weights, done. No retraining needed. Fast, cheap, needs only a small calibration dataset.

Downside: Rounding errors accumulate in ways the model wasn't trained to handle. Accuracy drops slightly, more noticeably at aggressive bit depths (4-bit).

Quantization-Aware Training (QAT) Simulate quantization during training. The model sees fake rounding errors as it learns and adjusts its weights to stay accurate despite them. More expensive — needs full training data and compute.

Upside: The model adapts to quantization. Best accuracy at low bit widths.

┌─────────────┬──────────────────┬──────────────────────┐
│             │ PTQ              │ QAT                  │
├─────────────┼──────────────────┼──────────────────────┤
│ When        │ After training   │ During training      │
│ Cost        │ Low, fast        │ High, needs full data│
│ Data needed │ Small samples    │ Full training set    │
│ Accuracy    │ Good             │ Best                 │
│ Ease of use │ Very easy        │ Harder               │
└─────────────┴──────────────────┴──────────────────────┘

For most use cases — deploying an existing open-source model — PTQ is the right call. QAT is for when you're training from scratch and want to ship an optimized model.

---

Weight-Only vs Weight + Activation Quantization

A model has two kinds of numbers during inference:

• Weights — fixed after training, stored on disk
• Activations — dynamic numbers produced by each layer as input flows through

Weight-only quantization: Quantize the weights (e.g. to INT4/INT8), but keep activations in FP16 or FP32. Saves memory massively. Most consumer-facing quantized models work this way.

Weight + Activation quantization: Quantize both weights AND activations to INT8. Gives more speed because the actual matrix multiplications run in INT8. More complex to implement. Common for edge inference and production deployments (think TensorRT, ONNX).

---

The Popular Tools and Formats

GPTQ (Generative Pre-Trained Quantization) — PTQ method that quantizes layer by layer, 4-bit. Works well for large models. Most quantized model releases on HuggingFace use GPTQ.

AWQ (Activation-Aware Weight Quantization) — smarter than GPTQ. Identifies which weights are most important to a model's accuracy by looking at activations, then gives those weights finer treatment before quantizing. Keeps 4-bit size with better accuracy.

BitsAndBytes — the easiest to use library. Load any HuggingFace model in 4-bit or 8-bit with load_in_4bit=True. Hugely popular for quickly trying quantized models without converting anything.

GGUF / llama.cpp — format + runtime for running quantized models on regular CPUs (your MacBook, a gaming PC, no GPU needed). The reason you can run LLaMA-3 8B on a laptop. Llama.cpp handles the inference engine.

Use case → right tool:
  Quick experiments on GPU     → BitsAndBytes
  Best 4-bit quality on GPU    → AWQ
  Standard 4-bit releases      → GPTQ
  Running on CPU / edge device → GGUF + llama.cpp

---

Key Takeaways

• Quantization = store weights at lower precision (INT8, INT4) to save memory and run faster
• Scale and zero-point are the two parameters that let you compress and decompress without losing the meaning
• Per-channel quantization is more accurate than per-tensor; per-batch is used for activations
• PTQ is fast and good enough for most use cases; QAT is better but expensive
• Weight-only quantization saves memory; weight + activation quantization gives actual compute speedups
• GPTQ and AWQ for GPU inference, BitsAndBytes for quick experiments, GGUF for CPU/edge

---

The gap between "this model needs an A100" and "this model runs on my laptop" is mostly quantization. It's one of the most practically important ideas in the entire LLM deployment stack — and now you understand exactly how it works, number by number.

---

Part of an ongoing series on how LLMs actually work under the hood.