Quantization

Quantization represents numbers using a limited set of values. In model deployment, that can reduce the storage and memory traffic of weights or activations, and sometimes allow faster arithmetic. Integer quantization, such as replacing float32 weights with int8 values, is one common case. Weight-only quantization, quantized activations, and lower-precision floating-point formats make different choices about what is stored and how computation proceeds.

From a real number to an integer grid

An affine integer quantizer uses a positive scale s and an integer zero point z. Encoding computes q = clip(round(x / s) + z, qmin, qmax); reconstruction computes x̂ = s(q − z). The scale is the distance between adjacent reconstructed values, the zero point identifies the integer representing real zero, and the clipping limits are the smallest and largest supported integers. This mapping is described in the ONNX Runtime quantization documentation.

For a hypothetical signed int8 grid with s = 0.1, z = 0, and codes from −128 to 127, the real value 0.26 rounds to code 3 and reconstructs as 0.30. Its error is 0.04. The representable range is −12.8 through 12.7; an input of 20 clips to 127 and reconstructs as 12.7, giving an error of 7.3. Rounding within the covered range is bounded by half a step under nearest rounding. Clipping outside that range can produce much larger error. Integer encoding therefore does not make arbitrary real values exactly recoverable.

A symmetric scheme centers its represented values around zero, often with zero point 0 for signed storage. An asymmetric scheme shifts the grid to use its codes across a range that is not centered on zero. Implementation conventions can reserve endpoints or choose slightly asymmetric signed ranges. What matters when reading an artifact is its actual scale, zero point, bounds, and rounding rule; bit width alone does not specify the mapping.

Why ranges and outliers matter

With a fixed number of codes, covering a wider range requires a larger step. Suppose unsigned 8-bit values span 0 to 1: 256 levels give a step of 1/255, about 0.00392. Expanding the same grid to cover 0 to 100 makes the step about 0.392. Most values near zero now receive much coarser resolution because of a distant extreme. Clipping the range can protect common values at the cost of distorting the extremes; the right compromise depends on their contribution to the task.

A per-tensor scale shares that compromise across the whole tensor. Per-channel scales, for example one for each output channel’s weights, let channels with different ranges use different grids. Group-wise or block-wise schemes share parameters over smaller groups. Finer groups can isolate outliers but add scale metadata and require compatible kernels. An outlier can still damage the resolution of values in its own group. These choices change both numerical error and the work required to use the representation.

Weights, activations, and arithmetic

Weights are the learned parameters of a model, described in model weights. Their ranges can be inspected after training. Activations are intermediate results that depend on incoming data, so their ranges must be estimated from examples or determined during inference. A calibration sample containing only short, easy inputs may poorly represent the activation ranges encountered on other workloads.

In a simple quantized dot product, integer codes have their zero points subtracted, corresponding values are multiplied, and the products are accumulated in a wider representation. The input and weight scales then determine the real scale of that sum. Converting the result into another quantized layer’s grid requires rescaling and rounding again. Wider accumulation reduces overflow risk but does not remove error introduced in the inputs or at layer boundaries. Actual accumulation types and supported operations depend on the runtime.

Weight-only storage may instead unpack or dequantize weights for higher-precision arithmetic. Thus a smaller file does not prove that inference uses integer multiplication throughout. Quantize/dequantize boundaries, unsupported operators, or repeated format conversion can offset the savings. Inspect the converted graph and the kernels used on the target device when explaining a speed claim.

Choosing how to obtain the quantized model

Post-training quantization starts from an already trained model. Static activation quantization uses representative calibration inputs to choose ranges before deployment; dynamic activation quantization determines parameters during inference. Static parameters avoid that runtime range calculation but can mismatch new inputs. Dynamic calculation adapts the grid to current values while adding overhead. Neither method has a universal accuracy or latency advantage.

Quantization-aware training exposes the model to simulated quantization effects during training or fine-tuning, allowing parameters to adapt before conversion. Rounding is not ordinarily differentiable, so training uses an approximation to its effect on gradients. This adds training and data requirements; it does not guarantee that the final runtime implements every simulated operation identically. Keeping sensitive tensors or operations at higher precision is another option when selective conversion meets the deployment goal.

Deciding whether conversion helped

Payload arithmetic is a starting point. One million float32 weights occupy 4 million bytes; int8 payload occupies 1 million bytes, and ideally packed 4-bit payload occupies 500,000 bytes. Scale metadata, padding, unconverted tensors, and the container add to the file. A 4-bit simulation stored in an int8 array still uses one byte per element until it is packed. Runtime memory also includes activations, buffers, and other state, so model-file reduction and peak-memory reduction need separate measurements.

Evaluate task quality on representative held-out inputs, including difficult cases and important segments. Agreement with the original model diagnoses the conversion but does not measure correctness: preserving an original mistake still counts as agreement. Small weight error can change a prediction near a decision boundary, while a larger error in an insensitive direction may not. Where quality falls, comparing intermediate tensors helps locate sensitive layers before changing ranges, group sizes, or precision.

Then measure latency, throughput, and memory using the intended device, runtime, input lengths, and batch sizes. Report which tensors were converted and compare equivalent workloads. A conversion restricted to a small subset of layers cannot deliver the payload reduction implied by quantizing every parameter. Quantization is one choice within model compression and deployment; its value depends on satisfying the task’s quality constraints while improving the actual deployment bottleneck.

References

• ONNX Runtime documentation, Quantize ONNX models. onnxruntime.ai
• PyTorch documentation, torch.ao.quantization.quantize_dynamic. docs.pytorch.org
• scikit-learn documentation, sklearn.linear_model.LogisticRegression. scikit-learn.org