ONNX (Open Neural Network Exchange)

ONNX is an open format for describing a trained model as a computation graph. Facebook and Microsoft introduced it in 2017; it is now maintained as an open standard with converters from most major frameworks and runtimes on most major platforms.

An ONNX graph names operators, the tensors they consume and produce, and any initializers stored with it. Initializers often hold trained weights, but can also hold other constants. Nodes resolve through imported operator sets, including supported custom domains. The model is serialized with protobuf; a runtime still needs implementations for the operators it uses.

What gets exchanged

Consider a small computation with a two-value input x, a weight matrix W, a bias b, and an output y = ReLU(xW + b). A graph can express this as MatMul consuming x and W, Add consuming that result and b, and Relu replacing negative values with zero. The output name from one node is the input name of the next. W and b can be stored as initializers; x is supplied for each prediction. Shapes and element types constrain which connections are valid.

For a concrete example, let x = [1, 2], W have rows [1, −1] and [2, 1], and b = [−6, 0]. The matrix product is [5, 1], adding b gives [−1, 1], and ReLU produces [0, 1]. A runtime follows those dependencies using implementations of the declared operators. The graph describes the computation; it is not itself a GPU kernel or a service endpoint. An implementation may fuse compatible nodes while preserving the required behavior within its numerical constraints.

Compare that with shipping a framework checkpoint. A typical PyTorch state_dict stores parameters and registered buffers rather than the forward computation — the code that arranges them into a forward pass has to travel separately, and has to keep working. ONNX moves the arrangement into the file. Both still require the weights and the preprocessing to be versioned together; A supported ONNX runtime can execute the exported graph without the original training framework, but preprocessing must be included in the graph or deployed consistently alongside it. A tokenizer, normalization rule, feature order, or label mapping left outside the graph remains part of the model’s behavior.

Opsets are the compatibility contract

In ONNX, portability is checked against the model IR version, imported operator sets and the target runtime implementation. Operators are grouped into versioned sets; the specification requires the set version to increase when its inventory or operator semantics change.

The resolution rule catches people out, so it is worth stating plainly: a graph's opset version selects, for each operator, the most recent specification at or below that version. The ONNX documentation gives the example of Add, which was updated at versions 6, 7, 13 and 14 — in a graph declaring opset 15, Add follows the version 14 specification. So “exported at opset 15” does not mean every operator has a version 15 definition, and two exports at different opsets can contain genuinely different operator semantics.

An ONNX operator equivalent is only part of a successful export. The exporter must capture the intended computation and shapes, and the target runtime must support its operator schemas, domains and types. Control flow can be represented, but a tracing route may freeze a branch or shape that was meant to remain dynamic. Validate the model structure and compare the exported graph with the original across representative inputs, boundary cases, dynamic shapes and branches, using stated numerical tolerances. Passing those tests establishes agreement on the tested cases, not a proof for every input.

From export to a usable deployment

Start from an evaluated source model in the intended inference configuration and choose a target runtime before selecting export settings. Export the computation with its input contract: names, types, layout, and which dimensions may vary. Structural checking can find an invalid graph, but a valid graph may still use an operator unsupported by the target. Runtime loading therefore tests a different boundary from format validation.

Run the same prepared inputs through the source and exported models and compare corresponding outputs with tolerances appropriate to their precision. Include batch sizes and sequence lengths the service will actually accept. A model that matches on one fixed example can still fail on another branch or shape. Also compare task metrics when numerical differences could change decisions near a threshold.

Finally, test the execution provider and device configuration used for serving. A graph may load yet execute some operations on the CPU, introducing transfers and different latency from an all-accelerator path. Package any external tensor data, preprocessing assets, and output mappings together, record versions, and retain a tested previous artifact for rollback. Choosing ONNX is useful when this portability and deployment path reduces a concrete integration constraint; export alone is not evidence of faster inference.

What the format is and is not for

The common summary — train in one framework, deploy in another — is accurate in one direction and optimistic in the other. Export from a training framework to ONNX is well traveled and widely supported. Importing ONNX back into a different training framework, to continue training there, is a much thinner path. Treat ONNX as an inference-deployment format unless you have verified otherwise for your specific pair of tools.

A graph also gives runtimes opportunities to fuse operations, fold constants and select hardware-specific kernels. Quantization can be applied as a graph transformation by a tool that reads and writes the representation. These optimizations depend on the graph and runtime: ONNX can contain control flow, and framework compilers can optimize graphs too.

References

• ONNX documentation, Concepts (graphs, nodes, initializers, protobuf serialization, opset versions). onnx.ai
• ONNX documentation, Versioning (operator sets, opset_import, compatibility rules). onnx.ai
• ONNX Runtime documentation, Quantize ONNX models. onnxruntime.ai