FLOPs

FLOPs are floating-point operations: a count of the arithmetic a model performs. The count is an estimate of work, not of time, cost, memory or energy. It is useful because it can be computed from an architecture on paper, before any hardware is booked.

Two conventions travel under the same name and differ by a factor of two, so any comparison has to fix one first. A count needs its convention, input dimensions, and the part of execution being counted before it can be compared with another result.

Counting, and the factor of two

Multiplying an \(m \times n\) matrix by an \(n \times p\) one costs \(mp(2n - 1)\) exactly, since each of the \(mp\) outputs needs \(n\) multiplications and \(n - 1\) additions. The familiar \(2mnp\) overstates that by 0.05% when the inner dimension is 1000 — and by 14.29% when it is 4. The relative difference depends on the inner dimension n, not simply on the overall matrix size. These counts assume the usual dot-product algorithm and separate multiply/add accounting.

One 4096→4096 dense layer is 16,777,216 multiply-accumulates, or 33,554,432 FLOPs if a multiply-accumulate counts as two. The same layer, same arithmetic, two numbers. A paper reporting MACs and a paper reporting FLOPs differ by 2× on identical work, which is why a reported count is meaningless without its convention, its input size and whether it covers a forward pass or a full training step.

For this dense operation, adding one bias per output exactly cancels the \(-mp\) term: \(mp(2n - 1) + mp = 2mnp\). This equality covers the matrix multiplication and bias only; an activation or other operation needs separate accounting.

Scaling the count to a training run

A common approximation estimates dense-model training work as roughly \(6ND\), for \(N\) active parameters and \(D\) tokens. In this approximation the forward pass costs \(2ND\), and the backward pass about \(4ND\): the full training computation is about three times the forward computation. It abstracts away architecture-specific work such as attention-score computation and optimizer updates. For the illustrative inputs of 174.6 billion parameters and 300 billion tokens it gives \(3.14\times10^{23}\) FLOPs, matching the scale reported in the GPT-3 compute table.

Dividing by an accelerator's peak rate gives an accelerator-day figure, and here the arithmetic has to be labeled honestly. An A100's peak dense BF16 Tensor Core throughput is 312 TFLOP/s, so the division yields about 11,659 accelerator-days. This is an idealized compute-time floor under the stated work and peak-rate assumptions, not a claim that the model fits on one device or that enough devices make the schedule feasible. At an assumed sustained 40% of that rate, the same arithmetic would require about 29,147 accelerator-days before adding work omitted from the model. Dividing by the device count further assumes effective parallel scaling; communication and pipeline bubbles can prevent it.

Note also what \(6ND\) ignores: attention's dependence on sequence length, activation recomputation, optimizer arithmetic, and everything that is not a parameter multiply. It is a good first estimate for dense transformers and a poor one outside that family.

Why the count does not predict the clock

Arithmetic intensity is the operation count divided by bytes transferred at a stated memory boundary. Consider idealized float32 operations that read each input once and write the output once, ignoring caches, repeated transfers, and temporary buffers. A 4096 × 4096 matrix times a 4096-element vector uses approximately 33,554,432 FLOPs and transfers 67,141,632 bytes, about 0.50 FLOPs per byte. Computing a × x + y over two arrays of 4096² elements uses the same operation count but reads two arrays and writes one, transferring 201,326,592 bytes: about 0.17 FLOPs per byte. Here a is a scalar whose transfer is ignored. The identical arithmetic count hides three times as much idealized traffic.

Multiplying two 4096 × 4096 matrices, under the same idealized assumptions, uses about 137.44 billion FLOPs and 201,326,592 bytes, or 682.67 FLOPs per byte. Each matrix value contributes to many outputs, making reuse possible. Actual traffic depends on how effectively kernels and memory hierarchies realize that reuse. An accelerator with a peak arithmetic rate far above its memory bandwidth reaches peak only on high-intensity work; a low-intensity operation spends its time waiting on memory and the arithmetic units idle. Two models with identical FLOP counts can differ severalfold in latency for this reason alone, before precision, kernel quality, batch shape, sparsity support or inter-device communication enter.

A useful performance bound connects these quantities: attainable arithmetic rate is at most the smaller of peak compute rate and memory bandwidth multiplied by arithmetic intensity at that memory boundary. The second term has units of FLOP/s. For a hypothetical 1 TB/s memory interface and intensity 0.5 FLOPs per byte, the bandwidth term is 0.5 TFLOP/s, however large the arithmetic peak may be. This is an upper bound under idealized overlap and traffic assumptions, not a latency prediction; launch overhead and dependencies can reduce performance further.

Keep FLOPs and FLOP/s apart. FLOPs is an operation count under a stated convention; FLOP/s is a rate. Smaller architectures or shorter sequences can reduce the count. Quantization changes precision, storage and available kernels without necessarily reducing the number of multiplies and adds; integer operations must be labeled separately if excluded from a FLOP count. Combine these quantities with measured latency, memory use and cost on the accelerator cluster you will actually use.

References

• Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165, Appendix D, Table D.1. arxiv.org
• NVIDIA, A100 Tensor Core GPU specifications. nvidia.com
• Google Cloud documentation, TPU v4. cloud.google.com