FLOPs

Definition

FLOPs means floating-point operations.

• It estimates the number of floating-point arithmetic operations—such as additions and multiplications—required by a model or algorithm.
• In machine learning, FLOPs are commonly used as a proxy for computational workload during training or inference.
• FLOPs do not directly measure execution time, monetary cost, memory usage, or energy consumption.

FLOP-counting conventions vary. For example, some tools count a multiply-accumulate operation as one operation, while others count the multiplication and addition separately as two FLOPs. Comparisons should therefore use the same convention.

---

Key Points

1. Operation Type
   • FLOPs count floating-point arithmetic operations.
   • They usually exclude memory access, data transfer, communication, control flow, and input/output operations.
   • Matrix multiplications, convolutions, and attention layers can account for a large share of a model’s FLOPs.
2. Metric Units
   • 1 FLOP = one floating-point operation.
   • 1 MFLOP = $10^6$ floating-point operations.
   • 1 GFLOP = $10^9$ floating-point operations.
   • 1 TFLOP = $10^{12}$ floating-point operations.
   • 1 PFLOP = $10^{15}$ floating-point operations.
3. Why FLOPs Matter in Machine Learning
   • Higher FLOPs generally indicate more arithmetic work under the same counting convention.
   • FLOPs help compare the computational requirements of models and configurations.
   • They help estimate training workload and inference requirements before deployment.

---

Example: Matrix Multiplication

Consider multiplying an $m \times n$ matrix by an $n \times p$ matrix.

• The output contains $m \times p$ elements.
• Each output element requires $n$ multiplications and $n-1$ additions using the classical algorithm.
• If multiplication and addition each count as one FLOP, the exact count is:

$\text{FLOPs} = m p (2n – 1) = 2mnp – mp$

For large $n$, this is commonly approximated as:

$\text{FLOPs} \approx 2mnp$

For two $1000 \times 1000$ matrices:

$\text{FLOPs} = 1000 \times 1000 \times (2 \times 1000 – 1) = 1.999 \times 10^9$

This is approximately 2 billion FLOPs under the two-operation convention.

---

FLOPs in Deep Learning

• Training: total FLOPs depend on model architecture, parameter count, number of training examples or tokens, input dimensions, batch configuration, number of epochs, and the cost of forward and backward passes.
• Inference: FLOPs are usually reported per forward pass, image, token, sequence, or generated output.
• Input dependence: FLOPs can change with image resolution, sequence length, batch size, sparsity, and generated-token count.
• Large language models: a common rough estimate for dense Transformer training is proportional to the number of parameters multiplied by the number of training tokens.

For example, the estimated training compute for the 175-billion-parameter GPT-3 model is approximately $3.14 \times 10^{23}$ FLOPs. Such estimates depend on the included operations and accounting method.

Reported FLOP counts for models such as ResNet or BERT should always specify the input size, sequence length, forward or training scope, and multiply-accumulate counting convention.

---

FLOPs vs. FLOP/s

• FLOPs = a count of floating-point operations.
• FLOP/s, often written as FLOPS = floating-point operations per second, a throughput rate.
• TFLOP/s = $10^{12}$ floating-point operations per second.
• Example: the NVIDIA A100 has a theoretical peak dense FP16 or BF16 Tensor Core throughput of approximately 312 TFLOP/s, with higher advertised effective throughput when supported structured sparsity is used.

Analogy:

• FLOPs describe the total amount of arithmetic work.
• FLOP/s describe how quickly hardware can perform arithmetic under specified conditions.

---

FLOPs vs. Real-World Performance

Two models with the same FLOP count can have different latency, throughput, memory usage, and energy consumption.

• Memory bandwidth: a model may spend substantial time moving data rather than performing arithmetic.
• Hardware utilization: theoretical peak throughput is rarely sustained by every workload.
• Operation structure: large matrix multiplications may run more efficiently than many small or irregular operations.
• Precision: FP64, FP32, TF32, BF16, and FP16 operations have different hardware throughput.
• Sparsity: sparse operations may reduce executed work, but practical gains depend on hardware and software support.
• Communication: distributed training may be limited by communication between accelerators.

For deployment decisions, FLOPs should be considered together with measured latency, throughput, memory consumption, energy use, and monetary cost on the target hardware.

---

Why It Is Important

• Helps estimate the computational workload of training and inference.
• Supports comparisons of model efficiency and predictive performance.
• Helps teams plan hardware capacity and experimentation budgets.
• Provides one component of environmental and energy-impact analysis.
• Helps identify whether an architecture is practical for edge, mobile, or real-time deployment.

---

Summary
FLOPs measure the number of floating-point arithmetic operations required by a model or algorithm under a stated counting convention.

• FLOPs estimate computational workload but do not directly measure latency or cost.
• Matrix multiplication of an $m \times n$ matrix and an $n \times p$ matrix requires approximately $2mnp$ FLOPs under the two-operation convention.
• FLOP counts depend on input dimensions, model configuration, and counting rules.
• FLOPs are different from FLOP/s, which measures hardware throughput.
• Real-world evaluation should combine FLOPs with latency, throughput, memory, energy, and cost measurements.