Inference Cost (Inference $)

Definition

• In machine learning, inference = running a trained model on new data to get predictions.
• Inference cost = the money (or resources) required to serve those predictions in production.
• Often written as Inference $ in monitoring dashboards or MLOps docs.

---

What Makes Up Inference Cost

1. Compute (GPU/CPU time)
   • Each prediction requires FLOPs → paid as GPU hours or vCPU time.
2. Memory & Storage
   • Hosting large models (like LLMs) needs lots of RAM.
3. Networking / Payload Transfer
   • Sending/receiving large inputs (e.g., video, long prompts).
4. Cloud Service Fees
   • If using SageMaker, Vertex AI, OpenAI API, you pay per request or per token.

---

Examples

• OpenAI API (GPT-4o)
  • Cost is per token processed (input + output).
  • Example: $0.005 per 1K input tokens.
• AWS SageMaker
  • If you deploy a model on ml.m5.xlarge → you pay hourly rate + request usage.
• On-Prem GPU
  • Inference cost = electricity + GPU amortization per prediction.

---

Why Important

• Inference cost is a key OpEx metric in production ML.
• Even if training is expensive, inference happens millions of times per day.
• Optimizing inference cost = critical for scalability & profitability.

---

Summary
Inference $ = Inference cost.
It represents the operational expense of running predictions in production (cloud fees, GPU usage, latency overhead).