Cloud Inference

Definition

Cloud inference means running a trained ML model on cloud infrastructure to generate predictions on new (unseen) data.

• “Inference” = using a trained model (not training it) to make predictions.
• Done in the cloud (AWS, GCP, Azure, etc.) instead of local/on-device hardware.

---

How It Works

1. Model Training
   • Train model locally or in the cloud.
   • Save trained weights (parameters).
2. Model Deployment
   • Upload model to a cloud service (e.g., AWS SageMaker, GCP Vertex AI, Azure ML).
   • Expose it via an API endpoint (REST/GRPC).
3. Inference
   • Client sends input data → API request.
   • Cloud service loads the model → runs forward pass → returns prediction.

---

Benefits

• Scalability: Handles millions of requests with autoscaling.
• Low maintenance: Cloud providers manage servers, GPUs, updates.
• Flexibility: Can serve multiple models, switch versions easily.
• Accessibility: Any device can call the API (web, mobile, IoT).

---

Challenges

• Latency: Network roundtrip may add delay vs on-device inference.
• Cost (OpEx): Cloud compute per prediction can be expensive at scale.
• Data privacy: Sending sensitive data to cloud requires compliance.
• Reliability: Needs monitoring to ensure uptime & fairness.

---

Examples

1. Image Recognition API
   • Upload photo to Google Cloud Vision → API returns detected objects.
2. LLM Cloud Inference (ChatGPT, Gemini, Claude)
   • User sends prompt → cloud server runs model → returns response.
3. Fraud Detection Service
   • Banking app sends transaction details → model in AWS SageMaker returns fraud probability.

---

Cloud vs. Edge Inference

Cloud Inference	Edge/On-device Inference
Runs on cloud servers	Runs locally (phone, IoT, laptop)
More compute power (GPUs, TPUs)	Limited compute, optimized models
Higher latency (network)	Low latency (real-time)
Easier to update models	Harder to update remotely

---

Summary
Cloud inference = serving ML model predictions from the cloud.

• Pros: scalable, easy to manage, powerful hardware.
• Cons: latency, cost, privacy.
• Common in APIs (LLMs, vision, fraud detection).