Definition
Cloud inference means running a trained ML model on cloud infrastructure to generate predictions on new (unseen) data.
- “Inference” = using a trained model (not training it) to make predictions.
- Done in the cloud (AWS, GCP, Azure, etc.) instead of local/on-device hardware.
How It Works
- Model Training
- Train model locally or in the cloud.
- Save trained weights (parameters).
- Model Deployment
- Upload model to a cloud service (e.g., AWS SageMaker, GCP Vertex AI, Azure ML).
- Expose it via an API endpoint (REST/GRPC).
- Inference
- Client sends input data → API request.
- Cloud service loads the model → runs forward pass → returns prediction.
Benefits
- Scalability: Handles millions of requests with autoscaling.
- Low maintenance: Cloud providers manage servers, GPUs, updates.
- Flexibility: Can serve multiple models, switch versions easily.
- Accessibility: Any device can call the API (web, mobile, IoT).
Challenges
- Latency: Network roundtrip may add delay vs on-device inference.
- Cost (OpEx): Cloud compute per prediction can be expensive at scale.
- Data privacy: Sending sensitive data to cloud requires compliance.
- Reliability: Needs monitoring to ensure uptime & fairness.
Examples
- Image Recognition API
- Upload photo to Google Cloud Vision → API returns detected objects.
- LLM Cloud Inference (ChatGPT, Gemini, Claude)
- User sends prompt → cloud server runs model → returns response.
- Fraud Detection Service
- Banking app sends transaction details → model in AWS SageMaker returns fraud probability.
Cloud vs. Edge Inference
| Cloud Inference | Edge/On-device Inference |
|---|---|
| Runs on cloud servers | Runs locally (phone, IoT, laptop) |
| More compute power (GPUs, TPUs) | Limited compute, optimized models |
| Higher latency (network) | Low latency (real-time) |
| Easier to update models | Harder to update remotely |
Summary
Cloud inference = serving ML model predictions from the cloud.
- Pros: scalable, easy to manage, powerful hardware.
- Cons: latency, cost, privacy.
- Common in APIs (LLMs, vision, fraud detection).