AWS SageMaker Endpoints

A SageMaker endpoint is a managed interface for invoking a deployed model. Real-time, serverless and asynchronous inference address different latency and traffic needs; batch transform runs jobs without a persistent endpoint.

AWS’s inference options guide distinguishes these modes. Real-time inference returns a synchronous response from provisioned capacity. Serverless inference suits intermittent synchronous requests, subject to cold starts and supported features. Asynchronous inference takes an S3 input location, queues processing and writes results to S3; configured autoscaling can reduce capacity to zero between jobs. Batch transform processes stored inputs as a job. The SageMaker workflow connects model artifacts, permissions and deployment configuration.

What gets deployed

In the conventional model-and-variant deployment path, the model resource describes the inference image, artifacts, and execution role. An endpoint configuration specifies the models and hosting settings. The endpoint is the running service created from that configuration. These are different resources: storing model metadata does not allocate a serving fleet. AWS’s CreateModel and CreateEndpoint references describe the relationship.

CreateEndpoint starts provisioning and returns before the endpoint is ready. Check DescribeEndpoint until it reports InService before sending prediction traffic. A provisioning failure needs investigation of the reported failure and container startup, artifact access, and selected capacity; repeatedly submitting prediction requests does not fix it. Updating the endpoint uses a new endpoint configuration, so retain the prior configuration and artifacts if they are needed for recovery.

For synchronous inference, the caller sends serialized model input through the runtime invocation API and receives the model’s result. The caller’s permission to invoke and the service role’s permission to load artifacts serve different purposes. An input can pass transport validation while having the wrong feature order, units, or encoding. Test a known example through the deployed service, including preprocessing and response interpretation, rather than testing the model file alone.

For an asynchronous example, upload a document to S3, pass its S3 location to InvokeEndpointAsync, and keep the returned request identifier and output location. Acceptance means the request was queued, not that processing succeeded. Retrieve the result when available or use the optional success/error notification. A timed-out caller should reconcile the request state before resubmitting work. AWS’s asynchronous workflow requires AsyncInferenceConfig and supports asynchronous invocations on that configured endpoint.

The mode is chosen by the traffic shape

A synchronous application needs a response within its caller’s deadline. Real-time capacity can remain available for that path; serverless can fit intermittent supported workloads when startup behavior is acceptable. An asynchronous document job can finish later and return a stored result. A nightly dataset often fits batch transform, which is a job rather than a live endpoint. Payload limits, processing duration, accelerators, and supported features must be checked for the selected mode; these modes are not interchangeable merely because all produce predictions.

Traffic timing matters within each mode. For an illustrative nightly job of four million records at 28 records per second per instance, forty instances require about 3,571 seconds, excluding startup and overhead. At an assumed 1.15 per instance-hour, that is about 45.63 of compute. Keeping the same forty instances for 24 hours costs 1,104. The difference is the chosen lifetime of the fleet; it is not an intrinsic batch discount or an asynchronous surcharge. Price the actual configuration, including startup, storage, and required completion time.

What always-on actually costs

For a fixed always-on fleet, utilization strongly affects cost per request; model throughput and resource requirements also set the capacity and price.

At an assumed 1.15 per instance-hour, two instances kept for 30 days cost 2 × 1.15 × 24 × 30 = 1,656. At a sustained 28 requests per second each, their idealized monthly capacity is 145,152,000 requests. Serving 50,000 requests allocates 0.03312 per request; serving 50 million allocates 0.00003312. This division describes cost allocation. It does not show whether short bursts meet latency requirements or whether throughput stays constant across payloads.

At the full assumed capacity, allocated cost approaches 0.0000114 per request. At 50,000 monthly requests, about 99.97% of that nominal capacity is unused. This is an allocation of the fixed bill, not a separate idle-time charge.

Interpret utilization together with cost per successful request, latency, errors, and traffic timing. Capacity allocation helps explain why a quoted inference cost and a realized one differ so much.

Scale-up delay and burst duration

Autoscaling reacts with a delay. The NumPy model below starts with two instances and adds one at each lag boundary, beginning at one lag. It assumes immediate detection, no queue and dropped requests whenever demand exceeds capacity; it is not a simulation of an AWS scaling policy. The endpoint starts at 2 instances serving 56 requests a second, one instance serves 28, and the spike asks for 190 a second, so seven instances would cover it. In burst, each array element represents one second. Subtracting available throughput from demand and summing positive shortfalls counts requests that this simplified model cannot serve; real systems may queue or reject them differently.

import numpy as np

rps_per_instance, start, spike_rps = 28.0, 2, 190.0
needed = np.ceil(spike_rps / rps_per_instance)  # 7 instances would cover the spike


def burst(lag, spike):
    seconds = np.arange(spike)                  # one element per second of the spike
    added = np.floor(seconds / lag)             # one instance per completed lag
    have = np.minimum(start + added, needed)
    capacity = have * rps_per_instance
    dropped = np.maximum(spike_rps - capacity, 0).sum()
    return have[-1], dropped


print("scale-up lag  spike length  instances at end  requests dropped  share dropped")
for lag in (60, 180, 300, 600):
    for spike in (120, 300, 900):
        have_end, dropped = burst(lag, spike)
        print("%12d %13d %17.0f %17s %14.1f%%"
              % (lag, spike, have_end, "{:,.0f}".format(dropped),
                 dropped / (spike_rps * spike) * 100))

scale-up lag  spike length  instances at end  requests dropped  share dropped
          60           120                 3            14,400           63.2%
          60           300                 6            23,400           41.1%
          60           900                 7            23,400           13.7%
         180           120                 2            16,080           70.5%
         180           300                 3            36,840           64.6%
         180           900                 6            70,200           41.1%
         300           120                 2            16,080           70.5%
         300           300                 2            40,200           70.5%
         300           900                 4            95,400           55.8%
         600           120                 2            16,080           70.5%
         600           300                 2            40,200           70.5%
         600           900                 3           112,200           65.6%

At a 60-second lag, one instance is added halfway through the two-minute spike, reducing dropped requests to 63.2%. With lags of 180 seconds or more, that spike ends before any addition and drops 70.5%. Longer bursts give scaling more time to catch up, as the table shows.

Autoscaling can help a burst that outlasts enough of its response delay. Warm headroom, admission control or a queue can cover the gap; asynchronous invocation changes the completion-time promise. Size these mechanisms against a measured traffic pattern rather than defining every spike as too short to scale.

References

• AWS, CreateModel
• AWS, CreateEndpoint
• AWS: Inference options in SageMaker AI
• AWS: Asynchronous inference