Concurrency and Request Rate

User concurrency counts users or sessions active at the same time; request concurrency counts requests that have started but not yet finished. Twenty active users do not necessarily mean twenty requests in flight. Request rate counts submissions per unit of time, often requests per second. The relationship depends on submission timing, whether a user waits for a response, and how long requests remain in the system. State which kind of concurrency a requirement means. It is the demand side of a performance statement; the response side is response latency, and neither is interpretable without the other.

Same 20 users, three times the load

Twenty clerks whose average interval from one submission to the next is 30 seconds generate about 20 / 30 ≈ 0.67 requests per second. If month-end pressure makes each of them submit every 10 seconds, the rate becomes 20 / 10 = 2 requests per second: the same user population and three times the submission rate. A 30-minute test at 2 requests per second contains 2 × 1,800 = 3,600 requests. The reverse also holds: 2 requests per second could come from 200 occasional users rather than 20 busy ones.

Opening 20 idle sessions does not establish performance under active lookup demand. Distinguish the interval between submission starts from a pause after a response. If each user waits for a response and then pauses for 30 seconds, a two-second response makes the cycle 32 seconds, giving about 20 / 32 = 0.625 requests per second. If responses slow down, that closed user loop submits fewer requests. An arrival schedule that submits two requests per second independently of completion keeps applying that demand, provided the load generator can sustain it, and requests may accumulate. Record the scheduling model and actual submitted rate.

Why capacity is planned below the limit

Capacity planning needs both the arrival rate and the number of requests in progress, together with a workload and latency target. When arrivals or service times vary, operating close to a bottleneck’s sustainable rate can increase queueing sharply. Headroom, admission control, and queue limits help manage that risk. The appropriate utilization target depends on the workload and service objective rather than a universal fraction of maximum capacity.

Neither measure alone determines whether to scale up or out. First separate time spent waiting from time spent doing work, then identify the limiting resource. A single analytical query may benefit from more workers if it can execute in parallel; additional workers may do little for a serial dependency, lock, or slow external service. More concurrent requests may require more capacity, lower per-request demand, or admission control. Compare measurements before and after the change rather than choosing capacity solely from the symptom.

State the variation, not only the average

Google’s site-reliability guidance notes that a rate averaged over a window can hide bursts several times higher that last only seconds. A useful load statement gives the average, the peak, the burst pattern, and the aggregation interval. Retries after errors and timeouts add requests of their own, so failures can raise the effective rate exactly when the system is weakest — the amplification that exponential backoff and retry budgets exist to bound, and that throttling caps from the server side.

User activity and submission rate describe demand; in-flight request concurrency also reflects how long requests remain unfinished. Throughput describes what the system actually completed per unit of time, and latency describes how long each request took. A requirement should fix the demand and the measurement boundary, then report all eligible requests, including failures and timeouts, against the agreed limit. Where several workloads share one system, the rate that matters is the combined one: a capacity sized for each in isolation still fails when peaks coincide, which is a workload isolation decision as much as a sizing one.

Figures here are illustrative.

References

• Google SRE: Service level objectives
• Grafana k6: Open and closed models