Kubernetes vs. Managed Services Comparison

A team that has a container image and a few services can choose among many runtimes, and the choice is not always made on evidence. Kubernetes is widely used, so it can become the default. A managed service sounds easier, so it can become the exception. A single host running a Compose file can sound too small to be serious, and be dismissed without anyone asking what it would take to outgrow it.

This post builds an evaluation procedure instead of a recommendation. It starts from the workload’s needs, estimates the operational burden of each option, reads the boundary of what each managed service operates, designs a benchmark and a cost comparison that can survive scrutiny, and lists the conditions under which moving from one runtime to another is justified. The workloads in the examples are illustrative: three invented services and their assumed numbers. No real system, price, or measurement is described, and any figure marked as assumed is an input to a model, not a result.

What this post assumes

It builds on operating shared clusters. From that post it assumes you have seen what running a shared Kubernetes cluster costs in operating work: tenant isolation settings, quotas, admission policy, version skew and the upgrade calendar, and the limits of rollback. This post turns those costs into inputs for a comparison. General computer science knowledge is assumed, including processes, containers as packaged processes, HTTP request handling, and basic queueing intuition.

Two notes on evidence. No cluster or cloud account was used, so nothing here was deployed, and no provider was benchmarked. The statements about products come from the providers’ documentation listed in the references, read at the time of writing, and product behavior and pricing change, so treat each as a claim to recheck before deciding. Two small Python scripts were run locally: a deterministic latency model for the benchmark discussion and a break-even model for the cost discussion. Their inputs are assumptions stated where they appear. The Python blocks are short calculation aids that make a documented rule or an assumed number concrete. They are excerpts: the output shown came from fuller local versions that also print headers and extra rows.

Start with the workload’s needs

Choosing a runtime first and fitting workloads to it reverses the dependency. The workload has requirements, and the runtime either meets them or imposes work to compensate.

Describe the workload

Write the description of each workload before comparing runtimes. The questions below are the ones that change the answer.

Question	Why it changes the choice
What protocol does it serve: HTTP requests, queue messages, TCP, UDP, batch runs?	Some serverless container platforms are built around HTTP requests; others have separate job or worker models. Non-HTTP protocols may need an orchestrator that exposes arbitrary ports.
Where does its state live?	A stateless process on ephemeral disk fits almost any runtime. A process that owns files on a volume needs volume lifecycle, identity, and backup, which the earlier post on stateful workloads described.
What is its traffic shape: steady, bursty, idle for long periods?	Idle periods favor runtimes that bill only while work is processed or that scale to zero. Steady load favors provisioned capacity.
How quickly must it start and scale?	Scale-from-zero adds startup delay on any platform. Latency-sensitive services may need minimum capacity.
Does it need privileged access, host paths, special devices, or a particular kernel feature?	Managed platforms restrict some of these. A restriction is a hard filter, not a cost.
How many teams and services share the runtime?	Shared-platform costs (isolation, policy, upgrades) amortize over more services. For a small number they may not.
What isolation or compliance rules apply?	They set how much sharing is acceptable and who must operate which layer.
How much portability matters, and why?	Portability is a requirement only with a stated reason, such as an exit condition or a customer deployment target. Without one it is a preference with a cost.

The table is a working checklist written for this post, not a standard taxonomy. The value comes from answering it per workload. Two workloads in the same company can have different answers, and a single runtime for all of them is an outcome to justify rather than assume.

Three illustrative workloads

To keep the rest of the post concrete, assume three invented services. They are hypothetical and chosen to span the questions above.

• Service A, a bursty HTTP API. It is stateless, receives traffic in short bursts, and is idle for much of the day. Assumed to be active about 10% of the time.
• Service B, a database-backed internal platform service with many dependents. It runs continuously with steady load, keeps data on a volume, and has a dozen other services calling it.
• Service C, a nightly batch job. It runs to completion a few times a day with no inbound traffic and needs more memory than the other two.

None of these descriptions is a measurement, and they exist so the evaluation can be followed step by step.

The option space

Runtime options sit on a spectrum of how much the provider operates.

Option	What the documentation says it is	Who operates what, at a high level
A host or VM running processes under a service manager	Not a product; the baseline	You operate everything: OS, process supervision, deploys, scaling
Docker Compose on a host	A YAML file that defines an application’s services, networks, volumes, configs, and secrets	You operate the host and its OS; Compose describes the multi-container application on it
A serverless container service such as Cloud Run	A managed platform that runs your container or code without you building clusters	The provider operates infrastructure and scaling; you supply a container that meets the platform’s contract
ECS on Fargate	ECS is an AWS container orchestrator; Fargate is its serverless compute option	AWS operates the orchestrator and the servers under Fargate; you define tasks and services
Managed Kubernetes, standard mode (EKS, GKE standard)	A Kubernetes control plane operated by the provider	The provider operates the control plane; you operate nodes (depending on mode), add-ons, workloads, and policies
Managed Kubernetes with automated nodes (EKS Auto Mode, GKE Autopilot)	A Kubernetes API with the provider also managing nodes	The provider also manages nodes; you manage workload specs, policies, and what the platform restricts
Self-managed Kubernetes (kubeadm and similar)	Kubernetes you install and operate	You operate the control plane, etcd, nodes, certificates, upgrades, and add-ons
Nomad	A workload orchestrator that runs containers and other workload types through task drivers	You operate the orchestrator, or use a managed offering if one exists

The descriptions come from the Compose, Cloud Run, ECS, EKS, GKE Autopilot, Kubernetes production, and Nomad documentation. Some details deserve a direct statement. The Cloud Run documentation describes services (HTTP request handling with autoscaling), jobs (tasks that run to completion), and worker pools (continuous pull-based workloads such as queue consumers), and states that containers must listen on a TCP port to handle stateless HTTP requests and that file systems are ephemeral. The ECS documentation distinguishes a task (an application such as a batch job that performs work and stops) from a service (a long-running stateless application) and describes Fargate as serverless compute for which you do not manage servers or capacity planning. The EKS documentation distinguishes standard mode, where AWS manages the Kubernetes control plane, from Auto Mode, where EKS also manages nodes. The Autopilot documentation says Google manages nodes, scaling, and security hardening, and that privileged workloads require allowlist approval. Nomad’s documentation presents it as a single-binary orchestrator for containers and non-container workloads. That description is the vendor’s own positioning, so check it against your needs rather than treating it as a comparison.

Operational burden: what you must still do

The previous post is an inventory of what running a shared Kubernetes cluster requires. The Kubernetes production documentation gives the top-level categories: availability (replicate the control plane, separate it from workers, load balance the API server), scale (mechanisms for growth), security and access management (RBAC and restrictions on users and workloads), and resource management (limits and policies that prevent exhaustion). For a self-managed cluster it adds certificate management, a load balancer for the API server, a separately backed-up etcd, and multi-zone deployment for critical workloads. The same page points out that partner solutions can take over parts of this: serverless offerings that need no cluster management, a managed control plane, managed worker nodes, and integrated services.

Burden is a list of duties with owners

Burden is easier to compare when written as duties and the party responsible for each. The table assigns the duties from the earlier posts to three illustrative arrangements. The assignments for managed products are summaries of the documentation and should be checked against the specific product and mode.

Duty	Self-managed Kubernetes	Managed Kubernetes (standard)	Serverless container service
Control plane availability and patching	You	Provider	Provider
Node OS patching and replacement	You	You, unless the mode manages nodes	Provider
Kubernetes minor-version upgrades	You, in sequence, with rehearsal	Provider performs or schedules the control-plane upgrade; node pools and add-ons depend on the mode	Not applicable in the same form
Add-on compatibility (network, storage, ingress)	You	You, with some add-ons offered as managed components	Mostly provider-defined
Tenant isolation, quotas, admission policy	You	You	Platform-defined; limited configuration
Workload sizing and scaling configuration	You	You	You, within the platform’s model
Backup of cluster state	You	Provider-specific; check what is offered	Not applicable
Backup of application data	You	You	You
Incident response for the workload	You	You	You

The last two rows are the same across options. Choosing a managed service moves some duties to the provider and leaves the application’s data and behavior with the team. A product’s marketing language can blur this. The EKS documentation, for example, says that the shared responsibility model applies to security, and the Autopilot documentation lists workload specifications, Pod configuration, network policies, and access controls as the customer’s.

Estimating burden without invented numbers

Burden can be estimated with a method that avoids made-up figures. List the duties for each workload and option as above. For duties you already perform, count the recurring tasks and measure the time they take. The glossary entry on toil describes the attributes of work that is manual, repetitive, automatable, reactive, of little enduring value, and growing with the service, and it can serve as a diagnostic for which duties to automate or move to a provider. For duties you have not performed, such as a first Kubernetes minor upgrade, estimate from a rehearsal instead of from an assumption, and record the estimate as an assumption. A rehearsal on a non-production cluster yields a real count of steps, a real list of incompatibilities, and a real duration, none of which can be obtained from a document.

Capability also matters. A duty that needs scarce expertise is a risk even if its average hours are low, because the hours arrive during incidents. Count the people who can perform each duty without help.

Managed service boundaries: what is and is not operated for you

A managed service is defined by its boundary. The questions below apply to any provider and turn the marketing statement into checks.

1. What does the provider operate, and what do you still operate? For EKS standard, the control plane is the provider’s, and nodes are yours unless you choose a mode that manages them. For Autopilot, nodes are managed, and the customer still defines workloads, network policies, and access controls.
2. What is constrained? Autopilot requires allowlist approval for privileged workloads. Cloud Run requires a container that listens on a port and keeps durable state outside its ephemeral file system. A constraint is a filter on workloads, so check each workload against it before costing.
3. Who controls upgrades and when? Autopilot upgrades through release channels with maintenance window controls according to its documentation. EKS offers standard and extended support periods for Kubernetes versions. A provider-driven upgrade removes work and also removes the ability to choose the day, so tenants must keep their manifests compatible continuously.
4. What can you not see or change? Whether you can change API server settings or read control-plane logs the way a self-managed cluster allows depends on the product, so check which settings and logs a given service exposes.
5. What are the limits? Providers set quotas on clusters, nodes, instances, and API calls. Find the limits that your largest workload approaches.
6. What happens when something fails? Read the service’s availability commitments and what it states about its own failure modes. A commitment covers the provider’s component, not your application.

Conformance is a related question for managed Kubernetes. The glossary entry on a Kubernetes distribution explains that CNCF conformance tests required Kubernetes APIs for a certified version, which supports interoperability, but certification does not mean every workload moves unchanged between products. StorageClasses, identity bindings, admission rules, and network integration differ, so a Deployment that is valid Kubernetes can behave differently at another destination. Test representative workloads in the target environment before treating two Kubernetes services as interchangeable.

Dependence and exit

Every managed boundary creates a dependency. The glossary entries on technical lock-in and exit cost describe dependence created by architecture, product capabilities, data, or skills, and an exit cost made of data movement, rewriting, rebuilt operations, running both systems, contracts, and people. Portability is not free either: avoiding every provider-specific feature can cost capability and effort. The evaluation should compare the value of a managed feature with an exit you could actually afford, using a representative migration trial where possible, and should record the dependency as a decision with a review condition.

Benchmark and cost comparison

A runtime comparison needs two kinds of evidence, performance under the workload and cost for the work done, and each is easy to get wrong.

What a benchmark must hold constant

A benchmark compares runtimes only if everything else is the same. The checklist below defines a comparison that can be defended.

• The same application artifact and configuration. The same container image, runtime settings, and dependencies in each environment. A difference in the image or in a database configuration will dominate the result.
• The same workload model. The request mix, payload sizes, concurrency, and arrival pattern should come from production data or from an explicitly stated model, and should include the burst shape for a bursty service.
• The same objective. Express the target as a service level objective, such as a latency percentile at a request rate, and compare the capacity and cost needed to meet it. Comparing raw speed at unequal resource sizes tells you about the sizes.
• Warm-up and cold-start measured separately. Startup delay after scaling from zero is a distinct result from steady-state latency.
• Repetition and variance. Repeat runs, report the spread, and avoid conclusions from differences smaller than the spread.
• Recorded conditions. Region, instance sizes, versions, network path between the load generator and the system, and the date. Provider performance changes over time.

A benchmark that reports average latency can mislead, and a load generator can hide stalls. The next script is a deterministic model rather than a measurement of any runtime. A single server takes 1 millisecond per request and stalls for one second at the five-second mark of a ten-second run. In the open-loop run, requests arrive every 10 milliseconds regardless of replies, as independent users would. In the closed-loop run, the generator sends the next request only after the previous one returns, as many simple benchmark tools do.

SERVICE, STALL_AT, STALL, DURATION, INTERVAL = 0.001, 5.0, 1.0, 10.0, 0.010

def finish(start):
    s = start
    if STALL_AT <= s < STALL_AT + STALL: s = STALL_AT + STALL
    elif s < STALL_AT < s + SERVICE:     s = STALL_AT + STALL
    return s + SERVICE

# open loop: arrivals every 10 ms regardless of replies
free, t, lat = 0.0, 0.0, []
while t < DURATION:
    end = finish(max(t, free)); free = end; lat.append(end - t); t += INTERVAL

# closed loop: next request is sent only after the previous reply
lat2, t = [], 0.0
while t < DURATION:
    end = finish(t); lat2.append(end - t); t = max(end, t + INTERVAL)

The full script printed these summaries, where p50 and p99 are the 50th and 99th percentile latencies.

open-loop   n=1001  p50=1.0 ms p99=911.0 ms max=1001.0 ms mean=57.0 ms
closed-loop n=901  p50=1.0 ms p99=1.0 ms max=1001.0 ms mean=2.1 ms

Both runs saw the same stall, and the closed-loop result reports a 99th percentile of 1 millisecond because the generator stopped sending while the server was stalled, so almost no request experienced the delay. The open-loop result keeps sending, queues the requests that arrive during the stall, and reports 911 milliseconds at the 99th percentile. This is the general behavior of a generator that waits for replies, and it is why a load test should drive the arrival rate independently of the replies, report percentiles, and state which model it used. The numbers above come from a simple model with assumed parameters and say nothing about any real runtime. Apply the lesson, not the numbers: when comparing runtimes, compare under an open-loop arrival pattern, and report tail latency during scaling events, deployments, and node replacement as well as in steady state.

A benchmark of failure behavior is as informative as one of speed. Kill an instance, drain a node, scale from zero, and roll out a new version during load, and record the error rate and the time to recover for each runtime. These are the events that differ most between a managed serverless platform and a self-operated cluster, and their results feed the objective the workload must meet.

A cost model with explicit assumptions

Cost comparisons fail most often by comparing unlike things: a node’s price against a request’s price, or a platform fee with no staff time. The model below uses a common unit, cost per effective vCPU-hour, and keeps every input as an explicit assumption. The prices are unitless, with a provisioned vCPU-hour defined as 1 price unit.

Provisioned capacity is paid for whether it is used or not. Let pack be the fraction of the provisioned capacity that workloads actually request, which captures bin-packing waste, headroom for failures and rollouts, and idle reserve. Then the cost per effective vCPU-hour is \( 1 / \mathit{pack} \). Pay-per-use capacity is billed only while an instance is active. Let \( k \) be the price multiple of an active vCPU-hour relative to the provisioned rate, and let \( u \) be the fraction of time the workload is active. Then the cost per effective vCPU-hour is \( k \cdot u \). The two are equal when \[ u^{*} = \frac{1}{\mathit{pack} \cdot k} \] and pay-per-use is cheaper when the active fraction is below \( u^{*} \). In words, the break-even depends on how well you pack provisioned capacity and how much extra you pay per active hour for the managed convenience. The model ignores request fees, minimum instances, discounts, networking, and staff time, all of which move the answer, and it treats both options as meeting the same objective, which a benchmark has to confirm. The script computed \( u^{*} \) for assumed values of \( k \) and \( \mathit{pack} \). A value of 1 or above means the pay-per-use option is cheaper at any active fraction, which the table prints as “always”.

k\pack      0.3     0.5     0.7     0.9
1.0      always  always  always  always
1.5      always  always    0.95    0.74
2.0      always  always    0.71    0.56
3.0      always    0.67    0.48    0.37
4.0        0.83    0.50    0.36    0.28

Read one row. With an assumed price multiple of 3, a workload that packs half of provisioned capacity into use breaks even at an active fraction of 0.67. Service A, assumed active about 10% of the time, is far below that, so under these inputs pay-per-use is cheaper, with a wide margin that survives errors in the assumptions. Service B, running continuously at steady load, is active nearly all the time, so provisioned capacity is cheaper under the same inputs unless its packing is poor. The method is the point. The inputs for your case come from billing data and the benchmark, and the conclusion holds only at the inputs. Cloud billing models also differ in structure: the Cloud Run documentation describes request-based billing, which charges while processing requests, and instance-based billing, which charges for the instance’s lifetime; the Autopilot documentation describes billing by Pod resource requests for general-purpose workloads; and the EKS documentation says the cluster has its own pricing and that you pay separately for the nodes and other resources. The serverless containers entry makes the same point: whether idle time is charged depends on the product, resource type, and settings, and utilization alone does not establish the cheaper design.

Fixed costs belong in the model too. A shared cluster has costs that do not depend on how many services run on it: the control plane fee where one applies, the platform engineering time to operate it, and the add-ons. Spread across services, they fall per service as the count grows. The script modeled this with assumed values of 200 price units of monthly fixed cost and 12 units of saving per service. Under those assumptions it printed a break-even at about 16.7 services, with a net of minus 140 at 5 services, plus 4 at 17, and plus 520 at 60.

services=  5 savings=   60.0 fixed= 200.0 net=  -140.0
services= 10 savings=  120.0 fixed= 200.0 net=   -80.0
services= 17 savings=  204.0 fixed= 200.0 net=     4.0
services= 18 savings=  216.0 fixed= 200.0 net=    16.0
services= 30 savings=  360.0 fixed= 200.0 net=   160.0
services= 60 savings=  720.0 fixed= 200.0 net=   520.0

The qualitative result is that shared platform costs favor sharing only above some number of services, and the number depends on the inputs. The 200 and 12 are assumptions with no empirical basis, so do not reuse the 16.7. The total cost of ownership entry gives the discipline to apply: price both options over the same period, workload, and scope, include staff time, show sensitivity to the assumptions, and remember that a cost comparison says nothing about whether an option meets mandatory requirements. A cheaper option that fails a hard constraint, such as privileged access, is not an option.

Migration conditions: when to move, and how

A runtime decision is not permanent, and a migration is itself a project with a cost. Define in advance the conditions that would justify moving, so that the decision is driven by evidence instead of fashion or frustration.

Triggers for moving toward Kubernetes

A simpler runtime has limits that a team can state in measurable terms. Triggers worth defining include a count of services or teams that makes per-service setup and per-host operation unmanageable, a need for multi-tenant primitives such as quotas, namespaces, and admission policy that the current runtime lacks, workloads that need scheduling features the current runtime cannot express, a dependency on operators or custom controllers, and a portability requirement with a stated reason. Each trigger should have a threshold and a measurement, for example how many hours per month the team spends on a duty that Kubernetes would automate. The thresholds themselves are decisions that belong to the team and are not set by this post. Without a threshold, a trigger is an opinion.

Triggers for moving away

The reverse also happens. A team that adopted Kubernetes may find that its workloads are a handful of stateless services whose operating duties are mostly cluster upkeep. Triggers include upgrade and add-on work that consumes more capacity than the workloads justify, a lack of staff who can operate the cluster during incidents, and workloads that fit a managed container service’s contract. Moving away has the same migration costs, and the sunk investment in the cluster is not a reason to continue. The glossary entry on exit cost lists what to include.

A migration plan has a fallback

A migration moves workloads, data, traffic, configuration, and operating practices, and each can fail separately. A plan that works should contain the following.

1. Inventory and compatibility. List each workload with its requirements from the earlier table and check it against the target’s constraints. Workloads that fail a hard constraint are handled separately.
2. Equivalent behavior in the target. Configuration, secrets handling, identity, networking, logging, and alerting need counterparts. A manifest that is valid at the target is not evidence the behavior matches.
3. State. For workloads with data, decide whether data moves by replication, by backup and restore, or stays where it is and is accessed from the new runtime. Practice the restore and measure the duration.
4. Traffic shifting in steps. Move a small share of traffic first, watch the agreed metrics, and keep the ability to return. The canary deployment entry explains why the share is an exposure limit, not proof.
5. A tested return path. The migration fallback entry describes a prepared procedure for returning clients to the source after switchover, and notes that problems may appear days or weeks later when a business cycle first exercises a feature. A backup supplies data and does not by itself supply a return procedure.
6. Retire the source only after the window. Keep the source runnable until the return path is no longer needed, and record the date and the person who decides.

The sequence applies whichever direction the migration goes, and each step costs staff time that should be in the comparison above. Migration has its own cost line in the total cost: a runtime that is cheaper at steady state can lose against one that is cheaper to enter if the horizon is short.

A decision procedure, applied to three illustrative workloads

The pieces above combine into an ordered procedure.

1. Describe each workload with the needs checklist, and mark hard constraints.
2. Remove options that fail a hard constraint.
3. For each remaining option, list the duties and owners, and estimate burden by rehearsal or measurement. Mark each estimate as measured or assumed.
4. Check the boundary questions for any managed service and record what remains yours.
5. Benchmark the finalists with an open-loop workload at equal objectives, including failure and scaling events.
6. Build the cost model with explicit assumptions, staff time, and a sensitivity range.
7. Record the decision, its assumptions, the migration triggers in both directions, and a date to review it.

Applied to the invented services, with the assumptions stated earlier, the reasoning might go as follows. This is an illustration of how to use the procedure, not a recommendation for any real system.

Service	Needs that matter	Plausible starting point under the assumptions	What would change it
A: bursty stateless HTTP API	HTTP, stateless, active about 10% of the time	A serverless container service that scales to zero, since the active fraction is far below the assumed break-even	Cold-start latency exceeding the objective; a platform constraint the service violates
B: steady stateful platform service with many dependents	Continuous load, volume-backed state, many callers	Provisioned capacity; Kubernetes is a candidate if the team can own volumes, backup, and upgrades, and a managed database is a candidate if the data layer is the main burden	Staff unable to operate the data layer; a managed data service that meets the objectives
C: nightly batch job needing more memory	Run to completion, large memory, no inbound traffic	A job-oriented managed service or a Kubernetes Job, depending on the memory ceilings and schedule features available	Memory limits of the managed option; job duration limits; cost at the job’s duty cycle

The same company can reasonably end up with more than one runtime. The cost of two runtimes is two sets of operating practices, and the decision should include that cost. One runtime for all three might still be right if a shared platform’s fixed costs fall across enough services, which is the second model above. Whether the number of services is high enough depends on inputs this post cannot supply.

A last point concerns the shared platform. The follow-on posts in this series cover a standard service contract for a platform’s users, the golden path service contract, and the measurement of platform costs, platform unit economics. The runtime decision feeds both: the golden path says what a team gets on the chosen runtime, and unit economics measures what that costs in practice, which is how the assumptions in this post become measured values.

References

• Kubernetes Documentation, Production environment
• Amazon EKS User Guide, What is Amazon EKS?
• Google Kubernetes Engine Documentation, Autopilot overview
• Amazon ECS Developer Guide, What is Amazon Elastic Container Service?
• Google Cloud Run Documentation, What is Cloud Run
• HashiCorp Nomad Documentation, What is Nomad?
• Docker Documentation, How Compose works: the Compose application model