Do You Actually Need a Cluster? When Distributed Processing Pays and When It Costs

A team has a nightly job that aggregates about forty million order rows. It takes thirty minutes on one machine, and the plan for next quarter is to move it to a cluster. The reasoning is familiar and rarely examined: the data is growing, clusters are what you use for data at scale, and the platform team already runs one. The example is invented, and the decision may well be right — but it is being made on a premise that stopped being reliable some years ago, which is that a single machine is small.

This article is about that judgment: what distribution actually buys, what it costs, which conditions genuinely force it, and how much a single node handles today. Product figures below were checked in September 2026 and move upward over time.

The problem the phrase “big data” pointed at has moved

When the vocabulary of big data formed, the constraint was concrete: datasets exceeded what one affordable machine could hold or read in reasonable time, so the work had to be spread across many cheap machines. Two things have changed since.

• One machine got much larger. AWS documents its U7i high-memory instances as offering up to 32 TiB of memory and up to 1,920 vCPUs, in models such as u7i-12tb.224xlarge (896 vCPUs, 12 TiB). These exist for in-memory databases rather than for analytics, but they set the ceiling: a dataset of a few terabytes is no longer automatically a multi-machine problem.
• Single-node engines stopped requiring the data to fit in memory. DuckDB’s documentation states that it supports “larger-than-memory workloads, i.e., it is able to process datasets that are larger than the available system memory (also known as out-of-core processing),” spilling to disk for blocking operators such as GROUP BY, JOIN, ORDER BY, and windowed aggregation.

Add columnar storage, vectorized execution, and the fact that most analytical queries read a few columns of a few partitions rather than everything, and the practical ceiling of one machine is far above where most people place it. What has grown faster than data volume, in many organizations, is the number of concurrent users, the number of pipelines, and the number of sources — which are different problems with different solutions.

The uncomfortable benchmark

In 2015 Frank McSherry, Michael Isard, and Derek Murray proposed a metric they called COST — “the Configuration that Outperforms a Single Thread.” Their definition: “The COST of a given platform for a given problem is the hardware configuration required before the platform outperforms a competent single-threaded implementation.” The point of measuring it is stated in their introduction as a question: “To what degree are these systems truly improving performance, as opposed to parallelizing overheads that they themselves introduce?”

Surveying data-parallel systems published at major systems conferences, they found many with “a surprisingly large COST, often hundreds of cores, or simply underperform one thread for all of their reported configurations” — several had unbounded COST, meaning no configuration beat a competent single-threaded implementation on a high-end laptop. Their epigraph, quoting Paul Barham, is the sentence worth keeping: “You can have a second computer once you’ve shown you know how to use the first one.”

The paper is about graph processing systems in 2015, not about your warehouse in 2026, and citing it as proof that clusters are useless would misread it. What transfers is the discipline: a scaling curve shows how a system behaves as you add resources, not whether the absolute performance is good. Twice as fast on sixteen machines is not impressive if one machine with a better data layout is faster than all of them.

What actually forces distribution

Four conditions are the usual reasons to reach for more than one machine. Two of them force it; two are choices about how you would rather solve the problem. Volume alone is rarely the first one to bite.

Condition	What it looks like	Why one node cannot do it
Throughput beyond one machine’s I/O	The job must read tens of terabytes within a fixed window	Aggregate read bandwidth is what scales with machines; a single node’s is bounded
Concurrency	Hundreds of simultaneous queries or many pipelines competing	A single node runs queries in parallel and can be given resource limits, so this forces distribution only once one node’s throughput or isolation targets are actually missed. Separate compute per workload is the usual answer, and that is workload isolation rather than distributing one query
Availability	The work must survive a machine failure without waiting for a restart	One node is one failure domain
Elasticity of cost	A ten-minute daily peak against a mostly idle day	Also a choice rather than a requirement: a single machine can be started for the run and stopped after it. Managed clusters make that automatic and finer-grained, which is why it is usually bought rather than built

Notice that three of the four are not about data size. That is the practical shift: distribution is usually bought for isolation, availability, and elasticity, and those are good reasons. It is worth keeping three different things apart while deciding, because they need different architectures: splitting one query across machines, giving each workload its own compute, and replicating for failover. “The dataset is large” is the reason people give, and it is the one that most often does not hold up when measured. For the mechanics of how distributed engines divide the work, see massively parallel processing.

What parallelism costs

Splitting work across machines introduces categories of expense that do not exist on one node.

• Moving data between machines. Any operation that regroups rows — a join on a new key, a global aggregation, a sort — has to send data across the network. Spark’s tuning guide notes that shuffle operations build a hash table within each task that “can often be large,” and that data locality “can have a major impact” on performance. This is the cost that grows fastest with cluster size, and the one that a single node simply does not pay.
• Serialization. Data that crosses a process boundary must be encoded and decoded. The same guide calls serialization something that “plays an important role in the performance of any distributed application” and often “the first thing you should tune.”
• Coordination. Scheduling tasks, tracking their status, and handling stragglers costs time that is pure overhead relative to a loop on one machine. Spark’s own recommendation of “2-3 tasks per CPU core” exists because task granularity is a tuning problem that single-node execution does not have.
• Skew. Parallel work finishes when the slowest partition finishes. One popular key, one enormous customer, one null-heavy join column, and forty machines wait for one — see partitioning, shuffles, and data skew.
• Operations and expertise. A cluster brings cluster sizing, version upgrades, dependency management, failure modes that only appear under load, and a debugging experience where the stack trace is on another machine. That burden is real whether or not it appears on an invoice.

None of this argues against clusters. It argues that the comparison is not “one machine, slowly” against “many machines, quickly” but against “many machines, quickly, minus the overheads the distribution itself introduced, plus the work of running it.”

How far one node goes, in practice

A single-node engine reading columnar files handles more than most estimates assume, because the data that matters is smaller than the data that exists:

• Partition pruning removes whole files before reading, so “three years of history” is often “eleven days” for the query at hand.
• Predicate pushdown and column projection mean a 200-column table read for four columns costs roughly four columns.
• Out-of-core processing removes the requirement that the working set fit in RAM, within documented limits — DuckDB notes that several blocking operators in one query can still exhaust memory, and that some aggregate functions cannot spill at all.

The honest way to find the boundary is to measure it rather than to reason about it: run the real query on the real data on one large machine, and see. Single-node analytics with DuckDB, Polars, and Arrow covers the tooling, and working with tabular data at scale covers the techniques that keep it viable.

A decision order that avoids the common mistake

1. State the requirement in time, not in size. “This must finish within twenty minutes of the source being available” is a testable constraint; “we have big data” is not.
2. Measure the honest baseline. One machine, columnar files, a sensible partition layout, and the real query. Most “we need a cluster” conversations have never seen this number.
3. Fix the data layout before adding machines. Small files, no partitioning, row formats, and reading columns nobody uses are the usual reasons a job is slow — and distribution multiplies those costs rather than removing them. File compaction is often the cheapest speedup available.
4. Name which of the four conditions applies. Throughput, concurrency, availability, or elasticity — with the number that shows it.
5. Compare the whole cost, including the operating burden and the skills required, not just the compute bill. That is ordinary total cost of ownership reasoning applied to an architecture choice.

Then leave the decision alone until something moves it. The trigger to revisit is a change of an order of magnitude in volume, concurrency, or the deadline itself — not a quarterly review, which produces churn without information.

And the reverse case deserves the same discipline. A team already running a cluster that works, with staff who know it, should not move a stable workload to a single node because a blog post said so. The cost of a migration is real, and “it would run on one machine” is not by itself a reason to make it.

References

Documentation and instance figures were checked in September 2026; hardware ceilings and engine capabilities change, so confirm current values before deciding.

• Frank McSherry, Michael Isard, and Derek G. Murray, Scalability! But at what COST?, HotOS XV, 2015
• DuckDB Documentation, How to tune workloads
• Amazon Web Services, Amazon EC2 U7i Instances
• Apache Spark 4.2.0 documentation, Tuning Spark