LLM Capabilities, Limits, and System Design

ByJu Yeon Eum

Consider a fictional finance team whose extraction pilot appears successful. The pipeline sends OCR text with document and page identifiers to the model. A language model read supplier invoices and returned the supplier, invoice number, dates, currency, line items, and total as JSON for the payables system. On 200 test invoices, mostly in English, 97% of fields matched what a clerk had entered. The team estimated cost by counting words, assumed about three words to every four tokens, and approved a rollout to 40,000 invoices a month.

Production went differently. About one response in two hundred was not valid JSON, usually a sentence of explanation before the opening brace, and the parser rejected it. The team switched on the service’s schema-constrained output, and complete, successful responses adhered to the supported schema; refusals and incomplete responses still needed separate handling. Three weeks later an auditor found invoices whose due dates did not appear anywhere on the invoice: the schema required a date, the invoices had none, and the model had supplied a plausible one. Multi-page invoices had been cut to fit the model’s input limit by dropping the end, which was where the totals were. The estimate omitted a repeated 3,000-token instruction block and used an English word-count proxy for a multilingual workload. Some requests took about twelve seconds because the requested JSON included a long explanation field. Those examples are not a measured p95 or a worst-case bound.

Each problem traces to how these models work and what they are charged by. The rest of this article follows those mechanics: what a language model does, why tokens are the unit that sets cost, limits, and speed, what structured output guarantees, and how to choose a model for a task.

What the model actually does

A large language model predicts text. Google’s introductory course explains that they predict individual tokens or longer sequences that can extend across several paragraphs. For an autoregressive decoder, the generation process is as follows: given its input and generated prefix, the model produces a probability for each possible next token; one is selected, appended, and the process repeats until a stop condition. The architecture that makes this work at scale is covered in Transformers from Scratch, and how models acquire their knowledge from training text in LLM Pretraining.

Several practical properties follow. The output is chosen, not looked up: the selection procedure, decoding, may be greedy or may involve sampling, and neither setting tells the model what is true. Its answer can use patterns and information learned in training together with the supplied context. A reported training cutoff is neither complete coverage before that date nor a guarantee of accurate recall. A stateless model call does not retain a previous conversation by itself; an application or stateful service can store history and supply it to later calls. And fluent text is not evidence; the same course notes that generated predictions can include mistaken claims. A hallucination is usually confident and well formed, which is what makes it costly.

For design, separate missing evidence from difficult computation. Supplied documents make extraction and summarization easier to check, but they do not guarantee correctness. Current company facts require an authorized source if they are absent from the input. Arithmetic and character counting are different: the necessary information may already be present, yet exact computation can still fail. Use deterministic code for totals and counts, and validate extraction against the correct document fields. The invoice task needs both evidence for dates and a calculation contract for totals.

Tokens set the cost, the limit, and the speed

For text input and output, the model processes tokens defined by its tokenization. Google’s Gemini token guide gives a text rule of thumb, “a token is equivalent to about 4 characters”, and roughly 60 to 80 English words per hundred tokens. The word ratio is an English approximation; even English documents can differ substantially. Token counts depend on the tokenizer and the actual text, including script, numbers, formatting, and code. The same invoice can cost noticeably more tokens in German or Korean than in English, and a table of part numbers more still. Use the selected model or service’s token-counting interface on the actual request, and reconcile it with billed usage. How tokenizers are built, and why they split text as they do, is explained in Tokenization Explained.

Many hosted services bill input and output tokens at separate rates. Gemini documents a combined context budget as well as model-specific input and output limits; check both when reserving space for an answer. This example counts OCR text. Sending PDF pages or images directly uses the service’s multimodal accounting instead of a text-only word ratio. Token volume affects cost and processing time, but it does not by itself determine latency. Measure representative inputs and outputs for every supported language and document format.

The context is a budget, not a memory

Each request has a context window that everything must fit into: instructions, examples, conversation history, the document, and the output. The window holds the explicit evidence available for this call; learned parameters also influence the answer. When material does not fit, something must be left out, and the application decides what. Cutting from the end, as the invoice pipeline did, removes whatever happens to be last, and on an invoice that is the total. For whole-invoice extraction, process all pages that can contain requested fields or line items. Split long documents into chunks with document, page, and row identifiers; reconcile continued rows and duplicates, and check page coverage before accepting an aggregate. First and totals pages suffice only for a restricted document format that puts every requested field there. A missing middle page can hide a different item even when the grand total matches.

Very large windows help but do not remove the problem. Material that fits is not necessarily used well. Google’s documentation on long context reports strong results for finding a single fact in a long input, then notes that when there are multiple pieces of information to find, “the model does not perform with the same accuracy.” Including an entire manual in every request also means paying for it every time. When the same long material starts every request, context caching lets a service reuse its processing of that prefix; the provider’s guidance is to put large, common content at the beginning of the prompt so the prefix matches. Moving a changing timestamp after stable instructions can improve prefix reuse. It does not guarantee a hit: the selected model and API must support the feature, the prefix must meet that model’s minimum size, and scope, lifetime, and matching rules must hold. A 3,000-token prefix is not eligible for every model. Check cached-token usage and end-to-end cost and latency before claiming savings.

A prompt changes the current computation, not the model. Instructions and examples in the input shape the answer without altering the model’s parameters; fine-tuning is one way to update parameters, with its own data and evaluation requirements, described in Fine-Tuning and Alignment.

Embeddings are a different use of a model

Not every language model call generates text. An embedding model turns a piece of text into a vector, a fixed-length list of numbers, placed so that texts with related meaning land near each other. Embeddings are used to search by meaning, group similar documents, detect duplicates, and supply features to classifiers. Applications often precompute and store document vectors, then refresh them when content or the embedding configuration changes. Relative cost depends on the models, input sizes, and workload; queries still need compatible embeddings.

Two limits matter for design. Nearness is not correctness: the closest passage to a question may be about a different product or an expired policy. Similarity comparisons require compatible embedding spaces and preprocessing. Google’s Gemini Embedding 001-to-2 migration is a specific incompatible change requiring re-embedding; it is not a rule that every pair of model versions is incompatible. Some systems deliberately train compatible query and document encoders. Validate compatibility, dimension, normalization, and retrieval quality for the actual migration. The embedding model is therefore a versioned dependency of every index built with it. For the invoice pipeline, embeddings could match an unknown supplier name to the closest known supplier record, under a validated acceptance policy using supplier identifiers and other independent evidence. A high similarity score can still point to the wrong supplier; a threshold alone does not establish identity.

Structured output: a format contract with completion conditions

When output feeds code, it needs a fixed shape. There are three levels of control. Asking for JSON in the prompt usually works and occasionally does not, which is the one-in-two-hundred failure the pipeline saw. Where supported, JSON mode constrains successfully completed output to JSON without enforcing a particular application schema. Structured output with a supported schema constrains successful output to that schema. Refusals, truncation, unsupported schemas, empty results, and transport failures remain separate outcomes to handle.

Schema compliance does not establish factual correctness. Google’s Gemini structured-output guide requires application-side value validation. A required non-null date gives a successful schema-compliant answer no way to express absence and can encourage fabrication; the API can also fail or refuse instead of returning a date. Allow null or an explicit not_found value and evaluate whether abstention is correct. First check response status and completion, then parse and validate the payload, then verify its meaning. Link each evidence span to its document, page, and field; a date appearing on an invoice could be the invoice date rather than the due date. Compare normalized dates and amounts using explicit rules. Reconcile subtotal, tax, discounts, shipping, and rounding rather than assuming raw line items always equal the final total. These checks detect classes of errors but do not prove that every accepted value is correct. The same limits apply to a well-formed tool call naming the wrong order.

Measure input work and output generation separately

Measure end-to-end request time separately from time to the first visible token (TTFT) and the intervals between later visible tokens. TTFT can include queueing, preprocessing, network delay, input processing, and internal reasoning before the answer. For N visible tokens, generation completion time is approximately TTFT plus N−1 inter-token intervals; a mean interval is useful only for the workload measured. Long inputs and high concurrency can make input work or queueing dominant. Autoregressive generation and serving optimizations are explained in LLM Inference Optimization.

For a simplified illustration, use 0.8 seconds TTFT and 15 milliseconds per visible output token, approximating N−1 intervals by N. Then 150 tokens take about 3.05 seconds and 750 take 12.05 seconds. Removing a 600-token explanation reduces this modeled latency by about 74.7% and visible output tokens by 80%; actual serving results must be measured. Internal reasoning before the first visible token is already in measured TTFT, so do not add that time again. Billable reasoning usage follows the selected service’s pricing rules. Streaming can reveal progress earlier without guaranteeing shorter completion time. As of September 2026, Google’s Gemini Batch API documents a 50% standard-rate discount for eligible processing and a target turnaround of 24 hours. This is not an overnight completion guarantee: a job submitted at 20:00 has only 12 hours before an 08:00 deadline. Allow time for partial failures and retries, reconcile per-request results, and prevent duplicate submissions or downstream writes. As with batch scoring, use batch only when its completion behavior fits the business deadline. Interactive requests need an explicit latency budget.

Calculate token charges by category, then add cache creation or storage, retries, and other applicable charges. In this hypothetical workload, 4,500 input and 750 output tokens replace an estimate of 1,500 input and 150 output. With hypothetical prices of 1 cost unit per thousand input tokens and 4 per thousand output tokens, the estimates are 7.5 versus 2.1 units per request, about 3.57 times as much. Removing the explanation gives 5.1 units before caching or batch discounts. Whether further savings reach the original budget depends on eligible cached tokens, hit rate, lifetime, storage, retries, and the actual discount combination. Check the current pricing contract for cache and batch together; do not assume their discounts multiply. Measure total cost per accepted invoice, including chunking and review, before approving the budget.

Choosing a model

The largest available model is rarely the right starting assumption. The choice should come from the task, measured on examples of the task. Build development and selection sets from real inputs, including other languages, long documents, missing fields, and poor scans. Keep regression fixtures for known failures and reserve independent data for final quality confirmation; repeatedly selecting against the same set can overfit it. Define what counts as correct for each field. Then run candidates and compare them on the measures that matter here.

Compare task accuracy by segment: published benchmarks cover other tasks, and an average can hide errors in difficult languages or fields. Check format compliance and handling of absent values together, since they determine the validation and review work that follows.

Test the context limit against the longest real inputs to decide whether documents need splitting or trimming. Measure p95 latency with realistic output lengths rather than relying on an average. Estimate cost per thousand completed tasks with measured token counts for each workload; word counts remain an uncertain proxy even in English.

Apply data-handling and deployment constraints before choosing a candidate: some inputs may have to remain in a region or require self-hosting. Check version stability and the deprecation schedule as well, because a model retired next quarter will require another evaluation.

For this extraction task, compare smaller, faster candidates with larger ones under the same quality margins, critical-error limits, cost, and latency requirements. Use the measured task and segment results to decide whether extra model capacity earns its cost; neither model size nor a general reputation for reasoning establishes that trade-off.

The course material on the subject describes distillation, which produces a smaller model from a larger one, as trading some quality for lower cost; judge that trade using prespecified quality margins, uncertainty, and critical errors by language and field. Fields within one invoice are correlated, so report document-level or appropriately clustered uncertainty. A one-point difference on 300 invoices alone does not establish equivalence. Model compression and serving are covered in Model Compression and Deployment, and the discipline of building an evaluation that tells the truth in Ninety-Seven on the Test Set, with a fixed evaluation protocol so that candidates are compared fairly.

Once chosen, a hosted model should be called by an exact version. Provider aliases can move to newer releases without an application configuration change. Google’s Gemini Models guide advises that “most production apps should use a specific stable model.” Model version pinning reduces changes caused by moving aliases; it does not guarantee identical answers or an unchanged serving environment. Record the provider’s version contract and deprecation schedule, monitor behavior, and evaluate upgrades using regression fixtures plus independent confirmation data. The model, prompt, schema, decoding settings, and embedding model together make one release, which is the practice described in LLMOps.

What a reviewer should ask

  • Is the necessary evidence present, and which parts require exact computation or independent identity checks?
  • Were token counts measured with the model’s tokenizer on real inputs in every language?
  • What happens when an input exceeds the context window, and what is dropped?
  • Can the output schema express that a value is absent?
  • Which values are checked in code against the input or reference data?
  • How long are outputs, and is every output token needed?
  • Was the model chosen on an evaluation set of this task, including hard segments?
  • Is the model called by a pinned version, and when is that version retired?

Three things are often offered as evidence that a model-based feature is ready and are not. A pilot on clean, English, typical examples, which measures the easy segment. Output that always parses, which shows the form is right and says nothing about the values. And a cost estimate from word counts or a price sheet, which is a guess until real token counts are measured.

The revised design for this fictional team accepts nullable fields, sends missing required business values to review, and checks evidence against the correct document and field. It processes all relevant pages, reconciles continued or duplicated rows, and validates totals under the invoice’s accounting rules. Prefix caching is enabled only after eligibility and measured savings are established. The explanation field is removed when unnecessary, and batch is used only where the deadline permits retries and incomplete jobs.

A multilingual selection set helps compare candidates; an independent confirmation set and prespecified quality margins determine whether the smaller model is acceptable. Pinning records the selected model identity while monitoring continues. These changes make errors and costs more observable. Continue checking production samples for invented values with the sampling scope and uncertainty stated, and investigate new failures as inputs and serving conditions change.

References:

Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.