MLOps and LLMOps Explained: From Experiment to Operated System

What changes after a successful experiment?

A shop tests two systems: a model that forecasts tomorrow’s orders and an assistant that explains return policies. The forecast looks accurate on a test file; the assistant answers a few examples correctly. Before either becomes a service, the team still needs to know which inputs and versions produced those results, who accepts the remaining errors, how users receive the result, and what happens when it stops working. These questions connect experimentation to operation.

MLOps applies engineering and operating practices to the lifecycle of machine-learning systems. LLMOps is commonly used for the corresponding work around large-language-model applications. The boundary is not a universal standard: LLMOps overlaps MLOps and software operations, while putting particular attention on prompts, retrieval, generated answers, and tool behavior. Neither term means merely installing a platform or automating model training.

This article follows the decisions between preparation, evaluation, registration, deployment, monitoring, and revision. The stages are responsibilities rather than mandatory tools or departments. You can follow them without knowing training algorithms or deployment commands. Google Cloud’s MLOps architecture describes how development, testing, delivery, and operation fit together.

The lifecycle is a loop with decisions

Define the task and acceptance rules
                 ↓
Prepare data and candidate configuration
                 ↓
Train or adapt if needed → evaluate the candidate
                                   ↓
                      record artifacts and evidence
                                   ↓
                         promotion decision
                         ├─ reject → revise and re-evaluate, or stop
                         └─ approve
                                ↓
                      deploy and verify behavior
                                ↓
                     monitor and investigate
                     ├─ continue operating
                     ├─ revise → return to preparation and evaluation
                     └─ retire

Investigated failures feed back into development evaluation cases.

A candidate is a proposed version of the system. Promotion is the decision to allow it into a more consequential environment or role; deployment is the technical work of making it run there. A failed evaluation can send the team back to the data, features, prompt, or application code. It does not necessarily call for more training. A working system can also be retired when its purpose ends, its data is no longer permitted, or a simpler approach serves the need.

Preparation must identify what was actually tested

For the forecast, record the input data version, the time range, the meaning of an order, the features, the training code, and the settings. A feature is a model input such as recent order volume. Separate examples used to fit or tune the model from examples used to assess it. For a future forecast, an evaluation must not use information that would only become available after the prediction time. Otherwise the reported accuracy answers an easier question than the deployed system faces.

Check that feature processing agrees between training and serving, the stage that produces predictions for use. Suppose training counts completed orders, but the live input also counts canceled orders. The field can have the same name and numeric type while meaning something different. This training-serving skew can undermine a model without changing its file. Test definitions, time boundaries, missing-value handling, and transformations using representative inputs in both paths.

For an assistant built on a hosted model, your team may do no model training. It still changes the application: the selected model version, instructions, retrieval settings, policy documents, tool definitions, and response checks. A new prompt is a candidate change even when the model name stays the same. If the provider does not expose or preserve an exact model snapshot, record the identifier and date you used and recognize that exact replay may be unavailable.

An experiment tracking record links a run to its inputs, configuration, outputs, and measurements. Preserve enough to explain and, where possible, reproduce the result within the agreed retention policy. Recording a random seed alone is insufficient: different data, software libraries, hardware behavior, or external services can change the outcome. Do not retain private prompts or training records indefinitely just because they help debugging.

Evaluate the task, not just a model score

A baseline is a reference approach, such as yesterday’s accepted model or a simple seasonal forecast. Compare a candidate with it on the same relevant cases. For order forecasting, examine error by time period or shop rather than relying only on a combined average. For the assistant, examine whether the answer preserves policy conditions, cites supporting material, and accurately reports tool outcomes. Also measure response time and the cost of providing a useful result.

Different checks answer different questions. A unit test checks a small piece of code. An integration test checks components together, such as the assistant calling a test order service. Behavioral evaluation checks whether the resulting answer or prediction meets the task’s criteria. A model used as a judge can help score some outputs, but its agreement with human reviewers and its failure cases need assessment too. Passing ordinary software tests does not establish answer quality.

Choose acceptance rules before selecting the winning candidate. A release gate uses those rules and the supporting evidence to decide whether a candidate may be promoted. Keep development cases for iteration and reserve independent cases for checking generalization. Include missing evidence, unusual inputs, and failures of dependencies. MLflow’s LLM evaluation documentation illustrates evaluation datasets and scoring for application behavior; the criteria themselves still belong to the team and its users.

When generation varies between runs, one answer per case can hide unstable behavior. Set a repeat count suited to the task before comparing candidates, record generation settings, and inspect repeated failures as well as overall scores. Repeating a question probes variability on that question; it does not create new independent user scenarios. Keep fresh held-out cases for later checks when familiar failures have become development examples.

Registration records a version; it does not approve it

An artifact is an output needed to run or inspect a version, such as a model file, a configuration bundle, or an evaluation report. A model registry organizes versions and their associated metadata. It can connect a model to the run that produced it and give teams a stable way to identify a candidate. Saving an artifact, labeling it “production,” or changing an alias does not by itself demonstrate that it passed evaluation or that the serving system loaded it.

An alias is a movable name pointing to a version. To investigate a particular request, log the resolved version actually loaded by the serving process; the alias’s current target may differ from what handled that request. Preserve the corresponding artifact or its content hash where available, rather than relying on the alias alone. MLflow’s registry documentation describes version management, lineage, and aliases. A registry is one implementation choice, not a substitute for the release decision or a record of actual execution.

For an LLM application, track a compatible release of the whole behavior. An illustrative release record could look like this. These names are invented; the record is not executable configuration. References must resolve to artifacts or evidence the team actually retains, and secrets should be managed separately.

release: support-assistant-r12
application_code: app-c42
model_identifier: provider-model-snapshot-A
prompt_version: returns-p8
retrieval_config: retrieve-r4
policy_index: policy-v3-index-b2
tool_contract: order-api-v2
evaluation: eval-set-v5 / report-r12
previous_compatible_release: support-assistant-r11
release_owner: support-platform-team

A better average can still be a rejected release

Consider a fictional evaluation of the returns assistant on 100 cases: 80 routine questions and 20 cases where claiming an unconfirmed action is a critical error. Each case passes only if it meets all of its assigned criteria. The team has agreed that every critical case must pass before this candidate is eligible for a limited rollout. This is an example policy, not a universal numerical threshold.

Version	Routine cases passed	Critical cases passed	Total passed	Decision under the stated rule
Accepted r11	70 / 80	20 / 20	90 / 100	Reference version; other release requirements still apply
Candidate r12	78 / 80	17 / 20	95 / 100	Reject promotion; investigate the three critical failures

The overall pass rate rises from 90% to 95%, a gain of five percentage points. Yet the candidate violates the critical-case rule. Keep the result and failure examples, correct the cause, and evaluate the changed candidate again. Passing all 20 critical examples would mean only that those examples passed; it would not establish zero risk for future requests. A small curated set is not an estimate of the production error rate unless its sampling and evaluation design support that interpretation.

Match the inference path to the work

Synchronous inference returns a result while the caller waits, as in a live recommendation. Asynchronous inference accepts a job and provides its outcome later, which can suit a long recording or large document. Batch inference processes a bounded dataset, such as a scheduled set of customer scores. These terms describe different aspects of timing and workload; a batch job can itself run asynchronously.

For a concrete service distinction, Amazon SageMaker AI offers real-time and asynchronous endpoints, while Batch Transform runs inference without requiring a persistent endpoint. Batch Transform is not a third kind of live endpoint. In its asynchronous path, an input can be stored in S3 and referenced in the request, with results written to storage. The Batch Transform and asynchronous inference documentation describe these paths. The application must distinguish job acceptance from successful completion and check failures, timeouts, and output identity.

Large inputs need an input-handling decision

A payload is the data carried in a request. A long video can be expensive to transfer even before inference begins. Sending a storage reference makes the request smaller but does not eliminate the upload, processing, or access requirements. The worker needs authorized access, a stable input version, and a way to associate the final result with the job.

Compression, image resizing, frame selection, chunking, or local feature extraction can reduce transfer, but each must preserve evidence the task needs and match the model’s expected input. Taking one frame per second can miss a brief event. An audio model expecting a waveform cannot simply receive arbitrary extracted features instead. More bytes do not always imply proportionally more model computation: encoding, decompression, tokenization, and model architecture intervene.

Streaming requires support along the processing path; it does not automatically bypass a request or context limit. Batching can reduce per-request overhead while increasing waiting time and memory use. Evaluate the transformed inputs, resource limits, retries, and failure recovery as part of the candidate release.

Managed services still need an operating plan

Managed ML offerings on AWS and Google Cloud provide facilities for training, tuning, model deployment, and operating workflows. Google Cloud’s managed machine-learning overview illustrates these responsibilities. Feature stores, registries, and pipeline tools are optional components selected for the workload, not a compulsory sequence for every application.

Teams still choose artifacts and compatible runtimes, configure scaling and permissions, validate predictions, and own recovery. Autoscaling has configuration, capacity, and startup limits; it does not promise instant handling of every traffic spike. Compare idle capacity, active compute, storage, transfer, and operating effort. A workflow scheduler coordinates steps, but task quality, fairness checks, and approval rules must be deliberately configured and reviewed.

Deployment changes exposure to users

Test that the candidate can load, accept the expected input, reach its dependencies, and produce a valid response in the target environment. A shadow run sends copies of inputs to a candidate without using its result for the user’s decision; side effects such as submitting a return must be disabled or isolated. A canary release serves a limited portion of real work with the candidate. It exposes some users to the new behavior, so define who receives it, what is measured, and when to stop.

A limited rollout reduces the scope of exposure while evidence accumulates; it does not prove safety or quality merely by remaining online. Choose a comparison period and enough relevant cases, and watch important user groups separately. The SRE Workbook’s canary guidance explains release comparison and limiting the impact of a bad change.

Rollback restores a known compatible release, not necessarily one model file. An older prompt may no longer work with a changed tool schema, and an obsolete policy index must not be restored just because it existed in the previous release. Verify compatibility and current data-access rules before switching back. Rollback also cannot undo emails sent or transactions already executed; those need separate investigation and, when appropriate, compensating actions.

Monitoring must connect a signal to a response

Layer	Example signal	What the signal does not establish
Service	Errors, timeouts, queue length, response time	A successful response can contain an incorrect answer
Inputs and dependencies	Missing features, changed input distribution, stale documents	A distribution change alone does not prove task quality fell
Task quality	Forecast error after actual orders arrive; reviewed answer failures	A small or biased feedback sample does not describe all users
Operations and cost	Tool failures, cost per task, fallback frequency	Low cost is not evidence that the task was completed correctly

Drift describes change relative to a reference, such as a different distribution of input values. It is an investigation signal. The change may come from seasonality, a new user group, or a broken data feed. Determine the cause and task impact before choosing retraining. For forecasting, actual outcomes arrive later, so immediate service metrics and delayed quality measurements answer different questions.

For an assistant, a request trace connects the input, selected evidence, model and prompt versions, tool calls, and final result. It helps identify where a failure arose. Capture only what is permitted and useful, with access restrictions, redaction where needed, and retention limits. Assign an owner and response to each actionable alert: investigate, disable a tool, use an approved fallback, or stop the service. Automatically retraining on every alert can amplify a bad input or reward a misleading feedback signal.

Automation needs owners and explicit handoffs

Continuous integration (CI) checks changes as they are integrated. Continuous delivery keeps a tested release ready to deploy; continuous deployment goes further by deploying eligible changes automatically. Continuous training automates a training path when its triggers and conditions are met. It need not mean training after every request, and training success must not silently bypass release evaluation. A small team can begin with repeatable scripts and recorded review before automating more steps.

Responsibility	A concrete handoff
Task or product owner	Defines acceptable outcomes, important user groups, and the cost of failure
Data owner and data engineer	Specify definitions, permitted use, data versions, and quality signals
Model or application developer	Supplies a reproducible candidate, evaluation results, and known limitations
Platform or release operator	Verifies packaging, access, rollout, actual version in use, and recovery
Service owner or on-call responder	Receives actionable alerts and has authority to limit harm or restore service

These are responsibilities, not a required organization chart. One person can hold several, but “the pipeline did it” cannot answer who approved a changed policy or who will respond tonight. A handoff is complete when the receiving person or system has the artifact, evidence, authority, and instructions needed for the next step.

Check whether you can make the release decision

1. A candidate passes 95 of 100 cases while the accepted version passes 90. Can it be promoted under the example’s rule?

Solution

No. The candidate fails three critical cases, while the rule requires all critical cases to pass. The aggregate improvement does not override the stated constraint. Keep the candidate’s artifacts and evidence for investigation, but do not label the evaluation as a release approval.

2. The model version did not change, but the assistant now gives wrong return advice after a policy-index refresh. Is rolling back the model the obvious fix?

Solution

No. Inspect the retrieved passages, source versions, permissions, prompt, and request trace. The changed index may have introduced missing or obsolete evidence. Correct the responsible component, test the compatible application release, and preserve current policy and access requirements during recovery.

3. Input drift is detected, but actual order counts will not arrive until tomorrow. What can the team conclude now?

Solution

It can conclude that the monitored input measure differs from its reference under the chosen detector. It cannot yet infer the forecast error from that signal alone. Check data validity and recent changes, apply the agreed interim response, and compare predictions with actual outcomes when they arrive before deciding on the remedy.