Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO

A pretrained language model can answer questions, imitate dialogue, or continue a list of questions, depending on its training and context. Post-training aims to make the desired behavior more reliable. SFT learns from demonstrations; preference methods learn from comparisons. LoRA specifies which parameters change and can be used with either objective. These are choices that can be combined, not a mandatory sequence of four stages.

Supervised fine-tuning

Supervised fine-tuning (SFT) applies next-token training to demonstrations such as instruction–response conversations. An assistant-only loss trains on assistant responses while retaining the prompt as context. Full-conversation losses are another training choice; including prompt tokens is not inherently an error. Specify which roles contribute, including whether assistant end-of-turn tokens are targets.

Use the checkpoint’s actual chat template and special tokens. A manually invented <|user|> string need not be a recognized role token. Tokenize the complete formatted conversation and obtain a response mask aligned to that tokenization; separately encoding prompt and answer can change boundary tokens or duplicate special tokens. Some templates can return assistant masks, but support depends on the template, so inspect a decoded example and the scored positions. The helper below starts from those IDs and masks and rejects overlong examples; shortening, splitting, or discarding them is an explicit data decision. Run the examples in order with PyTorch.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

def build_sft_example(input_ids, assistant_mask, max_len=2048):
    """Token IDs and a per-token boolean assistant mask from the same formatting pass."""
    if len(input_ids) != len(assistant_mask):
        raise ValueError("IDs and assistant mask must have equal lengths")
    if not isinstance(max_len, int) or max_len < 2 or not 2 <= len(input_ids) <= max_len:
        raise ValueError("Example must fit without silently truncating the response")
    if not all(isinstance(flag, bool) for flag in assistant_mask):
        raise ValueError("assistant_mask must contain booleans")
    labels = [token if flag else -100 for token, flag in zip(input_ids, assistant_mask)]
    if not any(label != -100 for label in labels[1:]):
        raise ValueError("No response target remains after the causal shift")
    return {"input_ids": list(input_ids), "labels": labels}

def response_logps(logits, labels):
    """Full-sequence (B, T, V) logits and aligned (B, T) labels; shift once here."""
    if logits.ndim != 3 or logits.shape[:2] != labels.shape:
        raise ValueError("Expected aligned logits and labels")
    shifted = labels[:, 1:]
    mask = shifted != -100
    counts = mask.sum(1)
    if not (counts > 0).all():
        raise ValueError("Each sequence needs a scored response token")
    safe_ids = shifted.masked_fill(~mask, 0)
    logps = logits[:, :-1].log_softmax(-1).gather(-1, safe_ids.unsqueeze(-1)).squeeze(-1)
    return logps.masked_fill(~mask, 0).sum(1), counts

example = build_sft_example([0, 1, 2, 3, 4], [False, False, False, True, True])
labels = torch.tensor([example["labels"]])
logits = torch.zeros(1, 5, 5, requires_grad=True)
sums, counts = response_logps(logits, labels)
sft_loss = -sums.sum() / counts.sum()
sft_loss.backward()
print("labels", example["labels"])
print("scored tokens", counts.tolist(), "mean loss", round(sft_loss.item(), 4))
# labels [-100, -100, -100, 3, 4]
# scored tokens [2] mean loss 1.6094

In the toy example, IDs 0–2 stand for the prompt and assistant prefix, and IDs 3–4 for the answer and its end marker. Labels remain aligned with the full input. The loss helper shifts once, so logits at positions 2 and 3 predict IDs 3 and 4. Uniform probabilities over five tokens give a mean loss of log(5), about 1.6094. Padding labels should be −100, with an appropriate attention mask supplied to the model. Masking prompt labels excludes prompt prediction targets; it does not prevent the response from attending to the prompt. Frameworks that shift internally must receive unshifted labels; do not shift a second time.

Small, carefully selected demonstration sets have worked well in particular studies, but data quantity and coverage still matter. SFT can teach task behavior, domain information, and refusal decisions from appropriate examples. It is not restricted to formatting knowledge already present in pretraining. Evaluate learning rate, duration, mixture, and loss masking on held-out tasks instead of treating a fixed number of epochs as a universal boundary.

Adaptation can improve targeted behavior while degrading other capabilities, but this trade-off is not inevitable and is not defined by exceeding three epochs. Compare with the base model on the tasks you need to retain. LoRA and full fine-tuning may require different learning rates, so a fixed ratio to the pretraining rate is not a sufficient tuning rule.

How these models are pretrained at scale is covered in LLM Pretraining: Objectives, Data, and Scaling Laws.

LoRA

Under a 16-byte-per-parameter mixed-precision Adam estimate, 7B parameters require about 112 decimal GB (104 GiB) for weights, gradients, and optimizer state, before activations and temporary storage. Actual dtypes and sharding change that estimate. LoRA freezes a base matrix and learns two smaller matrices whose product supplies an update:

\[W’=W+\frac{\alpha}{r}BA,\qquad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times k},\;r\ll\min(d,k)\]

For a base matrix \(W\) with \(d\) outputs and \(k\) inputs, \(A\) maps to \(r\) intermediate coordinates and \(B\) maps them to \(d\) outputs. The update has rank at most \(r\) and contains \(r(k+d)\) trainable entries. This restricts the adaptation; the best full update need not be low rank. With finite values, random \(A\) and zero \(B\) give an initially zero adapter output. At that initialization the loss gradient into \(A\) is zero, while \(B\) can receive a nonzero gradient and start learning.

from copy import deepcopy

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.05):
        super().__init__()
        if not isinstance(r, int) or not 1 <= r <= min(base.in_features, base.out_features):
            raise ValueError("Choose a positive rank no larger than either matrix dimension")
        if not math.isfinite(alpha) or alpha <= 0 or not 0 <= dropout < 1:
            raise ValueError("Use positive finite alpha and dropout in [0, 1)")
        self.base = base.requires_grad_(False)
        self.A = nn.Parameter(base.weight.new_empty(r, base.in_features).normal_(std=0.01))
        self.B = nn.Parameter(base.weight.new_zeros(base.out_features, r))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.scale * ((self.drop(x) @ self.A.T) @ self.B.T)

    @torch.no_grad()
    def merged_linear(self):
        if self.training:
            raise ValueError("Switch to eval mode before merging")
        merged = deepcopy(self.base)
        merged.weight.add_(self.scale * (self.B @ self.A))
        return merged

d, r = 4096, 16
trainable, base_weights = 2 * d * r, d * d
print(trainable, base_weights, f"{trainable / base_weights:.4%}")
# 131072 16777216 0.7812%

For the single 4096×4096 matrix above, the adapter has 0.7812% as many entries as the base weight matrix; this is not the trainable fraction of an entire model. Frozen weights need no new gradient or Adam moment tensors, but they still occupy memory and participate in computation. Backpropagation through them may still be needed to train earlier adapters. Activation memory and checkpointing remain relevant.

The implementation creates adapters on the base weight’s device and with its dtype. In evaluation mode, merging adds the update to an ordinary dense matrix and removes the separate adapter multiplications. With dropout active, that equivalence does not hold. Floating-point rounding can cause small differences, and quantized weights need a separate merge/requantization procedure. Adapters also depend on the exact base checkpoint they were trained against.

This small regression example exercises adapter learning and merging without loading a language model. Its base layer is random and has no linguistic capability. The first backward pass reaches B while A has a zero gradient; after B changes, gradients can also reach A. The final comparison checks the algebra of merging in evaluation mode.

torch.manual_seed(4)
adapter = LoRALinear(nn.Linear(4, 3).double(), r=2, alpha=4, dropout=0)
x_demo = torch.randn(8, 4, dtype=torch.float64)
with torch.no_grad():
    target_demo = adapter.base(x_demo) + 0.1
optimizer = torch.optim.Adam([adapter.A, adapter.B], lr=0.03)
for step in range(40):
    optimizer.zero_grad(set_to_none=True)
    loss_demo = F.mse_loss(adapter(x_demo), target_demo)
    loss_demo.backward()
    if step == 0:
        print("initial A gradient norm", adapter.A.grad.norm().item())
        print("initial B gradient nonzero", bool(adapter.B.grad.norm() > 0))
    optimizer.step()
adapter.eval()
with torch.no_grad():
    merged = adapter.merged_linear()
    print("merged output agrees", torch.allclose(adapter(x_demo), merged(x_demo), atol=1e-12, rtol=1e-12))
    print("base has gradients", any(p.grad is not None for p in adapter.base.parameters()))
# initial A gradient norm 0.0
# initial B gradient nonzero True
# merged output agrees True
# base has gradients False

Attention projections and MLP matrices are candidate adapter locations. Rank, target modules, dropout, scale, and training budget should be compared for the task. Setting alpha to twice the rank keeps alpha/r equal to two; it does not make update norms or learning dynamics invariant to rank. There is no rank range that guarantees a particular kind of adaptation, and increasing the scale is not generally identical to changing an optimizer’s learning rate.

QLoRA stores frozen base weights in a 4-bit representation and backpropagates through dequantized computations into adapters. Its reported method combines NF4 quantization, quantization of scale constants, and paged optimizers to manage memory. The paper demonstrated fine-tuning a 65B model on one 48GB GPU under its configuration. This is not a guarantee for arbitrary context lengths, batches, or implementations. Storage precision, computation precision, and adapter/optimizer dtypes are distinct choices; QLoRA does not make the entire training computation four-bit.

What preference comparisons add

A demonstration supplies a target response for a prompt. A preference pair supplies two responses to the same prompt and a judgment about which is better: \((x,y_w,y_l)\). The subscripts mean preferred and less preferred, not objectively correct and incorrect. Either response can contain errors. Comparisons can be easier to collect for some tasks, but specialist or ambiguous judgments may still be difficult and inconsistent.

RLHF

One common RLHF workflow starts from an SFT policy, collects response comparisons, and trains a reward model that assigns a scalar score to a prompt–response pair. Under a Bradley–Terry model, the probability that the preferred response wins is the sigmoid of the reward difference:

\[\mathcal{L}_{RM}=-\log\sigma\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right)\]

A reward gap of zero predicts preference probability 0.5; a gap of log(3) predicts 0.75. Adding the same prompt-dependent constant to both rewards leaves the comparison unchanged. Reward scores are learned predictors of preferences, not independently verified measures of truth or safety.

Next, sample responses from the current policy, score them with the reward model, and update the policy, often using PPO. A common objective for prompts \(x\sim\mathcal D\) is:

\[\max_\pi\;\mathbb E_{x\sim\mathcal D}\left[\mathbb E_{y\sim\pi(\cdot\mid x)}r_\phi(x,y)-\beta D_{KL}\bigl(\pi(\cdot\mid x)\Vert\pi_{\mathrm{ref}}(\cdot\mid x)\bigr)\right],\quad\beta>0.\]

The fixed reference is commonly the starting SFT policy. The KL penalty discourages moving too far from it, but does not guarantee that reward exploitation or capability loss is prevented. Other constraints are possible in other algorithms. Monitor generated outputs, reward, KL, task performance, and refusal behavior together; a favorable value of one metric is insufficient.

PPO typically uses a value estimator to predict future reward and form an advantage, which indicates whether a sampled response did better than expected. Its clipped surrogate objective reduces the incentive for large probability-ratio changes relative to the policy that generated the samples. This is not a hard bound on policy movement and is separate from the KL penalty to the fixed reference. Policy, reference, reward, and value are four roles, not necessarily four full models simultaneously resident on one device: implementations may share components, shard, offload, or cache results. Online generation and optimization still make this workflow more involved than a fixed-pair loss.

DPO

For a fixed reward and reference, the ideal optimizer of the KL-regularized objective has \(\pi^*(y\mid x)=\pi_{\mathrm{ref}}(y\mid x)\exp(r(x,y)/\beta)/Z(x)\), assuming positive reference support and a finite normalizing sum. Rearranging gives \(r(x,y)=\beta\log[\pi^*(y\mid x)/\pi_{\mathrm{ref}}(y\mid x)]+\beta\log Z(x)\). The final term cancels between responses to the same prompt. Substituting this into the Bradley–Terry loss motivates DPO’s policy-based parameterization:

\[\mathcal{L}_{DPO}=-\log\sigma\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right)\]

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    """Each tensor is (B,): summed conditional log-probability of response tokens."""
    values = (policy_chosen_logps, policy_rejected_logps,
              ref_chosen_logps, ref_rejected_logps)
    if values[0].ndim != 1 or values[0].numel() == 0 or any(v.shape != values[0].shape for v in values):
        raise ValueError("Use matching nonempty vectors")
    if not math.isfinite(beta) or beta <= 0:
        raise ValueError("beta must be positive and finite")
    policy_gap = policy_chosen_logps - policy_rejected_logps
    ref_gap = ref_chosen_logps.detach() - ref_rejected_logps.detach()
    return -F.logsigmoid(beta * (policy_gap - ref_gap)).mean()

Use response_logps above separately for preferred and less-preferred sequences, under the policy and reference. Score the response and its chosen end marker, exclude prompt and padding labels, and shift exactly once. Standard DPO sums token log-probabilities; replacing the sum with a length average changes this objective. Both models must use compatible tokenization and formatting. Reference scores can be computed without gradients and cached for a fixed dataset and fixed reference, so a live reference model is not required on every training batch.

The fixed-dataset DPO loop has no separate reward-model fit or on-policy generation step. It still depends on the preference data’s coverage, label quality, beta, and the reference policy. Iterative variants can collect new responses and train again; DPO is not inherently unable to use online data. The algebra does not guarantee that finite neural-network training reproduces an ideal RLHF optimum, or that either method wins on every task.

Choosing a method

Retrieval makes external evidence available in the context and can make updates easier than changing model weights. It does not guarantee that the retrieved evidence is correct or that the answer uses it faithfully. Fine-tuning can improve retrieval use, citation behavior, and domain tasks, so these approaches can complement each other. Compare them on representative queries rather than treating a method name or dataset size as a quality guarantee.

How to decide which improvement to attempt next is the subject of A Systematic Strategy for Improving Machine Learning Models.

Evaluation

Evaluate the behaviors you intend to change and the capabilities you need to retain. Perplexity alone does not measure conversational helpfulness, and a general benchmark suite can miss instruction-following or refusal failures. Keep evaluation prompts and near-duplicates out of the training examples, and report performance on relevant prompt groups.

Pairwise judgments can compare the tuned model against a baseline. Define the rubric, count ties explicitly, and report the number of prompts and uncertainty alongside win rate. Human and model judges can disagree; model judges can favor response length, presentation order, or familiar wording. Swap response positions or balance their order, keep the judge prompt fixed, and inspect a human-reviewed subset. Length and position controls alone do not remove all judge bias.

Measure desired refusals and mistaken refusals of answerable prompts separately. Check factual correctness, task success, and retained capabilities alongside preference scores: a more agreeable answer can still be wrong. When a trade-off appears, judge it against the intended application instead of collapsing all behavior into one alignment number.

Compressing and serving the finished model is covered in Model Compression and Deployment: Distillation, Pruning, and Serving.

The reinforcement learning this builds on is covered in Deep Reinforcement Learning: MDPs, DQN, Policy Gradients, and PPO.

Exercises

1. LoRA parameter accounting. Assume a nominal 7B model with 32 layers and square 4096×4096 query and value matrices in each layer. Adapt only those two matrices per layer at ranks 8 and 64. Compare their adapter counts with full fine-tuning of the nominal 7B parameters.

You should get: adapter counts tied to the stated target matrices, rather than a universal fraction for all 7B models.

d, L, total = 4096, 32, 7e9
for r in (8, 64):
    per_matrix = r * d + d * r
    trainable = L * 2 * per_matrix
    print(f"rank {r:3d}  trainable {trainable:>12,}  "
          f"{100 * trainable / total:.4f}% of the model")
# rank   8  trainable    4,194,304  0.0599% of the model
# rank  64  trainable   33,554,432  0.4793% of the model

2. DPO from the RLHF objective. Show that DPO uses a binary logistic loss on the implicit reward difference, and explain what this removes from the RLHF pipeline and what it does not.

You should get: a binary preference likelihood built from policy and fixed-reference response scores.

lp_chosen = torch.tensor([-12.0], requires_grad=True)
lp_rejected = torch.tensor([-14.0], requires_grad=True)
ref_chosen, ref_rejected = torch.tensor([-13.0]), torch.tensor([-13.5])
beta = 0.1
r_chosen = beta * (lp_chosen - ref_chosen)
r_rejected = beta * (lp_rejected - ref_rejected)
loss = dpo_loss(lp_chosen, lp_rejected, ref_chosen, ref_rejected, beta)
loss.backward()
print(f"implicit rewards {r_chosen.item():.4f} {r_rejected.item():.4f}")
print(f"loss {loss.item():.4f}")
print(f"log-prob gradients {lp_chosen.grad.item():.6f} {lp_rejected.grad.item():.6f}")
# implicit rewards 0.1000 -0.0500
# loss 0.6210
# log-prob gradients -0.046257 0.046257

3. Learning when to refuse. Can SFT learn different responses to prompts that warrant refusal and prompts that should be answered? Explain what paired preference data adds and how you would test mistaken refusals.

You should get: two forms of supervision whose usefulness depends on examples, coverage, and evaluation.

References

• Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences. NIPS.
• Schulman, Wolski, Dhariwal, Radford, and Klimov (2017). Proximal Policy Optimization Algorithms. arXiv preprint.
• Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint.
• Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
• Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS.
• Dettmers, Pagnoni, Holtzman, and Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.
• Rafailov et al. (2023). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS.
• Zhou et al. (2023). LIMA: Less Is More for Alignment. NeurIPS.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.