Datasets That Can Disagree With You

In this chapter

• Create grouped training and evaluation splits
• Explain supervised loss masks
• Keep ambiguous labels visible

A split is an experimental boundary

Suppose one fictional customer conversation produces five paraphrased training examples. If four enter training and one enters evaluation, the held-out example is not meaningfully independent. Group related messages by conversation, incident, source document, or generation seed before splitting. A random row split can look balanced while leaking almost identical content into both sides.

Define the label policy before collecting a large dataset. A message about an invoice page that will not load could be billing or reliability depending on the operational contract. Write the boundary explicitly, retain ambiguous cases for review, and record disagreements. A model cannot repair a training objective that tells it two incompatible things without explanation. Synthetic examples are useful for mechanics, but do not establish performance on real customer language.

Choose which positions contribute to loss

In supervised language-model adaptation, some token positions can be excluded from the training loss. For example, a prompt may provide context while only the desired response is scored. Padding positions should not become meaningful targets. Masking a position from loss does not remove it from the model's input; attention masking is a different operation. Accidentally scoring prompt text can change the training objective in ways that aggregate loss obscures.

import hashlib
import math

examples = [('thread-a', 'invoice issue'), ('thread-a', 'same incident'), ('thread-b', 'login issue')]
def split(group):
    number = int(hashlib.sha256(group.encode()).hexdigest()[:8], 16)
    return 'evaluation' if number % 5 == 0 else 'training'
assert split(examples[0][0]) == split(examples[1][0])

correct_token_probabilities = [0.9, 0.2, 0.8]
loss_mask = [0, 1, 1]
selected = [-math.log(p) for p, use in zip(correct_token_probabilities, loss_mask) if use]
print('Masked mean loss:', round(sum(selected) / len(selected), 4))
assert len(selected) == 2

Keep a genuinely independent check

The deterministic hash split makes grouping reproducible, but does not guarantee category balance. Inspect counts and adjust the dataset construction process if a rare category disappears from evaluation. Do not keep changing the split until the model looks good. For time-dependent use cases, a chronological holdout can reveal changes that a random split misses.

Store dataset version, group identifiers, label policy, and exclusion reasons in a manifest. Keep sensitive text out of casual experiment logs. When reporting results, distinguish a synthetic smoke test from a representative benchmark. If the final holdout influences prompt or hyperparameter choices, retire it as a final test and obtain another independent evaluation set before claiming generalization.

Build a grouped split manifest

• Create twelve synthetic conversations, with two paraphrases each and three support categories.
• Assign all paraphrases of one conversation to the same split and list category counts.
• Write a response-only loss-mask example and separately document how padding would be handled.

Expected checks

• No conversation identifier appears in both splits.
• All evaluated categories have examples or an explicit coverage limitation.
• The explanation distinguishes loss masking from removing input context and from attention masking.

Check your understanding

Why should paraphrases from the same conversation stay together?

• To prevent closely related content leaking across the evaluation boundary
• To guarantee equal category frequencies
• To make the loss exactly zero

Official tools & further reading

• PyTorch CrossEntropyLoss and ignored targets