From Tokens to Context

In this chapter

• Explain why tokens differ from words
• Compute a small attention-weighted result
• Recognize context-length and interpretation limits

Text becomes a representation

Signal Bench classifies fictional support messages such as invoice export fails into billing, access, or reliability. Before a language model can process a message, a tokenizer converts text into identifiers. A token can be a word, part of a word, punctuation, or another fragment. Two tokenizers can split the same sentence differently, so a word count is not a reliable context budget.

Those identifiers select learned vectors. Positional information helps the model distinguish ordered sequences, while attention lets a position combine information from other permitted positions. This is a numerical computation over representations, not a database lookup that guarantees factual accuracy. A context window limits what can be supplied during one computation; it is not the same as permanent memory or authorization to retain user data.

Calculate a small attention mixture

In a single attention head, query-key compatibility scores become weights after normalization, and those weights combine value vectors. The example starts from scores already calculated, so it is not a complete transformer implementation. Subtracting the maximum score before exponentiation preserves the normalized result while improving numerical stability. With scores two and zero, the first value receives approximately 88 percent of the weight.

import math

scores = [2.0, 0.0]
values = [(1.0, 0.0), (0.0, 1.0)]
maximum = max(scores)
exp_scores = [math.exp(score - maximum) for score in scores]
weights = [value / sum(exp_scores) for value in exp_scores]
context = tuple(sum(w * v[i] for w, v in zip(weights, values)) for i in range(2))
assert abs(sum(weights) - 1.0) < 1e-12
assert context[0] > context[1]
print(tuple(round(value, 4) for value in context))

What the weights do not explain

A large attention weight does not establish that a token caused a decision in a simple human-readable way. Many layers, heads, nonlinear operations, and residual paths contribute to the output. Avoid presenting a colorful attention diagram as proof that the model reasoned correctly. Instead, test behavior by changing relevant and irrelevant parts of the input and measuring the effect on a defined task.

For support classification, try adding an invoice number, a quoted previous message, or a long irrelevant footer. A truncation policy might accidentally remove the actual problem statement. Track input length and document which part is retained. When you later evaluate a real tokenizer or model, pin its identifier and version; changing tokenization can affect both model behavior and serving costs.

Inspect a context decision

• Run the mixture and replace the scores with equal values; explain the changed weights.
• Write three synthetic support messages where the important phrase appears at the beginning, middle, and end.
• Design a truncation test that detects loss of the category-defining phrase without using a model API.

Expected checks

• Attention weights sum to one within numerical tolerance.
• Your explanation distinguishes the toy mixture from a complete transformer.
• The input policy records what is truncated and does not claim that token counts equal word counts.

Check your understanding

What can a high attention weight establish on its own?

• That a factual claim is true
• That the model has a human-like explanation for its output
• That one value receives a larger weight in that particular attention computation

Official tools & further reading

• Transformers automatic model and tokenizer interfaces