Establish Search Baselines

In this chapter

• Explain where lexical and vector similarity differ
• Create a deterministic lexical baseline
• Avoid interpreting similarity as factual confidence

Ask two different questions

A technician searches for AX-4 reset. Another searches for restarting the inspection unit. The first query contains a precise identifier; the second describes an intention. A lexical search compares words or normalized terms, making the identifier a strong signal. A vector representation compares coordinates designed to capture aspects of meaning. Neither mechanism knows whether the document is current, authorized, or correct. Those are separate checks.

Start with a small baseline that you understand. The following token-overlap ranker is not BM25 and is not offered as a substitute for a production search engine. It deliberately exposes its scoring rule so that every result can be explained. Lowercasing is useful here, but blindly discarding punctuation could damage identifiers such as AX-4. Tokenization choices are part of the product, not invisible preprocessing.

Run the same questions through both representations

The example keeps hyphenated identifiers and returns only documents sharing a query token. A production lexical engine adds indexing, linguistic analysis, and more developed ranking. Record the baseline first so an integration can demonstrate an improvement rather than merely a larger dependency list. A zero-overlap query should produce no results, not an arbitrary document with a zero score.

import re

def tokens(text):
    return set(re.findall(r'[a-z0-9]+(?:-[a-z0-9]+)*', text.lower()))

docs = {
    'reset': 'AX-4 reset sequence for the inspection unit',
    'store': 'BX-2 storage temperature and packing',
    'check': 'AX-4 daily inspection checklist',
}
query = tokens('AX-4 reset')
ranked = sorted(
    ((len(query & tokens(body)), key) for key, body in docs.items()),
    key=lambda item: (-item[0], item[1]),
)
print([(key, score) for score, key in ranked if score > 0])
assert ranked[0] == (2, 'reset')

Interpret a vector honestly

Imagine two teaching coordinates: reset-related meaning and storage-related meaning. The query vector (1, 0) is closely aligned with a reset vector (0.9, 0.1), but not with storage (0.1, 0.9). Cosine similarity divides their dot product by the product of their lengths, comparing direction rather than magnitude. Real embedding dimensions are not generally named concepts like these, and this illustration is not a learned embedding model.

Semantic similarity can retrieve an older reset procedure because it discusses the same topic. That is a relevance success but a versioning failure. Build a test set containing exact product codes, paraphrases, unrelated questions, and near-identical revisions. Compare top results manually before attaching an answer generator. Otherwise a fluent sentence may hide the retrieval error you needed to see.

Build a six-query baseline report

• Add three fictional documents and six queries including an unknown identifier.
• Write the intended relevant document identifiers before examining scores.
• Record whether a relevant result appears in the first three positions and explain every miss.

Expected checks

• Identical runs produce identical ordering, including ties.
• An unrelated query returns an empty result set.
• Your report names one lexical failure and one plausible semantic failure without inventing accuracy claims.

Check your understanding

A high vector similarity score primarily indicates what?

• The passage is authorized and factually current
• The query and passage representations are close under the chosen metric
• The passage must answer every part of the question

Official tools & further reading

• PostgreSQL full-text search introduction