Skip to content
Interface language: English
Audit / Signal

AI embeddings: similar meaning cannot replace a part number

A similar card is not necessarily the right variant. Check codes, numbers, units and negations, and separate retrieval from approval.

Article language: English

Illustration: part of a stone column, with a thin line marking a crack.
In this article

The search engine found a product very like the one needed. The name matches, the description sounds familiar and the photograph looks convincing. Only the code ends in 12 rather than 21. In a conversation about furnishing style that might be minor. When ordering a specific variant, it is a different product. Similarity has no authority to swap digits.

An embedding is a numerical representation of information used, among other things, for searching by meaning. It helps find content described in different words from the query. Trouble starts when the result of such a comparison is treated as confirmation that every condition matches. A candidate for checking and an approved match need separate places in the process.

First establish what must match exactly

Imagine a fictional catalogue of dividers for desk organisers. A customer uses code PB-120-G and an employee looks for the right product card. The catalogue also contains PB-120-B and PB-210-G. We are not establishing technical-product specifications or permitted uses here. This simple example distinguishes a category name, an identifier and a variant attribute.

If a code uniquely identifies an item, the process should check it under an agreed rule. It need not guess the final letter's meaning from descriptions. A person asking for a specific item should receive that card or be told it could not be found. The nearest description can be useful later as a separately labelled suggestion.

Record which fields are selection conditions. These may be a code, unit, effective date or document version. Other elements may help find a similar use. Someone who knows the catalogue and ordering process must decide. Numbers in text alone do not reveal whether they are an identifier, dimension, price or page number.

Also prepare a rule for an incorrectly written code. A space, hyphen or letter case may be immaterial if the data owner says so. Removing a leading zero may already change an identifier. Do not apply one convenient normalisation to every field without checking its effect. Text clean-up can discard the very information the catalogue search needed.

In a company's knowledge architecture (in Polish), distinguish names people use from stable resource identifiers. A customer may ask for “a grey divider for a wide organiser”. The system must know whether that description indicates one variant, several candidates or no current item. Otherwise the interface will appear more decisive than the data.

A vector helps search but does not read the company's intent

Cohere's embeddings documentation describes representing information as lists of numbers and comparing their similarity. For semantic search it also distinguishes the input type for a query from that of a searched document. Configuration of that comparison belongs to the solution being assessed; knowing that a project “uses vectors” is not enough.

When a person asks for a divider that separates notebooks from small objects, semantic search may find a description using different words. Assess whether the material found meets the need. Do not demand that every word match: different vocabulary may be the reason to use the method.

When the same query includes PB-120-G, that condition must also be preserved. A similar PB-210-G description does not become correct merely because it ranks highly. Your own test should check what the solution does with the code rather than assuming success or failure for every embedding model.

A similarity score is not itself a percentage probability that an item meets an order's requirements. To interpret a particular solution's score that way, the relationship would need to be separately established and tested on an appropriate sample. Its meaning depends on the model, comparison method and preparation of material. If the application shows a number, explain what it means and does not establish. A high position in a list is not the signature of the person responsible for variant compliance.

Also check document preparation. If the comparison receives a common family description while variant identifiers are stored separately, the result may correctly identify the family. A later stage must still choose the variant. Assessment should account for that structure instead of penalising the first stage for a task never assigned to it.

A relevant passage must lead to the right card

A catalogue may be split into shorter passages before search. See whether the divider's description retained its connection to the code and title. If a passage includes the end of one card and the beginning of the next, the system may receive text about the right use with the neighbouring variant's identifier. The problem then lies in material preparation before comparison begins.

Follow the result to the original source. In the fictional catalogue a click should open the particular card or a clearly indicated passage. If it leads to the first page of a long file, the employee must still locate the variant. Assess the link and presentation of the basis too. Relevant excerpt text alone does not complete the task of preparing an answer for the customer.

Add a test after a catalogue change. A withdrawn card may remain in the index while its address now leads to another item. Check how old representations are updated and removed. The team needs to identify the source version used for the match, distinguishing poor comparison from successfully finding outdated material.

If the common family description is repeated for every variant, a list may contain many similar cards from one group. Establish whether the recipient needs variants or different use proposals. These are different tasks. Do not mechanically remove similar results when their differences are precisely what the customer still needs to choose.

Aeneia conceptual photography

Number, unit and negation change a condition

Prepare fictional descriptions that differ in one element important to selection. Width 120 and 210 are different conditions even though both cards describe an organiser. Likewise “with divider” and “without divider” do not describe the same set. Check whether the system preserves the difference during search and presentation. This is a hypothesis to test in your solution.

Check the unit with the value. Finding a number alone does not establish whether it refers to a piece, set, package or dimension. In the fictional catalogue a set of two dividers should not automatically answer a request for two sets. Similar descriptions require clarifying what the customer is actually ordering.

Add a case with insufficient information to choose. If a message says only “I need the same as last time” and the process has no order history, it should recognise the gap. Searching for a nearest card can prepare an auxiliary list, but cannot reconstruct a missing order. Asking for the code may be the shortest route to a correct result.

In AI answer regression tests (in Polish), retain these pairs with the expected meaning. Check whether a model or index update changed the chosen variant. If the application still returns the correct format and an attractive description, a decline in the correctness of the item itself can easily be missed.

Combine methods while retaining mandatory conditions

Elastic describes hybrid search as combining full-text and vector search in one request and a shared ranking. It combines word matching with similarity. Combining rankings alone does not mean an identifier is mandatorily checked under the company's rule. That condition must be designed and tested separately.

For the fictional catalogue consider separate paths for a code and a description of need. If a recognisable code is supplied, first check the specific item. If the query describes a use, find candidates and then show attributes needing selection. This is an example design decision, not a universal configuration for every search engine.

Establish what happens on conflict. A customer may provide the code of one divider and the dimension of another. The system should not silently choose whichever information was easier to match. You need a message identifying the discrepancy and a question that can resolve it. In order handling, conflict is information about the case, not an obstacle to hide beneath the best result.

Also see whether a filter preserves constraints across the whole path. If the result must contain only current items from a particular catalogue, checking before one stage and omitting the check after results are combined is not enough. Follow an example from input to the visible card. The material the user actually receives is what counts.

In source-based RAG (in Polish), separate finding the document from answering on its basis. You can have a good candidate and an incorrect description, or poor retrieval and a cautious refusal. Without this separation fixes may land in the wrong place: in a model instruction instead of the index, or vice versa.

A change in representation needs a new comparison

Record which model and data-preparation method built the index, the searchable set of representations. The query must be compared through a compatible mechanism. During an update, check the selected provider's requirements and the plan to rebuild the index. Changing a model name in one place does not establish compatibility of every stored record.

Include a code absent from the catalogue in the acceptance test. The system might return no result, suggest checking the spelling or display separately labelled similar items. Agree expected behaviour with the process owner. Without this case, a tool that always returns some card may look good on a set containing only available products.

Watch the person using the result. Can they tell they received a suggestion, not confirmation of the code? Can they see conditions the system could not check? A small “similar” label may not suffice if the adjacent button immediately approves the item. The meaning of a choice should be clear when it is made. If selection uses a suggestion field, also distinguish the typed text, suggestion and confirmed record (in Polish).

Keep a case where a person rejected the first card and chose the next. Identify what information they used. The description may indicate the right family, while a missing filter makes choosing a variant difficult. Such observations help prepare a specific fix instead of changing the similarity threshold after every complaint without knowing the cause.

Compare a code, a free-text description and conflicting conditions on the same test. Record candidate relevance and final selection separately. Do not change the whole method because of one difficult code if that harms natural-language search. See which task improved and which needs a separate rule.

If considering model fine-tuning (in Polish), first check whether the problem already arises when retrieving the card. If results are stored, inspect answer-cache freshness (in Polish). A correct selection from an old catalogue may no longer fit a new case even with the same query.

For a conversation with Aeneia Audit, prepare several cards, the query and a result where similarity won over a requirement. Discuss whether and how this question could be included in an AI audit. In your own investigation, identify which information should decide the choice and where it stopped being checked. If the right card never reaches the candidates, inspect retrieval. If it appears but loses to another code, inspect the final selection rule. The product number should decide where the company has assigned it that role.

Aeneia / Aeneia editorial

Editorial article. Source links are included in the text.

About Aeneia

Brands we have worked with

  • L’Oréal
  • Garnier
  • Netflix
  • Carrefour
  • Warner Bros. Discovery
  • Biedronka
  • HBO Max
  • Wolt
  • Żabka
  • COSRX
  • Embassy of China in Poland
  • Mixa
Aeneia / Initium

From knowledge to a decision

Tell us what you want to put in order. We will agree on where to start.

Contact Aeneia

We partner with and use solutions from:

  • Google
  • OpenAI
  • Microsoft
  • Stripe
  • Anthropic
  • Linear