Why Search Needs More Than Content Understanding

Similarity search and semantic recommendation are built on the same foundational idea: convert items and queries into vector representations, then find the closest matches in a large catalog. The modality can vary. Some systems encode text, some encode images, some encode both. But the underlying mechanism is the same, and so is the underlying problem.

Most of the models used to produce those vectors were trained to understand content: to recognize what something depicts or assign it to a category. That is a different task from search. The difference shows directly how useful the results are.

Understanding versus searching

A model trained to understand content organizes its embedding space around what items contain. It is good at answering “what is this?” But search requires an answer to a different question: “which other items would a user consider similar?”

Those two questions do not have the same answer. Two products with similar content might be nothing alike from a user’s perspective. Two products that feel intuitively similar to a shopper might look quite different in terms of raw content. When a model optimized for content understanding is deployed for search without further adaptation, the results feel almost right: technically in the right neighborhood, but not what the user was looking for. Searches that should convert don’t.

What a retrieval-optimized embedding space looks like

Retrieval requires an embedding space where closeness reflects perceived similarity, not just shared content features. Items a user would consider interchangeable should be near each other. Items that look superficially alike but feel different in context should be further apart. This structure should hold across the full catalog, including product types and styles that were never seen during training.

Achieving that requires fine-tuning specifically for retrieval, using diverse training data while deliberately excluding any overlap with the evaluation benchmarks. The objective is not to recognize content but to organize the space around similarity, so the model generalizes new items and domains rather than reflecting the patterns of a specific training set.

What the research shows

The chart below is drawn from Prof. Kai Uwe Barthel’s tutorial Advanced Methods for Visual Information Retrieval and Exploration in Large Multimedia Collections, presented at VISAPP 2026. It compares five widely used backbone architectures on a retrieval benchmark, before and after retrieval-specific fine-tuning.

Two findings stand out.

Retrieval-specific fine-tuning consistently improves every model tested. The same fine-tuning regime lifts all five backbones regardless of their starting point. This result holds across more than 60 evaluated encoders and 15 diverse benchmark sets. The consistent improvement confirms that the gap between standard pretraining and retrieval performance is structural, not architecture specific.

The fine-tuned models hold up on data they have never seen. The fine-tuned encoders improve not just on standard test sets but also on distribution shift benchmarks, data that looks different from anything in training. For a live catalog, that robustness means the model produces stable, meaningful representations for new products as they arrive, including new styles, categories, and photography conditions never seen during training.

What this means in practice

A catalog is not a static benchmark. It grows, absorbs new product ranges, and evolves with trends and seasons. A retrieval model fine-tuned for general purpose similarity search on diverse; domain agnostic data gives a strong foundation that works across many product types without per category tuning. Domain specific adaptation on top of that foundation can push performance further for specialized use cases.

At vviinn, we use such a fine-tuned model to power similarity search across millions of products. It makes results relevant when a customer uploads a photo or gets recommendations based on something they viewed, without us knowing in advance what the catalog will contain next season.

My research at HTW Berlin focused on exactly this problem, developing a fine-tuning approach that improves retrieval across diverse visual domains without overfitting any specific one. You can read the full methodology and benchmark results in Improving Image Encoders for General Purpose Nearest Neighbor Search and Classification; the paper, code, and model weights are all publicly available.

References

[1] K. Schall, K. U. Barthel, N. Hezel, and K. Jung, "Improving Image Encoders for General Purpose Nearest Neighbor Search and Classification," in Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), 2023.  DOI: 10.1145/3591106.3592266.

[2] K. U. Barthel, "Advanced Methods for Visual Information Retrieval and Exploration in Large Multimedia Collections,” Tutorial presented at the International Conference on Computer Vision Theory and Applications (VISAPP), 2026.