Historically, keyword search has been a staple of information retrieval. Type in a term or a phrase, using Boolean or not, and results appear. But keyword search alone has been a disappointment as it often misses things that are “nearly the same,” like related documents or items.

As a search specialists our job is to help searchers find what they are looking for – regardless of how they phrase their queries.
Embeddings in vector-based search solutions have emerged as a powerful means to overcome the limitations of keyword search, and fulfill our role as search specialists.

This article overviews search embeddings, underlying principles, and applications in modern search systems.

Keyword Search and Inverted Indexes 

The simple and most efficient process for searching across keywords has been to use an inverted index. An inverted index creates a list of terms (single words or phrases) and a corresponding list of “documents” that contain that term. The inverted index creates a data structure so that you can quickly find the term within a document. 

Information retrieval has used the inverted index to find terms being queried. The challenge has been when a term is not explicitly found. How can a search engine retrieve items that are similar? 

Search embeddings are key to solving this.

What Are Search Embeddings

Simply put, an embedding is a numerical representation (a vector) of an entity or attribute, that we use to map a concept. The closer two vectors are, the more likely they are to be semenatically related. Conversely, the greater the distance between the vectors, the less likely they are related.

An inverted index createsne an index that have one dimension per term. If you think of every single keyword in an index as binary — it exists you give it a one, it doesn’t it’s a zero. That’s not very semantically meaningful embedding though. So instead we map all our terms into dimensions in vector space. Each dimension represents a feature and in this way we can begin to group similar terms together.

Embeddings are useful for

• Search
• Product discovery
• Recommendations
• Anomaly detection 

The Evolution from Keyword-Based to Vector-Based Search

As noted, traditional search engines rely heavily on keyword matching. While this approach is effective, it often falls short of understanding the context and semantics of user queries. For example, a keyword-based search might struggle with synonyms, polysemy (words with multiple meanings), and contextually relevant results.

Vector-based search maps the parts of both queries and potential results to numerical representations called “vectors.” A vector is a list of values describing some attributes of an item. For example, if your items are houses, you may have a list of attributes like price, size, and number of bedrooms. If you have a home costing $100,000 with 1,000 square feet and 2 bedrooms, this could be represented as the vector [100000, 1000, 2].

Vector-based search, leverages embeddings to overcome limitations of keyword search. By mapping both queries and documents to a common vector space, it allows for the comparison of semantic similarity rather than mere keyword matching. This transition is facilitated by natural language processing (NLP) and machine learning advances.

The Difference Between an Inverted Index and Embeddings

Consider a term like “cheese pizza.” Converted to single dimension vectors (similar to most inverted indexes) it might look like the table below. When a query is executed, an exact lookup occurs in the index for any matched terms (post-text-analysis), and then a similarity score is calculated based on a comparison of the vector for the query and the vector for the document that is being scored relative to the query. 

There are obvious downsides to this approach. While it is great for finding documents with exact keyword matches, what happens when you want to find “related” things instead? 

For example, you’ll notice that the term soda appears in a query, but never in the index. Even though there are other kinds of drinks (apple juice, water, cappuccino, and latte), the search engine will always return zero results because it doesn’t understand that the user is searching for a drink. Similarly, you’ll notice that even though the term caffeine exists in the index, queries for latte, cappuccino, and green tea will never match the term caffeine, even though they are related.

For these reasons, it is now common practice to use something called “word embeddings” to model a semantic meaning for term sequences in your index and queries. A word embedding for a term is a vector of features that represents the term’s conceptual meaning in a semantic space. The table below demonstrates the terms now mapped to a dimensionally reduced vector that can serve as a word embedding.

With this method, instead of one dimension per term (exists or missing), now higher-level dimensions exist that score shared attributes across items such as “healthy”, contains “caffeine” or “bread” or “dairy”, or whether the item is “food” or a “drink”.

With a new word embedding vector now available for each term sequence in the leftmost column of the table above, we can now score the relationship between each pair of term sequences, leveraging the similarity between their vectors. 

In linear algebra, we use a cosine similarity function or another distance measure to score the relationship between two vectors, which is computed by performing a dot product between the two vectors and scaling it by the magnitudes (lengths) of each of the vectors. The figure below shows the results of scoring the similarity between several of these vectors.

While “green tea” may not pull back “latte” in keyword search, with the vector similarity model green tea is obviously more similar to a latte than to a donut. Similarly, a “donut” is closer to “cinnamon bread sticks” than to “cheese pizza” or “apple juice.” With this vector similarity model the results “closest” to the query can be retrieved rather than just results “with the same” or “similar words” as the query. 

Types of Embedding

Distributional semantics is a research area within the field of natural language processing that focuses on the semantic relationships between terms and phrases based on the distributional hypothesis. The distributional hypothesis is that words that occur in similar contexts tend to share similar meanings. It is summarized well by the popular quote, “You shall know a word by the company it keeps.”

These higher-level attribute vectors we’ve discussed might represent other term sequences in queries, within documents, or even in entire documents. It is commonplace to encode terms and term sequences into word embeddings, but sentence embeddings (encoding a vector for an entire sentence), paragraph embeddings (encoding a vector for an entire paragraph), and document embeddings (encoding a vector for an entire document) are also common techniques. 

It’s also very common that dimensions themselves are more abstract than our examples here. For example, deep learning models like large language models (LLMs) may detect seemingly unintelligible features from character sequences and the way that documents cluster together within the corpus. 

It wouldn’t be easy to apply a human-readable label to these dimensions in the embedding vector. But that is usually not a concern for most search teams – as long as it improves the predictive power of the model to increase relevance.

Ultimately, combining multiple models for harnessing the power of distributional semantics and word embeddings tends to create the best outcomes. The basic, dead-on keyword match is often the best result where it exists. If the corpus has “cheese pizza” as the title and the user searches on “cheese pizza,” then obviously that should be the first result. However, if those words are missing or we want to offer suggestions, vectors of embeddings are a good addition.

How to Create Search Embeddings

The creation of search embeddings involves several steps:

1. Tokenization: The text is divided into smaller units, such as words or subwords. This step is crucial for handling different granularity levels of text.
2. Embedding Model: Pre-trained models like Word2Vec, GloVe, or transformers such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) are used to convert tokens into dense vectors. These models are trained on large corpora to capture a wide range of semantic relationships.
3. Vector Representation: Each token is represented as a multi-dimensional vector. The dimensions capture various semantic attributes of the token, such as its meaning, context, and relationships with other tokens.
4. Similarity Measurement: The similarity between vectors is measured using techniques such as cosine similarity or Euclidean distance. This allows for the comparison of the semantic proximity between the query and potential results, enabling more relevant search outcomes.

Popular Embedding Models

Several models have been developed to generate embeddings, each with its unique approach and advantages:

• Word2Vec: Developed by Google, Word2Vec is one of the pioneering models in embedding techniques. It uses the Continuous Bag of Words (CBOW) or Skip-gram methods to predict a word based on its context or vice versa, effectively capturing semantic relationships. 

• GloVe: Short for Global Vectors for Word Representation, GloVe is a model developed by Stanford University. It focuses on capturing global statistical information from a corpus to create word embeddings, combining the benefits of global matrix factorization and local context window methods.

• BERT: Developed by Google, BERT represents a significant advancement by incorporating bidirectional context in its embeddings. This means BERT considers both the left and right context of a word, leading to more accurate representations and understanding of the text.

• GPT: OpenAI’s GPT model uses a transformer architecture to generate embeddings. Unlike BERT, GPT is primarily a generative model, but it also produces highly effective embeddings for search tasks, particularly in understanding and generating human-like text.

Conclusion

Search embeddings represent a significant improvement in the field of information retrieval. By capturing the semantic essence of text, they enable more accurate and contextually relevant search results. There are ever increasing new ways of capturing the meaning of text and its context, but all come down to establishing the characteristics and mapping them to numbers then finding out how “close” two vectors are to each other. 

Using these techniques it’s possible to not just match on what the requestor “said” but what they “meant.”

Dig Deeper

Read More about Search Embeddings in Chapter 3 of AI-Powered Search.