By Trey Grainger

Prior to November 2022, when OpenAI released or even unleashed ChatGPT to the world, the definition of “Artificial Intelligence” (AI) was a bit nebulous to the general public. It was understood to include things like self-driving cars, autonomous robots, and other futuristic technologies that made computers appear to be intelligent. Still, with everything fortified with AI, it appeared to be more of a marketing buzzword versus having a concrete definition. 

And the world of search, which was relegated to the bowels of a company as an infrastructure play is suddenly thrust into the parlance of board rooms – with the prefix of AI appended to it. Whether business or technical there is confusion. So the purpose of this article is to disambiguate a few things – and provide a foundation of understanding for stakeholders. 

In the software industry a more concrete definition has existed for years, however.

In the context of software development, the term Artificial Intelligence generally describes any computer program that can perform a task that previously required human intelligence. That program often includes machine learning techniques allowing it to learn from data and improve its performance over time. 

That said, even rules-based systems that do not leverage machine learning techniques – but which generate human-like feedback, have also traditionally been considered “AI” systems. 

In my book AI-Powered Search, both the general definition of AI and machine learning aspects of AI are discussed at length.

The term search (or search engine) is likewise considered by the general public to refer to web search engines like Google, Bing, or Baidu. In software development, the term “search” is used to describe any technology that enables users to query for and find information. It might be for site search, e-commerce search, knowledge management, whatever. The goal is to recall documents (the search result) that are as precise as possible to match the intention of the query.  

Basics of Traditional Search/Information Retrieval  

The field of information retrieval typically involves at least two critical steps – finding documents that match a query (matching) and then ordering those documents by relevance to the query (ranking). Search can also include many pre-processing steps to better understand the query, and post-processing steps to extract answers or summarize results from the matched documents. 

Search is often the primary way users find information, whether conducting general web search, product search, enterprise search, video/image search, or any of hundreds of other common use cases for finding and ranking information. It is also becoming the primary way Generative AI systems find updated factual content to use as context for their prompts.

A search engine is one of the most cross-functional kinds of systems within the software engineering world. Most underlying search engine technology is designed to operate in a massively scalable way, serving large volumes of queries against millions, billions, or even trillions of documents –  and delivering results in hundreds of milliseconds or less. In many cases, real-time processing and near-real-time searching on newly ingested data is required, and all of this must be parallelizable across numerous servers to scale out to meet such high-performance requirements.

Implementing search engines also requires substantial work building search-specific data structures like an inverted index or ANN-based vector store, an understanding of linear algebra and vector similarity scoring, experience with text analysis and natural language processing, and knowledge of numerous search-specific types of data models and capabilities (spell checking, autosuggest, faceting, text highlighting, embeddings, and so on).

For a search engine to fully interpret user intent, it’s critical that you combine a thorough understanding of your content, your users, and your domain. 

AI-Powered Search

So, what is AI-powered search, and how does it differ from traditional “search”? 

In a nutshell, AI-powered search is all the technologies and techniques at the intersection of the fields of Search / Information Retrieval and Artificial Intelligence, to provide the best possible user experience. It is a system that continues learning to become more efficient – and proficient when it comes to ranking. These technologies overlap heavily with and leverage the fields of data science, machine learning, and deep learning.

Many buzzwords such as AI, machine learning, data science, and deep learning are often thrown around interchangeably, and it’s important to understand their distinction and how they overlap with AI-powered search. 

Intersection of Search and AI

As you can see in the image above, machine learning is a subset of AI that focuses on using data to train models to perform tasks based on insights learned from the training data. Deep learning is a further subset of machine learning that focuses on training artificial neural networks – algorithms that partially mimic the structure of the human brain – to learn to solve complex problems. 

Notice that deep learning is a fully contained subset of machine learning, which is then a fully contained subset of artificial intelligence. Data science is a discipline that overlaps heavily with AI and Search, but it also contains other distinct focus areas, so is not completely a superset or subset of either.

I’ve highlighted the intersection of Search and AI, and in particular the application of machine learning and deep learning techniques, as how to improve the relevance of search results and to automate the process of tuning search configurations. Note, you can call something AI-powered search – even if there is no machine learning. The results will not be the same.

Building an AI search engine involves many well-known machine learning techniques, but also many that are specific to information retrieval and the search domain. As you might imagine, being proficient requires many different skill sets –  often not found in one person alone. 

The figure above, provides some categories of the key AI-powered search techniques that are applicable – broken down by whether they are deep learning techniques, other machine learning techniques not requiring deep learning, or other artificial intelligence techniques not requiring machine learning.

In the AI-only category, question-answering (QA) systems, virtual assistants, chatbots, and rules-based relevancy are all examples of AI techniques that are often built using machine learning –  but which do not require machine learning.

Many teams have built chatbots based entirely on rules to understand different user utterances and intents. Likewise, QA systems can be built solely on rules and ontologies. That said, machine learning is often used to learn these kinds of rules and ontologies, so the lines between these categories are often blurred.

When algorithms begin to use data to train models, we enter into the machine learning subcategory of AI-powered search. It’s here that we use behavioral signals from search engine users (clicks, likes, add-to-carts, purchases, etc.) to build models that can learn to better rank documents. 

Select Types of AI Search Models

These models include:

• signals boosting (top documents per query or category)
• collaborative filtering to generate recommendations or personalize search results
• ranking classifiers (learning to rank) that learn from content and behavioral signals to better rank results

Machine learning is also used to learn knowledge graphs, which are graphs of entities and concepts and their relationships, that can be used to better understand the domain and to better interpret user queries. You can enable semantic search (search on meaning, not just keywords) by using these knowledge graphs, along with traditional natural language processing approaches, query intent classification, document clustering, and other techniques driven by user queries, documents, and user behavioral signals.

Finally, in the deep learning subcategory of AI-powered search, we see the use of neural networks to build models to understand user queries and documents, as well as to rank and summarize search results. Here, text is used to train Large Language Models (LLMs) to understand the meaning of words and phrases, to generate answers to questions, and to generate summaries of documents. 

LLMs are a type of foundation model that can interpret text content and are often trained on massive amounts of text from the internet. Foundation models can also be trained on other types of content beyond just text (images, audio, video) to enable multimodal search across different content types: text-to-image search, text-to-audio, image-to-video, and so on. LLMs are also used to generate embeddings, which are vector representations of content used to compare the similarity of different content. 

Since a search engine’s primary job is to find and provide search results similar to an incoming query, these embeddings enable a sophisticated ability to search on the meaning of content and significantly improve query understanding and ranking. Further fine-tuning foundation models on specific goals or domain-specific data sets will also make them significantly better at understanding the nuances of those domains or use cases.

Foundation models compress a large amount of human knowledge (often much of the internet), providing them with a broad understanding across most domains. This compression of knowledge, however, is a lossy compression – the original data is not stored, and specific facts and concepts can be easily confused. Therefore, foundation models are well known to hallucinate answers to questions, making them generally unreliable for answering factual questions. 

As a result, in addition to search engines leveraging foundation models to improve query understanding and ranking, we’re also seeing the use of search engines as a knowledge source for foundation models to rely on for accurate information. This technique, known as Retrieval Augmented Generation (RAG), is one of the most reliable techniques for improving the reliability of generative AI models today.

AI-powered search is way more than just leveraging the latest LLM to interpret queries. It’s about engineering an end-to-end system for continuous learning. Ultimately, you’ll end up with a system that receives constant streams of document changes and user signals, continually processes those streams to improve models, and is then constantly adjusting future search results and measuring the impact of changes to deliver more intelligent results. 

That is the key behind AI-powered search: the processes of continual learning and improvement, based upon real user interactions and evolving content patterns and language models, to fully understand user intent and deliver an ever-improving search experience.