How To: Dynamically Creating Suggestion Fields for Elasticsearch Using C# and N-grams

Recently I had to implement a comprehensive autocomplete solution built on Elasticsearch.

The original implementation performed a match against the content title.

Whilst this worked with relative success, the user input was tightly bound to the title field for each document that existed in the Elasticsearch index.

A better user experience was to implement a Google-type autocomplete suggester where similar input phrases or combinations of related phrases could be suggested in real-time whilst the user typed.

This went beyond a regular autocomplete feature and would help users find the information they required quicker.

~

Elasticsearch Suggestion Fields

Elasticsearch lets you configure suggestion fields to provide a rapid autocomplete / query suggestion feature.  Suggestion fields are applied against each document that has been ingested in your index.

 

Consider the following title: The TickProvider API.  An example of typical suggestion fields for this string might be the following:

"suggest": {
                        "input": [
                            "The TickProvider API",
                            "The",
                            "TickProvider",
                            "API",
                            "The TickProvider",
                            "TickProvider API"
                        ]
                    }

 

 

But how to generate batches of suggestion fields when there are thousands of documents to process?

~

Ngrams to the Rescue

What is an ngram?

An n-gram is a sequence of n items from a given text or speech sample and are often used in natural language processing (NLP) to analyse language patterns.

Items can be characters, syllables, or words.  The length of the sequence (n) defines the size of the n-gram.

(image source: botpenguin)

 

For example, in a text analysis context, a “unigram” would look at individual words, a “bigram” would look at two-word combinations, and a “trigram” would focus on three-word sequences.

N-grams are used for classification tasks such as sentiment analysis and machine translation because they help you capture contextual language patterns.

By breaking down text into these manageable sequences, n-grams help models to make educated guesses about language use, improving text analysis and predictive accuracy.

 

You can write your own ngram generator or use an open-source SDK.  I wrote my own.  Some publicly available generators include, but are not limited to:

 

• OpenNLPSharp – An open-source C# wrapper for OpenNLP, useful for tokenization and potentially for generating N-grams.
• TextBlob for Python – TextBlob simplifies processing text data with support for basic NLP tasks, including generating N-grams using its ngrams method.
• Tokenizers by Hugging Face – This library offers fast tokenization and subword generation with flexibility for creating N-grams, ideal for NLP applications.

~

An Example

Here we can see a simple example of a ngram generator (I leveraged AI to create this):

using System;
using System.Collections.Generic;



public class NgramGenerator
{
    public static List<string> GenerateAllNgrams(string input)
    {
        List<string> ngrams = new List<string>();

        string[] words = input.Split(' ');

        // Generate n-grams for all sizes from 1 to the number of words
        for (int n = 1; n <= words.Length; n++)
        {
            for (int i = 0; i <= words.Length - n; i++)
            {
                // Join n words to form an n-gram
                ngrams.Add(string.Join(" ", words, i, n));
            }
        }

        return ngrams;
    }


    public static void Main(string[] args)
    {
        string text = "Elasticsearch N-gram generator example";

        List<string> ngrams = GenerateAllNgrams(text);

        Console.WriteLine("Generated N-grams:");

        foreach (var ngram in ngrams)
        {
            Console.WriteLine(ngram);
        }
    }
}

 

We can pass in the string The TickProvider API.   The result is as follows:

 

In the above, we can see 6 ngrams were created from the supplied text.  The generated ngrams can then be used as additional suggestion fields by Elasticsearch when implementing a query suggester.

 

This provides a better auto complete solution in that users can be shown different yet related suggestions as they type.

~

 

Summary

In this blog post, we have seen how to generate ngrams.

We saw how these can be used to supplement input field creation when implementing completion suggester experience with Elasticsearch.

Learn more about completion suggesters and Elasticsearch here.