Custom Analyzers: Synonyms, Stemming and Folding

Building your own

Define analyzers in index settings under analysis, then reference them from the mapping. Common building blocks: lowercase; asciifolding so "café" matches "cafe"; a stemmer (for example light_english or porter_stem) so "running" matches "run"; stop for very common words; and synonym_graph so "tv" matches "television". Synonyms are usually best applied at search time via search_analyzer, because you can then update the synonym list without reindexing (search analyzers can be reloaded, and newer versions offer a synonyms API; check the docs for your version). Stemming improves recall but can hurt precision, so keep an unstemmed sub-field for exact-ish matching and boost it.

Index-time and search-time analyzers

Kibana Dev Tools console syntax; send the same requests with curl or a client library.

PUT /catalog
{
  "settings": {
    "analysis": {
      "filter": {
        "en_stem":     { "type": "stemmer", "language": "light_english" },
        "my_synonyms": {
          "type": "synonym_graph",
          "synonyms": [ "tv, television", "laptop, notebook computer" ]
        }
      },
      "analyzer": {
        "product_index": {
          "type": "custom",
          "char_filter": [ "html_strip" ],
          "tokenizer": "standard",
          "filter": [ "lowercase", "asciifolding", "en_stem" ]
        },
        "product_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [ "lowercase", "asciifolding", "my_synonyms", "en_stem" ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "product_index",
        "search_analyzer": "product_search",
        "fields": { "exact": { "type": "text", "analyzer": "standard" } }
      }
    }
  }
}

Keep index and search analyzers compatible

If the index side stems and folds but the search side does not, queries produce terms that were never indexed. Change both deliberately, and remember that changing an index-time analyzer requires a reindex.

Quick check: Why are synonyms often applied only in the search analyzer?

• It makes indexing slower
• Index-time synonyms are not supported
• The synonym list can be updated without reindexing documents
• Search analyzers cannot stem