8.1.2-elasticsearch文本解析之自定义分词器及分词器匹配规则

创建自定义analyzer

在具体的业务场景当中可能内置的analyzer并不能满足需求,这就需要能够自定义analyzer,前文已经说过analyzer由3部分组成,自定义analyzer就是通过配置以下三部分内容来实现的:

配置参数具体说明

以下举例进行说明
(1)简单示例

//定义char_filter、tokenizer、filter,均使用默认配置
PUT custom_analyzer_simple_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer":{
          "type":"custom",
          "tokenizer":"standard",
          "char_filter":["html_strip"],
          "filter":["lowercase","asciifolding"]
        }
      }
    }
  }
}


POST custom_analyzer_simple_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": [
    "<p>Transmission Control Protocol is a transport layer</p>"
  ]
}

以上示例将生成以下关键字

[transmission,control,protocol,is,a,transport,layer]

(2)复杂示例

//自定义char_filter(自定义k-v结构)、tokenizer(模式匹配当中有空格)、filter
PUT custom_analyzer_complex_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "type": "custom",
          "char_filter": [
            "emoticons"
          ],
          "tokenizer": "punctuation",
          "filter": [
            "lowercase",
            "english_stop"
          ]
        }
      },
      "char_filter": {
        "emoticons": {
          "type": "mapping",
          "mappings": [
            ":) => _happy_",
            ":( => _sad_"
          ]
        }
      },
      "tokenizer": {
        "punctuation": {
          "type": "pattern",
          "pattern": "[ .,!?]"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        }
      }
    }
  }
}

//1、请求参数
POST custom_analyzer_complex_index/_analyze
{
  "analyzer": "custom_analyzer",
  "text": "The good weather make me :)"
}

//2、返回结果:':)'转换成'_happy_'
{
  "tokens" : [
    {
      "token" : "good",
      "start_offset" : 4,
      "end_offset" : 8,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "weather",
      "start_offset" : 9,
      "end_offset" : 16,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "make",
      "start_offset" : 17,
      "end_offset" : 21,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "me",
      "start_offset" : 22,
      "end_offset" : 24,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "_happy_",
      "start_offset" : 25,
      "end_offset" : 27,
      "type" : "word",
      "position" : 5
    }
  ]
}

以上示例将生成以下关键字

[good,weather,make,me,_happy_]

指定analyzer的一点思考

es提供多种指定analyzer的方式,不过本质上都是针对text类型的字段进行处理,不管是text字段级别还是文档级别;
一般出于配置及使用简单性上的考虑,针对text类型字段统一采用相同的analyzer进行处理,较少针对每个text类型字段分别制定(除非特殊必要的场景);

I、在索引时多个级别均配置analyzer的情况下,es如何选择analyzer呢?

在这种情况下,es主要根据以下几个参数来顺序决定使用何种analyzer:
1)、查看字段上是否指定analyzer参数;
2)、查看索引上是否设置参数analysis.analyzer.default;
3)、若以上均未设置,则analyzer为standard;

1a)、字段级别指定analyzer

PUT part_field_analyzer_index
{
  "mappings": {
    "properties": {
      "title":{
        "type": "text",
        "analyzer": "whitespace"
      }
    }
  }
}

2a)、索引级别指定analyzer

PUT full_field_analyzer_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default":{
          "type":"simple"
        }
      }
    }
  }
}

II、在多个级别均配置查询analyzer的情况下,es如何选择索引的analyzer呢?

大多数情况下不需要针对字段或索引指定不同的analyzer,并且这种做法也不推荐(若一定要使用最好经过充足的测试);
1)、查看查询时字段上是否指定analyzer参数;
2)、查看字段上是否指定search_analyzer参数;
3)、查看索引上是否设置参数analysis.analyzer.default_search;
4)、查看字段上是否指定analyzer参数;
5)、若以上均未设置,则analyzer为standard;

1a)、查询时字段上指定analyzer

//1、新增文档
PUT part_field_analyzer_index/_doc/1
{
  "title":"The nice day"
}

//2、文档查询
GET part_field_analyzer_index/_search
{
  "query": {
    "match": {
      "title": {
        "query": "the nice day",
        "analyzer": "stop"
      }
    }
  }
}

1b)、建立索引时字段指定search_analyzer字段

//指定search_analyzer参数时同时一定要指定analyzer
PUT full_field_search_analyzer_index
{
  "mappings": {
    "properties": {
      "title":{
        "type": "text",
        "analyzer": "whitespace",
        "search_analyzer": "simple"
      }
    }
  }
}

1c)、建立索引时指定默认的查询analyzer
可以在建立索引时指定索引默认的查询analyzer,参数:analysis.analyzer.default_search;
若默认的查询analyzer指定,则同时也需要指定默认索引analyzer,参数:analysis.analyzer.default;

PUT full_default_search_analyzer_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default":{
          "type":"simple"
        },
        "default_search":{
          "type":"whitespace"
        }
      }
    }
  }
}