Logstash + ElasticSearch: initial type mapping results in missing log lines -
we have classic logstash + elasticsearch + kibana setup log aggregation. use aggregate logs across servers , applications, , we've stumbled upon following problem: first time es receives log line (a json document in our case), creates mapping document (see http://bit.ly/1h3qwc9). of time, properties mapped strings, in cases they're mapped dates or numbers. in latter case, if log line (from different application) has same field, string value, es fail index (throwing exception log , continue usual). workaround we've configured es ignore malformed documents (index.mapping.ignore_malformed: true), feels more of hack.
any ideas how solve issue in elegant way?
it sounds guys don't care date types, or type. think best solution define dynamic template define types string:
{ "_default_" : { "dynamic_templates" : [ { "long_to_string" : { "match" : "*", "match_mapping_type": "long", "mapping" : { "type" : "string", "index" : "analyzed" } } }, { "double_to_string" : { "match" : "*", "match_mapping_type": "double", "mapping" : { "type" : "string", "index" : "analyzed" } } }, { "float_to_string" : { "match" : "*", "match_mapping_type": "float", "mapping" : { "type" : "string", "index" : "analyzed" } } }, { "integer_to_string" : { "match" : "*", "match_mapping_type": "integer", "mapping" : { "type" : "string", "index" : "analyzed" } } }, { "date_to_string" : { "match" : "*", "match_mapping_type": "date", "mapping" : { "type" : "string", "index" : "analyzed" } } }, { "boolean_to_string" : { "match" : "*", "match_mapping_type": "boolean", "mapping" : { "type" : "string", "index" : "analyzed" } } } ] } }
from here.
Comments
Post a Comment