Scale Your Metrics with Elasticsearch

Scale Your Metrics with Elasticsearch Philipp Krenn @xeraa

$ curl http://localhost:9200 { "name": "elasticsearch", "cluster_name": "docker-cluster", "cluster_uuid": "l6wfwv8XSniiI_Fx6qqrcw", "version": { "number": "5.6.9", "build_hash": "877a590", "build_date": "2018-04-12T16:25:14.838Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }

https://db-engines.com/en/ ranking

Full-Text Search

Logs

Metrics

I'm not going to use a search engine for metrics. — Too o!en

Developer

Agenda Building Blocks Tuning Delivering

Building Blocks

Only accept features that scale. — https://github.com/elastic/engineering/blob/ master/development_constitution.md

Horizontal Scaling Shards Replication Writes & Reads

Cluster, Node, Index, Shard

Write

Read

Append-Only Optimization IDs assigned on coordinating node Fast add docs instead of the slow update docs

Storage Compression LZ4 (default), DEFLATE (best_compression)

BKD Trees Points in Lucene

Half & Scaled Floats

Pipeline Aggregations

Tuning

_all Removal https://www.elastic.co/guide/en/elasticsearch/ reference/current/mapping-all-field.html

Doc Values Replaced Fielddata https://www.elastic.co/guide/en/elasticsearch/ guide/current/_deep_dive_on_doc_values.html

Delivering

Architecture Hot — Warm — Cold (Frozen)

bin/elasticsearch -Enode.attr.rack=rack1 -Enode.attr.size=hot PUT today/_settings { "index.routing.allocation.include.size": "hot" }

Time-Based Indices

Rollover Indices

Rollups

PUT _xpack/rollup/job/metrics { "index_pattern": "metrics-", "rollup_index": "metrics_rollup", "cron": "/30 * * * * ?", "page_size" :1000, "groups" : { "date_histogram": { "field": "timestamp", "interval": "1h", "delay": "7d" }, "terms": { "fields": ["node"] } }, "metrics": [ { "field": "cpu", "metrics": ["min", "max", "sum"] }, { "field": "memory", "metrics": ["avg"] } ] }

Index Lifecycle Management Currently https://github.com/elastic/curator

Conclusion

Benchmarks Fair Reproducible Close to Production

Thank You Questions? Philipp Krenn @xeraa