Scale Your Metrics with Elasticsearch Philipp Krenn @xeraa
Slide 1
Slide 2
Slide 3
$ curl http://localhost:9200 { "name": "elasticsearch", "cluster_name": "docker-cluster", "cluster_uuid": "l6wfwv8XSniiI_Fx6qqrcw", "version": { "number": "5.6.9", "build_hash": "877a590", "build_date": "2018-04-12T16:25:14.838Z", "build_snapshot": false, "lucene_version": "6.6.1" }, "tagline": "You Know, for Search" }
Slide 4
https://db-engines.com/en/ ranking
Slide 5
Slide 6
Slide 7
Full-Text Search
Slide 8
Slide 9
Slide 10
Logs
Slide 11
Slide 12
Slide 13
Slide 14
Metrics
Slide 15
I'm not going to use a search engine for metrics. — Too o!en
Slide 16
Developer
Slide 17
Agenda Building Blocks Tuning Delivering
Slide 18
Building Blocks
Slide 19
Only accept features that scale. — https://github.com/elastic/engineering/blob/ master/development_constitution.md
Slide 20
Horizontal Scaling Shards Replication Writes & Reads
Slide 21
Cluster, Node, Index, Shard
Slide 22
Write
Slide 23
Write
Slide 24
Write
Slide 25
Write
Slide 26
Write
Slide 27
Write
Slide 28
Read
Slide 29
Read
Slide 30
Read
Slide 31
Read
Slide 32
Append-Only Optimization IDs assigned on coordinating node Fast add docs instead of the slow update docs
Slide 33
Storage Compression LZ4 (default), DEFLATE (best_compression)
Slide 34
BKD Trees Points in Lucene
Slide 35
Half & Scaled Floats
Slide 36
Pipeline Aggregations
Slide 37
Tuning
Slide 38
_all Removal https://www.elastic.co/guide/en/elasticsearch/ reference/current/mapping-all-field.html
Slide 39
Doc Values Replaced Fielddata https://www.elastic.co/guide/en/elasticsearch/ guide/current/_deep_dive_on_doc_values.html
Slide 40
Delivering
Slide 41
Architecture Hot — Warm — Cold (Frozen)
Slide 42
bin/elasticsearch -Enode.attr.rack=rack1 -Enode.attr.size=hot PUT today/_settings { "index.routing.allocation.include.size": "hot" }
Slide 43
Time-Based Indices
Slide 44
Rollover Indices
Slide 45
Rollups
Slide 46
Slide 47
PUT _xpack/rollup/job/metrics { "index_pattern": "metrics-", "rollup_index": "metrics_rollup", "cron": "/30 * * * * ?", "page_size" :1000, "groups" : { "date_histogram": { "field": "timestamp", "interval": "1h", "delay": "7d" }, "terms": { "fields": ["node"] } }, "metrics": [ { "field": "cpu", "metrics": ["min", "max", "sum"] }, { "field": "memory", "metrics": ["avg"] } ] }
Slide 48
Slide 49
Index Lifecycle Management Currently https://github.com/elastic/curator
Slide 50
Slide 51
Conclusion
Slide 52
Benchmarks Fair Reproducible Close to Production
Slide 53
Slide 54
Slide 55
Slide 56
Thank You Questions? Philipp Krenn @xeraa
