Scale Your Metrics with Elasticsearch

Slide 1

Scale Your Metrics with Elasticsearch Philipp Krenn @xeraa

Slide 2

Slide 3

$ curl http://localhost:9200 { “name” : “elasticsearch-hot”, “cluster_name” : “metrics-cluster”, “cluster_uuid” : “06nHPLLgTrmZEpYli6JW5w”, “version” : { “number” : “6.5.0”, “build_flavor” : “default”, “build_type” : “tar”, “build_hash” : “c53b7d3”, “build_date” : “2018-11-08T21:28:50.577384Z”, “build_snapshot” : false, “lucene_version” : “7.5.0”, “minimum_wire_compatibility_version” : “5.6.0”, “minimum_index_compatibility_version” : “5.0.0” }, “tagline” : “You Know, for Search” }

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

I’m not going to use a search engine for metrics. — Too often

Slide 13

Developer

Slide 14

Agenda Building Blocks Architecture Demo

Slide 15

Building Blocks

Slide 16

Only accept features that scale. — https://github.com/elastic/engineering/blob/master/ development_constitution.md

Slide 17

Horizontal Scaling Shards Replication Writes & Reads

Slide 18

Cluster, Node, Index, Shard

Slide 19

Write Coordinating Node, Hash, Primary, Replica(s)

Slide 20

Get Coordinating Node, Hash, Shard

Slide 21

Search Coordinating Node, Query then Fetch

Slide 22

Append-Only Optimization IDs assigned by coordinating node Fast add instead of the slow update https://github.com/elastic/elasticsearch/issues/19813

Slide 23

Lucene Segments index.refresh_interval: 1s index.search.idle.after: 30s Iff default refresh (added in 7.0)

Slide 24

Storage Compression LZ4 (default) DEFLATE (best_compression)

Slide 25

BKD Trees Points in Lucene (added in 6.0)

Slide 26

Integer (1D 4 byte point) vs legacy IntField

Slide 27

Half & Scaled Floats

Slide 28

Slide 29

https://github.com/elastic/beats/blob/master/metricbeat/module/system/load/_meta/ fields.yml - name: load type: group description: > CPU load averages. release: ga fields: - name: “1” type: scaled_float scaling_factor: 100 description: > Load average for the last minute. - name: “5” type: scaled_float scaling_factor: 100 description: > Load average for the last 5 minutes. …

Slide 30

_all Removal https://www.elastic.co/guide/en/elasticsearch/reference/ current/mapping-all-field.html

Slide 31

Doc Values Replaced Fielddata https://www.elastic.co/guide/en/elasticsearch/guide/ current/_deep_dive_on_doc_values.html

Slide 32

Architecture

Slide 33

Time Based Indices index: “metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}”

Slide 34

Rollover Indices Condition when to switch

Slide 35

Slide 36

Rollups

Slide 37

Nodes! “

Slide 38

$ bin/elasticsearch -Enode.attr.rack=rack1 -Enode.attr.size=hot PUT /metricbeat/_settings { “index.routing.allocation.include.size”: “hot” }

Slide 39

Frozen Indices Ratio Heap : Storage Read-only No memory

Slide 40

Frozen Indices Throttled Thread Pool 1 parallel search / node 100 in queue

Slide 41

Index Lifecycle Management https://github.com/elastic/curator for snapshots

Slide 42

Features & Order https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME );