Reaching Zen in Your Cluster Coordination Layer

A presentation at Voxxed Days Minsk in May 2019 in Minsk, Belarus by Philipp Krenn

Slide 1

Slide 1

Reaching Zen in Your Cluster Coordination Layer Philipp Krenn @xeraa

Slide 2

Slide 2

Developer

Slide 3

Slide 3

Slide 4

Slide 4

Concepts Cluster, Node, Index, Shard, Document, ID

Slide 5

Slide 5

Cluster Coordination?

Slide 6

Slide 6

Cluster State?

Slide 7

Slide 7

== Cluster Metadata Cluster Settings Index Metadata Lots more

Slide 8

Slide 8

GET _cluster/state Only move forward Do not lose data

Slide 9

Slide 9

Demo Cluster Cluster State

Slide 10

Slide 10

3 Main Components Discovery Master Election Cluster State Publication

Slide 11

Slide 11

Zen Zen to Zen2 Not pluggable

Slide 12

Slide 12

Slide 13

Slide 13

Discovery Where are master-eligible nodes? Is there a master already?

Slide 14

Slide 14

Slide 15

Slide 15

Settings discovery.zen.ping.unicast.hosts -> discovery.seed_hosts static discovery.zen.hosts_provider -> discovery.seed_providers dynamic (File, EC2, GCE,…)

Slide 16

Slide 16

Master Election Agree which node should be master Form a cluster

Slide 17

Slide 17

Slide 18

Slide 18

Slide 19

Slide 19

Slide 20

Slide 20

Slide 21

Slide 21

Slide 22

Slide 22

Slide 23

Slide 23

Slide 24

Slide 24

discovery.zen. minimum_master_node s Trust users?

Slide 25

Slide 25

Demo Upgrade 6.7 -> 7.0, 6.8 -> 7.1

Slide 26

Slide 26

Demo Full Cluster Restart [CTRL] + [C]

Slide 27

Slide 27

cluster. initial_master_node s Nodes for the very first election

Slide 28

Slide 28

List of node names OK to set on multiple nodes as long as they are all consistent Ignored once node has joined a cluster even if restarted Unnecessary when joining new node to existing cluster

Slide 29

Slide 29

Upgrade 6 to 7 Full cluster restart: Set cluster.initial_master_nodes Rolling upgrade: cluster.initial_master_nodes not required

Slide 30

Slide 30

Demo Cluster Rebuild

Slide 31

Slide 31

elasticsearch2 | {“type”: “server”, “timestamp”: “2019-05-24T14:02:51,173+0000”, “level”: “WARN”, “component”: “o.e.c.c.ClusterFormationFailureHelper”, “cluster.name”: “docker-cluster”, “node.name”: “elasticsearch2”, “message”:

Slide 32

Slide 32

“master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [ {elasticsearch1}{pSUJ60tSRWSrcWkRevLfyA}{_jIaabgyTQOHAOjcwUruIQ} {192.168.112.3}{192.168.112.3:9300} {ml.machine_memory=1073741824, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch3}{ngaTCze8QHSHydCXsttXyw}{mbIad-A4SLOJvP7Ava5dEw} {192.168.112.4}{192.168.112.4:9300} {ml.machine_memory=1073741824, ml.max_open_jobs=20, xpack.installed=true} ]; discovery will continue using [192.168.112.3:9300, 192.168.112.4:9300] from hosts providers and [ {elasticsearch2}{iANt64LESxqjJv8tHV5KKw}{K0bYEuQ2TnamsiOefTUXgQ} {192.168.112.2}{192.168.112.2:9300} {ml.machine_memory=1073741824, xpack.installed=true, ml.max_open_jobs=20} ] from last-known cluster state; node term 0, last-accepted version 0 in term 0”

Slide 33

Slide 33

Result: Safer Cluster Operations

Slide 34

Slide 34

Dynamic Cluster Scaling Master-ineligible: same as before Adding master-eligible: Just do it Removing master-eligible: Just do it, as long as you remove less than half

Slide 35

Slide 35

Cluster State Publication Agree on cluster state updates Broadcast updates to all nodes

Slide 36

Slide 36

Slide 37

Slide 37

Slide 38

Slide 38

Conclusion

Slide 39

Slide 39

Zen to Zen2 Faster, safer, more debuggable

Slide 40

Slide 40

Reaching Zen in Your Cluster Coordination Layer Philipp Krenn @xeraa