Reaching Zen in Your Cluster Coordination Layer
A presentation at Voxxed Days Minsk in in Minsk, Belarus by Philipp Krenn
Reaching Zen in Your Cluster Coordination Layer Philipp Krenn @xeraa
Developer
Concepts Cluster, Node, Index, Shard, Document, ID
Cluster Coordination?
Cluster State?
== Cluster Metadata Cluster Settings Index Metadata Lots more
GET _cluster/state Only move forward Do not lose data
Demo Cluster Cluster State
3 Main Components Discovery Master Election Cluster State Publication
Zen Zen to Zen2 Not pluggable
Discovery Where are master-eligible nodes? Is there a master already?
Settings discovery.zen.ping.unicast.hosts -> discovery.seed_hosts static discovery.zen.hosts_provider -> discovery.seed_providers dynamic (File, EC2, GCE,…)
Master Election Agree which node should be master Form a cluster
discovery.zen. minimum_master_node s Trust users?
Demo Upgrade 6.7 -> 7.0, 6.8 -> 7.1
Demo Full Cluster Restart [CTRL] + [C]
cluster. initial_master_node s Nodes for the very first election
List of node names OK to set on multiple nodes as long as they are all consistent Ignored once node has joined a cluster even if restarted Unnecessary when joining new node to existing cluster
Upgrade 6 to 7 Full cluster restart: Set cluster.initial_master_nodes Rolling upgrade: cluster.initial_master_nodes not required
Demo Cluster Rebuild
elasticsearch2 | {“type”: “server”, “timestamp”: “2019-05-24T14:02:51,173+0000”, “level”: “WARN”, “component”: “o.e.c.c.ClusterFormationFailureHelper”, “cluster.name”: “docker-cluster”, “node.name”: “elasticsearch2”, “message”:
“master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [ {elasticsearch1}{pSUJ60tSRWSrcWkRevLfyA}{_jIaabgyTQOHAOjcwUruIQ} {192.168.112.3}{192.168.112.3:9300} {ml.machine_memory=1073741824, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch3}{ngaTCze8QHSHydCXsttXyw}{mbIad-A4SLOJvP7Ava5dEw} {192.168.112.4}{192.168.112.4:9300} {ml.machine_memory=1073741824, ml.max_open_jobs=20, xpack.installed=true} ]; discovery will continue using [192.168.112.3:9300, 192.168.112.4:9300] from hosts providers and [ {elasticsearch2}{iANt64LESxqjJv8tHV5KKw}{K0bYEuQ2TnamsiOefTUXgQ} {192.168.112.2}{192.168.112.2:9300} {ml.machine_memory=1073741824, xpack.installed=true, ml.max_open_jobs=20} ] from last-known cluster state; node term 0, last-accepted version 0 in term 0”
Result: Safer Cluster Operations
Dynamic Cluster Scaling Master-ineligible: same as before Adding master-eligible: Just do it Removing master-eligible: Just do it, as long as you remove less than half
Cluster State Publication Agree on cluster state updates Broadcast updates to all nodes
Conclusion
Zen to Zen2 Faster, safer, more debuggable
Reaching Zen in Your Cluster Coordination Layer Philipp Krenn @xeraa
Distributed systems are hard and unfortunately often less perfect than we would hope. This talk shows a concrete example: Elasticsearch’s cluster coordination system, called Zen Discovery, got a rewrite in version 7.0. Starting from a formal model, the coordination layer was rebuilt to address multiple issues discovered over the years. We discuss the main improvements of the new implementation: Master elections are much faster, the infamous minimum_master_nodes setting has been removed, growing and shrinking clusters becomes safer and easier, and leaves less room to misconfigure the system. Let us join the new, more Zen way of cluster coordination.
