Elasticsearch (R)Evolution — You Know, for Search...
A presentation at InfoShare in in Gdańsk, Poland by Philipp Krenn
(R) Evolution Philipp Krenn 44444444 @xeraa
Revolution
https://db-engines.com/en/ranking
https://www.reddit.com/r/SQL/comments/7i828i/ms_sql_query_is_too_complex/
Who uses Elasticsearch?
Infrastructure | Developer Advocate
http://thedudeabides.com/articles/ the_birth_of_compass
Terms Cluster, Node, Index, Shard, Document, ID
$ curl http: //localhost:9200 {
"name" : "elasticsearch1" ,
"cluster_name" : "docker-cluster" ,
"cluster_uuid" : "l6wfwv8XSniiI_Fx6qqrcw" ,
"version" : {
"number" : "5.6.9" ,
"build_hash" : "877a590" ,
"build_date" : "2018-04-12T16:25:14.838Z" ,
"build_snapshot" : false ,
"lucene_version" : "6.6.1" },
"tagline" : "You Know, for Search" }
$ curl http: //localhost:9200 {
"name" : "elasticsearch1" ,
"cluster_name" : "docker-cluster" ,
"cluster_uuid" : "l6wfwv8XSniiI_Fx6qqrcw" ,
"version" : {
"number" : "5.6.9" ,
"build_hash" : "877a590" ,
"build_date" : "2018-04-12T16:25:14.838Z" ,
"build_snapshot" : false ,
"lucene_version" : "6.6.1" },
"tagline" : "You Know, for Search" }
Evolution
Questions: https://sli.do/xeraa Answers: https://twitter.com/xeraa
Strictness * 5.0 * Demo
Bootstrap Checks https://www.elastic.co/guide/en/elasticsearch/ reference/current/bootstrap-checks.html
Parameters & Configs
Rolling Upgrades * 6.0 * Demo
Floodstage Watermark * 6.0 * Demo
Low 85% High 90% Floodstage 95%
Sequence Numbers * 6.0 * Demo
63 bits ought to be enough for anyone.
Tradeoff index.translog.retention.size: 512MB index.translog.retention.age: 12h
Cross Datacenter Replication 6.x or 7.x
Types * 5.6 to 8.0 * Demo
Why Data types Sparsity Scoring
How 5.6 opt-in single type 6.x single type 7.x type optional in API 8.x no more types https://www.elastic.co/guide/en/elasticsearch/reference/current/ removal-of-types.html#_schedule_for_removal_of_mapping_types
Automatic Queue Resizing 6.0
Reject and Retry Instead of Long Queues thread_pool.search.target_response_rate: 2s Serving 50 requests/s Queue size:
Adaptive Replica Selection 6.1 (enabled by default in 7.0)
Pick Best Shard Exponentially Weighted Moving Average (EWMA) Piggyback on requests
Shrink & Split * 4 5.0 444 6.1 * Demo
Shrink Combine shards by a factor
Split Split into a factor of number_of_routing_shards Not required in 7.0+
Shards 7.0
Default: 1 shard per index Oversharding
Benchmarks ∞
Rally https://elasticsearch-benchmarks.elastic.co
Conclusion
Rolling Upgrades Floodstage Watermark Sequence Numbers Types
Automatic Queue Resizing Adaptive Replica Selection Shrink & Split Shard Number
Questions? Philipp Krenn 4444444 @xeraa
Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. After the initial release in 2010 it has become the most widely used full-text search engine, but it is not stopping there.
The revolution happened and now it is time for evolution. We dive into the following questions:
- How did leniency help the initial adoption, but why and how do we lean more on strictness today?
- How can upgrades be improved to avoid any downtime even when changing major versions?
- How can new resiliency features improve recovery scenarios and add totally new features?
- Why are types finally disappearing and how are we are trying to avoid the upgrade pain as much as possible?
- What are examples for some clever performance improvements?
- How can you shrink and (finally) split shards in a highly efficient way?
Attendees learn both about new and upcoming features as well as the motivation and engineering challenges behind them.
Buzz and feedback
Here’s what was said about this presentation on social media.
