Elasticsearch (R)Evolution — You Know, for Search...
A presentation at Berlin Buzzwords in in Berlin, Germany by Philipp Krenn
(R) Evolution Philipp Krenn 44444444 @xeraa
Developer !
Questions: https://sli.do/xeraa Answers: https://twitter.com/xeraa
Revolution
https://db-engines.com/en/ranking
Evolution
Strictness * 5.0 * Demo
Bootstrap Checks https://www.elastic.co/guide/en/elasticsearch/ reference/current/bootstrap-checks.html
Parameters & Configs
Rolling Upgrades * 6.0 * Demo
Floodstage Watermark * 6.0 * Demo
Low 85% High 90% Floodstage 95%
Sequence Numbers * 6.0 * Demo
63 bits ought to be enough for anyone.
Tradeoff index.translog.retention.size: 512MB index.translog.retention.age: 12h
Cross Datacenter Replication 6.x or 7.x
Types * 5.6 to 8.0 * Demo
Why Data types Sparsity Scoring
How 5.6 opt-in single type 6.x single type 7.x type optional in API 8.x no more types https://www.elastic.co/guide/en/elasticsearch/reference/current/ removal-of-types.html#_schedule_for_removal_of_mapping_types
Automatic Queue Resizing 6.0
Reject and Retry Instead of Long Queues thread_pool.search.target_response_rate: 2s Serving 50 requests/s Queue size:
Adaptive Replica Selection 6.1 (enabled by default in 7.0)
Pick Best Shard Exponentially Weighted Moving Average (EWMA) Piggyback on responses to coordinating node
Shrink & Split * 4 5.0 444 6.1 * Demo
Shrink Combine shards by a factor
Split Split into a factor of number_of_routing_shards Not required in 7.0+
Open Code 6.3
Shards 7.0
Default: 1 Shard per Index Oversharding
Simon Says Use a single shard until it blows up
Beats: Rollover Planning phase: size based, depends on ILM
JDK 11 ? 7.0
Can users upgrade? Version for Java clients?
Low-level REST: independent (JDK 7) High-level REST: JDK 11 Transport: deprecated
Benchmarks ∞
Rally https://elasticsearch-benchmarks.elastic.co
Conclusion
Don't be Shay to change
Strictness Rolling Upgrades Floodstage Watermark Sequence Numbers Types
Automatic Queue Resizing & Adaptive Replica Selection Shrink & Split Open Code Default Shard Number JDK 11
https://www.meetup.com/Elasticsearch-Berlin/ Thursday, June 14, 19:00, Springer Nature
Questions? Philipp Krenn 4444444 @xeraa
Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. After the initial release in 2010 it has become the most widely used full-text search engine, but it is not stopping there.
The revolution happened and now it is time for evolution. We dive into the following questions:
- How did leniency help the initial adoption, but why and how do we lean more on strictness today?
- How can upgrades be improved to avoid any downtime even when changing major versions?
- How can new resiliency features improve recovery scenarios and add totally new features?
- Why are types finally disappearing and how are we are trying to avoid the upgrade pain as much as possible?
- What are examples for some clever performance improvements?
- How can you shrink and (finally) split shards in a highly efficient way?
Attendees learn both about new and upcoming features as well as the motivation and engineering challenges behind them.
Video
Code
The following code examples from the presentation can be tried out live.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Crate.io at Berlin Buzzwords
Quick recap of the talk and others at Berlin Buzzwords.
Buzz and feedback
Here’s what was said about this presentation on social media.
