Official Docker Images – Lessons Learned
A presentation at XP Days in in Kyiv, Ukraine, 02000 by Philipp Krenn
Official Docker Images Lessons Learned Philipp Krenn @xeraa
Developer
Who uses Docker?
Who uses Docker in production?
Who uses stateful Docker images?
Who uses our images?
Who uses our stack with other images?
Docker: the world's most heavily funded college project Internal quote from Slack
Content "Official" Base Images Release Policy Security & Stability Customization Orchestration
Docker Hub "official"
What do you get? docker pull elasticsearch:X Same for Kibana and Logstash
It's Complicated Docker Inc Deprecated Replicated
Custom Registry docker.elastic.co
I’m surprised more people don’t just host their own container registries since doing that is faster than every cloud offering and docker hub https://twitter.com/jessfraz/status/978449365261082625
Our Motivation Download statistics Speed & reliability
https://www.docker.elastic.co
Problems Some broken tooling like automated builds, Kitematic,... China IPv6
https://hub.docker.com/r/elastic/ elasticsearch/ kibana/ *beat/ logstash/ apm-server/
Base Images
Elasticsearch Alpine
Kibana Beats Logstash Ubuntu
Common base image in 5.4+ CentOS 7
https://github.com/elastic/elasticsearch-docker/blob/master/templates/Dockerfile.j2 FROM centos:7 AS prep_es_files ENV PATH /usr/share/elasticsearch/bin:$PATH RUN curl -s https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz | \ tar -C /opt -zxf ENV JAVA_HOME /opt/jdk-11.0.1 RUN groupadd -g 1000 elasticsearch && \ adduser -u 1000 -g 1000 -d /usr/share/elasticsearch elasticsearch
PS: Moving Repos in 6.6 / 7.0 https://github.com/elastic/elasticsearch/pull/36246
Upside Similar setup Shared layers JVM large anyway
Downside Size
$ docker images REPOSITORY docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat TAG 6.5.0 6.5.0 6.5.0 6.2.4 6.2.4 6.2.4 5.6.13 5.6.13 5.6.13 5.3.3 5.3.3 5.3.3 IMAGE ID fcc1f039f61c ff171d17e77c aee067f4a241 327c6538ba4c 7cb69da7148d 26a00abcde82 59fcc69d2cc6 21673573a265 2aec30f6b3fc ffe778f7e489 5857f98b5920 c01be8a8f630 SIZE 727MB 774MB 299MB 933MB 515MB 319MB 653MB 525MB 284MB 679MB 165MB 232MB
5.3 5.6 6.2 6.5 ES 165MB 525MB 515MB 774MB Kibana 679MB 653MB 933MB 727MB Filebeat 232MB 284MB 319MB 299MB
Does it matter? stateful vs stateless
What to include?
Single image for 5.x Platinum trial
Three flavors 6.0 to 6.2 * Basic , OSS, Platinum trial * Default
Two flavors 6.3+ * Basic / Platinum trial , OSS * Default
Include Plugins?
Future
Multiple base images?
Even Windows?!
Release Policy
No :latest
Zombies ideas that should have been killed by evidence, but keep shambling along
6 and 6.5?
What's in a tag? docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana 5.3.3 5.3.3 5.3.3 5857f98b5920 c01be8a8f630 ffe778f7e489 4 months ago 5 months ago 5 months ago
Currently Overwrite tag
Label Schema LABEL org.label-schema.schema-version="1.0" \ org.label-schema.vendor="Elastic" \ org.label-schema.name="elasticsearch" \ org.label-schema.version="{{ elastic_version }}" \ org.label-schema.url="https://www.elastic.co/products/elasticsearch" \ org.label-schema.vcs-url="https://github.com/elastic/elasticsearch-docker" \ {% if image_flavor == 'oss' -%} license="Apache-2.0" {% else -%} license="Elastic License" {% endif -%}
Base image & JVM direct dependencies
Future Add image version?
Security & Stability
Run Elasticsearch as root
Cockroaches claims that disappear for a while when proved wrong, but just keep on coming back
Mode Production: Clusterable Development: Local network
"Docker" mode discovery.type=single-node
Bootstrap checks are here to stay
6.0+ no more default credentials
6.x Security non-trial requires certificates
The container runs Elasticsearch as user elasticsearch using uid:gid 1000:1000. https://www.elastic.co/guide/en/elasticsearch/reference/current/ docker.html
Don't mutate the bind mounted local directory
Those who do not understand Unix are condemned to reinvent it, poorly. — Henry Spencer
Docker default value? LimitNOFILE & LimitNPROC
infinity https://github.com/moby/moby/commit/ 8db61095a3d0bcb0733580734ba5d54bc27a614d (July 2016)
Test $ docker run --rm centos:7 /bin/bash -c \ 'ulimit -Hn && ulimit -Sn && ulimit -Hu && ulimit -Su' 1048576 1048576 unlimited unlimited
Limit for virtual memory? vm.max_map_count
Test $ docker run --rm centos:7 /bin/bash -c \ 'sysctl vm.max_map_count' vm.max_map_count = 262144 Must be set on the host
Combine two new systems to get chaos & despair
Docker is a leaky abstraction
Customization
Feature Request if [ -f /custom/user_init.sh ] then . /custom/user_init.sh fi Or customization through environment variables
No Runtime Mutation
Dockerfile ARG ELASTIC_VERSION FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} RUN bin/elasticsearch-plugin install analysis-phonetic --batch ARG ELASTIC_VERSION RUN bin/elasticsearch-plugin install \ https://github.com/spinscale/elasticsearch-ingest-langdetect/releases/download/ ${ELASTIC_VERSION}.1/ingest-langdetect-${ELASTIC_VERSION}.1.zip --batch
Generate Keystore $ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" \ -v /Users/philipp/Desktop/demo/config/:/usr/share/elasticsearch/config/ \ -it docker.elastic.co/elasticsearch/elasticsearch:6.5.2 /bin/bash [root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore create Created elasticsearch keystore in /usr/share/elasticsearch/config [root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore add test Enter value for test: [root@1006ed50b646 elasticsearch]# exit exit $ cat config/elasticsearch.keystore ??lelasticsearch.keystore?@g?o!?$?K?Lf?w?VAEŠԨm?[?a6?B??? y?,!В}??Ħ?ǣ?AU=?C?:?o? ?W?O8?}U?;p?ӷ???cQ????7?JY? 2A?:???ZUY??2V?9?ϧ??(??0?q\
Mount Keystore (Docker Compose) elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} secrets: - source: elasticsearch.keystore target: /usr/share/elasticsearch/config/elasticsearch.keystore
Orchestration
Who uses Kubernetes?
Who uses Swarm?
Who uses Mesos?
Who uses Nomad?
No orchestration yet
Kubernetes 1.8 allows dots in env vars https://github.com/kubernetes/kubernetes/issues/2707
Kubernetes has made huge improvements in the ability to run stateful workloads including databases and message queues, but I still prefer not to run them on Kubernetes. https://twitter.com/kelseyhightower/status/963413508300812295
Conclusion
"Docker is disrupting the industry"
"Can I run Elasticsearch on Docker?"
"Should I run Elasticsearch on Docker?"
Even when stateful services do the right things managing state is still hard. Mixing stateful and stateless applications on the same cluster elevates the complexity of the entire cluster. Cluster security and upgrades become much harder. https://twitter.com/kelseyhightower/status/963417215608369153
Quick Quiz What do you get?
docker pull elastic/elasticsearch
$ docker pull elastic/elasticsearch Using default tag: latest Error response from daemon: manifest for elastic/elasticsearch:latest not found
docker pull logstash:alpine
docker pull metricbeat:6.5.2
$ docker pull metricbeat:6.5.2 Error response from daemon: pull access denied for metricbeat, repository does not exist or may require 'docker login'
docker pull elastic/metricbeat:6.5.2
docker pull docker.elastic.co/apm/apm-server:6.5.2
Questions & Discussion Philipp Krenn @xeraa
If you want to be taken seriously, you need to provide containers to your users. It’s easy — everybody is uploading containers to Docker Hub, right? Unfortunately, reality is never as easy as it sounds at first. This talk gives an overview of Elastic’s ongoing journey to providing official Docker images:
- Docker Hub: What “official” really means and why we are using our own registry.
- Base image: Just use Alpine — it is small and the perfect fit for containers. We tried that and reconsidered…
- Release policy: What do you actually get in a specific tag and how are we releasing our images?
- Support: Combine two complex systems like Elasticsearch and Docker — and you will get a lot of questions.
- Orchestration: Our current approach for orchestration and how we are treating feature requests.
