Official Docker Images – Lessons Learned

Official Docker Images Lessons Learned Philipp Krenn @xeraa

Developer

Who uses Docker?

Who uses Docker in production?

Who uses stateful Docker images?

Who uses our images?

Who uses our stack with other images?

Docker: the world's most heavily funded college project Internal quote from Slack

Content "Official" Base Images Release Policy Security & Stability Customization Orchestration

Docker Hub "official"

What do you get? docker pull elasticsearch:X Same for Kibana and Logstash

It's Complicated Docker Inc Deprecated Replicated

Custom Registry docker.elastic.co

I’m surprised more people don’t just host their own container registries since doing that is faster than every cloud offering and docker hub https://twitter.com/jessfraz/status/978449365261082625

Our Motivation Download statistics Speed & reliability

https://www.docker.elastic.co

Problems Some broken tooling like automated builds, Kitematic,... China IPv6

https://hub.docker.com/r/elastic/ elasticsearch/ kibana/ *beat/ logstash/ apm-server/

Base Images

Elasticsearch Alpine

Kibana Beats Logstash Ubuntu

Common base image in 5.4+ CentOS 7

https://github.com/elastic/elasticsearch-docker/blob/master/templates/Dockerfile.j2 FROM centos:7 AS prep_es_files ENV PATH /usr/share/elasticsearch/bin:$PATH RUN curl -s https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz | \ tar -C /opt -zxf ENV JAVA_HOME /opt/jdk-11.0.1 RUN groupadd -g 1000 elasticsearch && \ adduser -u 1000 -g 1000 -d /usr/share/elasticsearch elasticsearch

PS: Moving Repos in 6.6 / 7.0 https://github.com/elastic/elasticsearch/pull/36246

Upside Similar setup Shared layers JVM large anyway

Downside Size

$ docker images REPOSITORY docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat TAG 6.5.0 6.5.0 6.5.0 6.2.4 6.2.4 6.2.4 5.6.13 5.6.13 5.6.13 5.3.3 5.3.3 5.3.3 IMAGE ID fcc1f039f61c ff171d17e77c aee067f4a241 327c6538ba4c 7cb69da7148d 26a00abcde82 59fcc69d2cc6 21673573a265 2aec30f6b3fc ffe778f7e489 5857f98b5920 c01be8a8f630 SIZE 727MB 774MB 299MB 933MB 515MB 319MB 653MB 525MB 284MB 679MB 165MB 232MB

5.3 5.6 6.2 6.5 ES 165MB 525MB 515MB 774MB Kibana 679MB 653MB 933MB 727MB Filebeat 232MB 284MB 319MB 299MB

Does it matter? stateful vs stateless

What to include?

Single image for 5.x Platinum trial

Three flavors 6.0 to 6.2 * Basic , OSS, Platinum trial * Default

Two flavors 6.3+ * Basic / Platinum trial , OSS * Default

Include Plugins?

Future

Multiple base images?

Even Windows?!

Release Policy

No :latest

Zombies ideas that should have been killed by evidence, but keep shambling along

6 and 6.5?

What's in a tag? docker.elastic.co/elasticsearch/elasticsearch docker.elastic.co/beats/filebeat docker.elastic.co/kibana/kibana 5.3.3 5.3.3 5.3.3 5857f98b5920 c01be8a8f630 ffe778f7e489 4 months ago 5 months ago 5 months ago

Currently Overwrite tag

Label Schema LABEL org.label-schema.schema-version="1.0" \ org.label-schema.vendor="Elastic" \ org.label-schema.name="elasticsearch" \ org.label-schema.version="{{ elastic_version }}" \ org.label-schema.url="https://www.elastic.co/products/elasticsearch" \ org.label-schema.vcs-url="https://github.com/elastic/elasticsearch-docker" \ {% if image_flavor == 'oss' -%} license="Apache-2.0" {% else -%} license="Elastic License" {% endif -%}

Base image & JVM direct dependencies

Future Add image version?

Security & Stability

Run Elasticsearch as root

Cockroaches claims that disappear for a while when proved wrong, but just keep on coming back

Mode Production: Clusterable Development: Local network

"Docker" mode discovery.type=single-node

Bootstrap checks are here to stay

6.0+ no more default credentials

6.x Security non-trial requires certificates

The container runs Elasticsearch as user elasticsearch using uid:gid 1000:1000. https://www.elastic.co/guide/en/elasticsearch/reference/current/ docker.html

Don't mutate the bind mounted local directory

Those who do not understand Unix are condemned to reinvent it, poorly. — Henry Spencer

Docker default value? LimitNOFILE & LimitNPROC

infinity https://github.com/moby/moby/commit/ 8db61095a3d0bcb0733580734ba5d54bc27a614d (July 2016)

Test $ docker run --rm centos:7 /bin/bash -c \ 'ulimit -Hn && ulimit -Sn && ulimit -Hu && ulimit -Su' 1048576 1048576 unlimited unlimited

Limit for virtual memory? vm.max_map_count

Test $ docker run --rm centos:7 /bin/bash -c \ 'sysctl vm.max_map_count' vm.max_map_count = 262144 Must be set on the host

Combine two new systems to get chaos & despair

Docker is a leaky abstraction

Customization

Feature Request if [ -f /custom/user_init.sh ] then . /custom/user_init.sh fi Or customization through environment variables

No Runtime Mutation

Dockerfile ARG ELASTIC_VERSION FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} RUN bin/elasticsearch-plugin install analysis-phonetic --batch ARG ELASTIC_VERSION RUN bin/elasticsearch-plugin install \ https://github.com/spinscale/elasticsearch-ingest-langdetect/releases/download/ ${ELASTIC_VERSION}.1/ingest-langdetect-${ELASTIC_VERSION}.1.zip --batch

Generate Keystore $ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" \ -v /Users/philipp/Desktop/demo/config/:/usr/share/elasticsearch/config/ \ -it docker.elastic.co/elasticsearch/elasticsearch:6.5.2 /bin/bash [root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore create Created elasticsearch keystore in /usr/share/elasticsearch/config [root@1006ed50b646 elasticsearch]# ./bin/elasticsearch-keystore add test Enter value for test: [root@1006ed50b646 elasticsearch]# exit exit $ cat config/elasticsearch.keystore ??lelasticsearch.keystore?@g?o!?$?K?Lf?w?VAEŠԨm?[?a6?B??? y?,!В}??Ħ?ǣ?AU=?C?:?o? ?W?O8?}U?;p?ӷ???cQ????7?JY? 2A?:???ZUY??2V?9?ϧ??(??0?q\

Mount Keystore (Docker Compose) elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} secrets: - source: elasticsearch.keystore target: /usr/share/elasticsearch/config/elasticsearch.keystore

Orchestration

Who uses Kubernetes?

Who uses Swarm?

Who uses Mesos?

Who uses Nomad?

No orchestration yet

Kubernetes 1.8 allows dots in env vars https://github.com/kubernetes/kubernetes/issues/2707

Kubernetes has made huge improvements in the ability to run stateful workloads including databases and message queues, but I still prefer not to run them on Kubernetes. https://twitter.com/kelseyhightower/status/963413508300812295

Conclusion

"Docker is disrupting the industry"

"Can I run Elasticsearch on Docker?"

"Should I run Elasticsearch on Docker?"

Even when stateful services do the right things managing state is still hard. Mixing stateful and stateless applications on the same cluster elevates the complexity of the entire cluster. Cluster security and upgrades become much harder. https://twitter.com/kelseyhightower/status/963417215608369153

Quick Quiz What do you get?

docker pull elastic/elasticsearch

$ docker pull elastic/elasticsearch Using default tag: latest Error response from daemon: manifest for elastic/elasticsearch:latest not found

docker pull logstash:alpine

docker pull metricbeat:6.5.2

$ docker pull metricbeat:6.5.2 Error response from daemon: pull access denied for metricbeat, repository does not exist or may require 'docker login'

docker pull elastic/metricbeat:6.5.2

docker pull docker.elastic.co/apm/apm-server:6.5.2

Questions & Discussion Philipp Krenn @xeraa