Monitor Your Containers

Monitor Your Containers with the Stack Philipp Krenn 44444 @xeraa

Infrastructure | Developer Advocate

$ curl http: //localhost:9200 {

"name" : "zDODSc4" ,

"cluster_name" : "docker-cluster" ,

"cluster_uuid" : "qbx3DVATRfWOgHB6uiLtNw" ,

"version" : {

"number" : "6.3.0" ,

"build_flavor" : "default" ,

"build_type" : "tar" ,

"build_hash" : "424e937" ,

"build_date" : "2018-06-11T23:38:03.357887Z" ,

"build_snapshot" : false ,

"lucene_version" : "7.3.1" ,

"minimum_wire_compatibility_version" : "5.6.0" ,

"minimum_index_compatibility_version" : "5.0.0" },

"tagline" : "You Know, for Search" }

Filebeat

tail -f 4 4

tail -f over the network 4

tail -f over the network on !

Parse & Enrich Logstash or Ingest-Node

34.253.145.46

- [ 06 /Sep/ 2017 : 22 : 33 : 30

0000 ]

"GET /server-status HTTP/1.1"

200

97

"-"

"Go-http-client/1.1"

"-" "remote_ip" : "34.253.145.46" , "method" : "GET" , "url" : "/server-status" , "http_version" : "1.1" , "response_code" : 200 ,

"remote_ip" : "34.253.145.46" "geoip" : {

"continent_name" : "North America" ,

"city_name" : "Houston" ,

"country_iso_code" : "US" ,

"region_name" : "Texas" ,

"location" : {

"lon" : -95.5858 ,

"lat" : 29.6997 } }

At-Least-Once Backpressure Graceful Downtime

Filtering include_lines 44 exclude_lines 44 exclude_files filebeat.prospectors:

input_type:

log paths: -

/var/log/myapp/*.log include_lines:

["^ERR",

"^WARN"]

Multiline Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author.getBookIds(Author.java:38) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Caused by: java.lang.NullPointerException at com.example.myproject.Book.getId(Book.java:22) at com.example.myproject.Author.getBookIds(Author.java:35) ... 1 more multiline.pattern:

'^[[:space:]]+|^Caused by:' multiline.negate:

false multiline.match:

after

JSON Decode

Filebeat Modules Apache2, Auditd, Icinga, IIS, Kafka, Logstash, MongoDB, MySQL, Nginx, Osquery, PostgreSQL, Redis, System, Traefik

Logging with Docker 101 options

https://docs.docker.com/engine/admin/logging/overview/

001 JSON-File Filebeat for JSON ➕ Simple, default, well integrated Metadata (name, labels,...) docker logs ➖ Potentially slow By default unlimited file size

010 Syslog Local Syslog server and Filebeat ➕ Configurable path, rotation,... ➖ Custom Syslog server Metadaten serialized and deserialized Multiline

011 Journald Filebeat ➕ Widely available Metadata docker logs ➖ Not yet supported by Filebeat (Community Beat: Journalbeat)

100 GELF Logstash-GELF-Input ➕ Direct Logstash connection ➖ UDP — no ACK, no backpressure

101 Volume Filebeat ➕ Simple installation (if app rotates logs) Scalable ➖ Metadata

! Today: JSON, Syslog, Volume Future: Journald

Docker Metadata

input_type:

log paths: -

/var/lib/docker/containers//-json.log document_type:

docker

json.message_key:

log processors:

add_docker_metadata:

~

Kubernetes Metadata processors:

add_kubernetes_metadata: in_cluster:

true

Metricbeat

Metricbeat System

Metricbeat Service Many: https://www.elastic.co/guide/en/ beats/metricbeat/current/metricbeat- modules.html

Read cgroup data from /proc/

Part of the system module

No Docker API access required Security

All containers Docker, rkt, runC, LXD,...

Enriches process information automatically with cgroup data

No container names or labels

But Docker...

Dockerbeat https://github.com/Ingensi/dockerbeat

Dockbeat https://github.com/Ingensi/dockbeat

Metricbeat 5.1+

System Permissions $ docker run
--volume=/proc:/hostfs/proc:ro
--volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
--volume=/:/hostfs:ro
--net=host docker.elastic.co/beats/metricbeat:6.3.0 -system.hostfs=/hostfs

Service Permissions $ docker run
--link some-mysql:mysql
-e MYSQL_PASSWORD=secret
docker.elastic.co/beats/metricbeat:6.3.0

Metricbeat and Docker

Docker Metadata processors:

add_docker_metadata:

~

Kubernetes Metadata processors:

add_kubernetes_metadata: in_cluster:

true

Kubernetes Metrics

module:

kubelet metricsets:

["node",

"container",

"volume",

"pod",

"system"] hosts:

["localhost:10255"]

Packetbeat

Protocols

Flows Application layer: Unsupported or encrypted protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow

Packetbeat and Docker

Auditbeat

Linux Kernel File Integrity

Heartbeat

Winlogbeat

https://github.com/elastic/elasticsearch-docker https://github.com/elastic/kibana-docker https://github.com/elastic/logstash-docker https://github.com/elastic/beats-docker

version:

'2' services: kibana: image:

docker.elastic.co/kibana/kibana:6.3.0 links: -

elasticsearch ports: -

5601 :5601 elasticsearch: image:

docker.elastic.co/elasticsearch/elasticsearch:6.3.0 volumes: - esdata: /usr/share/elasticsearch/data ports: -

9200 :9200 volumes: esdata: driver:

local

Demo https://github.com/xeraa/elastic-docker/ tree/master/full_stack Elasticsearch, Kibana, Filebeat, Heartbeat, Metricbeat, Packetbeat, nginx, MySQL

Conclusion

Questions? Philipp Krenn 44444 @xeraa PS: Sticker