Monitor Your Containers

A presentation at Barcamp Armenia in June 2018 in Yerevan, Armenia by Philipp Krenn

Slide 1

Monitor Your Containers with the Stack Philipp Krenn 44444 @xeraa

Slide 2

Infrastructure | Developer Advocate

Slide 3

Slide 4

Slide 5

$ curl http: //localhost:9200 {

"name" : "zDODSc4" ,

"cluster_name" : "docker-cluster" ,

"cluster_uuid" : "qbx3DVATRfWOgHB6uiLtNw" ,

"version" : {

"number" : "6.3.0" ,

"build_flavor" : "default" ,

"build_type" : "tar" ,

"build_hash" : "424e937" ,

"build_date" : "2018-06-11T23:38:03.357887Z" ,

"build_snapshot" : false ,

"lucene_version" : "7.3.1" ,

"minimum_wire_compatibility_version" : "5.6.0" ,

"minimum_index_compatibility_version" : "5.0.0" },

"tagline" : "You Know, for Search" }

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Filebeat

Slide 16

tail -f 4 4

Slide 17

tail -f over the network 4

Slide 18

tail -f over the network on !

Slide 19

Parse & Enrich Logstash or Ingest-Node

Slide 20

34.253.145.46

- [ 06 /Sep/ 2017 : 22 : 33 : 30

0000 ]

"GET /server-status HTTP/1.1"

200

"-"

"Go-http-client/1.1"

"-" "remote_ip" : "34.253.145.46" , "method" : "GET" , "url" : "/server-status" , "http_version" : "1.1" , "response_code" : 200 ,

Slide 21

"remote_ip" : "34.253.145.46" "geoip" : {

"continent_name" : "North America" ,

"city_name" : "Houston" ,

"country_iso_code" : "US" ,

"region_name" : "Texas" ,

"location" : {

"lon" : -95.5858 ,

"lat" : 29.6997 } }

Slide 22

At-Least-Once Backpressure Graceful Downtime

Slide 23

Slide 24

Slide 25

Slide 26

Filtering include_lines 44 exclude_lines 44 exclude_files filebeat.prospectors:

input_type:

log paths: -

/var/log/myapp/*.log include_lines:

["^ERR",

"^WARN"]

Slide 27

Multiline Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author.getBookIds(Author.java:38) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Caused by: java.lang.NullPointerException at com.example.myproject.Book.getId(Book.java:22) at com.example.myproject.Author.getBookIds(Author.java:35) ... 1 more multiline.pattern:

'^[[:space:]]+|^Caused by:' multiline.negate:

false multiline.match:

after

Slide 28

JSON Decode

Slide 29

Filebeat Modules Apache2, Auditd, Icinga, IIS, Kafka, Logstash, MongoDB, MySQL, Nginx, Osquery, PostgreSQL, Redis, System, Traefik

Slide 30

Logging with Docker 101 options

Slide 31

https://docs.docker.com/engine/admin/logging/overview/

Slide 32

001 JSON-File Filebeat for JSON ➕ Simple, default, well integrated Metadata (name, labels,...) docker logs ➖ Potentially slow By default unlimited file size

Slide 33

010 Syslog Local Syslog server and Filebeat ➕ Configurable path, rotation,... ➖ Custom Syslog server Metadaten serialized and deserialized Multiline

Slide 34

011 Journald Filebeat ➕ Widely available Metadata docker logs ➖ Not yet supported by Filebeat (Community Beat: Journalbeat)

Slide 35

100 GELF Logstash-GELF-Input ➕ Direct Logstash connection ➖ UDP — no ACK, no backpressure

Slide 36

101 Volume Filebeat ➕ Simple installation (if app rotates logs) Scalable ➖ Metadata

Slide 37

! Today: JSON, Syslog, Volume Future: Journald

Slide 38

Docker Metadata

input_type:

log paths: -

/var/lib/docker/containers//-json.log document_type:

docker

json.message_key:

log processors:

add_docker_metadata:

Slide 39

Kubernetes Metadata processors:

add_kubernetes_metadata: in_cluster:

true

Slide 40

Metricbeat

Slide 41

Metricbeat System

Slide 42

Metricbeat Service Many: https://www.elastic.co/guide/en/ beats/metricbeat/current/metricbeat- modules.html

Slide 43

Read cgroup data from /proc/

Slide 44

Part of the system module

Slide 45

No Docker API access required Security

Slide 46

All containers Docker, rkt, runC, LXD,...

Slide 47

Enriches process information automatically with cgroup data

Slide 48

No container names or labels

Slide 49

But Docker...

Slide 50

Slide 51

Dockerbeat https://github.com/Ingensi/dockerbeat

Slide 52

Dockerbeat https://github.com/Ingensi/dockerbeat

Slide 53

Dockbeat https://github.com/Ingensi/dockbeat

Slide 54

Metricbeat 5.1+

Slide 55

System Permissions $ docker run
--volume=/proc:/hostfs/proc:ro
--volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
--volume=/:/hostfs:ro
--net=host docker.elastic.co/beats/metricbeat:6.3.0 -system.hostfs=/hostfs

Slide 56

Service Permissions $ docker run
--link some-mysql:mysql
-e MYSQL_PASSWORD=secret
docker.elastic.co/beats/metricbeat:6.3.0

Slide 57

Metricbeat and Docker

Slide 58

Docker Metadata processors:

add_docker_metadata:

Slide 59

Kubernetes Metadata processors:

add_kubernetes_metadata: in_cluster:

true

Slide 60

Kubernetes Metrics

module:

kubelet metricsets:

["node",

"container",

"volume",

"pod",

"system"] hosts:

["localhost:10255"]

Slide 61

Slide 62

Packetbeat

Slide 63

Protocols

Slide 64

Flows Application layer: Unsupported or encrypted protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow

Slide 65

Packetbeat and Docker

Slide 66

Auditbeat

Slide 67

Linux Kernel File Integrity

Slide 68

Heartbeat

Slide 69

Winlogbeat

Slide 70

Slide 71

https://github.com/elastic/elasticsearch-docker https://github.com/elastic/kibana-docker https://github.com/elastic/logstash-docker https://github.com/elastic/beats-docker