Monitor Your Containers
A presentation at Barcamp Armenia in in Yerevan, Armenia by Philipp Krenn
Monitor Your Containers with the Stack Philipp Krenn 44444 @xeraa
Infrastructure | Developer Advocate
$ curl http: //localhost:9200 {
"name" : "zDODSc4" ,
"cluster_name" : "docker-cluster" ,
"cluster_uuid" : "qbx3DVATRfWOgHB6uiLtNw" ,
"version" : {
"number" : "6.3.0" ,
"build_flavor" : "default" ,
"build_type" : "tar" ,
"build_hash" : "424e937" ,
"build_date" : "2018-06-11T23:38:03.357887Z" ,
"build_snapshot" : false ,
"lucene_version" : "7.3.1" ,
"minimum_wire_compatibility_version" : "5.6.0" ,
"minimum_index_compatibility_version" : "5.0.0" },
"tagline" : "You Know, for Search" }
Filebeat
tail -f 4 4
tail -f over the network 4
tail -f over the network on !
Parse & Enrich Logstash or Ingest-Node
34.253.145.46
- [ 06 /Sep/ 2017 : 22 : 33 : 30
0000 ]
"GET /server-status HTTP/1.1"
200
97
"-"
"Go-http-client/1.1"
"-" "remote_ip" : "34.253.145.46" , "method" : "GET" , "url" : "/server-status" , "http_version" : "1.1" , "response_code" : 200 ,
"remote_ip" : "34.253.145.46" "geoip" : {
"continent_name" : "North America" ,
"city_name" : "Houston" ,
"country_iso_code" : "US" ,
"region_name" : "Texas" ,
"location" : {
"lon" : -95.5858 ,
"lat" : 29.6997 } }
At-Least-Once Backpressure Graceful Downtime
Filtering include_lines 44 exclude_lines 44 exclude_files filebeat.prospectors:
- input_type:
log paths: -
/var/log/myapp/*.log include_lines:
["^ERR",
"^WARN"]
Multiline Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author.getBookIds(Author.java:38) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Caused by: java.lang.NullPointerException at com.example.myproject.Book.getId(Book.java:22) at com.example.myproject.Author.getBookIds(Author.java:35) ... 1 more multiline.pattern:
'^[[:space:]]+|^Caused by:' multiline.negate:
false multiline.match:
after
JSON Decode
Filebeat Modules Apache2, Auditd, Icinga, IIS, Kafka, Logstash, MongoDB, MySQL, Nginx, Osquery, PostgreSQL, Redis, System, Traefik
Logging with Docker 101 options
https://docs.docker.com/engine/admin/logging/overview/
001 JSON-File Filebeat for JSON ➕ Simple, default, well integrated Metadata (name, labels,...) docker logs ➖ Potentially slow By default unlimited file size
010 Syslog Local Syslog server and Filebeat ➕ Configurable path, rotation,... ➖ Custom Syslog server Metadaten serialized and deserialized Multiline
011 Journald Filebeat ➕ Widely available Metadata docker logs ➖ Not yet supported by Filebeat (Community Beat: Journalbeat)
100 GELF Logstash-GELF-Input ➕ Direct Logstash connection ➖ UDP — no ACK, no backpressure
101 Volume Filebeat ➕ Simple installation (if app rotates logs) Scalable ➖ Metadata
! Today: JSON, Syslog, Volume Future: Journald
Docker Metadata
- input_type:
log paths: -
/var/lib/docker/containers//-json.log document_type:
docker
json.message_key:
log processors:
- add_docker_metadata:
~
Kubernetes Metadata processors:
- add_kubernetes_metadata: in_cluster:
true
Metricbeat
Metricbeat System
Metricbeat Service Many: https://www.elastic.co/guide/en/ beats/metricbeat/current/metricbeat- modules.html
Read cgroup data from /proc/
Part of the system module
No Docker API access required Security
All containers Docker, rkt, runC, LXD,...
Enriches process information automatically with cgroup data
No container names or labels
But Docker...
Dockerbeat https://github.com/Ingensi/dockerbeat
Dockerbeat https://github.com/Ingensi/dockerbeat
Dockbeat https://github.com/Ingensi/dockbeat
Metricbeat 5.1+
System Permissions
$ docker run
--volume=/proc:/hostfs/proc:ro
--volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
--volume=/:/hostfs:ro
--net=host
docker.elastic.co/beats/metricbeat:6.3.0 -system.hostfs=/hostfs
Service Permissions
$ docker run
--link some-mysql:mysql
-e MYSQL_PASSWORD=secret
docker.elastic.co/beats/metricbeat:6.3.0
Metricbeat and Docker
Docker Metadata processors:
- add_docker_metadata:
~
Kubernetes Metadata processors:
- add_kubernetes_metadata: in_cluster:
true
Kubernetes Metrics
- module:
kubelet metricsets:
["node",
"container",
"volume",
"pod",
"system"] hosts:
["localhost:10255"]
Packetbeat
Protocols
Flows Application layer: Unsupported or encrypted protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow
Packetbeat and Docker
Auditbeat
Linux Kernel File Integrity
Heartbeat
Winlogbeat
https://github.com/elastic/elasticsearch-docker https://github.com/elastic/kibana-docker https://github.com/elastic/logstash-docker https://github.com/elastic/beats-docker
version:
'2' services: kibana: image:
docker.elastic.co/kibana/kibana:6.3.0 links: -
elasticsearch ports: -
5601 :5601 elasticsearch: image:
docker.elastic.co/elasticsearch/elasticsearch:6.3.0 volumes: - esdata: /usr/share/elasticsearch/data ports: -
9200 :9200 volumes: esdata: driver:
local
Demo https://github.com/xeraa/elastic-docker/ tree/master/full_stack Elasticsearch, Kibana, Filebeat, Heartbeat, Metricbeat, Packetbeat, nginx, MySQL
Conclusion
Questions? Philipp Krenn 44444 @xeraa PS: Sticker
Containers are quickly gaining popularity as the preferred tool for deploying and running services. While being easier to deploy and isolate, containerized applications are creating new challenges for the logging and monitoring systems.
This talk will look into the details about how the Elastic Stack, and in particular Beats — lightweight shippers — are gathering data from containers.
The session shows our way to:
- fetch logs from containers
- collect different measurements from cgroups
- collect metrics using the Docker API
- enhance the data with the metadata of the containers
- monitor the network traffic exchanged between containers
- collect metrics from the underlying host
- enrich data with Docker and Kubernetes metadata
We conclude the talk with a live demo of all the components in action.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub: Demo Code
Running the Elastic Stack on Docker and monitoring it plus nginx and MySQL.
