Centralized Logging Patterns
A presentation at DevOpsDays Riga 2018 in in Riga, Latvia by Philipp Krenn
Centralized Logging Patterns Philipp Krenn @xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Developer @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Apache License 2.0 @xeraa
Disclaimer I build highly monitored Hello World apps @xeraa
Example: Java SLF4J, Logback, MDC @xeraa
.NET: NLog PHP: Monolog JavaScript: Winston ... @xeraa
Anti-Pattern: print System.out.println("Oops"); @xeraa
Anti-Pattern: Coupling @xeraa
Parse @xeraa
@xeraa
Pro: No change Con: RegEx, timestamp, multiline @xeraa
Send @xeraa
@xeraa
Pro: No files Con: Outages & coupling @xeraa
Structure @xeraa
@xeraa
Pro: Right format Con: JSON serialization overhead @xeraa
Containerize @xeraa
@xeraa
@xeraa
https://turnoff.us/geek/beforedevops-after-devops/
Where to put Filebeat? Sidecar @xeraa
Default JSON log filebeat.prospectors: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - add_docker_metadata: ~ @xeraa
Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa
Mount log path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa
Registry file filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa
..-__ ''-._ _.-.. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'_.-'| |-.._ / _.-' |-._ -._-./ .-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'.-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'_.-' _.-' -._-..-' _.-' -._ _.-'-..-'
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Configuration templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa
Drop asciiart lines
Pro: Hot Con: Complexity @xeraa
Orchestrate @xeraa
@xeraa
Where to put Filebeat? DaemonSet @xeraa
Metadata processors: - add_kubernetes_metadata: in_cluster: true @xeraa
Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa
Configuration templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa
Customize indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa
Pro: Hot Con: Complexity++ @xeraa
Conclusion @xeraa
Examples https://github.com/xeraa/java-logging @xeraa
Parse Send Structure Containerize Orchestrate @xeraa
Questions? Philipp Krenn @xeraa @xeraa
"Do you also feel need to centralize logs? Choose one of the approaches and patterns that suits you the best. I will talk about parsing, sending, structuring, containerizing and orchestrating. Each pattern has its own demo, so you can easily try out the different approaches in your environment," Philipp Krenn from Vienna, Austria introduces with his talk. He is part of the infrastructure team and a developer advocate at Elastic. Additionally, Philipp is organizing multiple meetups in Vienna.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub: Demo Code
How to log from a Java application to the Elastic Stack. This repository gives an overview of five different logging patterns:
- Parse: Take the log files of your applications and extract the relevant pieces of information.
- Send: Add a log appender to send out your events directly without persisting them to a log file.
- Structure: Write your events in a structured file, which you can then centralize.
- Containerize: Keep track of short lived containers and configure their logging correctly.
- Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.
