Aggregierte Logging-Patterns
A presentation at Continuous Lifecycle in in Mannheim, Germany by Philipp Krenn
Aggregierte Logging-Patterns Philipp Krenn @xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Developer @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Licensing Open Source Apache-2.0 Basic free Commercial @xeraa
Disclaimer I build highly monitored Hello World apps @xeraa
Example: Java SLF4J, Logback, MDC @xeraa
And Everywhere Else .NET: NLog PHP: Monolog JavaScript: Winston Python: structlog @xeraa
Anti-Pattern: print System.out.println("Oops"); @xeraa
Anti-Pattern: Coupling @xeraa
Parse @xeraa
@xeraa
[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}[%{WORD:thread}] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa
Elastic Common Schema https://github.com/elastic/ecs @xeraa
Helpers Dev Tools: Grok Debugger Machine Learning: Data Visualizer @xeraa
Log UI @xeraa
Pro: No change Con: RegEx, timestamp, multiline @xeraa
Send @xeraa
@xeraa
Pro: No files Con: Outages & coupling @xeraa
Structure @xeraa
@xeraa
Pro: Right format Con: JSON serialization overhead @xeraa
Containerize @xeraa
@xeraa
Where to put Filebeat? Sidecar @xeraa
Default JSON log filebeat.prospectors: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - add_docker_metadata: ~ - add_host_metadata: ~ @xeraa
Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa
Mount log path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa
Registry file filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa
..-__ ''-._ _.-.. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'_.-'| |-.._ / _.-' |-._ -._-./ .-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'.-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'_.-' _.-' -._-..-' _.-' -._ _.-'-..-'
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Configuration templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa
Drop asciiart lines
Infrastructure UI @xeraa
Pro: Hot Con: Complexity @xeraa
Orchestrate @xeraa
@xeraa
Where to put Filebeat? DaemonSet @xeraa
Metadata processors: - add_kubernetes_metadata: in_cluster: true @xeraa
Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa
Configuration templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa
Customize indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa
Pro: Hot Con: Complexity++ @xeraa
Moar @xeraa
Architecture @xeraa ! "
Frozen Indices https://github.com/elastic/elasticsearch/issues/34352 @xeraa
Centralized Logstash & Beats Management @xeraa
Conclusion @xeraa
Examples https://github.com/xeraa/java-logging @xeraa
Parse Send Structure Containerize Orchestrate @xeraa
Questions? Philipp Krenn @xeraa @xeraa
In verteilten Applikationen besteht immer der Bedarf, Logs zu zentralisieren – sobald man mehr als ein paar Container hat, reichen cat, tail oder less nicht mehr aus. Das übliche Problem ist aber, wie man möglichst effizient zu einer zentralisierten oder aggregierten Log-Lösung kommt.
Dieser Vortrag stellt fünf Ansätze und Patterns für zentrales Logging mit ihren Vor- und Nachteilen vor, sodass die Zuhörer den für sie passenden Ansatz wählen können.
Jedes der fünf Patterns hat sein eigenes Demo mit dem Open-Source Elastic Stack (früher ELK Stack), wobei die Muster generell bei jedem zentralisierten Logging-System anwendbar sind.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub Demo
How to log from a Java application to the Elastic Stack.
Buzz and feedback
Here’s what was said about this presentation on social media.
