Aggregierte Logging Patterns

A presentation at Magdeburger Developer Days in May 2019 in Magdeburg, Germany by Philipp Krenn

Disclaimer I build highly monitored Hello World apps @xeraa

Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout @xeraa

And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa

Anti-Pattern: print System.out.println(“Oops”); @xeraa

Bind Mount Logs java_app: volumes: - ‘./logs-docker/:/logs/’ … filebeat_for_logstash: volumes: - ‘./logs-docker/:/mnt/logs/:ro’ … @xeraa

Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log @xeraa

Metadata processors: - add_host_metadata: ~ @xeraa

Test Multiline Pattern https://www.elastic.co/guide/ en/beats/filebeat/current/ *test_your_regexp_pattern_for* multiline.html @xeraa

Grok https://github.com/logstashplugins/logstash-patternscore/blob/master/patterns/ grok-patterns @xeraa

[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime… at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}] %{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE} %{GREEDYDATA:message}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa

Logstash Key Value Filter for MDC kv { source => “labels” field_split => “,” trim_key => ” ” } @xeraa

Pro: No change Con: Regular expression, multiline, format changes @xeraa

logback.xml <appender name=”logstash” class=”net.logstash.logback.appender.LogstashAccessTcpSocketAppender”> <destination>logstash:4560</destination> <encoder class=”net.logstash.logback.encoder.LogstashEncoder”/> </appender> @xeraa

Pro: No files Con: Outages & coupling @xeraa

Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true @xeraa

Bonues: Multi-Index output.elasticsearch: hosts: [“http://localhost:9200”] indices: - index: “warning-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “WARN” - index: “error-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “ERR” @xeraa

Pro: Right format Con: JSON serialization overhead @xeraa

https://github.com/elastic/beats/tree/ master/deploy/docker @xeraa

Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ @xeraa

Metadata No Docker metadata with the other methods @xeraa

“docker”: { “container”: { “labels”: { “app”: “fizzbuzz”, “co_elastic_logs/multiline_match”: “after”, “com_docker_compose_config-hash”: “41520c6cf2b6a1f3dae4f16d0a6fd76760cdfc38fbfe43a3a3be2e09bdd1b8b5”, “environment”: “production”, “co_elastic_logs/multiline_pattern”: “^\[“, “co_elastic_logs/multiline_negate”: “true”, “com_docker_compose_oneoff”: “False”, “com_docker_compose_project”: “java-logging”, “com_docker_compose_service”: “java_app”, “com_docker_compose_container-number”: “1”, “com_docker_compose_version”: “1.23.2” } } } @xeraa

Missing the Last Line Waiting for the newline @xeraa

Hints labels: - “app=fizzbuzz” - “co.elastic.logs/multiline.pattern=^\[” - “co.elastic.logs/multiline.negate=true” - “co.elastic.logs/multiline.match=after” @xeraa

Registry File filebeat.registry.path: /usr/share/filebeat/data/registry @xeraa

Ingest Pipeline output.elasticsearch: hosts: [“http://elasticsearch:9200”] index: “docker” pipelines: - pipeline: “parse_java” when.contains: container.name: “java_app” @xeraa

Ingest Pipeline { “description” : “Parse Java log lines”, “processors”: [ { “grok”: { “field”: “message”, “patterns”: [ “^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\[%{WORD:log.method}\]%{SPACE}%{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\n+(?<stacktrace>(?:.|\r|\n)+))?” ], “ignore_failure”: true } } ] } @xeraa

ASCII Art ..-`__ ''-._ _.-.`. ”-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|’`_.-'| |`-.`._ / _.-' |`-._ `-._`-./ *.-’ *.-’ |`-._`-.*`-.__.-' _.-'_.-'| |`-.*`-._ _.-'_.-' |`-._ `-._`-..-‘*.-’ *.-’ |`-._`-.*`-.__.-' _.-'_.-'| |`-.*`-._ _.-'_.-' |`-._ `-._`-..-‘_.-’ _.-’ `-._`-..-’ _.-’ `-._ _.-'`-..-’ Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757 http://redis.io @xeraa

Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa

Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true @xeraa

https://github.com/elastic/beats/tree/ master/deploy/kubernetes @xeraa

Metadata Either in cluster or outside processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa

{ “host”: “172.17.0.21”, “port”: 9090, “kubernetes”: { “container”: { “id”: “382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51”, “image”: “my-java:1.0.0”, “name”: “my-java” }, “labels”: { “app”: “java”, }, “namespace”: “default”, “node”: { “name”: “minikube” }, “pod”: { “name”: “java-2657348378-k1pnh” } }, } @xeraa

More Metadata Add: Cloud, local timezone, process Drop: Events, fields Rename: Fields Dissect, DNS reverse lookup @xeraa

Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - “${data.kubernetes.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa

Customize Indices output.elasticsearch: index: “%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}” @xeraa

Examples https://github.com/xeraa/java-logging @xeraa

Parse Send Structure Containerize Orchestrate @xeraa

Philipp Krenn
@xeraa

1 / 78

In verteilten Applikationen besteht immer der Bedarf Logs zu zentralisieren - sobald man mehr als ein paar Server oder Container hat, reichen SSH und cat, tail oder less nicht mehr aus. Das übliche Problem ist aber, wie man möglichst effizient zu einer zentralisierten oder aggregierten Log-Lösung kommt. Dieser Vortrag stellt mehrere Ansätze und Patterns mit ihren Vor- und Nachteilen vor, so dass die Zuhörer den auswählen können, der am besten zu Ihrer Organisation passt: parsen, senden strukturieren, containerisieren, orchestrieren Jedes Muster hat sein eigenes Demo mit dem Open Source Elastic Stack (früher ELK Stack), so dass die Zuhörer die unterschiedlichen Ansätze in ihrer Umgebung einfach ausprobieren können. Wobei die Muster generell bei jedem zentralisierten Logging-System anwendbar sind.

Resources

The following resources were mentioned during the presentation or are useful additional information.

GitHub Demo Code

How to log from a Java application to the Elastic Stack.

Buzz and feedback

Here’s what was said about this presentation on social media.

Tipps und Tricks zum Thema Logging gibt es bei den #mddevdays von @xeraa. pic.twitter.com/BZN0YUBoBG
— Christian Nicklisch (@ChrisNicklisch) May 22, 2019

Aggregierte Logging Patterns

Link for this presentation:

HTML code for embedding:

Share on social media:

Resources

GitHub Demo Code

Buzz and feedback