Aggregierte Logging Patterns
A presentation at devday.19 in in Dresden, Germany by Philipp Krenn
Aggregierte Logging Patterns Philipp Krenn @xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Developer @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Disclaimer I build highly monitored Hello World apps @xeraa
Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout @xeraa
And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa
Anti-Pattern: print System.out.println(“Oops”); @xeraa
Anti-Pattern: Coupling @xeraa
Parse @xeraa
@xeraa
Bind Mount Logs java_app: volumes: - ‘./logs-docker/:/logs/’ … filebeat_for_logstash: volumes: - ‘./logs-docker/:/mnt/logs/:ro’ … @xeraa
Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log @xeraa
Metadata processors: - add_host_metadata: ~ @xeraa
Test Multiline Pattern https://www.elastic.co/guide/en/beats/filebeat/current/ _test_your_regexp_pattern_for_multiline.html @xeraa
Grok https://github.com/logstash-plugins/logstash-patterns-core/blob/ master/patterns/grok-patterns @xeraa
Dev Tools Grok Debugger @xeraa
[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime… at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}] %{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE} %{GREEDYDATA:message}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa
Elastic Common Schema https://github.com/elastic/ecs @xeraa
Machine Learning Data Visualizer @xeraa
Logstash Key Value Filter for MDC kv { source => “labels” field_split => “,” trim_key => ” ” } @xeraa
Monitoring: Logstash Pipeline Plus other components @xeraa
Pro: No change Con: Regular expression, multiline, format changes @xeraa
Send @xeraa
@xeraa
logback.xml <appender name=”logstash” class=”net.logstash.logback.appender.LogstashAccessTcpSocketAppender”> <destination>logstash:4560</destination> <encoder class=”net.logstash.logback.encoder.LogstashEncoder”/> </appender> @xeraa
Pro: No files Con: Outages & coupling @xeraa
Structure @xeraa
@xeraa
Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true @xeraa
Stack(trace) Hash @xeraa
Bonues: Multi-Index output.elasticsearch: hosts: [“http://localhost:9200”] indices: - index: “warning-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “WARN” - index: “error-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “ERR” @xeraa
Pro: Right format Con: JSON serialization overhead @xeraa
Containerize @xeraa
@xeraa
Where to put Filebeat? Sidecar @xeraa
@xeraa
https://github.com/elastic/beats/tree/ master/deploy/docker @xeraa
Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ @xeraa
Metadata No Docker metadata with the other methods @xeraa
“docker”: { “container”: { “labels”: { “app”: “fizzbuzz”, “co_elastic_logs/multiline_match”: “after”, “com_docker_compose_config-hash”: “41520c6cf2b6a1f3dae4f16d0a6fd76760cdfc38fbfe43a3a3be2e09bdd1b8b5”, “environment”: “production”, “co_elastic_logs/multiline_pattern”: “^\[“, “co_elastic_logs/multiline_negate”: “true”, “com_docker_compose_oneoff”: “False”, “com_docker_compose_project”: “java-logging”, “com_docker_compose_service”: “java_app”, “com_docker_compose_container-number”: “1”, “com_docker_compose_version”: “1.23.2” } } } @xeraa
Missing the Last Line Waiting for the newline @xeraa
Hints labels: - “app=fizzbuzz” - “co.elastic.logs/multiline.pattern=^\[” - “co.elastic.logs/multiline.negate=true” - “co.elastic.logs/multiline.match=after” @xeraa
Registry File filebeat.registry.path: /usr/share/filebeat/data/registry @xeraa
Ingest Pipeline output.elasticsearch: hosts: [“http://elasticsearch:9200”] index: “docker” pipelines: - pipeline: “parse_java” when.contains: container.name: “java_app” @xeraa
Ingest Pipeline { “description” : “Parse Java log lines”, “processors”: [ { “grok”: { “field”: “message”, “patterns”: [ “^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\[%{WORD:log.method}\]%{SPACE}%{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\n+(?<stacktrace>(?:.|\r|\n)+))?” ], “ignore_failure”: true } } ] } @xeraa
Unknown Fields @xeraa
ASCII Art ..-__ ''-._ _.-.. ”-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|’_.-'| |-.._ / _.-' |-._ -._-./ .-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘.-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘_.-’ _.-’ -._-..-’ _.-’ -._ _.-'-..-’
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa
Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true @xeraa
Pro: Hot Con: Complexity @xeraa
Orchestrate @xeraa
@xeraa
Where to put Filebeat? DaemonSet @xeraa
https://github.com/elastic/beats/tree/ master/deploy/kubernetes @xeraa
Metadata Either in cluster or outside processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa
{ “host”: “172.17.0.21”, “port”: 9090, “kubernetes”: { “container”: { “id”: “382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51”, “image”: “my-java:1.0.0”, “name”: “my-java” }, “labels”: { “app”: “java”, }, “namespace”: “default”, “node”: { “name”: “minikube” }, “pod”: { “name”: “java-2657348378-k1pnh” } }, } @xeraa
More Metadata Add: Cloud, local timezone, process Drop: Events, fields Rename: Fields Dissect, DNS reverse lookup @xeraa
Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - “${data.kubernetes.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa
Customize Indices output.elasticsearch: index: “%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}” @xeraa
Pro: Hot Con: Complexity++ @xeraa
Moar @xeraa
Index Patterns Time based (default: daily) Versioned @xeraa
Sizing Daily volume * Retention * Replication Number of shards @xeraa
Index Lifecycle Management ! ” @xeraa
Order https://github.com/elastic/elasticsearch/blob/7.1/x-pack/plugin/core/src/main/java/org/elasticsearch/ xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME ); @xeraa
Frozen Indices https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ frozen-indices.html @xeraa
Ratio Heap : Storage Index > Frozen Index > Closed Index Read-only @xeraa
Throttled Thread Pool 1 parallel search / node 100 in queue @xeraa
Conclusion @xeraa
Examples https://github.com/xeraa/java-logging @xeraa
Parse Send Structure Containerize Orchestrate @xeraa
Questions? Philipp Krenn @xeraa @xeraa
In verteilten Applikationen besteht immer der Bedarf Logs zu zentralisieren - sobald man mehr als ein paar Server oder Container hat, reichen SSH und cat, tail oder less nicht mehr aus. Das übliche Problem ist aber, wie man möglichst effizient zu einer zentralisierten oder aggregierten Log-Lösung kommt.
Dieser Vortrag stellt mehrere Ansätze und Patterns mit ihren Vor- und Nachteilen vor, so dass die Zuhörer den auswählen können, der am besten zu Ihrer Organisation passt:
- Parsen: Bestehende Logdaten können weiterverwendet werden und die relevanten Informationen werden per Regular Expression extrahiert.
- Senden: Mit einem geeigneten Log-Appender werden die Ereignisse direkt gesendet, ohne dass sie in einer Logdatei gespeichert werden müssen.
- Strukturieren: Ereignisse in einem strukturierten Format abspeichern, das dann direkt zentralisiert werden kann.
- Containerisieren: Das Logging mit Containern erfordert zusätzliche Automatismen, um effizient mit Logs arbeiten zu können.
- Orchestrieren: Auch wenn Dienste nur kurzfristig laufen und dynamisch verteilt werden, gibt mit Kubernetes Möglichkeiten den Überblick zu bewahren.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub Workshop Code
How to log from a Java application to the Elastic Stack.
Buzz and feedback
Here’s what was said about this presentation on social media.
