Centralized Logging Patterns

A presentation at DEVOPS.BARCELONA in June 2019 in Barcelona, Spain by Philipp Krenn

Slide 1

Centralized Logging Patterns Philipp Krenn @xeraa @xeraa

Slide 2

@xeraa

Slide 3

@xeraa

Slide 4

@xeraa

Slide 5

@xeraa

Slide 6

@xeraa

Slide 7

@xeraa

Slide 8

@xeraa

Slide 9

@xeraa

Slide 10

Slide 11

Slide 12

Developer @xeraa

Slide 13

@xeraa

Slide 14

@xeraa

Slide 15

@xeraa

Slide 16

@xeraa

Slide 17

@xeraa

Slide 18

Slide 19

Disclaimer I build highly monitored Hello World apps @xeraa

Slide 20

Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout @xeraa

Slide 21

And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa

Slide 22

Anti-Pattern: print System.out.println(“Oops”); @xeraa

Slide 23

Anti-Pattern: Coupling @xeraa

Slide 24

Parse @xeraa

Slide 25

@xeraa

Slide 26

Bind Mount Logs java_app: volumes: - ‘./logs-docker/:/logs/’ … filebeat_for_logstash: volumes: - ‘./logs-docker/:/mnt/logs/:ro’ … @xeraa

Slide 27

Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log @xeraa

Slide 28

Metadata processors: - add_host_metadata: ~ @xeraa

Slide 29

Slide 30

Test Multiline Pattern https://www.elastic.co/guide/ en/beats/filebeat/current/ test_your_regexp_pattern_for multiline.html @xeraa

Slide 31

Grok https://github.com/logstashplugins/logstash-patternscore/blob/master/patterns/ grok-patterns @xeraa

Slide 32

Dev Tools Grok Debugger @xeraa

Slide 33

[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime… at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}] %{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE} %{GREEDYDATA:message}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa

Slide 34

Machine Learning Data Visualizer @xeraa

Slide 35

Logstash Key Value Filter for MDC kv { source => “labels” field_split => “,” trim_key => ” ” } @xeraa

Slide 36

Pro: No change Con: Regular expression, multiline, format changes @xeraa

Slide 37

Send @xeraa

Slide 38

@xeraa

Slide 39

logback.xml <appender name=”logstash” class=”net.logstash.logback.appender.LogstashAccessTcpSocketAppender”> <destination>logstash:4560</destination> <encoder class=”net.logstash.logback.encoder.LogstashEncoder”/> </appender> @xeraa

Slide 40

Pro: No files Con: Outages & coupling @xeraa

Slide 41

Structure @xeraa

Slide 42

@xeraa

Slide 43

Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true @xeraa

Slide 44

Stack(trace) Hash @xeraa

Slide 45

Bonues: Multi-Index output.elasticsearch: hosts: [“http://localhost:9200”] indices: - index: “warning-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “WARN” - index: “error-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “ERR” @xeraa

Slide 46

Pro: Right format Con: JSON serialization overhead @xeraa

Slide 47

Containerize @xeraa

Slide 48

@xeraa

Slide 49

Where to put Filebeat? Sidecar @xeraa

Slide 50

@xeraa

Slide 51

https://github.com/elastic/beats/tree/ master/deploy/docker @xeraa

Slide 52

Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ @xeraa

Slide 53

Metadata No Docker metadata with the other methods @xeraa

Slide 54

“docker”: { “container”: { “labels”: { “app”: “fizzbuzz”, “co_elastic_logs/multiline_match”: “after”, “com_docker_compose_config-hash”: “41520c6cf2b6a1f3dae4f16d0a6fd76760cdfc38fbfe43a3a3be2e09bdd1b8b5”, “environment”: “production”, “co_elastic_logs/multiline_pattern”: “^\[“, “co_elastic_logs/multiline_negate”: “true”, “com_docker_compose_oneoff”: “False”, “com_docker_compose_project”: “java-logging”, “com_docker_compose_service”: “java_app”, “com_docker_compose_container-number”: “1”, “com_docker_compose_version”: “1.23.2” } } } @xeraa

Slide 55

Missing the Last Line Waiting for the newline @xeraa

Slide 56

Hints labels: - “app=fizzbuzz” - “co.elastic.logs/multiline.pattern=^\[” - “co.elastic.logs/multiline.negate=true” - “co.elastic.logs/multiline.match=after” @xeraa

Slide 57

Registry File filebeat.registry.path: /usr/share/filebeat/data/registry @xeraa

Slide 58

Ingest Pipeline output.elasticsearch: hosts: [“http://elasticsearch:9200”] index: “docker” pipelines: - pipeline: “parse_java” when.contains: container.name: “java_app” @xeraa

Slide 59

Ingest Pipeline { “description” : “Parse Java log lines”, “processors”: [ { “grok”: { “field”: “message”, “patterns”: [ “^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\[%{WORD:log.method}\]%{SPACE}%{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\n+(?<stacktrace>(?:.|\r|\n)+))?” ], “ignore_failure”: true } } ] } @xeraa

Slide 60

Unknown Fields @xeraa

Slide 61

ASCII Art ..-__ ''-._ _.-.. ”-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|’_.-'| |-.._ / _.-' |-._ -._-./ .-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘.-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘_.-’ _.-’ -._-..-’ _.-’ -._ _.-'-..-’ Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757 http://redis.io @xeraa

Slide 62

Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa

Slide 63

Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true @xeraa

Slide 64

Pro: Hot Con: Complexity @xeraa

Slide 65

Orchestrate @xeraa

Slide 66

@xeraa

Slide 67

Where to put Filebeat? DaemonSet @xeraa

Slide 68

https://github.com/elastic/beats/tree/ master/deploy/kubernetes @xeraa

Slide 69

Metadata Either in cluster or outside processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa

Slide 70

{ “host”: “172.17.0.21”, “port”: 9090, “kubernetes”: { “container”: { “id”: “382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51”, “image”: “my-java:1.0.0”, “name”: “my-java” }, “labels”: { “app”: “java”, }, “namespace”: “default”, “node”: { “name”: “minikube” }, “pod”: { “name”: “java-2657348378-k1pnh” } }, } @xeraa

Slide 71

More Metadata Add: Cloud, local timezone, process Drop: Events, fields Rename: Fields Dissect, DNS reverse lookup @xeraa

Slide 72

Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - “${data.kubernetes.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa

Slide 73

Customize Indices output.elasticsearch: index: “%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}” @xeraa

Slide 74

Pro: Hot Con: Complexity++ @xeraa

Slide 75

Conclusion @xeraa

Slide 76

Examples https://github.com/xeraa/java-logging @xeraa

Slide 77

Parse Send Structure Containerize Orchestrate @xeraa

Slide 78

Questions? Philipp Krenn @xeraa @xeraa