Centralized Logging Patterns Philipp Krenn @xeraa @xeraa
Slide 1
Slide 2
@xeraa
Slide 3
@xeraa
Slide 4
@xeraa
Slide 5
@xeraa
Slide 6
@xeraa
Slide 7
@xeraa
Slide 8
@xeraa
Slide 9
@xeraa
Slide 10
Slide 11
Developer @xeraa
Slide 12
@xeraa
Slide 13
@xeraa
Slide 14
@xeraa
Slide 15
@xeraa
Slide 16
@xeraa
Slide 17
Slide 18
Apache License 2.0 @xeraa
Slide 19
Disclaimer I build highly monitored Hello World apps @xeraa
Slide 20
Example: Java SLF4J, Logback, MDC @xeraa
Slide 21
.NET: NLog PHP: Monolog JavaScript: Winston ... @xeraa
Slide 22
Anti-Pattern: print System.out.println("Oops"); @xeraa
Slide 23
Anti-Pattern: Coupling @xeraa
Slide 24
Parse @xeraa
Slide 25
@xeraa
Slide 26
[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] - user_experience= , session=46, loop=15 - Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}[%{WORD:thread}] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa
Slide 27
Pro: No change Con: RegEx, timestamp, multiline @xeraa
Slide 28
Send @xeraa
Slide 29
@xeraa
Slide 30
Pro: No files Con: Outages & coupling @xeraa
Slide 31
Structure @xeraa
Slide 32
@xeraa
Slide 33
Pro: Right format Con: JSON serialization overhead @xeraa
Slide 34
Containerize @xeraa
Slide 35
@xeraa
Slide 36
Slide 37
https://turnoff.us/geek/beforedevops-after-devops/
Slide 38
Where to put Filebeat? Sidecar @xeraa
Slide 39
Default JSON log filebeat.prospectors: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - add_docker_metadata: ~ @xeraa
Slide 40
Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa
Slide 41
Mount log path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa
Slide 42
Registry file filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa
Slide 43
..-__ ''-._ _.-.. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'_.-'| |-.._ / _.-' |-._ -._-./ .-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'.-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'_.-' _.-' -._-..-' _.-' -._ _.-'-..-'
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Slide 44
Configuration templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa
Drop asciiart lines
Slide 45
Pro: Hot Con: Complexity @xeraa
Slide 46
Orchestrate @xeraa
Slide 47
@xeraa
Slide 48
Where to put Filebeat? DaemonSet @xeraa
Slide 49
Metadata processors: - add_kubernetes_metadata: in_cluster: true @xeraa
Slide 50
Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa
Slide 51
Configuration templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa
Slide 52
Customize indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa
Slide 53
Pro: Hot Con: Complexity++ @xeraa
Slide 54
Moar @xeraa
Slide 55
Architecture @xeraa ! "
Slide 56
Slide 57
Slide 58
Slide 59
Frozen Indices https://github.com/elastic/elasticsearch/issues/34352 @xeraa
Slide 60
InfraUI & LogUI Centralized Beats Management @xeraa
Slide 61
Conclusion @xeraa
Slide 62
Examples https://github.com/xeraa/java-logging @xeraa
Slide 63
Parse Send Structure Containerize Orchestrate @xeraa
Slide 64
Questions? Philipp Krenn @xeraa @xeraa
