Centralized Logging Patterns
A presentation at Devoxx Ukraine 2018 in in Kyiv, Ukraine, 02000 by Philipp Krenn
Centralized Logging-Patterns Philipp Krenn @xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Developer @xeraa
Questions: https://sli.do/xeraa Answers: https://twitter.com/xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Disclaimer I build highly monitored Hello World apps @xeraa
Example: Java SLF4J, Logback, MDC @xeraa
And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa
Anti-Pattern: print System.out.println("Oops"); @xeraa
Anti-Pattern: Coupling @xeraa
Parse @xeraa
@xeraa
Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log #exclude_lines: ['^TRACE'] @xeraa
[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}[%{WORD:thread}] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa
Elastic Common Schema https://github.com/ elastic/ecs @xeraa
Grok https://github.com/logstash-plugins/logstash-patterns-core/blob/ master/patterns/grok-patterns @xeraa
Dev Tools Grok Debugger @xeraa
Machine Learning Data Visualizer @xeraa
Visualize For example session @xeraa
Log UI @xeraa
Monitoring: Logstash Pipeline Plus other components @xeraa
Pro: No change Con: RegEx, timestamp, multiline @xeraa
Send @xeraa
@xeraa
Pro: No files Con: Outages & coupling @xeraa
Structure @xeraa
@xeraa
Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json fields_under_root: true json: message_key: message keys_under_root: true processors: - add_host_metadata: ~ @xeraa
Pro: Right format Con: JSON serialization overhead @xeraa
Containerize @xeraa
@xeraa
Where to put Filebeat? Sidecar @xeraa
@xeraa
Default JSON Log filebeat.input: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - decode_json_fields: fields: ["message"] target: "" overwrite_keys: true - add_docker_metadata: ~ - add_host_metadata: ~ @xeraa
Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa
@xeraa
Mount Log Path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa
Registry File filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa
..-__ ''-._ _.-.. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'_.-'| |-.._ / _.-' |-._ -._-./ .-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'.-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'_.-' _.-' -._-..-' _.-' -._ _.-'-..-'
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa
Drop asciiart lines
Infrastructure UI @xeraa
Pro: Hot Con: Complexity @xeraa
Orchestrate @xeraa
@xeraa
Where to put Filebeat? DaemonSet @xeraa
Metadata Either in cluster or not processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa
Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa
Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa
Customize Indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa
Pro: Hot Con: Complexity++ @xeraa
Moar @xeraa
Architecture @xeraa ! "
Frozen Indices https://github.com/elastic/elasticsearch/issues/34352 @xeraa
Centralized Logstash & Beats Management @xeraa
Conclusion @xeraa
Examples https://github.com/xeraa/java-logging @xeraa
Parse Send Structure Containerize Orchestrate @xeraa
Questions? Philipp Krenn @xeraa @xeraa
Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.
This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
- Parse: Take the log files of your applications and extract the relevant pieces of information.
- Send: Add a log appender to send out your events directly without persisting them to a log file.
- Structure: Write your events in a structured file, which you can then centralize. * Containerize: Keep track of short lived containers and configure their logging correctly.
- Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.
Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.
Video
Code
The following code examples from the presentation can be tried out live.
