Centralized Logging Patterns

A presentation at Voxxed Days Cluj in November 2018 in Cluj-Napoca, Romania by Philipp Krenn

Licensing Open Source Apache-2.0 Basic free Commercial @xeraa

Disclaimer I build highly monitored Hello World apps @xeraa

Example: Java SLF4J, Logback, MDC @xeraa

And Everywhere Else .NET: NLog PHP: Monolog JavaScript: Winston Python: structlog @xeraa

Anti-Pattern: print System.out.println("Oops"); @xeraa

Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log #exclude_lines: ['^TRACE'] @xeraa

[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}[%{WORD:thread}] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa

Elastic Common Schema https://github.com/ elastic/ecs @xeraa

Monitoring: Logstash Pipeline Plus other components @xeraa

Pro: No change Con: RegEx, timestamp, multiline @xeraa

Pro: No files Con: Outages & coupling @xeraa

Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json fields_under_root: true json: message_key: message keys_under_root: true processors: - add_host_metadata: ~ @xeraa

Pro: Right format Con: JSON serialization overhead @xeraa

Default JSON Log filebeat.input: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - add_docker_metadata: ~ - add_host_metadata: ~ @xeraa

Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa

Logging as JSON - decode_json_fields: fields: ["message"] target: "" overwrite_keys: true @xeraa

Mount Log Path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa

Registry File filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa

..-`__ ''-._ _.-.`. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'`_.-'| |`-.`._ / _.-' |`-._ `-._`-./ *.-' *.-' |`-._`-.*`-.__.-' _.-'_.-'| |`-.*`-._ _.-'_.-' |`-._ `-._`-..-'*.-' *.-' |`-._`-.*`-.__.-' _.-'_.-'| |`-.*`-._ _.-'_.-' |`-._ `-._`-..-'_.-' _.-' `-._`-..-' _.-' `-._ _.-'`-..-' Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757 http://redis.io @xeraa

Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa
Drop asciiart lines

Metadata Either in cluster or not processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa

Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa

Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa

Customize Indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa

Frozen Indices https://github.com/elastic/elasticsearch/issues/34352 @xeraa

Centralized Logstash & Beats Management @xeraa

Examples https://github.com/xeraa/java-logging @xeraa

Parse Send Structure Containerize Orchestrate @xeraa

Philipp Krenn
@xeraa

1 / 75

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:

Parse: Take the log files of your applications and extract the relevant pieces of information.
Send: Add a log appender to send out your events directly without persisting them to a log file.
Structure: Write your events in a structured file, which you can then centralize.
Containerize: Keep track of short lived containers and configure their logging correctly.
Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Video

Resources

The following resources were mentioned during the presentation or are useful additional information.

GitHub Demo Code

How to log from a Java application to the Elastic Stack.

Centralized Logging Patterns

Drop asciiart lines

Link for this presentation:

HTML code for embedding:

Share on social media:

Video

Resources

GitHub Demo Code