Centralized Logging Patterns
A presentation at DevOpsDays Amsterdam 2019 in in Amsterdam, Netherlands by Philipp Krenn
Centralized Logging Patterns Philipp Krenn @xeraa @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Developer @xeraa
@xeraa
@xeraa
@xeraa
@xeraa
@xeraa
Disclaimer I build highly monitored Hello World apps @xeraa
Setup (optional) $ git clone https://github.com/xeraa/java-logging.git $ docker load -i <image>.tar (optional) $ docker-compose up @xeraa
Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout @xeraa
And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog @xeraa
Anti-Pattern: print System.out.println(“Oops”); @xeraa
Anti-Pattern: Coupling @xeraa
Parse @xeraa
@xeraa
Bind Mount Logs java_app: volumes: - ‘./logs-docker/:/logs/’ … filebeat_for_logstash: volumes: - ‘./logs-docker/:/mnt/logs/:ro’ … @xeraa
Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log @xeraa
Metadata processors: - add_host_metadata: ~ @xeraa
Test Multiline Pattern https://www.elastic.co/guide/en/beats/filebeat/current/ _test_your_regexp_pattern_for_multiline.html @xeraa
Grok https://github.com/logstash-plugins/logstash-patterns-core/blob/ master/patterns/grok-patterns @xeraa
Dev Tools Grok Debugger @xeraa
[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime… at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:@timestamp}]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}[%{WORD:log.method}] %{SPACE}-%{SPACE}%{GREEDYDATA:log.labels}%{SPACE}-%{SPACE} %{GREEDYDATA:message}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa
Elastic Common Schema https://github.com/elastic/ecs @xeraa
Machine Learning Data Visualizer @xeraa
Logstash Key Value Filter for MDC kv { source => “labels” field_split => “,” trim_key => ” ” } @xeraa
Monitoring: Logstash Pipeline Plus other components @xeraa
Pro: No change Con: Regular expression, multiline, format changes @xeraa
Send @xeraa
@xeraa
logback.xml <appender name=”logstash” class=”net.logstash.logback.appender.LogstashAccessTcpSocketAppender”> <destination>logstash:4560</destination> <encoder class=”net.logstash.logback.encoder.LogstashEncoder”/> </appender> @xeraa
Pro: No files Con: Outages & coupling @xeraa
Structure @xeraa
@xeraa
Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true @xeraa
Stack(trace) Hash @xeraa
Bonues: Multi-Index output.elasticsearch: hosts: [“http://localhost:9200”] indices: - index: “warning-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “WARN” - index: “error-%{[agent.version]}-%{+yyyy.MM.dd}” when.contains: message: “ERR” @xeraa
Pro: Right format Con: JSON serialization overhead @xeraa
Containerize @xeraa
@xeraa
Where to put Filebeat? Sidecar @xeraa
@xeraa
https://github.com/elastic/beats/tree/ master/deploy/docker @xeraa
Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ @xeraa
Metadata No Docker metadata with the other methods @xeraa
“docker”: { “container”: { “labels”: { “app”: “fizzbuzz”, “co_elastic_logs/multiline_match”: “after”, “com_docker_compose_config-hash”: “41520c6cf2b6a1f3dae4f16d0a6fd76760cdfc38fbfe43a3a3be2e09bdd1b8b5”, “environment”: “production”, “co_elastic_logs/multiline_pattern”: “^\[“, “co_elastic_logs/multiline_negate”: “true”, “com_docker_compose_oneoff”: “False”, “com_docker_compose_project”: “java-logging”, “com_docker_compose_service”: “java_app”, “com_docker_compose_container-number”: “1”, “com_docker_compose_version”: “1.23.2” } } } @xeraa
Missing the Last Line Waiting for the newline @xeraa
Hints labels: - “app=fizzbuzz” - “co.elastic.logs/multiline.pattern=^\[” - “co.elastic.logs/multiline.negate=true” - “co.elastic.logs/multiline.match=after” @xeraa
Registry File filebeat.registry.path: /usr/share/filebeat/data/registry @xeraa
Ingest Pipeline output.elasticsearch: hosts: [“http://elasticsearch:9200”] index: “docker” pipelines: - pipeline: “parse_java” when.contains: container.name: “java_app” @xeraa
Ingest Pipeline { “description” : “Parse Java log lines”, “processors”: [ { “grok”: { “field”: “message”, “patterns”: [ “^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\[%{WORD:log.method}\]%{SPACE}%{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\n+(?<stacktrace>(?:.|\r|\n)+))?” ], “ignore_failure”: true } } ] } @xeraa
Unknown Fields @xeraa
ASCII Art ..-__ ''-._ _.-.. ”-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|’_.-'| |-.._ / _.-' |-._ -._-./ .-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘.-’ .-’ |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-‘_.-’ _.-’ -._-..-’ _.-’ -._ _.-'-..-’
Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757
http://redis.io
@xeraa
Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa
Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true @xeraa
Pro: Hot Con: Complexity @xeraa
Orchestrate @xeraa
@xeraa
Where to put Filebeat? DaemonSet @xeraa
https://github.com/elastic/beats/tree/ master/deploy/kubernetes @xeraa
Metadata Either in cluster or outside processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa
{ “host”: “172.17.0.21”, “port”: 9090, “kubernetes”: { “container”: { “id”: “382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51”, “image”: “my-java:1.0.0”, “name”: “my-java” }, “labels”: { “app”: “java”, }, “namespace”: “default”, “node”: { “name”: “minikube” }, “pod”: { “name”: “java-2657348378-k1pnh” } }, } @xeraa
More Metadata Add: Cloud, local timezone, process Drop: Events, fields Rename: Fields Dissect, DNS reverse lookup @xeraa
Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - “${data.kubernetes.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] @xeraa
Customize Indices output.elasticsearch: index: “%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}” @xeraa
Pro: Hot Con: Complexity++ @xeraa
Moar @xeraa
Index Patterns Time based (default: daily) Versioned @xeraa
Sizing Daily volume * Retention * Replication Number of shards @xeraa
Rollover New index on condition Size Time Number of documents @xeraa
Index Lifecycle Management ! ” @xeraa
Order https://github.com/elastic/elasticsearch/blob/7.1/x-pack/plugin/core/src/main/java/org/elasticsearch/ xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME ); @xeraa
Frozen Indices https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ frozen-indices.html @xeraa
Ratio Heap : Storage Index > Frozen Index > Closed Index Read-only @xeraa
Throttled Thread Pool 1 parallel search / node 100 in queue @xeraa
Conclusion @xeraa
Examples https://github.com/xeraa/java-logging @xeraa
Helm & Operator https://github.com/elastic/helm-charts https://github.com/elastic/cloud-on-k8s @xeraa
PS: Free Security https://www.elastic.co/blog/security-for-elasticsearch-is-now-free TLS, file & native realm, RBAC @xeraa
Parse Send Structure Containerize Orchestrate @xeraa
And More Metrics APM / Tracing Health @xeraa
Questions? Philipp Krenn @xeraa @xeraa
Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.
This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
- Parse: Take the log files of your applications and extract the relevant pieces of information.
- Send: Add a log appender to send out your events directly without persisting them to a log file.
- Structure: Write your events in a structured file, which you can then centralize.
- Containerize: Keep track of short lived containers and configure their logging correctly.
- Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.
Each pattern has its own lab demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in the workshop. Though the general patterns are applicable with any centralized logging system.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub Demo Code
How to log from a Java application to the Elastic Stack.
Buzz and feedback
Here’s what was said about this presentation on social media.
