Centralized Logging Patterns
A presentation at WeAreDevelopers World Congress in in Vienna, Austria by Philipp Krenn
Centralized Logging Patterns Philipp Krenn 44444 @xeraa
Developer Advocate
Apache License 2.0
Disclaimer I build highly monitored Hello World apps
Example: Java
.NET: NLog PHP: Monolog JavaScript: Winston ...
Parse ! 44
Pro: No change Con: RegEx
Send ✉ 44
Pro: No more files Con: Outages & coupling
Structure ! 44
Pro: Right format Con: Special log appender
Containerize ! 44
https://turnoff.us/geek/before- devops-after-devops/
Where to put Filebeat? Sidecar
Default JSON log filebeat.prospectors:
- type:
log paths:
"/var/lib/docker/containers//.log"
json.message_key:
log
json.keys_under_root:
true processors:
- add_docker_metadata:
~
Metadata {
"host" : "10.4.15.9" ,
"port" : 6379 ,
"docker" : {
"container" : {
"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,
"name" : "my-java" ,
"image" : "my-java:1.0.0" ,
"labels" : {
"app" : "java" } } } }
Mount log path my-java: container_name:
my-java hostname:
my-java build:
${PWD}/config/my-java networks:
['stack'] command:
java
-jar
my-java.jar volumes: -
./logs/my-java/:/opt/my-java/logs/ filebeat: container_name:
filebeat hostname:
filebeat image:
"docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: -
./logs/my-java/:/var/log/my-java/ -
./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command:
filebeat
-e networks:
['stack']
Configuration templates filebeat.autodiscover: providers: - type:
docker templates: - condition: equals:
docker.container.image:
redis config: - type:
docker
containers.ids: -
"${data.docker.container.id}" exclude_lines:
["^\s+[\-`('.|_]"]
Drop asciiart lines
Pro: Hot ! Con: Complexity
Orchestrate ! 44
Where to put Filebeat? DaemonSet
Metadata processors:
- add_kubernetes_metadata: in_cluster:
true
Metadata {
"host" : "172.17.0.21" ,
"port" : 9090 ,
"kubernetes" : {
"container" : {
"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,
"image" : "my-java:1.0.0" ,
"name" : "my-java" },
"labels" : {
"app" : "java" , },
"namespace" : "default" ,
"node" : {
"name" : "minikube" },
"pod" : {
"name" : "java-2657348378-k1pnh" } }, }
Configuration templates filebeat.autodiscover: providers: - type:
kubernetes templates: - condition: equals:
kubernetes.namespace:
kube-system config: - type:
docker
containers.ids: -
"${data.kubernetes.container.id}" exclude_lines:
["^\s+[\-`('.|_]"]
Drop asciiart lines
Customize indices output.elasticsearch: index:
"%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
Pro: Hot ! ! ! Con: Complexity++
Conclusion 44
Examples https://github.com/xeraa/java-logging
Parse ! Send ✉ Structure ! Containerize ! Orchestrate !
Questions? 44 Philipp Krenn 44444 @xeraa
Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.
This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
- Parse: Take the log files of your applications and extract the relevant pieces of information.
- Send: Add a log appender to send out your events directly without persisting them to a log file.
- Structure: Write your events in a structured file, which you can then centralize.
- Containerize: Keep track of short lived containers and configure their logging correctly.
- Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.
Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
GitHub: Demo Code
How to log from a Java application to the Elastic Stack
Buzz and feedback
Here’s what was said about this presentation on social media.
