Centralized Logging Patterns Philipp Krenn 44444 @xeraa
Slide 1
Slide 2
Slide 3
Slide 4
Slide 5
Slide 6
Slide 7
Slide 8
Slide 9
Slide 10
Developer Advocate
Slide 11
Slide 12
Slide 13
Slide 14
Slide 15
Slide 16
Slide 17
Slide 18
Slide 19
Slide 20
Slide 21
Apache License 2.0
Slide 22
Disclaimer I build highly monitored Hello World apps
Slide 23
Example: Java
Slide 24
.NET: NLog PHP: Monolog JavaScript: Winston ...
Slide 25
Parse ! 44
Slide 26
Slide 27
Pro: No change Con: RegEx
Slide 28
Send ✉ 44
Slide 29
Slide 30
Pro: No more files Con: Outages & coupling
Slide 31
Structure ! 44
Slide 32
Slide 33
Pro: Right format Con: Special log appender
Slide 34
Containerize ! 44
Slide 35
https://turnoff.us/geek/before- devops-after-devops/
Slide 36
Where to put Filebeat? Sidecar
Slide 37
Default JSON log filebeat.prospectors:
- type:
log paths:
"/var/lib/docker/containers//.log"
json.message_key:
log
json.keys_under_root:
true processors:
- add_docker_metadata:
~
Slide 38
Metadata {
"host" : "10.4.15.9" ,
"port" : 6379 ,
"docker" : {
"container" : {
"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,
"name" : "my-java" ,
"image" : "my-java:1.0.0" ,
"labels" : {
"app" : "java" } } } }
Slide 39
Mount log path my-java: container_name:
my-java hostname:
my-java build:
${PWD}/config/my-java networks:
['stack'] command:
java
-jar
my-java.jar volumes: -
./logs/my-java/:/opt/my-java/logs/ filebeat: container_name:
filebeat hostname:
filebeat image:
"docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: -
./logs/my-java/:/var/log/my-java/ -
./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command:
filebeat
-e networks:
['stack']
Slide 40
Configuration templates filebeat.autodiscover: providers: - type:
docker templates: - condition: equals:
docker.container.image:
redis config: - type:
docker
containers.ids: -
"${data.docker.container.id}" exclude_lines:
["^\s+[\-`('.|_]"]
Drop asciiart lines
Slide 41
Pro: Hot ! Con: Complexity
Slide 42
Orchestrate ! 44
Slide 43
Where to put Filebeat? DaemonSet
Slide 44
Metadata processors:
- add_kubernetes_metadata: in_cluster:
true
Slide 45
Metadata {
"host" : "172.17.0.21" ,
"port" : 9090 ,
"kubernetes" : {
"container" : {
"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,
"image" : "my-java:1.0.0" ,
"name" : "my-java" },
"labels" : {
"app" : "java" , },
"namespace" : "default" ,
"node" : {
"name" : "minikube" },
"pod" : {
"name" : "java-2657348378-k1pnh" } }, }
Slide 46
Configuration templates filebeat.autodiscover: providers: - type:
kubernetes templates: - condition: equals:
kubernetes.namespace:
kube-system config: - type:
docker
containers.ids: -
"${data.kubernetes.container.id}" exclude_lines:
["^\s+[\-`('.|_]"]
Drop asciiart lines
Slide 47
Customize indices output.elasticsearch: index:
"%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
Slide 48
Pro: Hot ! ! ! Con: Complexity++
Slide 49
Conclusion 44
Slide 50
Examples https://github.com/xeraa/java-logging
Slide 51
Parse ! Send ✉ Structure ! Containerize ! Orchestrate !
Slide 52
Questions? 44 Philipp Krenn 44444 @xeraa
