Centralized Logging Patterns Philipp Krenn 44444 @xeraa 44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

Developer Advocate 44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

44 @xeraa

Apache License 2.0 44 @xeraa

Disclaimer I build highly monitored Hello World apps 44 @xeraa

Example: Java SLF4J, Logback, MDC 44 @xeraa

.NET: NLog PHP: Monolog JavaScript: Winston ... 44 @xeraa

Anti-Pattern: print System.out.println("Oops"); 44 @xeraa

Anti-Pattern: Coupling 44 @xeraa

Parse ! 44 44 @xeraa

44 @xeraa

Pro: No change Con: RegEx, timestamp, multiline 44 @xeraa

Send ✉ 44 44 @xeraa

44 @xeraa

Pro: No files Con: Outages & coupling 44 @xeraa

Structure ! 44 44 @xeraa

44 @xeraa

Pro: Right format Con: JSON serialization overhead 44 @xeraa

Containerize ! 44 44 @xeraa

44 @xeraa

44 @xeraa

https://turnoff.us/geek/before- devops-after-devops/

Where to put Filebeat? Sidecar 44 @xeraa

Default JSON log filebeat.prospectors:

  • type:

log paths:

"/var/lib/docker/containers//.log"

json.message_key:

log

json.keys_under_root:

true processors:

  • add_docker_metadata:

~ 44 @xeraa

Metadata {

"host" : "10.4.15.9" ,

"port" : 6379 ,

"docker" : {

"container" : {

"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,

"name" : "my-java" ,

"image" : "my-java:1.0.0" ,

"labels" : {

"app" : "java" } } } } 44 @xeraa

Mount log path my-java: container_name:

my-java hostname:

my-java build:

${PWD}/config/my-java networks:

['stack'] command:

java

-jar

my-java.jar volumes: -

./logs/my-java/:/opt/my-java/logs/ filebeat: container_name:

filebeat hostname:

filebeat image:

"docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: -

./logs/my-java/:/var/log/my-java/ -

./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command:

filebeat

-e networks:

['stack'] 44 @xeraa

Registry file filebeat.registry_file:

/usr/share/filebeat/data/registry 44 @xeraa

            _._
       _.-

__ ''-._ _.- .. ''-. Redis 4.0.9 (00000000/0) 64 bit .- `` .-

.

/ ., ''-._ ( ' , .-|, ) Running in stand alone mode |-._-...-__...-. `` -._|' .-'| Port: 6379 | -._. / .-' | PID: 55757 -._-. -./ _.-' _.-' |-.-._-.__.-' .-'.-'| | -._-. .-'.-' | http://redis.io -._-.-.__.-'_.-' _.-' |-.-._-..-' .-'.-'| | -._-._ .-'.-' | -._-.-.__.-'_.-' _.-'-. -.__.-' _.-'-._ _.-' `-..-' 44 @xeraa

Configuration templates filebeat.autodiscover: providers: - type:

docker templates: - condition: equals:

docker.container.image:

redis config: - type:

docker

containers.ids: -

"${data.docker.container.id}" exclude_lines:

["^\s+[\-`('.|_]"]

Drop asciiart lines

44 @xeraa

Pro: Hot ! Con: Complexity 44 @xeraa

Orchestrate ! 44 44 @xeraa

44 @xeraa

Where to put Filebeat? DaemonSet 44 @xeraa

Metadata processors:

  • add_kubernetes_metadata: in_cluster:

true 44 @xeraa

Metadata {

"host" : "172.17.0.21" ,

"port" : 9090 ,

"kubernetes" : {

"container" : {

"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,

"image" : "my-java:1.0.0" ,

"name" : "my-java" },

"labels" : {

"app" : "java" , },

"namespace" : "default" ,

"node" : {

"name" : "minikube" },

"pod" : {

"name" : "java-2657348378-k1pnh" } }, } 44 @xeraa

Configuration templates filebeat.autodiscover: providers: - type:

kubernetes templates: - condition: equals:

kubernetes.namespace:

redis config: - type:

docker

containers.ids: -

"${data.kubernetes.container.id}" exclude_lines:

["^\s+[\-`('.|_]"]

Drop asciiart lines

44 @xeraa

Customize indices output.elasticsearch: index:

"%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" 44 @xeraa

Pro: Hot ! ! ! Con: Complexity++ 44 @xeraa

Conclusion 44 44 @xeraa

Examples https://github.com/xeraa/java-logging 44 @xeraa

Parse ! Send ✉ Structure ! Containerize ! Orchestrate ! 44 @xeraa

Questions? 44 Philipp Krenn 44444 @xeraa 44 @xeraa