Centralized Logging Patterns

A presentation at Riga DevDays in May 2018 in Riga, Latvia by Philipp Krenn

Slide 1

Slide 1

Centralized Logging Patterns Philipp Krenn 44444 @xeraa 44 @xeraa

Slide 2

Slide 2

44 @xeraa

Slide 3

Slide 3

44 @xeraa

Slide 4

Slide 4

44 @xeraa

Slide 5

Slide 5

44 @xeraa

Slide 6

Slide 6

44 @xeraa

Slide 7

Slide 7

44 @xeraa

Slide 8

Slide 8

44 @xeraa

Slide 9

Slide 9

44 @xeraa

Slide 10

Slide 10

Slide 11

Slide 11

Developer Advocate 44 @xeraa

Slide 12

Slide 12

44 @xeraa

Slide 13

Slide 13

44 @xeraa

Slide 14

Slide 14

44 @xeraa

Slide 15

Slide 15

44 @xeraa

Slide 16

Slide 16

44 @xeraa

Slide 17

Slide 17

44 @xeraa

Slide 18

Slide 18

44 @xeraa

Slide 19

Slide 19

44 @xeraa

Slide 20

Slide 20

44 @xeraa

Slide 21

Slide 21

Slide 22

Slide 22

Apache License 2.0 44 @xeraa

Slide 23

Slide 23

Disclaimer I build highly monitored Hello World apps 44 @xeraa

Slide 24

Slide 24

Example: Java SLF4J, Logback, MDC 44 @xeraa

Slide 25

Slide 25

.NET: NLog PHP: Monolog JavaScript: Winston ... 44 @xeraa

Slide 26

Slide 26

Anti-Pattern: print System.out.println("Oops"); 44 @xeraa

Slide 27

Slide 27

Anti-Pattern: Coupling 44 @xeraa

Slide 28

Slide 28

Parse ! 44 44 @xeraa

Slide 29

Slide 29

44 @xeraa

Slide 30

Slide 30

Pro: No change Con: RegEx, timestamp, multiline 44 @xeraa

Slide 31

Slide 31

Send ✉ 44 44 @xeraa

Slide 32

Slide 32

44 @xeraa

Slide 33

Slide 33

Pro: No files Con: Outages & coupling 44 @xeraa

Slide 34

Slide 34

Structure ! 44 44 @xeraa

Slide 35

Slide 35

44 @xeraa

Slide 36

Slide 36

Pro: Right format Con: JSON serialization overhead 44 @xeraa

Slide 37

Slide 37

Containerize ! 44 44 @xeraa

Slide 38

Slide 38

44 @xeraa

Slide 39

Slide 39

44 @xeraa

Slide 40

Slide 40

https://turnoff.us/geek/before- devops-after-devops/

Slide 41

Slide 41

Where to put Filebeat? Sidecar 44 @xeraa

Slide 42

Slide 42

Default JSON log filebeat.prospectors:

  • type:

log paths:

"/var/lib/docker/containers//.log"

json.message_key:

log

json.keys_under_root:

true processors:

  • add_docker_metadata:

~ 44 @xeraa

Slide 43

Slide 43

Metadata {

"host" : "10.4.15.9" ,

"port" : 6379 ,

"docker" : {

"container" : {

"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,

"name" : "my-java" ,

"image" : "my-java:1.0.0" ,

"labels" : {

"app" : "java" } } } } 44 @xeraa

Slide 44

Slide 44

Mount log path my-java: container_name:

my-java hostname:

my-java build:

${PWD}/config/my-java networks:

['stack'] command:

java

-jar

my-java.jar volumes: -

./logs/my-java/:/opt/my-java/logs/ filebeat: container_name:

filebeat hostname:

filebeat image:

"docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: -

./logs/my-java/:/var/log/my-java/ -

./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command:

filebeat

-e networks:

['stack'] 44 @xeraa

Slide 45

Slide 45

Registry file filebeat.registry_file:

/usr/share/filebeat/data/registry 44 @xeraa

Slide 46

Slide 46 

            _._
       _.-

__ ''-._ _.-.. ''-. Redis 4.0.9 (00000000/0) 64 bit .- `` .-

.

/ ., ''-._ ( ' , .-|, ) Running in stand alone mode |-._-...-__...-. `` -._|'.-'| Port: 6379 | -._. / .-' | PID: 55757 -._-.-./ _.-' _.-' |-.-._-.__.-' .-'.-'| | -._-..-'.-' | http://redis.io -._-.-.__.-'_.-' _.-' |-.-._-..-' .-'.-'| | -._-._ .-'.-' | -._-.-.__.-'_.-' _.-'-.-.__.-' _.-'-._ _.-' `-..-' 44 @xeraa

Slide 47

Slide 47

Configuration templates filebeat.autodiscover: providers: - type:

docker templates: - condition: equals:

docker.container.image:

redis config: - type:

docker

containers.ids: -

"${data.docker.container.id}" exclude_lines:

["^\s+[\-`('.|_]"]

Drop asciiart lines

44 @xeraa

Slide 48

Slide 48

Pro: Hot ! Con: Complexity 44 @xeraa

Slide 49

Slide 49

Orchestrate ! 44 44 @xeraa

Slide 50

Slide 50

44 @xeraa

Slide 51

Slide 51

Where to put Filebeat? DaemonSet 44 @xeraa

Slide 52

Slide 52

Metadata processors:

  • add_kubernetes_metadata: in_cluster:

true 44 @xeraa

Slide 53

Slide 53

Metadata {

"host" : "172.17.0.21" ,

"port" : 9090 ,

"kubernetes" : {

"container" : {

"id" : "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" ,

"image" : "my-java:1.0.0" ,

"name" : "my-java" },

"labels" : {

"app" : "java" , },

"namespace" : "default" ,

"node" : {

"name" : "minikube" },

"pod" : {

"name" : "java-2657348378-k1pnh" } }, } 44 @xeraa

Slide 54

Slide 54

Configuration templates filebeat.autodiscover: providers: - type:

kubernetes templates: - condition: equals:

kubernetes.namespace:

redis config: - type:

docker

containers.ids: -

"${data.kubernetes.container.id}" exclude_lines:

["^\s+[\-`('.|_]"]

Drop asciiart lines

44 @xeraa

Slide 55

Slide 55

Customize indices output.elasticsearch: index:

"%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" 44 @xeraa

Slide 56

Slide 56

Pro: Hot ! ! ! Con: Complexity++ 44 @xeraa

Slide 57

Slide 57

Conclusion 44 44 @xeraa

Slide 58

Slide 58

Examples https://github.com/xeraa/java-logging 44 @xeraa

Slide 59

Slide 59

Parse ! Send ✉ Structure ! Containerize ! Orchestrate ! 44 @xeraa

Slide 60

Slide 60

Questions? 44 Philipp Krenn 44444 @xeraa 44 @xeraa