Centralized Logging Patterns

A presentation at Voxxed Days Cluj in November 2018 in Cluj-Napoca, Romania by Philipp Krenn

Slide 1

Slide 1

Centralized Logging-Patterns Philipp Krenn @xeraa @xeraa

Slide 2

Slide 2

@xeraa

Slide 3

Slide 3

@xeraa

Slide 4

Slide 4

@xeraa

Slide 5

Slide 5

@xeraa

Slide 6

Slide 6

@xeraa

Slide 7

Slide 7

@xeraa

Slide 8

Slide 8

@xeraa

Slide 9

Slide 9

@xeraa

Slide 10

Slide 10

Slide 11

Slide 11

Developer @xeraa

Slide 12

Slide 12

@xeraa

Slide 13

Slide 13

@xeraa

Slide 14

Slide 14

@xeraa

Slide 15

Slide 15

@xeraa

Slide 16

Slide 16

@xeraa

Slide 17

Slide 17

Slide 18

Slide 18

Licensing Open Source Apache-2.0 Basic free Commercial @xeraa

Slide 19

Slide 19

Disclaimer I build highly monitored Hello World apps @xeraa

Slide 20

Slide 20

Example: Java SLF4J, Logback, MDC @xeraa

Slide 21

Slide 21

And Everywhere Else .NET: NLog PHP: Monolog JavaScript: Winston Python: structlog @xeraa

Slide 22

Slide 22

Anti-Pattern: print System.out.println("Oops"); @xeraa

Slide 23

Slide 23

Anti-Pattern: Coupling @xeraa

Slide 24

Slide 24

Slide 25

Slide 25

Slide 26

Slide 26

Parse @xeraa

Slide 27

Slide 27

@xeraa

Slide 28

Slide 28

Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log #exclude_lines: ['^TRACE'] @xeraa

Slide 29

Slide 29

Slide 30

Slide 30

[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] user_experience= , session=46, loop=15 Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^[%{TIMESTAMP_ISO8601:timestamp}]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}[%{WORD:thread}] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? @xeraa

Slide 31

Slide 31

Elastic Common Schema https://github.com/ elastic/ecs @xeraa

Slide 32

Slide 32

Dev Tools Grok Debugger @xeraa

Slide 33

Slide 33

Machine Learning Data Visualizer @xeraa

Slide 34

Slide 34

Visualize For example session @xeraa

Slide 35

Slide 35

Log UI @xeraa

Slide 36

Slide 36

Monitoring: Logstash Pipeline Plus other components @xeraa

Slide 37

Slide 37

Pro: No change Con: RegEx, timestamp, multiline @xeraa

Slide 38

Slide 38

Send @xeraa

Slide 39

Slide 39

@xeraa

Slide 40

Slide 40

Pro: No files Con: Outages & coupling @xeraa

Slide 41

Slide 41

Structure @xeraa

Slide 42

Slide 42

@xeraa

Slide 43

Slide 43

Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json fields_under_root: true json: message_key: message keys_under_root: true processors: - add_host_metadata: ~ @xeraa

Slide 44

Slide 44

Pro: Right format Con: JSON serialization overhead @xeraa

Slide 45

Slide 45

Containerize @xeraa

Slide 46

Slide 46

@xeraa

Slide 47

Slide 47

Where to put Filebeat? Sidecar @xeraa

Slide 48

Slide 48

Default JSON Log filebeat.input: - type: log paths: - "/var/lib/docker/containers//.log" json.message_key: log json.keys_under_root: true processors: - add_docker_metadata: ~ - add_host_metadata: ~ @xeraa

Slide 49

Slide 49

Metadata { "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "name": "my-java", "image": "my-java:1.0.0", "labels": { "app": "java" } } } } @xeraa

Slide 50

Slide 50

Logging as JSON - decode_json_fields: fields: ["message"] target: "" overwrite_keys: true @xeraa

Slide 51

Slide 51

Mount Log Path my-java: container_name: my-java hostname: my-java build: ${PWD}/config/my-java networks: ['stack'] command: java -jar my-java.jar volumes: - ./logs/my-java/:/opt/my-java/logs/ filebeat: container_name: filebeat hostname: filebeat image: "docker.elastic.co/beats/filebeat:${ELASTIC_VERSION}" volumes: - ./logs/my-java/:/var/log/my-java/ - ./docker-compose/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro command: filebeat -e networks: ['stack'] @xeraa

Slide 52

Slide 52

Registry File filebeat.registry_file: /usr/share/filebeat/data/registry @xeraa

Slide 53

Slide 53

..-__ ''-._ _.-.. ''-._ .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) |`-._`-...-` __...-.-.|'_.-'| |-.._ / _.-' |-._ -._-./ .-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'.-' .-' |-._-.-.__.-' _.-'_.-'| |-.-._ _.-'_.-' |-._ -._-..-'_.-' _.-' -._-..-' _.-' -._ _.-'-..-' Redis 4.0.9 (00000000/0) 64 bit Running in stand alone mode Port: 6379 PID: 55757 http://redis.io @xeraa

Slide 54

Slide 54

Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] @xeraa

Drop asciiart lines

Slide 55

Slide 55

Infrastructure UI @xeraa

Slide 56

Slide 56

Pro: Hot Con: Complexity @xeraa

Slide 57

Slide 57

Orchestrate @xeraa

Slide 58

Slide 58

@xeraa

Slide 59

Slide 59

Where to put Filebeat? DaemonSet @xeraa

Slide 60

Slide 60

Metadata Either in cluster or not processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config @xeraa

Slide 61

Slide 61

Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } @xeraa

Slide 62

Slide 62

Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\s+[\-`('.|_]"] # Drop asciiart lines @xeraa

Slide 63

Slide 63

Customize Indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" @xeraa

Slide 64

Slide 64

Pro: Hot Con: Complexity++ @xeraa

Slide 65

Slide 65

Moar @xeraa

Slide 66

Slide 66

Architecture @xeraa ! "

Slide 67

Slide 67

Slide 68

Slide 68

Slide 69

Slide 69

Slide 70

Slide 70

Frozen Indices https://github.com/elastic/elasticsearch/issues/34352 @xeraa

Slide 71

Slide 71

Centralized Logstash & Beats Management @xeraa

Slide 72

Slide 72

Conclusion @xeraa

Slide 73

Slide 73

Examples https://github.com/xeraa/java-logging @xeraa

Slide 74

Slide 74

Parse Send Structure Containerize Orchestrate @xeraa

Slide 75

Slide 75

Questions? Philipp Krenn @xeraa @xeraa