Logging #3: Structured Logging

Structured Logging, also known as Semantic Logging, is probably the most powerful logging pattern I know.

Let’s assume that our log is normally emitted as follows (code – uses Go standard library):

2009/11/10 23:00:00.000000 Hello, Robert!

Structured log differs from “regular” plain-text log, that is formatted in a consisted way. Usually in some form of key/value representation like:

Formatted text (code – uses logrus):

time="2009-11-10T23:00:00Z" level=info msg="Hello, Robert!" user=Robert

JSON (code – uses logrus):

{"level":"info","msg":"Hello, Robert!","time":"2009-11-10T23:00:00Z","user":"Robert"}

GELF (code – uses logrus and logrus-gelf-formatter):

{"_pid":3,"_user":"Robert","level":"info","level_name":"info","short_message":"Hello, Robert!","timestamp":1257894000,"version":"1.1"}

Compact JSON (not supported by any Go package, maybe I will create one…):

{"@t":"2009-11-10T23:00:00Z","@mt":"Hello, {User}!","User":"Robert"}

In theory it is possible to render such logs using log package from Go standard library. However, I feel that would be very sub-optimal. There are good structured logging libraries decouples the way how the logs are:

• captured: log.WithField("user", "Robert").Info("Hello, Robert!")
• formatted: log.SetFormatter(&log.JSONFormatter{})
• outputted: log.SetOutput(os.Stdout)

Structured logs are easier to process for machines and even often by human (at least for me). It is very common to send structured logs to systems such us ELK, Graylog, Loggly, Splunk, Seq to help you search, analyze and react on logs.

In my previous post, I have linked Peter Bourgon’s article, where he suggested to use log package from Go standard library. However, after a few years Peter praises structured logging libraries.

Structured logging make decoding logs a lot easier.

What is the drawback of structured logging? When you do not need any system to analyze or process the logs, then if is clearly just adding unnecessary complexity.

I also encourage you to read about Message Templates, a language-neutral specification for 1) capturing, and 2) rendering, structured log events in a format that’s both human-friendly and machine-readable.

Logging series

• #1: Demo Application
• #2: It Depends
• #3: Structured Logging
• #4: Log Levels
• #5: Log Entry Builder
• #6: Correlation Log Field
• #7: Contextual Error Data Log Field
• #8: Logger Output
•  #9: Observability via Logging