Log Rotation in Docker

What is Docker?

Docker is a containerization platform that can be used to develop, deploy and manage apps in virtualized environments known as containers.

The main use of Docker is in software development, where it aids in packaging and distributing applications that can work on all environments irrespective of their underlying OS.

Docker logs

Like most applications, Docker keeps a log of activities that occur in the container. Docker containers provide logs via stdout and stderr. The generated logs can be accessed via different logging drivers that can be configured in the Docker daemon.

Why Do We Need Logs?

Logs are very essential pieces of telemetry data that are used for debugging performance issues in applications. Logs can provide us all the events in our application if configured correctly. They can then be used to troubleshoot the occurred issues and identify the root cause of the problem.

The complexity of operating a container-based infrastructure has increased along with the easiness of scaling the application up and down in seconds. Containers are ephemeral, and the changing container environments are very challenging to monitor.

This is the part where logs come to our aid, assisting in discovering underlying issues and optimizing performance

Types of Docker Logs

Docker produces two types of logs which are:

1. Docker Daemon Logs

Logs generated by the Docker daemon and present in the host system.

2. Docker Container Logs

Container logs cover the logs related to running applications in a container. By default, size restrictions are not imposed on the log files. The log file size will increase over time and need pruning at some point. The more containers you have running, the more logs consume more storage.

Performing log rotation enables us to manage logs efficiently. Old logs are removed after a certain log file threshold is reached, which can be specified in the Docker daemon configuration.

Log Storage

All logs are stored in the host as the containers themselves are ephemeral. Thus, the logs will be instantly purged upon deletion of said container.

The logs are stored in /var/lib/Docker/containers/$id/$id.log unless otherwise specified in the daemon.json file.

Different logging drivers are provided other than JSON. For example,splunk, fluentd, greylog, syslog, etc.

The following command can check the current logging driver where JSON-file is the default.

docker info | grep -iw Logging\ Driver

Output:

Logging Driver: json-file

Setting Up Log Rotation

Configure Log Driver

First, we will need to create a daemon.json file in /etc/Docker/ because the installation doesn't ship the daemon.json file by default.

All the things we add in the daemon.json will get combined by the Docker daemon along with the pre-existing settings and then be used to configure itself.

We can also specify the config file location during the Docker daemon runtime via the --config-file flag. By default, it looks for a daemon.json file in /etc/Docker/

Use the configuration below to configure a different logging driver than the json-file:

{
  "log-driver": "local" 
  #can be replaced with syslog, fluentd, json-file, splunk, etc
}

Configure Log Rotation

Append the contents of the previously created daemon.json with the following config options to enable log rotation:

{
  "log-opts": {
    "max-size": "15m",
    "max-file": "5"
  }
}

After appending the file, it should look something like this:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "15m",
    "max-file": "5"
  }
}

Now we need to restart the Docker daemon for changes to be applied.

systemctl restart docker

Log Rotation for a Single Container

If you want a specific container to use a different driver than the one specified in the daemon, then the --log-driver flag can be used.

docker run --log-driver json-file -d -p 127.0.0.1:80:80 httpd

To find the logging driver of an already running container, run the following command:

docker inspect -f '{{.HostConfig.LogConfig.Type}}' 1e9c943d26d4

Output:

local

To configure log rotation for a specific container, run the commands below:

docker run  --log-opt max-size=5m --log-opt max-file=3 httpd:latest

To simplify the above command:

--log-opt max-size=5m tells Docker to limit the size of the Docker file to 5 MB

--log-opt max-file=5 tells Docker to store only three log files. When the threshold is met, Docker purges the old files from the host system.

Summary

The default logging settings of Docker work fine for normal use cases. However, it can be very hard to keep track of the logs and narrow down issues in the infrastructure if you run many containers, especially in a production environment.

Log rotation helps to achieve that by organizing logs automatically and removing older logs to free up space for fresh ones.

Conclusion

This article taught us about logging in Docker, logging driver types, managing logs and implementing log rotation.

Thank you for reading. Stay tuned for more blogs on Docker and Kubernetes. Please comment down below if you have any questions.