In the world of DevOps, effective log management is crucial for monitoring, debugging, and maintaining applications. Loki, a log aggregation system inspired by Prometheus, is designed to be cost-effective and easy to operate. It doesn’t index the contents of the logs but rather indexes the metadata, making it a lightweight and efficient solution for log management. In this article, we’ll explore how to push Docker logs to Loki using Promtail, an agent that ships the contents of local logs to a Loki instance.

Prerequisites

Before we begin, ensure you have the following:

1. Docker installed on your system.
2. A running Loki instance. You can set it up using Docker or any other preferred method.
3. Promtail installed on your system.

Setting Up Loki

If you haven’t set up Loki yet, you can quickly get it running using Docker. Here’s a basic command to start Loki:

docker run -d --name=loki -p 3100:3100 grafana/loki:latest

This command pulls the latest Loki Docker image and runs it, exposing it on port 3100.

Installing Promtail

Promtail is the agent responsible for gathering logs and sending them to Loki. You can install Promtail by downloading the binary from the official releases page or by using Docker.

To run Promtail using Docker, use the following command:

docker run -d --name=promtail -v /var/log:/var/log -v /etc/promtail:/etc/promtail grafana/promtail:latest -config.file=/etc/promtail/promtail-config.yaml

Configuring Promtail

Promtail requires a configuration file to know which logs to read and where to send them. Here’s a basic configuration example for Promtail (promtail-config.yaml):

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_image]
        target_label: image

Key Configuration Sections

• server: Defines the ports Promtail listens on.
• positions: Keeps track of the last read position in each log file to avoid re-reading logs.
• clients: Specifies the Loki server URL where logs are sent.
• scrape_configs: Defines the log sources. In this example, it includes system logs and Docker logs.

Running Promtail

With the configuration file in place, start Promtail:

docker run -d --name=promtail -v /var/log:/var/log -v /etc/promtail:/etc/promtail grafana/promtail:latest -config.file=/etc/promtail/promtail-config.yaml

Ensure that the paths in the -v options match the paths on your host system where the logs and configuration file are located.

Verifying the Setup

To verify that logs are being pushed to Loki, you can use Grafana to visualize them. Add Loki as a data source in Grafana and create a dashboard to query and display logs.

Conclusion

By using Promtail to push Docker logs to Loki, you can efficiently manage and monitor your logs without the overhead of indexing log content. This setup is scalable and integrates seamlessly with Grafana for visualization, providing a comprehensive observability solution.

References

• Loki Documentation
• Promtail Configuration
• Docker Logging