Loki is an open-source log aggregation system developed by Grafana Labs that is designed to be highly efficient and easy to use. Unlike traditional log systems like Elasticsearch, Loki is designed to work seamlessly with Prometheus and has a unique feature where it indexes only the metadata of logs, making it lightweight and scalable. This article will guide you through setting up a Loki server using Docker Compose and creating a Systemd service for easier management.

Prerequisites

Hardware Recommendations

The hardware requirements for your Loki server can vary based on your use case and data volume. Here are some general recommendations:

1. Small-scale deployments (Development and testing)

   • CPU: 1 vCPU
   • RAM: 2 GB
   • Storage: 20 GB SSD

2. Medium-scale deployments (Small production level).

   • CPU: 2 vCPUs
   • RAM: 4 GB
   • Storage: 50 GB SSD

3. Large-scale deployments (High log volume, multiple services).

   • CPU: 4+ vCPUs
   • RAM: 16+ GB
   • Storage: 200+ GB SSD (or more depending on retention period)

Note: Always consider future growth and possibly scale up your hardware as needed.

Step 1: Setting Up Docker and Docker Compose

If you haven’t already installed Docker and Docker Compose, you can do so with the following commands (for Debian/Ubuntu systems):

# Update package database
sudo apt-get update

# Install Docker
sudo apt-get install -y docker.io

# Start Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Install Docker Compose
sudo apt-get install -y docker-compose

Step 2: Create the Docker Compose File

1. Create a new directory for Loki configuration and navigate to it.

mkdir loki-docker && cd loki-docker

2. Create a docker-compose.yml file with the following content:

version: '3.7'

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki.yaml
    command: -config.file=/etc/loki/loki.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yml
      - /var/log:/var/log
      - /etc/hostname:/etc/hostname
      - /etc/hosts:/etc/hosts
    command: -config.file=/etc/promtail/config.yml

3. Create a basic configuration file for Loki called loki-config.yaml:

auth_enabled: false

server:
  http_port: 3100

ingester:
  wal:
    enabled: true
    dir: /loki/wal

# storage_config defines how to handle data persistence
storage_config:
  boltdb:
    directory: /loki/index
  filesystem:
    directory: /loki/chunks

schema_config:
  configs:
    - from: 2020-10-01
      store: boltdb
      schema: v11
      index:
        prefix: index_
        period: 24h

4. Create a configuration file for Promtail called promtail-config.yaml to scrape logs:

server:
  http_listen_port: 9080

positions:
  filename: /var/log/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

Step 3: Start Loki with Docker Compose

To start the services, run:

docker-compose up -d

Verify that both Loki and Promtail services are running:

docker-compose ps

Step 4: Create Systemd Service for Docker Compose

To manage your Loki server using Systemd, create a new service file:

1. Create the service file:

sudo nano /etc/systemd/system/loki.service

2. Add the following content:

[Unit]
Description=Loki log aggregation service
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStart=/usr/local/bin/docker-compose -f /path/to/your/loki-docker/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /path/to/your/loki-docker/docker-compose.yml down

[Install]
WantedBy=multi-user.target

Note: Make sure to replace /path/to/your/loki-docker with the actual path where the docker compose file resides.

3. Enable the service and start it:

sudo systemctl enable loki
sudo systemctl start loki

Step 5: Optimize Your Loki Server

1. Retention Policy: Choose an appropriate retention period based on log importance. Generally, 7-30 days is a good start for most applications.

2. Indexing Strategy: Adjust the schema for indexing based on your environment. For critical production logs, consider using more storage but increasing reliability.

3. Log Compression: Enable storage compression for long-term storage to save disk space.

4. Monitoring and Scaling: Monitor Loki’s performance using Grafana. If you notice performance degradation, horizontally scale the Loki instances or increase resources.

Happy logging!