Grafana, Loki, Tempo, and OTEL Collector Installation

This chapter extends the Docker Compose stack with Grafana (dashboards), Loki (logs), Tempo (traces), Promtail (log shipper), and the OTEL Collector. At the end you connect all of them as Grafana data sources.

Services Being Added

Service	Role	Port
Grafana	Dashboard UI	3000
Loki	Log storage and query	3100
Promtail	Ships PM2 log files to Loki	9080 internal
Tempo	Trace storage and query	3200
OTEL Collector	Receives app traces, forwards to Tempo	4317 4318

Step 1 - Create Loki Config

File location: ~/observability/loki/loki-config.yaml

Linux/macOS:

nano ~/observability/loki/loki-config.yaml

Windows:

notepad "$HOME\observability\loki\loki-config.yaml"

Content:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

Key settings explained:

auth_enabled: false - no separate Loki login, Grafana controls access
retention_period: 168h - auto-delete logs older than 7 days (168 hours)
boltdb-shipper - the index store, appropriate for single-node setups

Step 2 - Create Promtail Config

Promtail reads PM2 log files from disk and ships them into Loki.

File: ~/observability/promtail/promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

  - job_name: healthtune-out
    static_configs:
      - targets:
          - localhost
        labels:
          app: healthtune_api
          level: info
          env: development
          __path__: /var/log/pm2/healthtune-dev-api-out.log

  - job_name: healthtune-error
    static_configs:
      - targets:
          - localhost
        labels:
          app: healthtune_api
          level: error
          env: development
          __path__: /var/log/pm2/healthtune-dev-api-error.log

  - job_name: trackx-out
    static_configs:
      - targets:
          - localhost
        labels:
          app: trackx_api
          level: info
          env: development
          __path__: /var/log/pm2/trackx-dev-api-out.log

  - job_name: trackx-error
    static_configs:
      - targets:
          - localhost
        labels:
          app: trackx_api
          level: error
          env: development
          __path__: /var/log/pm2/trackx-dev-api-error.log

Important: The path values must match your actual PM2 log file paths. Check them:

ls -la ~/.pm2/logs/

Adjust paths in promtail-config.yaml to match exactly.

Step 3 - Create Tempo Config

File: ~/observability/tempo/tempo-config.yaml

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

search_enabled: true

Key settings:

grpc endpoint 0.0.0.0:4317 - receives traces from your apps or OTEL Collector
block_retention: 48h - keep traces for 2 days (increase for production)
backend: local - stores trace data on disk (local filesystem)
search_enabled: true - allows searching traces by service name, status, duration

Step 4 - Create OTEL Collector Config

File: ~/observability/otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true

  logging:
    loglevel: warn

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp, logging]

Step 5 - Extend docker-compose.yaml

Add these services to your existing docker-compose.yaml (append under the node-exporter service):

  # GRAFANA
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - observability
    depends_on:
      - prometheus
      - loki
      - tempo

  # LOKI
  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    volumes:
      - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - observability

  # PROMTAIL
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yaml:/etc/promtail/config.yaml:ro
      - /home/wenawa/.pm2/logs:/var/log/pm2:ro
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - observability
    depends_on:
      - loki

  # TEMPO
  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    restart: unless-stopped
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    volumes:
      - ./tempo/tempo-config.yaml:/etc/tempo.yaml:ro
      - tempo_data:/tmp/tempo
    command: -config.file=/etc/tempo.yaml
    networks:
      - observability

  # OTEL COLLECTOR
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    restart: unless-stopped
    ports:
      - "4317:4317"
      - "4318:4318"
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
    command: --config /etc/otelcol-contrib/config.yaml
    networks:
      - observability
    depends_on:
      - tempo

IMPORTANT NOTE on ports 4317/4318: Tempo and OTEL Collector both want to listen on 4317 and 4318. You have two choices:

Option A - Use OTEL Collector as the entry point (recommended):

OTEL Collector exposes 4317 and 4318 to the host
Tempo does NOT expose these ports to the host
Apps send to localhost:4317 (Collector), Collector forwards to tempo:4317 (internal)
Comment out the 4317/4318 port lines from the Tempo service block

Option B - Send directly to Tempo (simpler):

Remove the otel-collector service entirely
Tempo exposes 4317 and 4318 directly
Apps send traces directly to Tempo

This guide uses Option A (OTEL Collector in front of Tempo) for flexibility.

Step 6 - Fix Promtail Log Path for Your Server

The Promtail service mounts your PM2 logs directory:

volumes:
  - /home/wenawa/.pm2/logs:/var/log/pm2:ro

Change /home/wenawa to your actual username:

# Find your PM2 logs directory
ls ~/.pm2/logs/

# Then update docker-compose.yaml - replace wenawa with your username
sed -i "s|/home/wenawa|/home/$USER|g" ~/observability/docker-compose.yaml

Windows and macOS - PM2 logs are in different locations. Update the volume path accordingly.

Step 7 - Start the Full Stack

cd ~/observability

# Start all services
docker compose up -d

# Wait 30 seconds then check status
sleep 30
docker compose ps

All containers should show Up or healthy:

NAME             STATUS          PORTS
prometheus       Up (healthy)    0.0.0.0:9090->9090/tcp
node-exporter    Up              0.0.0.0:9100->9100/tcp
grafana          Up              0.0.0.0:3000->3000/tcp
loki             Up              0.0.0.0:3100->3100/tcp
promtail         Up
tempo            Up              0.0.0.0:3200->3200/tcp
otel-collector   Up              0.0.0.0:4317->4317/tcp, 0.0.0.0:4318->4318/tcp

Step 8 - Quick Health Checks

# Prometheus
curl http://localhost:9090/-/healthy

# Loki
curl http://localhost:3100/ready

# Tempo
curl http://localhost:3200/ready

# Grafana
curl -I http://localhost:3000

All should return 200 OK or a ready message.

Open in browser: http://localhost:3000

Or if on a remote server, SSH tunnel first:

ssh -L 3000:localhost:3000 username@YOUR_SERVER_IP

Username: admin
Password: admin123

Change the password immediately after first login. Go to Profile > Change Password.

Step 10 - Add Data Sources in Grafana

This is the critical step that connects Grafana to your data.

Add Prometheus

In Grafana sidebar, click Connections > Data Sources
Click Add new data source
Choose Prometheus
Set URL to: http://prometheus:9090
Leave everything else as default
Click Save & Test — should show green checkmark

Add Loki

Click Add new data source
Choose Loki
Set URL to: http://loki:3100
Click Save & Test — should show green checkmark

Add Tempo

Click Add new data source
Choose Tempo
Set URL to: http://tempo:3200
Click Save & Test — should show green checkmark

Optional - Link Tempo to Loki (trace-to-log correlation): In the Tempo data source settings, scroll to Trace to logs section:

Toggle enabled ON
Data source: Loki
Tags: app (so it finds logs from the same service as the trace)

This allows you to click a trace in Grafana and jump directly to related logs.

Stack Is Running

At this point:

Prometheus is collecting host metrics from Node Exporter
Loki is ready to receive logs from Promtail
Tempo is ready to receive traces from your apps
Grafana is connected to all three data sources
You can log in to Grafana and start building dashboards

Next: Connect your Node.js apps to send logs and traces

Services Being Added​

Step 1 - Create Loki Config​

Step 2 - Create Promtail Config​

Step 3 - Create Tempo Config​

Step 4 - Create OTEL Collector Config​

Step 5 - Extend docker-compose.yaml​

Step 6 - Fix Promtail Log Path for Your Server​

Step 7 - Start the Full Stack​

Step 8 - Quick Health Checks​

Step 9 - First Login to Grafana​

Step 10 - Add Data Sources in Grafana​

Add Prometheus​

Add Loki​

Add Tempo​

Stack Is Running​