Prometheus Installation

Prometheus is the metrics backbone. This covers installing Prometheus and Node Exporter via Docker Compose, configuring scrape targets, and verifying everything works.

Project Directory Setup

Create a dedicated folder for the entire observability stack. All config files live here.

Linux / macOS:

mkdir -p ~/observability/{prometheus,loki,tempo,grafana,promtail}
cd ~/observability

Windows PowerShell:

New-Item -ItemType Directory -Force -Path "$HOME\observability\prometheus","$HOME\observability\loki","$HOME\observability\tempo","$HOME\observability\grafana","$HOME\observability\promtail"
Set-Location "$HOME\observability"

Step 1 - Create prometheus.yml

This file tells Prometheus WHAT to scrape and HOW OFTEN.

Linux/macOS:

nano ~/observability/prometheus/prometheus.yml

Windows:

notepad "$HOME\observability\prometheus\prometheus.yml"

Paste this content:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]

  - job_name: node-exporter
    static_configs:
      - targets: [node-exporter:9100]

  - job_name: healthtune-api
    static_configs:
      - targets: [host.docker.internal:3001]
    metrics_path: /metrics
    scrape_interval: 30s

  - job_name: trackx-api
    static_configs:
      - targets: [host.docker.internal:7001]
    metrics_path: /metrics
    scrape_interval: 30s

Explanation of each section:

global.scrape_interval - how often Prometheus visits each target to collect numbers
job_name - a label grouping targets of the same type
targets - the host:port that exposes /metrics (or your custom metrics_path)
host.docker.internal - Docker magic hostname that lets a container reach services running on the host machine itself

Step 2 - Create docker-compose.yaml

File: ~/observability/docker-compose.yaml

This is the master file that launches all services. We add Prometheus and Node Exporter here now, and append the other services in later chapters.

version: 3.8

networks:
  observability:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:
  loki_data:
  tempo_data:

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle
    networks:
      - observability
    extra_hosts:
      - host.docker.internal:host-gateway

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - 9100:9100
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - --path.procfs=/host/proc
      - --path.rootfs=/rootfs
      - --path.sysfs=/host/sys
      - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)(34|/)
    networks:
      - observability

What each Prometheus flag does:

--config.file - path to your prometheus.yml inside the container
--storage.tsdb.path - where to store metric data on disk
--storage.tsdb.retention.time=30d - auto-delete data older than 30 days (prevents disk filling up)
--web.enable-lifecycle - lets you reload config without restarting (POST /-/reload)
extra_hosts: host.docker.internal:host-gateway - Linux only: lets the container reach your host apps

Note on Node Exporter on Windows/macOS: Node Exporter is a Linux-only tool. On local Windows/macOS dev machines, comment out the node-exporter service. It works on any Linux VM (GCP, AWS, local Ubuntu).

Step 3 - Start Prometheus

cd ~/observability

# Start only Prometheus and Node Exporter
docker compose up -d prometheus node-exporter

# Check status
docker compose ps

Expected:

NAME            STATUS          PORTS
prometheus      Up (healthy)    0.0.0.0:9090->9090/tcp
node-exporter   Up              0.0.0.0:9100->9100/tcp

Step 4 - Verify Prometheus Is Working

Check Prometheus is healthy:

curl http://localhost:9090/-/healthy
# Expected: Prometheus Server is Healthy.

curl http://localhost:9090/-/ready
# Expected: Prometheus Server is Ready.

Check Node Exporter is exposing metrics:

curl http://localhost:9100/metrics | head -20

You should see lines like:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu=0,mode=idle} 4823.12

Step 5 - Open the Prometheus UI

Open in your browser: http://localhost:9090

If this is a remote server, use SSH tunnel first:

ssh -L 9090:localhost:9090 username@YOUR_SERVER_IP

Then open: http://localhost:9090

Check That Scrape Targets Are UP

Go to Status > Targets in the Prometheus UI
You should see prometheus and node-exporter with state UP
If your apps expose /metrics they appear here too

If a target shows DOWN, check:

Is the service running? (pm2 list, docker compose ps)
Is the port correct in prometheus.yml?
Is host.docker.internal resolving? (Linux may need extra_hosts line)

Step 6 - Run Your First PromQL Query

In the Prometheus UI, go to the Graph tab and try these queries:

Host CPU usage (percentage):

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=idle}[5m])) * 100)

Available memory in GB:

node_memory_MemAvailable_bytes / 1024 / 1024 / 1024

Disk usage percentage:

100 - ((node_filesystem_avail_bytes{mountpoint=/} / node_filesystem_size_bytes{mountpoint=/}) * 100)

Number of running processes:

node_procs_running

Click Execute and switch to the Graph tab to see the chart.

Step 7 - Add Your App Metrics (Optional but Recommended)

For Prometheus to scrape your Node.js app, your app needs to expose a /metrics endpoint.

Install the Prometheus client for Node.js:

cd /home/wenawa/healthtune_api
yarn add prom-client

Add to your app (example for Express):

const client = require('prom-client');

// Auto-collect default metrics (CPU, memory, event loop)
client.collectDefaultMetrics({ prefix: 'healthtune_' });

// Expose /metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

Then restart your app:

pm2 restart healthtune_dev_api

Test it:

curl http://localhost:3001/metrics | head -30

Once /metrics responds, Prometheus will automatically scrape it based on your prometheus.yml config.

Hot Reload (Apply Config Changes Without Restart)

If you change prometheus.yml, apply it without stopping Prometheus:

curl -X POST http://localhost:9090/-/reload

Only works because we added --web.enable-lifecycle in Step 2.

Managing Prometheus

# Stop Prometheus (keeps data)
docker compose stop prometheus

# Start again
docker compose start prometheus

# View logs
docker compose logs prometheus -f

# Check disk usage of stored metrics
docker system df -v | grep prometheus

# How much data is stored
du -sh 

Prometheus Is Ready

At this point:

Prometheus is running at http://localhost:9090
Node Exporter is exposing host metrics
Scrape targets are visible in Status > Targets
You can run PromQL queries

Next: Install Grafana and connect it to Prometheus

Project Directory Setup​

Step 1 - Create prometheus.yml​

Step 2 - Create docker-compose.yaml​

Step 3 - Start Prometheus​

Step 4 - Verify Prometheus Is Working​

Step 5 - Open the Prometheus UI​

Check That Scrape Targets Are UP​

Step 6 - Run Your First PromQL Query​

Step 7 - Add Your App Metrics (Optional but Recommended)​

Hot Reload (Apply Config Changes Without Restart)​

Managing Prometheus​

Prometheus Is Ready​