🏗️ Architecture — How the Stack Connects

Understanding the architecture before installing prevents confusion later. This chapter explains every connection, every port, and every data flow in plain language.

🗺️ Full Stack Architecture

┌─────────────────────────────────────────────────────────────┐
│                     YOUR SERVER / VM                        │
│                                                             │
│  ┌─────────────┐    ┌─────────────┐                         │
│  │ healthtune  │    │   trackx    │  ← Your Node.js apps    │
│  │  API (PM2)  │    │  API (PM2)  │    managed by PM2       │
│  └──────┬──────┘    └──────┬──────┘                         │
│         │ OTEL SDK          │ OTEL SDK                      │
│         └────────┬──────────┘                               │
│                  │ gRPC :4317                               │
│                  ▼                                          │
│         ┌────────────────┐                                  │
│         │ OTEL Collector │  receives traces from apps       │
│         └────────┬───────┘                                  │
│                  │ forwards traces                          │
│                  ▼                                          │
│         ┌────────────────┐                                  │
│         │     Tempo      │ :3200  stores traces             │
│         └────────┬───────┘                                  │
│                  │                                          │
│  ┌───────────────┼───────────────────┐                      │
│  │               │                   │                      │
│  ▼               ▼                   ▼                      │
│ ┌──────┐  ┌────────────┐  ┌──────────────────┐             │
│ │Prom. │  │    Loki    │  │     Grafana      │ :3000       │
│ │:9090 │  │   :3100    │  │  (Dashboards)    │   Browser   │
│ └──┬───┘  └─────┬──────┘  └──────────────────┘             │
│    │ scrapes    │ receives logs                              │
│    ▼            ▼                                           │
│ ┌──────┐  ┌────────────┐                                    │
│ │Node  │  │  Promtail  │  reads PM2 log files               │
│ │Expo. │  └────────────┘                                    │
│ │:9100 │                                                    │
│ └──────┘                                                    │
└─────────────────────────────────────────────────────────────┘

📦 Component Roles

Prometheus (Metrics)

Pull model — Prometheus goes out and fetches metrics from targets
Scrapes /metrics endpoint on your apps and Node Exporter
Stores data in its own time-series database (TSDB) on disk
Query language: PromQL

Node Exporter (Host Metrics)

A small agent that exposes system metrics at http://localhost:9100/metrics
Prometheus scrapes this to get CPU, RAM, disk, network stats
You never interact with it directly — just let it run

Loki (Logs)

Push model — Promtail pushes logs to Loki
Does NOT index full log content (much cheaper than Elasticsearch)
Indexes only labels (app, level, host, etc.)
Query language: LogQL

Promtail (Log Shipper)

Reads log files from disk (PM2 logs, Docker logs, syslog)
Attaches labels to each log line and pushes to Loki at http://loki:3100
Similar role to Filebeat in the ELK stack

Tempo (Traces)

Push model — apps send traces via OTEL protocol
Stores traces efficiently on disk (designed for high volume)
Can receive traces via OTLP gRPC (4317) or HTTP (4318)

OTEL Collector (Optional but Recommended)

Acts as a middleman between your apps and Tempo
Lets you add processors (sampling, filtering, enriching)
Without it: apps send directly to Tempo on port 4317
With it: apps → Collector → Tempo (more control)

Grafana (Visualization)

Connects to all data sources: Prometheus, Loki, Tempo
You build dashboards with panels (graphs, tables, logs, trace views)
The only tool you open in a browser

🔄 Data Flow by Type

Metrics Flow

App exposes /metrics endpoint
         ↓  (Prometheus scrapes every 15s)
     Prometheus stores in TSDB
         ↓  (Grafana queries via PromQL)
     Grafana dashboard panel

Logs Flow

PM2 writes logs to /home/user/.pm2/logs/*.log
         ↓  (Promtail tails the files)
     Promtail adds labels and pushes
         ↓  (HTTP push to Loki port 3100)
     Loki indexes labels + stores log lines
         ↓  (Grafana queries via LogQL)
     Grafana logs panel

Traces Flow

User HTTP request hits your app
         ↓  (OTEL SDK creates a trace)
     tracing.js sends to port 4317
         ↓  (OTEL Collector → Tempo)
     Tempo stores trace
         ↓  (Grafana Tempo data source)
     Grafana trace visualization

🔌 Complete Port Map

Port	Container	Protocol	Direction	Used By
`3000`	Grafana	HTTP	Inbound	Your browser
`9090`	Prometheus	HTTP	Inbound	Browser / Grafana
`3100`	Loki	HTTP	Inbound	Promtail push / Grafana
`3200`	Tempo	HTTP	Inbound	Grafana queries
`4317`	OTEL Collector / Tempo	gRPC	Inbound	Your Node.js apps
`4318`	OTEL Collector	HTTP	Inbound	Apps (HTTP alternative)
`9100`	Node Exporter	HTTP	Inbound	Prometheus scrape
`9080`	Promtail	HTTP	Internal	Metrics about Promtail

Minimum ports to expose externally (firewall rules):

3000 — open Grafana in your browser
4317 — apps send traces (if apps are on a different server)

All other ports can stay internal (container-to-container via Docker network).

🐳 Docker Networking Rule

When containers talk to each other inside Docker Compose, use the service name — not localhost:

From	To	Use This URL
Grafana → Prometheus	`http://prometheus:9090`
Promtail → Loki	`http://loki:3100`
OTEL Collector → Tempo	`http://tempo:4317`
Prometheus → Node Exporter	`http://node-exporter:9100`

But from your PM2 apps on the host machine, use localhost:

// In tracing.js (app runs on host, not in Docker)
url: 'http://localhost:4317'

💾 Data Storage (What Lives Where)

All persistent data uses Docker named volumes:

Tool	What It Stores	Volume
Prometheus	Time-series metrics	`prometheus_data`
Loki	Log chunks + index	`loki_data`
Tempo	Trace blocks	`tempo_data`
Grafana	Dashboards, users, settings	`grafana_data`

⚠️ docker compose down -v permanently deletes all volumes. Use only for a clean wipe. Use docker compose down (no -v) to stop containers while keeping all data.

🧱 Deployment Options

Option	Setup	Best For
Docker Compose on one VM	This guide	Small-medium teams, simplest
Separate VMs per tool	Manual config	Scaling independently
Kubernetes with Helm	`kube-prometheus-stack` chart	K8s environments
Grafana Cloud (managed)	Zero ops	Teams avoiding self-hosting

This documentation covers Docker Compose on one server.

🗺️ Full Stack Architecture​

📦 Component Roles​

Prometheus (Metrics)​

Node Exporter (Host Metrics)​

Loki (Logs)​

Promtail (Log Shipper)​

Tempo (Traces)​

OTEL Collector (Optional but Recommended)​

Grafana (Visualization)​

🔄 Data Flow by Type​

Metrics Flow​

Logs Flow​

Traces Flow​

🔌 Complete Port Map​

🐳 Docker Networking Rule​

💾 Data Storage (What Lives Where)​

🧱 Deployment Options​