🏗️ Architecture — How the Stack Connects

Understanding the architecture before installing prevents confusion later. This chapter explains every connection, every port, and every data flow in plain language.

---

🗺️ Full Stack Architecture

┌─────────────────────────────────────────────────────────────┐
│                     YOUR SERVER / VM                        │
│                                                             │
│  ┌─────────────┐    ┌─────────────┐                         │
│  │ healthtune  │    │   trackx    │  ← Your Node.js apps    │
│  │  API (PM2)  │    │  API (PM2)  │    managed by PM2       │
│  └──────┬──────┘    └──────┬──────┘                         │
│         │ OTEL SDK          │ OTEL SDK                      │
│         └────────┬──────────┘                               │
│                  │ gRPC :4317                               │
│                  ▼                                          │
│         ┌────────────────┐                                  │
│         │ OTEL Collector │  receives traces from apps       │
│         └────────┬───────┘                                  │
│                  │ forwards traces                          │
│                  ▼                                          │
│         ┌────────────────┐                                  │
│         │     Tempo      │ :3200  stores traces             │
│         └────────┬───────┘                                  │
│                  │                                          │
│  ┌───────────────┼───────────────────┐                      │
│  │               │                   │                      │
│  ▼               ▼                   ▼                      │
│ ┌──────┐  ┌────────────┐  ┌──────────────────┐             │
│ │Prom. │  │    Loki    │  │     Grafana      │ :3000       │
│ │:9090 │  │   :3100    │  │  (Dashboards)    │   Browser   │
│ └──┬───┘  └─────┬──────┘  └──────────────────┘             │
│    │ scrapes    │ receives logs                              │
│    ▼            ▼                                           │
│ ┌──────┐  ┌────────────┐                                    │
│ │Node  │  │  Promtail  │  reads PM2 log files               │
│ │Expo. │  └────────────┘                                    │
│ │:9100 │                                                    │
│ └──────┘                                                    │
└─────────────────────────────────────────────────────────────┘

---

📦 Component Roles

Prometheus (Metrics)

• Pull model — Prometheus goes out and fetches metrics from targets
• Scrapes /metrics endpoint on your apps and Node Exporter
• Stores data in its own time-series database (TSDB) on disk
• Query language: PromQL

Node Exporter (Host Metrics)

• A small agent that exposes system metrics at http://localhost:9100/metrics
• Prometheus scrapes this to get CPU, RAM, disk, network stats
• You never interact with it directly — just let it run

Loki (Logs)

• Push model — Promtail pushes logs to Loki
• Does NOT index full log content (much cheaper than Elasticsearch)
• Indexes only labels (app, level, host, etc.)
• Query language: LogQL

Promtail (Log Shipper)

• Reads log files from disk (PM2 logs, Docker logs, syslog)
• Attaches labels to each log line and pushes to Loki at http://loki:3100
• Similar role to Filebeat in the ELK stack

Tempo (Traces)

• Push model — apps send traces via OTEL protocol
• Stores traces efficiently on disk (designed for high volume)
• Can receive traces via OTLP gRPC (4317) or HTTP (4318)

OTEL Collector (Optional but Recommended)

• Acts as a middleman between your apps and Tempo
• Lets you add processors (sampling, filtering, enriching)
• Without it: apps send directly to Tempo on port 4317
• With it: apps → Collector → Tempo (more control)

Grafana (Visualization)

• Connects to all data sources: Prometheus, Loki, Tempo
• You build dashboards with panels (graphs, tables, logs, trace views)
• The only tool you open in a browser

---

🔄 Data Flow by Type

Metrics Flow

App exposes /metrics endpoint
         ↓  (Prometheus scrapes every 15s)
     Prometheus stores in TSDB
         ↓  (Grafana queries via PromQL)
     Grafana dashboard panel

Logs Flow

PM2 writes logs to /home/user/.pm2/logs/*.log
         ↓  (Promtail tails the files)
     Promtail adds labels and pushes
         ↓  (HTTP push to Loki port 3100)
     Loki indexes labels + stores log lines
         ↓  (Grafana queries via LogQL)
     Grafana logs panel

Traces Flow

User HTTP request hits your app
         ↓  (OTEL SDK creates a trace)
     tracing.js sends to port 4317
         ↓  (OTEL Collector → Tempo)
     Tempo stores trace
         ↓  (Grafana Tempo data source)
     Grafana trace visualization

---

🔌 Complete Port Map

Port	Container	Protocol	Direction	Used By
3000	Grafana	HTTP	Inbound	Your browser
9090	Prometheus	HTTP	Inbound	Browser / Grafana
3100	Loki	HTTP	Inbound	Promtail push / Grafana
3200	Tempo	HTTP	Inbound	Grafana queries
4317	OTEL Collector / Tempo	gRPC	Inbound	Your Node.js apps
4318	OTEL Collector	HTTP	Inbound	Apps (HTTP alternative)
9100	Node Exporter	HTTP	Inbound	Prometheus scrape
9080	Promtail	HTTP	Internal	Metrics about Promtail

Minimum ports to expose externally (firewall rules):

• 3000 — open Grafana in your browser
• 4317 — apps send traces (if apps are on a different server)

All other ports can stay internal (container-to-container via Docker network).

---

🐳 Docker Networking Rule

When containers talk to each other inside Docker Compose, use the service name — not localhost:

From	To	Use This URL
Grafana → Prometheus	http://prometheus:9090
Promtail → Loki	http://loki:3100
OTEL Collector → Tempo	http://tempo:4317
Prometheus → Node Exporter	http://node-exporter:9100

But from your PM2 apps on the host machine, use localhost:

// In tracing.js (app runs on host, not in Docker)
url: 'http://localhost:4317'

---

💾 Data Storage (What Lives Where)

All persistent data uses Docker named volumes:

Tool	What It Stores	Volume
Prometheus	Time-series metrics	prometheus_data
Loki	Log chunks + index	loki_data
Tempo	Trace blocks	tempo_data
Grafana	Dashboards, users, settings	grafana_data

⚠️ docker compose down -v permanently deletes all volumes. Use only for a clean wipe. Use docker compose down (no -v) to stop containers while keeping all data.

---

🧱 Deployment Options

Option	Setup	Best For
Docker Compose on one VM	This guide	Small-medium teams, simplest
Separate VMs per tool	Manual config	Scaling independently
Kubernetes with Helm	kube-prometheus-stack chart	K8s environments
Grafana Cloud (managed)	Zero ops	Teams avoiding self-hosting

This documentation covers Docker Compose on one server.