Application Integration - Connecting Your Apps

This chapter instruments your Node.js apps (HealthTune and TrackX) to send traces to Tempo and logs to Loki, completing the full observability loop.

---

What We Are Setting Up

Telemetry	How It Gets to Grafana
Traces	App -> OTEL SDK -> OTEL Collector (port 4317) -> Tempo -> Grafana
Logs	PM2 log files -> Promtail -> Loki -> Grafana
Metrics	App /metrics endpoint -> Prometheus (scrape) -> Grafana

---

Part 1: Traces - Connect Apps to Tempo via OTEL

Install OpenTelemetry Packages

HealthTune API:

cd /home/wenawa/healthtune_api
yarn add @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc

TrackX API:

cd /home/wenawa/trackx_api/backend
yarn add @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc

Using npm instead of yarn:

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc

Package explanation:

• @opentelemetry/sdk-node - the core OpenTelemetry SDK for Node.js
• @opentelemetry/auto-instrumentations-node - automatically instruments Express, HTTP, DB clients without manual code changes
• @opentelemetry/exporter-trace-otlp-grpc - sends trace data via gRPC to port 4317

Create tracing.js

Create this file in both projects. It is loaded before your app starts.

HealthTune: /home/wenawa/healthtune_api/tracing.js TrackX: /home/wenawa/trackx_api/backend/tracing.js

'use strict';

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const traceExporter = new OTLPTraceExporter({
  // Sends to OTEL Collector which forwards to Tempo
  // Change localhost to your observability server IP if apps are on a different machine
  url: 'http://localhost:4317',
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'unknown-service',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV || 'development',
  }),
  traceExporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable noisy fs instrumentation
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
});

sdk.start();

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

Start Apps with PM2 and Tracing

Stop existing PM2 processes first:

pm2 delete healthtune_dev_api trackx_dev_api

Start HealthTune with tracing:

OTEL_SERVICE_NAME=healthtune_api \
pm2 start "node -r /home/wenawa/healthtune_api/tracing.js dist/main.js" \
  --name healthtune_dev_api \
  --cwd /home/wenawa/healthtune_api

Start TrackX with tracing:

NODE_ENV=development \
OTEL_SERVICE_NAME=trackx_api \
pm2 start "node -r /home/wenawa/trackx_api/backend/tracing.js server.js" \
  --name trackx_dev_api \
  --cwd /home/wenawa/trackx_api/backend

Save PM2 process list for auto-restart on reboot:

pm2 save
pm2 startup
# Run the command that pm2 startup prints

Verify Traces Are Reaching Tempo

Make some API calls to your app to generate trace data:

curl http://localhost:3000/
curl http://localhost:3000/api/users
curl http://localhost:7000/

Then in Grafana:

1. Click Explore (compass icon in sidebar)
2. Select Tempo as the data source
3. Set Query type to Search
4. In Service Name dropdown, your app names should appear
5. Click Run Query - traces should appear within 30 seconds

---

Part 2: Logs - Verify Promtail is Shipping Logs

Promtail was configured in the previous chapter. Verify it is working:

Check Promtail logs:

docker compose logs promtail -f

Look for lines like:

level=info msg="Tailing new file" path=/var/log/pm2/healthtune-dev-api-out.log

If you see "permission denied" errors, fix PM2 log file permissions:

# Give Promtail container read access
chmod o+r ~/.pm2/logs/*.log
# Or for new log files automatically:
chmod o+r ~/.pm2/logs/

Verify logs appear in Grafana:

1. Grafana > Explore
2. Select Loki as data source
3. In Label filters, set app = healthtune_api
4. Click Run Query
5. You should see your PM2 logs

Sample LogQL queries to try:

# All logs from HealthTune
{app="healthtune_api"}

# Only errors from any app
{level="error"}

# Search for specific text in HealthTune logs
{app="healthtune_api"} |= "database"

# Count error log rate
rate({level="error"}[5m])

---

Part 3: Metrics - Add /metrics to Your Node.js Apps

For Prometheus to scrape application-level metrics (not just host metrics), your app needs to expose a /metrics endpoint.

Install Prometheus client:

# HealthTune
cd /home/wenawa/healthtune_api
yarn add prom-client

# TrackX
cd /home/wenawa/trackx_api/backend
yarn add prom-client

Add to your Express app (example):

const client = require('prom-client');
const express = require('express');
const app = express();

// Collect default Node.js metrics (CPU, memory, event loop lag, garbage collection)
const register = new client.Registry();
client.collectDefaultMetrics({ register, prefix: 'healthtune_' });

// Custom metric example - track HTTP request duration
const httpRequestDuration = new client.Histogram({
  name: 'healthtune_http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [register],
});

// Middleware to record each request
app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route?.path || req.path, status: res.statusCode });
  });
  next();
});

// Expose /metrics endpoint for Prometheus to scrape
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Test the metrics endpoint:

curl http://localhost:3001/metrics | grep healthtune_

You should see output like:

# HELP healthtune_process_cpu_user_seconds_total Total user CPU time
healthtune_process_cpu_user_seconds_total 0.123456
# HELP healthtune_nodejs_heap_size_used_bytes Process heap size used
healthtune_nodejs_heap_size_used_bytes 42345678

Once /metrics responds correctly, Prometheus will automatically begin scraping it based on your prometheus.yml scrape_configs.

---

Part 4: Correlate Traces, Logs, and Metrics

This is the most powerful feature of the Grafana stack - clicking through from one signal to another.

Trace to Logs

When you view a trace in Grafana Tempo, you can click "Logs for this span" to see the logs from the exact same app and time window. This requires the Tempo data source to have Loki linked (configured in Step 10 of the previous chapter).

Logs to Traces

In Loki/Explore, if your log lines contain a traceId field, Grafana automatically makes it a clickable link that opens the trace in Tempo.

Add trace ID to your app logs:

const { trace, context } = require('@opentelemetry/api');

// In your Express middleware or request handler:
app.use((req, res, next) => {
  const span = trace.getActiveSpan();
  if (span) {
    const traceId = span.spanContext().traceId;
    // Add traceId to your logger so it appears in log output
    req.traceId = traceId;
  }
  next();
});

Then include req.traceId in your log messages. Grafana will detect the traceID pattern automatically.

Exemplars - Metrics to Traces

Add trace IDs to Prometheus metrics so clicking a spike on a chart opens the trace from that moment:

const histogram = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'status'],
  enableExemplars: true,  // enables exemplar support
  registers: [register],
});

// When recording a metric, attach the traceId as an exemplar
histogram.observe(
  { method: req.method, status: res.statusCode },
  durationSeconds,
  { traceId: req.traceId }  // links this metric observation to a specific trace
);

---

PM2 Log Rotation (Prevent Disk Fill)

Set up log rotation so PM2 log files do not grow forever:

pm2 install pm2-logrotate
pm2 set pm2-logrotate:max_size 50M
pm2 set pm2-logrotate:retain 7
pm2 set pm2-logrotate:compress true

This keeps at most 7 rotated files of 50MB each per process.

---

Integration Checklist

# Apps running with tracing?
pm2 list

# Generate traffic
curl http://localhost:3000/ && curl http://localhost:7000/

# Traces in Tempo? (check Grafana > Explore > Tempo > Search)
# Logs in Loki? (check Grafana > Explore > Loki > Label filters)

# Collector receiving data?
docker compose logs otel-collector --tail=20

# Promtail shipping logs?
docker compose logs promtail --tail=20