Zum Hauptinhalt springen

📡 OpenTelemetry Collector

OTEL Collector Flow in SigNoz

The OTEL Collector is the bridge between your apps and SigNoz. Your app sends raw telemetry data to the collector, the collector processes it, and then sends it to SigNoz.

🏗️ Architecture Overview​

Your Node.js App
    ↓  (sends via gRPC port 4317 or HTTP port 4318)
OTEL Collector (running as service or Docker)
    ↓  (forwards to SigNoz backend)
SigNoz ClickHouse DB
    ↓
SigNoz Dashboard (your browser)

🔧 The Three Parts of OTEL​

PartWhat It DoesWhere It Runs
SDK (in your app)Generates trace/metric/log dataInside your Node.js / Python code
CollectorReceives, processes, and forwards dataAs a system service (otelcol-contrib)
Backend (SigNoz)Stores and displays the dataDocker container

📦 Install the OTEL Collector​

On Linux (Ubuntu/Debian) — Recommended​

Download and install otelcol-contrib (the full-featured version):

📌 Versioning note: Use the latest stable collector release for new deployments. Keep one tested version pinned in your internal runbook so upgrades are controlled.

# Check latest version at https://github.com/open-telemetry/opentelemetry-collector-releases

# Download (replace version as needed)
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.97.0/otelcol-contrib_0.97.0_linux_amd64.deb

# Install
sudo dpkg -i otelcol-contrib_0.97.0_linux_amd64.deb

# Verify
otelcol-contrib --version

On Windows​

Download the .exe from: https://github.com/open-telemetry/opentelemetry-collector-releases/releases

Choose: otelcol-contrib_X.X.X_windows_amd64.tar.gz

Extract and run:

.\otelcol-contrib.exe --config config.yaml

Via Docker (alternative)​

docker run -d \
  --name otel-collector \
  -p 4317:4317 \
  -p 4318:4318 \
  -v $(pwd)/otel-config.yaml:/etc/otelcol-contrib/config.yaml \
  otel/opentelemetry-collector-contrib:latest

📝 Collector Config File​

The config file tells the collector:

  1. What data to receive (from your apps)
  2. How to process it (batching, filtering)
  3. Where to export it (SigNoz)

Location on Linux: /etc/otelcol-contrib/config.yaml

Here is the full config used in this project:

receivers:
  # Receive traces and metrics from your apps via OTLP protocol
  otlp:
    protocols:
      grpc:   # port 4317 — for Node.js, Python etc.
      http:   # port 4318 — alternative HTTP protocol

  # Collect host-level metrics (CPU, RAM) every 30 seconds
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}

  # Read log files from PM2-managed apps
  filelog/healthtune:
    include:
      - /home/youruser/.pm2/logs/healthtune-dev-api-out.log
      - /home/youruser/.pm2/logs/healthtune-dev-api-error.log

  filelog/trackx:
    include:
      - /home/youruser/.pm2/logs/trackx-dev-api-out.log
      - /home/youruser/.pm2/logs/trackx-dev-api-error.log

processors:
  # Batch data before sending — better performance, fewer requests
  batch:

exporters:
  # Send everything to SigNoz
  otlp:
    endpoint: "YOUR_SIGNOZ_VM_IP:4317"   # ← Change this to your SigNoz server IP
    tls:
      insecure: true   # Dev/internal only. Use TLS certs for internet-facing traffic.

service:
  pipelines:
    # Traces pipeline: receive → batch → send to SigNoz
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

    # Logs pipeline: read log files → batch → send to SigNoz
    logs:
      receivers: [filelog/healthtune, filelog/trackx]
      processors: [batch]
      exporters: [otlp]

✏️ Customize the Config for Your Setup​

Replace placeholders before start​

  • YOUR_SIGNOZ_VM_IP: IP/DNS where SigNoz OTLP endpoint is reachable
  • /home/youruser/.pm2/logs/...: replace with your PM2 log directory and filenames

Change the SigNoz endpoint​

exporters:
  otlp:
    endpoint: "34.31.206.197:4317"   # Replace with YOUR SigNoz server IP

If SigNoz is on the same server as the collector:

endpoint: "localhost:4317"

Add your own log files​

receivers:
  filelog/myapp:
    include:
      - /home/youruser/.pm2/logs/myapp-out.log
      - /home/youruser/.pm2/logs/myapp-error.log

Then add it to the logs pipeline:

service:
  pipelines:
    logs:
      receivers: [filelog/healthtune, filelog/trackx, filelog/myapp]
      processors: [batch]
      exporters: [otlp]

🛡️ Fix Log File Permissions​

The collector runs as the otelcol-contrib system user and may not have permission to read your PM2 log files. Fix this:

# Install ACL tools
sudo apt install -y acl

# Grant read access to otelcol-contrib user for all PM2 logs
sudo setfacl -m u:otelcol-contrib:r /home/youruser/.pm2/logs/*

# Verify the permission was set
getfacl /home/youruser/.pm2/logs/healthtune-dev-api-out.log

▶️ Start the Collector​

# Enable auto-start on reboot
sudo systemctl enable otelcol-contrib

# Start now
sudo systemctl start otelcol-contrib

# Or restart if already running
sudo systemctl restart otelcol-contrib

Check if It's Running​

sudo systemctl status otelcol-contrib

Expected output includes Active: active (running).

View Collector Logs (Useful for Debugging)​

sudo journalctl -u otelcol-contrib -f

Press Ctrl+C to stop following logs.

🧪 How Tracing Works Inside Your App​

When your app receives a request, the OTEL SDK:

  1. Creates a trace (a unique ID for this entire request)
  2. Creates spans for each step (database query, external API call, etc.)
  3. Measures the time for each span
  4. Sends everything to the OTEL Collector

Example of a trace:

Trace: user-login-request
  ├── span: parse-jwt-token        [3ms]
  ├── span: query-users-table      [45ms]
  ├── span: hash-password-check    [12ms]
  └── span: send-response          [2ms]
  Total: 62ms

In SigNoz, you can click on any trace and see exactly this breakdown.

🧪 Test the Collector Is Receiving Data​

Send a test trace manually:

# Install grpcurl
sudo apt install -y grpcurl   # or download from https://github.com/fullstorydev/grpcurl

# Send a test to the collector
grpcurl -plaintext localhost:4317 opentelemetry.proto.collector.trace.v1.TraceService/Export

Or just make some real API calls to your app and check the SigNoz dashboard within 30–60 seconds.

✅ Collector Checklist​

# Is the collector running?
sudo systemctl status otelcol-contrib

# Is it listening on the right ports?
sudo ss -ltnp | grep ':4317'
sudo ss -ltnp | grep ':4318'

# Any errors?
sudo journalctl -u otelcol-contrib --since "5 minutes ago"

Expected result:

  • Collector service is active (running)
  • Port 4317 is listening
  • No repeated export errors in recent logs

Next: Connect Your Production Apps →

Read in Sequence​