Caddy SimpleTrace Plugin

A lightweight Caddy module for W3C Trace Context propagation and structured logging without the overhead of full OpenTelemetry integration.

Features

• Parses incoming traceparent HTTP headers (W3C Trace Context specification)
• Generates new trace IDs when none exist
• Creates unique span IDs for each request
• Propagates traceparent headers to upstream/proxied requests
• Adds trace context to Caddy’s structured logs
• Supports Google Cloud Logging (Stackdriver) format
• Respects and propagates sampling flags
• Zero external dependencies beyond Caddy

Why SimpleTrace?

Caddy’s built-in tracing directive includes the full OpenTelemetry stack, which can be heavy if you only need trace context in logs. SimpleTrace provides just the essentials:

• Trace ID propagation across service boundaries
• Structured logging with trace context
• Cloud logging platform integration
• Minimal performance overhead

Installation

Using xcaddy

xcaddy build --with github.com/yourusername/caddy-simpletrace

Using Docker

FROM caddy:builder AS builder
RUN xcaddy build \
    --with github.com/yourusername/caddy-simpletrace

FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Usage

Handler Ordering

Before using simpletrace, you must define its position in Caddy’s middleware chain using the global order directive:

{
    order simpletrace before rewrite
}

example.com {
    simpletrace
    reverse_proxy backend:8080
}

Recommended positions:

• order simpletrace first - Run before all other handlers (captures everything)
• order simpletrace before rewrite - Run early, before URL modifications
• order simpletrace before reverse_proxy - Run just before proxying requests

Basic Configuration

{
    order simpletrace before rewrite
}

example.com {
    simpletrace
    reverse_proxy backend:8080
}

This will:

1. Parse or generate trace IDs
2. Add trace context to logs in OpenTelemetry format (default)
3. Propagate traceparent headers to backend

Log output (OpenTelemetry format - default):

{
  "level": "info",
  "ts": 1702123456.789,
  "msg": "handled request",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "trace_sampled": true
}

Supported Log Formats

SimpleTrace supports multiple log format conventions to integrate with different observability platforms:

OpenTelemetry (default)

simpletrace {
    format otel
}

Fields: trace_id, span_id, trace_sampled, parent_span_id

Compatible with: OpenTelemetry Collector, Jaeger, Grafana Tempo, most modern observability tools

Grafana Tempo

simpletrace {
    format tempo
}

Fields: traceID, spanID, traceSampled, parentSpanID (camelCase)

Log output:

{
  "traceID": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanID": "00f067aa0ba902b7",
  "traceSampled": true
}

Compatible with: Grafana Tempo, Grafana Loki with Tempo integration

Google Cloud Logging (Stackdriver) Format

{
    order simpletrace before rewrite
}

example.com {
    simpletrace {
        format stackdriver
        project_id your-gcp-project-id
    }
    reverse_proxy backend:8080
}

Using environment variables:

simpletrace {
    format stackdriver
    project_id {env.GOOGLE_CLOUD_PROJECT}
}

Auto-detect from environment (default):

simpletrace {
    format stackdriver
    # project_id automatically defaults to {env.GOOGLE_CLOUD_PROJECT}
}

When project_id is omitted and format is stackdriver or gcp, the plugin automatically uses the GOOGLE_CLOUD_PROJECT environment variable, which is the canonical environment variable set by Google Cloud Platform in Cloud Run, GKE, App Engine, and other services.

Log output:

{
  "level": "info",
  "ts": 1702123456.789,
  "msg": "handled request",
  "logging.googleapis.com/trace": "projects/your-gcp-project-id/traces/4bf92f3577b34da6a3ce929d0e0e4736",
  "logging.googleapis.com/spanId": "00f067aa0ba902b7",
  "logging.googleapis.com/trace_sampled": true
}

Benefits with Stackdriver format:

• Automatic trace correlation in Google Cloud Logging
• Integration with Cloud Trace (for sampled traces)
• Clickable trace links in GCP Console

Elastic Common Schema (ECS)

simpletrace {
    format ecs
}

Fields: trace.id, span.id, trace.sampled, span.parent_id (dot notation)

Log output:

{
  "trace.id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span.id": "00f067aa0ba902b7",
  "trace.sampled": true
}

Compatible with: Elasticsearch, Kibana, Elastic APM

Datadog

simpletrace {
    format datadog
}

Aliases: dd

Fields: dd.trace_id, dd.span_id, dd.sampled, dd.parent_id

Log output:

{
  "dd.trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "dd.span_id": "00f067aa0ba902b7",
  "dd.sampled": true
}

Compatible with: Datadog APM

Configuration Options

Format Options

simpletrace {
    format <format-name>
    project_id <gcp-project-id>  # Only for stackdriver/gcp format
}

Available formats:

• otel (default) - OpenTelemetry semantic conventions
• tempo - Grafana Tempo camelCase format
• stackdriver or gcp - Google Cloud Logging format
• ecs - Elastic Common Schema
• datadog or dd - Datadog APM format

Standard Format Fields

OpenTelemetry (default):

• trace_id - 32-character hex trace identifier
• span_id - 16-character hex span identifier
• trace_sampled - boolean indicating if trace should be recorded
• parent_span_id - (optional) parent span ID from incoming request

Tempo:

• traceID, spanID, traceSampled, parentSpanID (camelCase variants)

Stackdriver:

• logging.googleapis.com/trace - Full trace resource path (when project_id provided)
• logging.googleapis.com/spanId - Span identifier
• logging.googleapis.com/trace_sampled - Sampling flag
• parent_span_id - (optional) parent span ID

ECS:

• trace.id, span.id, trace.sampled, span.parent_id (dot notation)

Datadog:

• dd.trace_id, dd.span_id, dd.sampled, dd.parent_id (dd prefix)

How It Works

Trace Context Flow

1. Incoming Request

• If traceparent header exists: parse trace ID, parent span ID, and flags
• If no header: generate new trace ID with random sampling

1. Request Processing

• Generate new span ID for this request
• Add trace context fields to Caddy’s log context
• Create new traceparent header with current span as parent

1. Outgoing Request

• Propagate traceparent header to upstream services
• Downstream services can continue the trace

1. Logging

• All Caddy access logs for this request include trace context
• Enables correlation across service boundaries

Traceparent Header Format

Follows W3C Trace Context specification:

traceparent: 00-{trace-id}-{parent-span-id}-{flags}

Example:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

• 00 - Version
• 4bf92f3577b34da6a3ce929d0e0e4736 - Trace ID (32 hex chars)
• 00f067aa0ba902b7 - Parent Span ID (16 hex chars)
• 01 - Flags (01 = sampled, 00 = not sampled)

Sampling Flag

The least significant bit of the flags byte indicates sampling:

• 01 - Sampled: Trace should be recorded by tracing backends
• 00 - Not sampled: Trace context propagated but not recorded

When using Google Cloud Logging, the trace_sampled field controls whether traces are sent to Cloud Trace for analysis.

Example Configurations

Using with Snippets

{
    order simpletrace before rewrite
}

(common) {
    simpletrace {
        format tempo
    }
    encode gzip
}

example.com {
    import common
    reverse_proxy frontend:3000
}

api.example.com {
    import common
    reverse_proxy api:8080
}

Multi-service Setup

{
    order simpletrace before rewrite
}

# Frontend service
frontend.example.com {
    simpletrace {
        format stackdriver
        project_id my-project
    }
    reverse_proxy frontend-app:3000
}

# API service
api.example.com {
    simpletrace {
        format stackdriver
        project_id my-project
    }
    reverse_proxy api-app:8080
}

All services in your architecture can use SimpleTrace to maintain trace context across the entire request flow.

Development Setup

{
    order simpletrace before rewrite
}

localhost:8080 {
    simpletrace
    log {
        output stdout
        format json
    }
    reverse_proxy localhost:3000
}

Production with Cloud Logging

{
    order simpletrace before rewrite
}

example.com {
    simpletrace {
        format stackdriver
        project_id production-project-123
    }
    
    log {
        output stdout
        format json
    }
    
    reverse_proxy backend:8080
}

Grafana Loki + Tempo Setup

{
    order simpletrace before rewrite
}

example.com {
    simpletrace {
        format tempo
    }
    
    log {
        output stdout
        format json
    }
    
    reverse_proxy backend:8080
}

With this configuration, logs sent to Loki will automatically link to traces in Tempo when both use the same trace IDs.

When running on Google Cloud (GKE, Cloud Run, etc.), logs are automatically ingested by Cloud Logging and traces are correlated.

Comparison with Built-in Tracing

Feature	SimpleTrace	Built-in tracing
Trace ID propagation	✅	✅
Log augmentation	✅	✅
OpenTelemetry SDK	❌	✅
OTLP export	❌	✅
Span export	❌	✅
Performance overhead	Minimal	Moderate
Configuration complexity	Simple	Complex
Use case	Logging only	Full observability

Use SimpleTrace when you only need trace context in logs. Use the built-in tracing directive when you need full distributed tracing with span export to collectors like Jaeger, Zipkin, or Cloud Trace.

Troubleshooting

“directive ‘simpletrace’ is not an ordered HTTP handler” error

Add the order directive to your global options:

{
    order simpletrace before rewrite
}

This is required when using simpletrace in snippets or when Caddy cannot automatically determine the handler order.

Logs don’t include trace fields

Ensure you’re using JSON log format:

log {
    format json
}

Traces not appearing in Cloud Trace

1. Verify stackdriver directive includes your project ID
2. Check that trace_sampled is true in logs
3. Ensure Cloud Logging API is enabled
4. Verify service account has roles/cloudtrace.agent permission

Invalid traceparent headers

SimpleTrace validates incoming headers. Invalid formats are rejected and new trace IDs are generated. Check logs for trace ID generation patterns - consistent new traces may indicate malformed incoming headers.

Resources

• W3C Trace Context Specification
• Google Cloud Logging Structured Logging
• Caddy Documentation
• Caddy Module Development