[Demo] Add OpAMP for Collector Monitoring

[strawgate]

One a penny, two a penny, Op Amp Server

Dashboard

opamp-dashboard.ndjson.zip

Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

extensions:
  opamp:
    instance_uid: 
    capabilities:
      reports_effective_config: true
    server:
      http:
        endpoint: http://host.docker.internal:8220/v1/opamp
        tls:
          insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
  extensions: [opamp]

Notes

See docs/opamp-howto.md for setup.

Using content-* indices for now because i dont know how to give fleet service token access to new indices

[mergify]

This pull request does not have a backport label. Could you fix it @strawgate? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[michel-laterman]

Cool, we has a previous OpAMP PoC: #3944 (which is now very dated)
We tried wiring some things to allow an enrolled agent to checkin with opamp and found some areas of the spec lacking (I've worked to close the gaps in the spec).

As far as implementation goes, the current challenge will likely be a lack of long-poll support for http connections: open-telemetry/opamp-spec#245.

[github-actions]

🔍 Preview links for changed docs

• docs/opamp-howto.md

[strawgate]

Example Document:

{
    "@timestamp": "2025-12-18T03:42:39.799568Z",
    "opamp": {
      "agent": {
        "instance_uid": "8c7e94238d62461f87e23d233e292650",
        "type": "otelcol-contrib",
        "version": "0.142.0",
        "identifying_attributes": {
          "service.instance.id": "8c7e9423-8d62-461f-87e2-3d233e292650",
          "service.name": "otelcol-contrib",
          "service.version": "0.142.0"
        },
        "non_identifying_attributes": {
          "host.arch": "arm64",
          "host.name": "99eed58a9539",
          "os.description": " ",
          "os.type": "linux"
        }
      },
      "status": "degraded",
      "health": {
        "healthy": false,
        "status": "StatusOK",
        "status_time_unix_nano": 1766029305389184500,
        "component_health_map": {
          "extensions": {
            "healthy": true,
            "status": "StatusOK",
            "status_time_unix_nano": 1766029305388237600,
            "component_health_map": {
              "extension:opamp": {
                "healthy": true,
                "status": "StatusOK",
                "status_time_unix_nano": 1766029305388237600
              }
            }
          },
          "pipeline:logs": {
            "healthy": true,
            "status_time_unix_nano": 1766029305389184500,
            "component_health_map": {
              "exporter:debug": {
                "healthy": true,
                "status_time_unix_nano": 1766029305389056000,
                "status": "StatusOK"
              },
              "receiver:otlp": {
                "healthy": true,
                "status_time_unix_nano": 1766029305389184500,
                "status": "StatusOK"
              }
            },
            "status": "StatusOK"
          }
        },
        "start_time_unix_nano": 1766029305389184500
      },
      "capabilities": [
        "ReportsStatus",
        "ReportsEffectiveConfig",
        "ReportsHealth"
      ],
      "remote_config_status": {
        "status": "RemoteConfigStatuses_UNSET"
      },
      "sequence_num": 3,
      "effective_config": {
        "config_map": "....base64 config removed...,
        "hash": "...removed"
      }
    },
    "connection": {
      "last_seen": "2025-12-18T03:42:39.799568Z",
      "connected_at": "0001-01-01T00:00:00Z",
      "protocol": "http"
    },
    "agent": {
      "id": "8c7e94238d62461f87e23d233e292650",
      "name": "otelcol-contrib",
      "type": "opamp-agent",
      "version": "0.142.0"
    },
    "host": {
      "hostname": "99eed58a9539",
      "os": {
        "type": "linux"
      }
    }
  }

[cmacknz]

Put this in draft please :) This is very cool, but I don't want anyone thinking we are going to merge this as is and call it a day.

I would like us to figure out how much of the existing .fleet-agents and .fleet-policies structure we can use to try to avoid having to build and maintain central monitoring and management twice.

For central monitoring, we possibly do need a new index to contain the new op-amp data like effective config, but this also means the UI will have to be updated to query both that and .fleet-agents. The alternative is we add an opamp subobject to .fleet-agents and try to converge the agent protocol to fleet server to look more like opamp over time.

There is also no reason we can't do monitoring only Fleet with Elastic Agent today so we can try to benefit both use cases with any new index, e.g. agent could also start sending up effective config.