Skip to content

Using otlphttp exporter for self metrics #2093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rafaelwestphal wants to merge 50 commits into
base: master
Choose a base branch
from
Open

Using otlphttp exporter for self metrics #2093

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
b066750
Using otlphttp exporter for self metrics
rafaelwestphal Oct 15, 2025
3e29132
Adding the necessary transformers for agent metric
rafaelwestphal Oct 16, 2025
0969945
Removing extensions when unnecessar
rafaelwestphal Oct 16, 2025
06ae486
Include selfmetrics
rafaelwestphal Oct 17, 2025
6b8674d
Fixing golden
rafaelwestphal Oct 17, 2025
8901822
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Oct 17, 2025
486f335
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Oct 21, 2025
2ca0535
MErging file
rafaelwestphal Oct 22, 2025
68f9c4d
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Oct 22, 2025
27f1487
Fix lint
rafaelwestphal Oct 22, 2025
7c6ab48
Refactoring how to run featureflag tests
rafaelwestphal Oct 23, 2025
0203366
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Oct 31, 2025
fb747cc
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 4, 2025
3d60aa6
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 5, 2025
75b3426
Resuing some processor
rafaelwestphal Nov 5, 2025
dad0225
Refactoring code and removing comments
rafaelwestphal Nov 12, 2025
d06c53e
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 12, 2025
4b1d06d
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 12, 2025
958fb37
Copying host.id to instance_id to bypass bug 459468648
rafaelwestphal Nov 12, 2025
49d5d56
upDating golden
rafaelwestphal Nov 13, 2025
0f1b0d2
Fixing tranformation test
rafaelwestphal Nov 13, 2025
72f0623
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 19, 2025
193dd21
Adding system specific changes
rafaelwestphal Nov 19, 2025
edfe554
formating
rafaelwestphal Nov 19, 2025
db9427a
Readjust when the change type from dobule to int
rafaelwestphal Nov 27, 2025
a87cb54
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 27, 2025
87f07ee
update golden
rafaelwestphal Nov 27, 2025
d85b391
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Nov 28, 2025
aab9b17
Fixing merge
rafaelwestphal Nov 28, 2025
ecbc31f
Removing ProjectName since we are obatinng inside the aux funt
rafaelwestphal Nov 28, 2025
939da8e
Removing unnecessary ProjectNaME
rafaelwestphal Dec 1, 2025
9f32772
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Dec 1, 2025
460ca97
update golden
rafaelwestphal Dec 1, 2025
1cf2746
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Dec 5, 2025
06a6d3f
Reverting change to scaler transformation
rafaelwestphal Dec 5, 2025
765ab8d
Add the other changes
rafaelwestphal Dec 5, 2025
3b0a878
Reveting bad scalar changew
rafaelwestphal Dec 5, 2025
0d45cb4
Adding metricstarttime when using otlphtrp
rafaelwestphal Dec 8, 2025
d4224ac
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Dec 8, 2025
936ae3d
Adding function
rafaelwestphal Dec 8, 2025
d387e0a
Added metricstarttime exporter to the end of the pipelines
rafaelwestphal Dec 11, 2025
498bb89
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Jan 27, 2026
7bb8400
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Jan 27, 2026
6653367
reslve bad merge
rafaelwestphal Jan 27, 2026
c50b4a7
Fix golden files
rafaelwestphal Jan 27, 2026
429f8c7
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Feb 3, 2026
42c84d3
Fix the golden testdata
rafaelwestphal Feb 3, 2026
ed49fd2
Fixing golden
rafaelwestphal Feb 3, 2026
5b6fd75
Removing workaround for bug
rafaelwestphal Feb 3, 2026
9c7dc7f
Merge branch 'master' into westphalrafael-agent-otlp
rafaelwestphal Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions confgenerator/agentmetrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
package confgenerator

import (
"context"
"fmt"
"path/filepath"
"time"
Expand Down Expand Up @@ -56,9 +57,9 @@ var grpcToHTTPStatus = map[string]string{
"DEADLINE_EXCEEDED": "504",
}

func (r AgentSelfMetrics) AddSelfMetricsPipelines(receiverPipelines map[string]otel.ReceiverPipeline, pipelines map[string]otel.Pipeline) {
func (r AgentSelfMetrics) AddSelfMetricsPipelines(receiverPipelines map[string]otel.ReceiverPipeline, pipelines map[string]otel.Pipeline, ctx context.Context) {
// Receiver pipelines names should have 1 underscore to avoid collision with user configurations.
receiverPipelines["agent_prometheus"] = r.PrometheusMetricsPipeline()
receiverPipelines["agent_prometheus"] = r.PrometheusMetricsPipeline(ctx)

// Pipeline names should have no underscores to avoid collision with user configurations.
pipelines["otel"] = otel.Pipeline{
Expand All @@ -79,15 +80,15 @@ func (r AgentSelfMetrics) AddSelfMetricsPipelines(receiverPipelines map[string]o
Processors: r.LoggingMetricsPipelineProcessors(),
}

receiverPipelines["ops_agent"] = r.OpsAgentPipeline()
receiverPipelines["ops_agent"] = r.OpsAgentPipeline(ctx)
pipelines["opsagent"] = otel.Pipeline{
Type: "metrics",
ReceiverPipelineName: "ops_agent",
}
}

func (r AgentSelfMetrics) PrometheusMetricsPipeline() otel.ReceiverPipeline {
return otel.ReceiverPipeline{
func (r AgentSelfMetrics) PrometheusMetricsPipeline(ctx context.Context) otel.ReceiverPipeline {
return ConvertGCMSystemExporterToOtlpExporter(otel.ReceiverPipeline{
Receiver: otel.Component{
Type: "prometheus",
Config: map[string]interface{}{
Expand Down Expand Up @@ -128,7 +129,7 @@ func (r AgentSelfMetrics) PrometheusMetricsPipeline() otel.ReceiverPipeline {
),
},
},
}
}, ctx)
}

func (r AgentSelfMetrics) OtelPipelineProcessors() []otel.Component {
Expand Down Expand Up @@ -303,18 +304,18 @@ func (r AgentSelfMetrics) LoggingMetricsPipelineProcessors() []otel.Component {
}
}

func (r AgentSelfMetrics) OpsAgentPipeline() otel.ReceiverPipeline {
receiver_config := map[string]any{
func (r AgentSelfMetrics) OpsAgentPipeline(ctx context.Context) otel.ReceiverPipeline {
receiverConfig := map[string]any{
"include": []string{
filepath.Join(r.OtelRuntimeDir, "enabled_receivers_otlp.json"),
filepath.Join(r.OtelRuntimeDir, "feature_tracking_otlp.json")},
"replay_file": true,
"poll_interval": time.Duration(60 * time.Second).String(),
}
return otel.ReceiverPipeline{
return ConvertGCMSystemExporterToOtlpExporter(otel.ReceiverPipeline{
Receiver: otel.Component{
Type: "otlpjsonfile",
Config: receiver_config,
Config: receiverConfig,
},
ExporterTypes: map[string]otel.ExporterType{
"metrics": otel.System,
Expand All @@ -324,7 +325,7 @@ func (r AgentSelfMetrics) OpsAgentPipeline() otel.ReceiverPipeline {
otel.Transform("metric", "datapoint", []ottl.Statement{"set(time, Now())"}),
},
},
}
}, ctx)
}

// intentionally not registered as a component because this is not created by users
9 changes: 4 additions & 5 deletions confgenerator/confgenerator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,6 @@ func ConvertGCMSystemExporterToOtlpExporter(pipeline otel.ReceiverPipeline, ctx

func ConvertToOtlpExporter(pipeline otel.ReceiverPipeline, ctx context.Context, isPrometheus bool, isSystem bool) otel.ReceiverPipeline {
expOtlpExporter := experimentsFromContext(ctx)["otlp_exporter"]
resource, _ := platform.FromContext(ctx).GetResource()
if !expOtlpExporter {
return pipeline
}
Expand All @@ -91,7 +90,6 @@ func ConvertToOtlpExporter(pipeline otel.ReceiverPipeline, ctx context.Context,
return pipeline
}
pipeline.ExporterTypes["metrics"] = otel.OTLP
pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.GCPProjectID(resource.ProjectName()))
if isSystem {
pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.MetricsRemoveInstrumentationLibraryLabelsAttributes())
pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.MetricsRemoveServiceAttributes())
Expand All @@ -100,9 +98,7 @@ func ConvertToOtlpExporter(pipeline otel.ReceiverPipeline, ctx context.Context,
// b/476109839: For prometheus metrics using the OTLP exporter. The dots "." in the metric name are NOT replaced with underscore "_".
// This is diffrent from the GMP endpoint.
if isPrometheus {

pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.MetricUnknownCounter())
pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.MetricStartTime())
// If a metric already has a domain, it will not be considered a prometheus metric by the UTR endpoint unless we add the prefix.
// This behavior is the same as the GCM/GMP exporters.
pipeline.Processors["metrics"] = append(pipeline.Processors["metrics"], otel.MetricsTransform(otel.AddPrefix("prometheus.googleapis.com")))
Expand Down Expand Up @@ -191,7 +187,8 @@ func (uc *UnifiedConfig) GenerateOtelConfig(ctx context.Context, outDir, stateDi
OtelRuntimeDir: outDir,
OtelLogging: uc.Logging.Service.OTelLogging,
}
agentSelfMetrics.AddSelfMetricsPipelines(receiverPipelines, pipelines)
agentSelfMetrics.AddSelfMetricsPipelines(receiverPipelines, pipelines, ctx)
resource, _ := p.GetResource()

otelConfig, err := otel.ModularConfig{
LogLevel: uc.getOTelLogLevel(),
Expand All @@ -214,6 +211,8 @@ func (uc *UnifiedConfig) GenerateOtelConfig(ctx context.Context, outDir, stateDi
// The OTLP exporter doesn't batch by default like the googlecloud.* exporters.
// We need this to avoid the API point limits.
"metrics": {
otel.GCPProjectID(resource.ProjectName()),
otel.MetricStartTime(),
otel.BatchProcessor(200, 200, "200ms"),
},
// Batching logs improves log export performance.
Expand Down
118 changes: 76 additions & 42 deletions confgenerator/testdata/goldens/combined-receiver_otlp_otlphttp/golden/linux-gpu/otel.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,4 @@
exporters:
googlecloud:
metric:
instrumentation_library_labels: false
prefix: ""
resource_filters: []
service_resource_labels: false
skip_create_descriptor: true
user_agent: Google-Cloud-Ops-Agent-Metrics/latest (BuildDistro=build_distro;Platform=linux;ShortName=linux_platform;ShortVersion=linux_platform_version)
googlecloud/otel:
metric:
instrumentation_library_labels: true
Expand All @@ -27,7 +19,7 @@ processors:
agentmetrics/hostmetrics_0:
blank_label_metrics:
- system.cpu.utilization
batch/otlphttp/otlp_metrics_0:
batch/otlphttp/otlp_metrics_2:
send_batch_max_size: 200
send_batch_size: 200
timeout: 200ms
Expand Down Expand Up @@ -103,7 +95,7 @@ processors:
- location
interval/loggingmetrics_7:
interval: 1m
metricstarttime/otlp_5:
metricstarttime/otlphttp/otlp_metrics_1:
strategy: subtract_initial_point
metricstransform/fluentbit_1:
transforms:
Expand Down Expand Up @@ -604,23 +596,13 @@ processors:
include: ^(.*)$$
match_type: regexp
new_name: agent.googleapis.com/$${1}
metricstransform/otlp_6:
metricstransform/otlp_4:
transforms:
- action: update
include: ^(.*)$$
match_type: regexp
new_name: prometheus.googleapis.com/$${1}
resource/hostmetrics_1_1:
attributes:
- action: insert
key: gcp.project_id
value: test-project
resource/hostmetrics_3:
attributes:
- action: insert
key: gcp.project_id
value: test-project
resource/otlp_3:
resource/otlphttp/otlp_metrics_0:
attributes:
- action: insert
key: gcp.project_id
Expand All @@ -645,13 +627,13 @@ processors:
- delete_key(resource.attributes, "service.instance.id")
- delete_key(resource.attributes, "server.port")
- delete_key(resource.attributes, "url.scheme")
transform/hostmetrics_1_2:
transform/agent_prometheus_1:
metric_statements:
- context: scope
statements:
- set(name, "")
- set(version, "")
transform/hostmetrics_1_3:
transform/agent_prometheus_2:
metric_statements:
- context: resource
error_mode: silent
Expand All @@ -660,13 +642,28 @@ processors:
- delete_key(attributes, "service.instance.id")
- delete_key(attributes, "service.namespace")
- delete_key(attributes, "service.version")
transform/hostmetrics_4:
transform/hostmetrics_1_1:
metric_statements:
- context: scope
statements:
- set(name, "")
- set(version, "")
transform/hostmetrics_5:
transform/hostmetrics_1_2:
metric_statements:
- context: resource
error_mode: silent
statements:
- delete_key(attributes, "service.name")
- delete_key(attributes, "service.instance.id")
- delete_key(attributes, "service.namespace")
- delete_key(attributes, "service.version")
transform/hostmetrics_3:
metric_statements:
- context: scope
statements:
- set(name, "")
- set(version, "")
transform/hostmetrics_4:
metric_statements:
- context: resource
error_mode: silent
Expand Down Expand Up @@ -702,6 +699,21 @@ processors:
- context: datapoint
statements:
- set(time, Now())
transform/ops_agent_1:
metric_statements:
- context: scope
statements:
- set(name, "")
- set(version, "")
transform/ops_agent_2:
metric_statements:
- context: resource
error_mode: silent
statements:
- delete_key(attributes, "service.name")
- delete_key(attributes, "service.instance.id")
- delete_key(attributes, "service.namespace")
- delete_key(attributes, "service.version")
transform/otel_0:
error_mode: ignore
metric_statements:
Expand All @@ -718,7 +730,7 @@ processors:
- set(attributes["cluster"], "__gce__") where "__gce__" != nil and resource.attributes["cloud.platform"] == "gcp_compute_engine"
- set(attributes["instance_name"], resource.attributes["host.name"]) where resource.attributes["host.name"] != nil and resource.attributes["cloud.platform"] == "gcp_compute_engine"
- set(attributes["machine_type"], resource.attributes["host.type"]) where resource.attributes["host.type"] != nil and resource.attributes["cloud.platform"] == "gcp_compute_engine"
transform/otlp_4:
transform/otlp_3:
error_mode: ignore
metric_statements:
- context: metric
Expand Down Expand Up @@ -779,42 +791,51 @@ service:
- agentmetrics/hostmetrics_0
- filter/hostmetrics_1
- metricstransform/hostmetrics_2
- resource/hostmetrics_3
- transform/hostmetrics_3
- transform/hostmetrics_4
- transform/hostmetrics_5
- filter/default__pipeline_hostmetrics_0
- resourcedetection/_global_0
- batch/otlphttp/otlp_metrics_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- hostmetrics/hostmetrics
metrics/default__pipeline_hostmetrics_1:
exporters:
- otlphttp/otlp
processors:
- metricstransform/hostmetrics_1_0
- resource/hostmetrics_1_1
- transform/hostmetrics_1_1
- transform/hostmetrics_1_2
- transform/hostmetrics_1_3
- filter/default__pipeline_hostmetrics_1_0
- resourcedetection/_global_0
- batch/otlphttp/otlp_metrics_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- nvml/hostmetrics_1
metrics/fluentbit:
exporters:
- googlecloud
- otlphttp/otlp
processors:
- transform/agent_prometheus_0
- transform/agent_prometheus_1
- transform/agent_prometheus_2
- filter/fluentbit_0
- metricstransform/fluentbit_1
- resourcedetection/_global_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- prometheus/agent_prometheus
metrics/loggingmetrics:
exporters:
- googlecloud
- otlphttp/otlp
processors:
- transform/agent_prometheus_0
- transform/agent_prometheus_1
- transform/agent_prometheus_2
- transform/loggingmetrics_0
- filter/loggingmetrics_1
- filter/loggingmetrics_2
Expand All @@ -825,26 +846,39 @@ service:
- interval/loggingmetrics_7
- metricstransform/loggingmetrics_8
- resourcedetection/_global_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- prometheus/agent_prometheus
metrics/opsagent:
exporters:
- googlecloud
- otlphttp/otlp
processors:
- transform/ops_agent_0
- transform/ops_agent_1
- transform/ops_agent_2
- resourcedetection/_global_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- otlpjsonfile/ops_agent
metrics/otel:
exporters:
- googlecloud
- otlphttp/otlp
processors:
- transform/agent_prometheus_0
- transform/agent_prometheus_1
- transform/agent_prometheus_2
- transform/otel_0
- filter/otel_1
- filter/otel_2
- metricstransform/otel_3
- resourcedetection/_global_0
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- prometheus/agent_prometheus
metrics/otlp_otlp:
Expand All @@ -854,11 +888,11 @@ service:
- resourcedetection/otlp_0
- transform/otlp_1
- groupbyattrs/otlp_2
- resource/otlp_3
- transform/otlp_4
- metricstarttime/otlp_5
- metricstransform/otlp_6
- batch/otlphttp/otlp_metrics_0
- transform/otlp_3
- metricstransform/otlp_4
- resource/otlphttp/otlp_metrics_0
- metricstarttime/otlphttp/otlp_metrics_1
- batch/otlphttp/otlp_metrics_2
receivers:
- otlp/otlp
traces/traces_otlp_otlp:
Expand Down
Loading
Loading