Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 53 additions & 60 deletions docs/reference/opentelemetry-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,8 @@ applies_to:
Integration with the OpenTelemetry Tracing API was added as experimental in v3.34.0. Integration with the OpenTelemetry Metrics API was added as experimental in v3.45.0.
::::


The Elastic APM OpenTelemetry bridge allows one to use the vendor-neutral [OpenTelemetry API](https://opentelemetry.io/docs/instrumentation/js/) ([`@opentelemetry/api`](https://www.npmjs.com/package/@opentelemetry/api)) in your code, and have the Elastic Node.js APM agent handle those API calls. This allows one to use the Elastic APM agent for tracing and metrics without any vendor lock-in to the APM agent’s own [public API](/reference/api.md) when adding manual tracing or custom metrics.


## Using the OpenTelemetry Tracing API [otel-tracing-api]

① First, you will need to add the Elastic APM agent and OpenTelemetry API dependencies to your project. The minimum required OpenTelemetry API version is 1.0.0; see [the OpenTelemetry compatibility section](/reference/supported-technologies.md#compatibility-opentelemetry) for the current maximum supported API version. For example:
Expand Down Expand Up @@ -52,33 +50,33 @@ require('elastic-apm-node').start({

// Application code ...
```

1. Alternatively, you can use `apiKey: '<your API key>'`.

See [the full APM agent configuration reference](/reference/configuration.md) for other configuration options.

③ Finally, you can use the [OpenTelemetry API](https://open-telemetry.github.io/opentelemetry-js/modules/_opentelemetry_api.html) for any manual tracing in your code. For example, the following script uses [Tracer#startActiveSpan()](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.Tracer.html#startactivespan) to trace an outgoing HTTPS request:

```js
const https = require('https')
const otel = require('@opentelemetry/api')
const tracer = otel.trace.getTracer('trace-https-request')
const https = require('https');
const otel = require('@opentelemetry/api');
const tracer = otel.trace.getTracer('trace-https-request');

tracer.startActiveSpan('makeRequest', span => {
tracer.startActiveSpan('makeRequest', (span) => {
https.get('https://httpstat.us/200', (response) => {
console.log('STATUS:', response.statusCode)
const body = []
response.on('data', (chunk) => body.push(chunk))
console.log('STATUS:', response.statusCode);
const body = [];
response.on('data', (chunk) => body.push(chunk));
response.on('end', () => {
console.log('BODY:', body.toString())
span.end()
})
})
})
console.log('BODY:', body.toString());
span.end();
});
});
});
```

The APM agent source code repository includes [some examples using the OpenTelemetry tracing bridge](https://github.com/elastic/apm-agent-nodejs/tree/main/examples/opentelemetry-bridge).


## Using the OpenTelemetry Metrics API [otel-metrics-api]

① As above, install the needed dependencies. The minimum required OpenTelemetry API version is 1.3.0 (the version when metrics were added); see [the OpenTelemetry compatibility section](/reference/supported-technologies.md#compatibility-opentelemetry) for the current maximum supported API version. For example:
Expand All @@ -100,62 +98,62 @@ node my-app.js

```js
// otel-metrics-hello-world.js <1>
const { createServer } = require('http')
const otel = require('@opentelemetry/api')
const { createServer } = require('http');
const otel = require('@opentelemetry/api');

const meter = otel.metrics.getMeter('my-meter')
const numReqs = meter.createCounter('num_requests', { description: 'number of HTTP requests' })
const meter = otel.metrics.getMeter('my-meter');
const numReqs = meter.createCounter('num_requests', {
description: 'number of HTTP requests',
});

const server = createServer((req, res) => {
numReqs.add(1)
req.resume()
numReqs.add(1);
req.resume();
req.on('end', () => {
res.end('pong\n')
})
})
res.end('pong\n');
});
});
server.listen(3000, () => {
console.log('listening at http://127.0.0.1:3000/')
})
console.log('listening at http://127.0.0.1:3000/');
});
```

1. The full example is [here](https://github.com/elastic/apm-agent-nodejs/blob/main/examples/opentelemetry-metrics/otel-metrics-hello-world.js).



### Using the OpenTelemetry Metrics SDK [otel-metrics-sdk]

The Elastic APM agent also supports exporting metrics to APM server when the OpenTelemetry Metrics **SDK** is being used directly. You might want to use the OpenTelemetry Metrics SDK to use a [`View`](https://opentelemetry.io/docs/reference/specification/metrics/sdk/#view) to configure histogram bucket sizes, to setup a Prometheus exporter, or for other reasons. For example:

```js
// use-otel-metrics-sdk.js <1>
const otel = require('@opentelemetry/api')
const { MeterProvider } = require('@opentelemetry/sdk-metrics')
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus')
const otel = require('@opentelemetry/api');
const { MeterProvider } = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

const exporter = new PrometheusExporter({ host: '127.0.0.1', port: 3001 })
const meterProvider = new MeterProvider()
meterProvider.addMetricReader(exporter)
otel.metrics.setGlobalMeterProvider(meterProvider)
const exporter = new PrometheusExporter({ host: '127.0.0.1', port: 3001 });
const meterProvider = new MeterProvider({
readers: [exporter],
});
otel.metrics.setGlobalMeterProvider(meterProvider);

const meter = otel.metrics.getMeter('my-meter')
const latency = meter.createHistogram('latency', { description: 'Response latency (s)' })
const meter = otel.metrics.getMeter('my-meter');
const latency = meter.createHistogram('latency', {
description: 'Response latency (s)',
});
// ...
```

1. The full example is [here](https://github.com/elastic/apm-agent-nodejs/blob/main/examples/opentelemetry-metrics/use-otel-metrics-sdk.js).



### OpenTelemetry Metrics configuration [otel-metrics-conf]

A few configuration options can be used to control OpenTelemetry Metrics support.

* Specific metrics names can be filtered out via the [`disableMetrics`](/reference/configuration.md#disable-metrics) configuration option.
* Integration with the OpenTelemetry Metrics API can be disabled via the [`disableInstrumentations: '@opentelemetry/api'`](/reference/configuration.md#disable-instrumentations) configuration option.
* Integration with the OpenTelemetry Metrics SDK can be disabled via the [`disableInstrumentations: '@opentelemetry/sdk-metrics'`](/reference/configuration.md#disable-instrumentations) configuration option.
* All metrics support in the APM agent can be disabled via the [`metricsInterval: '0s'`](/reference/configuration.md#metrics-interval) configuration option.
* The default histogram bucket boundaries are different from the OpenTelemetry default, to provide better resolution. The boundaries used by the APM agent can be configured with the [`customMetricsHistogramBoundaries`](/reference/configuration.md#custom-metrics-histogram-boundaries) configuration option.

- Specific metrics names can be filtered out via the [`disableMetrics`](/reference/configuration.md#disable-metrics) configuration option.
- Integration with the OpenTelemetry Metrics API can be disabled via the [`disableInstrumentations: '@opentelemetry/api'`](/reference/configuration.md#disable-instrumentations) configuration option.
- Integration with the OpenTelemetry Metrics SDK can be disabled via the [`disableInstrumentations: '@opentelemetry/sdk-metrics'`](/reference/configuration.md#disable-instrumentations) configuration option.
- All metrics support in the APM agent can be disabled via the [`metricsInterval: '0s'`](/reference/configuration.md#metrics-interval) configuration option.
- The default histogram bucket boundaries are different from the OpenTelemetry default, to provide better resolution. The boundaries used by the APM agent can be configured with the [`customMetricsHistogramBoundaries`](/reference/configuration.md#custom-metrics-histogram-boundaries) configuration option.

## Bridge architecture [otel-architecture]

Expand All @@ -168,33 +166,28 @@ The only difference, from the user’s point of view, is in the setup of tracing
<hr>
The OpenTelemetry Metrics support, is slightly different. If your code uses just the Metrics **API**, then the APM agent provides a full MeterProvider so that metrics are accumulated and sent to APM server. If your code uses the Metrics **SDK**, then the APM agents adds a MetricReader to your MeterProvider to send metrics on to APM server. This allows you to use the APM agent as either an easy setup for using metrics or in conjunction with your existing OpenTelemetry Metrics configuration.


## Caveats [otel-caveats]

Not all features of the OpenTelemetry API are supported. This section describes any limitations and differences.


#### Tracing [otel-caveats-tracing]

* Span Link Attributes. Adding links when [starting a span](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.Tracer.html) is supported, but any added span link **attributes** are silently dropped.
* Span events ([`Span#addEvent()`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.Span.html#addevent)) are not currently supported. Events will be silently dropped.
* [Propagating baggage](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_api._opentelemetry_api.PropagationAPI.html) within or outside the process is not supported. Baggage items are silently dropped.

- Span Link Attributes. Adding links when [starting a span](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.Tracer.html) is supported, but any added span link **attributes** are silently dropped.
- Span events ([`Span#addEvent()`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.Span.html#addevent)) are not currently supported. Events will be silently dropped.
- [Propagating baggage](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_api._opentelemetry_api.PropagationAPI.html) within or outside the process is not supported. Baggage items are silently dropped.

#### Metrics [otel-caveats-metrics]

* Metrics [exemplars](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exemplars) are not supported.
* [Summary metrics](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#summary-legacy) are not supported.
* [Exponential Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exponentialhistogram) are not yet supported.
* The `sum`, `count`, `min` and `max` within the OpenTelemetry histogram data are discarded.
* The default histogram bucket boundaries are different from the OpenTelemetry default. They provide better resolution. They can be configured with the [`customMetricsHistogramBoundaries`](/reference/configuration.md#custom-metrics-histogram-boundaries) configuration option.
* Metrics label names are dedotted (`s/\./_/g`) in APM server to avoid possible mapping collisions in Elasticsearch.
* The default [Aggregation Temporality](https://github.com/elastic/apm/blob/main/specs/agents/metrics-otel.md#aggregation-temporality) used differs from the OpenTelemetry default — preferring **delta**-temporality (nicer for visualizing in Kibana) to cumulative-temporality.
- Metrics [exemplars](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exemplars) are not supported.
- [Summary metrics](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#summary-legacy) are not supported.
- [Exponential Histograms](https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exponentialhistogram) are not yet supported.
- The `sum`, `count`, `min` and `max` within the OpenTelemetry histogram data are discarded.
- The default histogram bucket boundaries are different from the OpenTelemetry default. They provide better resolution. They can be configured with the [`customMetricsHistogramBoundaries`](/reference/configuration.md#custom-metrics-histogram-boundaries) configuration option.
- Metrics label names are dedotted (`s/\./_/g`) in APM server to avoid possible mapping collisions in Elasticsearch.
- The default [Aggregation Temporality](https://github.com/elastic/apm/blob/main/specs/agents/metrics-otel.md#aggregation-temporality) used differs from the OpenTelemetry default — preferring **delta**-temporality (nicer for visualizing in Kibana) to cumulative-temporality.

Metrics support requires an APM server >=7.11 — for earlier APM server versions, metrics with label names including `.`, `*`, or `"` will get dropped.


#### Logs [otel-caveats-logs]

The OpenTelemetry Logs API is currently not support — only the Tracing and Metrics APIs.

6 changes: 3 additions & 3 deletions examples/opentelemetry-metrics/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"version": "1.0.0",
"private": true,
"dependencies": {
"@opentelemetry/api": "^1.4.1",
"@opentelemetry/exporter-prometheus": ">=0.41.0 <2",
"@opentelemetry/sdk-metrics": "^1.12.0",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/exporter-prometheus": ">=0.200.0",
"@opentelemetry/sdk-metrics": "^2.0.0",
"elastic-apm-node": "file:../.."
}
}
24 changes: 13 additions & 11 deletions examples/opentelemetry-metrics/use-otel-metrics-sdk.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,26 +29,28 @@ const { performance } = require('perf_hooks');
const otel = require('@opentelemetry/api');
const {
MeterProvider,
View,
ExplicitBucketHistogramAggregation,
AggregationType,
} = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

const exporter = new PrometheusExporter({ host: '127.0.0.1', port: 3001 });
const meterProvider = new MeterProvider({
views: [
new View({
{
instrumentName: 'latency',
aggregation: new ExplicitBucketHistogramAggregation(
// Use the same default buckets as in `prom-client` for comparison.
// This is to demonstrate using a View. The default buckets used by the
// APM agent would suffice as well.
[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
),
}),
aggregation: {
type: AggregationType.EXPLICIT_BUCKET_HISTOGRAM,
options: {
// Use the same default buckets as in `prom-client` for comparison.
// This is to demonstrate using a View. The default buckets used by the
// APM agent would suffice as well.
boundaries: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
},
},
},
],
readers: [exporter],
});
meterProvider.addMetricReader(exporter);
otel.metrics.setGlobalMeterProvider(meterProvider);
console.log('Prometheus metrics at http://127.0.0.1:3001/metrics');

Expand Down
12 changes: 5 additions & 7 deletions lib/instrumentation/modules/@opentelemetry/sdk-metrics.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ module.exports = function (mod, agent, { version, enabled }) {
// Minimum supported version is 1.11.0 because that version included the fix
// for side-effects from having two MetricReaders.
// https://github.com/open-telemetry/opentelemetry-js/issues/3664
if (!semver.satisfies(version, '>=1.11.0 <2', { includePrerelease: true })) {
if (!semver.satisfies(version, '>=2.0.0 <3', { includePrerelease: true })) {
log.debug(
'@opentelemetry/sdk-metrics@%s is not supported, skipping @opentelemetry/sdk-metrics instrumentation',
version,
Expand All @@ -56,15 +56,13 @@ module.exports = function (mod, agent, { version, enabled }) {
}

class ApmMeterProvider extends mod.MeterProvider {
constructor(...args) {
super(...args);
// We create a new metric reader for each new MeterProvider instance,
// because they shutdown independently -- they cannot be shared between
// multiple MeterProviders.
constructor(options = {}) {
const readers = options.readers ? [...options.readers] : [];
readers.push(createOTelMetricReader(agent));
log.trace(
'@opentelemetry/sdk-metrics ins: create Elastic APM MetricReader',
);
this.addMetricReader(createOTelMetricReader(agent));
super({ ...options, readers });
}
}
Object.defineProperty(mod, 'MeterProvider', {
Expand Down
20 changes: 10 additions & 10 deletions lib/opentelemetry-metrics/ElasticApmMetricExporter.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,8 @@ const { ExportResultCode } = require('@opentelemetry/core');
const {
AggregationTemporality,
InstrumentType,
ExplicitBucketHistogramAggregation,
SumAggregation,
LastValueAggregation,
DropAggregation,
DataPointType,
AggregationType,
} = require('@opentelemetry/sdk-metrics');
const { LRUCache } = require('lru-cache');

Expand Down Expand Up @@ -96,12 +93,15 @@ function fillIntakeHistogramSample(sample, otelDataPoint) {
class ElasticApmMetricExporter {
constructor(agent) {
this._agent = agent;
this._histogramAggregation = new ExplicitBucketHistogramAggregation(
this._agent._conf.customMetricsHistogramBoundaries,
);
this._sumAggregation = new SumAggregation();
this._lastValueAggregation = new LastValueAggregation();
this._dropAggregation = new DropAggregation();
this._histogramAggregation = {
type: AggregationType.EXPLICIT_BUCKET_HISTOGRAM,
options: {
boundaries: this._agent._conf.customMetricsHistogramBoundaries,
},
};
this._sumAggregation = { type: AggregationType.SUM };
this._lastValueAggregation = { type: AggregationType.LAST_VALUE };
this._dropAggregation = { type: AggregationType.DROP };
this._attrDropWarnCache = new LRUCache({ max: 1000 });
this._dataPointTypeDropWarnCache = new LRUCache({ max: 1000 });
}
Expand Down
5 changes: 3 additions & 2 deletions lib/opentelemetry-metrics/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,9 @@ function createOTelMetricReader(agent) {
}

function createOTelMeterProvider(agent) {
const meterProvider = new MeterProvider();
meterProvider.addMetricReader(createOTelMetricReader(agent));
const meterProvider = new MeterProvider({
readers: [createOTelMetricReader(agent)],
});
return meterProvider;
}

Expand Down
Loading
Loading