diff --git a/.gitbook/assets/otel/open-telemetry-kubernetes-operator.png b/.gitbook/assets/otel/open-telemetry-kubernetes-operator.png new file mode 100644 index 000000000..9315f857d Binary files /dev/null and b/.gitbook/assets/otel/open-telemetry-kubernetes-operator.png differ diff --git a/SUMMARY.md b/SUMMARY.md index 53159727c..d7dba026d 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -96,7 +96,8 @@ * [Overview](setup/otel/overview.md) * [Getting started](setup/otel/getting-started/README.md) * [Concepts](setup/otel/concepts.md) - * [Rancher & Kubernetes](setup/otel/getting-started/getting-started-k8s.md) + * [Kubernetes](setup/otel/getting-started/getting-started-k8s.md) + * [Kubernetes Operator](setup/otel/getting-started/getting-started-k8s-operator.md) * [Linux](setup/otel/getting-started/getting-started-linux.md) * [AWS Lambda](setup/otel/getting-started/getting-started-lambda.md) * [Open telemetry collector](setup/otel/collector.md) diff --git a/setup/otel/collector.md b/setup/otel/collector.md index d436187ba..97a7b80e8 100644 --- a/setup/otel/collector.md +++ b/setup/otel/collector.md @@ -81,7 +81,7 @@ exporters: compression: snappy ``` -{% hint type="warning" %} +{% hint style="warning" %} The OTLP HTTP endpoint for SUSE Observability is different from the OTLP endpoint. Use the [OTLP APIs](./otlp-apis.md) to find the correct URL. {% endhint %} diff --git a/setup/otel/getting-started/getting-started-k8s-operator.md b/setup/otel/getting-started/getting-started-k8s-operator.md new file mode 100644 index 000000000..4e9beb972 --- /dev/null +++ b/setup/otel/getting-started/getting-started-k8s-operator.md @@ -0,0 +1,306 @@ +--- +description: SUSE Observability +--- + +# Getting Started with Open Telemetry operator on Kubernetes + +Here is the setup we'll be creating, for an application that needs to be monitored: + +* The monitored application / workload running in cluster A, auto-instrumented by the operator +* The Open Telemetry operator in cluster A +* A collector created by the operator +* SUSE Observability running in cluster B, or SUSE Cloud Observability + +![Container instrumentation with Open Telemetry operator auto-instrumentation](/.gitbook/assets/otel/open-telemetry-kubernetes-operator.png) + +## Install the operator + +The Open Telemetry operator offers some extra features over the normal Kubernetes setup: +* It can auto-instrument your application pods for supported languages (Java, .NET, Python, Golang, Node.js), without having to modify the applications or docker images at all +* It can be dropped in as a replacement for the Prometheus operator and start scraping Prometheus exporter endpoints based on service and pod monitors + +### Create the namespace and a secret for the API key + +We'll install in the `open-telemetry` namespace and use the receiver API key generated during installation (see [here](/use/security/k8s-ingestion-api-keys.md#api-keys) where to find it): + +```bash +kubectl create namespace open-telemetry +kubectl create secret generic open-telemetry-collector \ + --namespace open-telemetry \ + --from-literal=API_KEY='' +``` + +### Configure & Install the operator + +The operator is installed with a Helm chart, so first configure the chart repository. + +```bash +helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts +``` + +Let's create a `otel-operator.yaml` file to configure the operator: + +{% code title="otel-operator.yaml" lineNumbers="true" %} +```yaml +# Add image pull secret for private registries +imagePullSecrets: [] +manager: + image: + # Uses chart.appVersion for the tag + repository: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator + collectorImage: + # find the latest collector releases at https://github.com/open-telemetry/opentelemetry-collector-releases/releases + repository: otel/opentelemetry-collector-k8s + tag: 0.123.0 + targetAllocatorImage: + repository: "" + tag: "" + # Only needed when overriding the image repository, make sure to always specify both the image and tag: + autoInstrumentationImage: + java: + repository: "" + tag: "" + nodejs: + repository: "" + tag: "" + python: + repository: "" + tag: "" + dotnet: + repository: "" + tag: "" + # The Go instrumentation support in the operator is disabled by default. + # To enable it, use the operator.autoinstrumentation.go feature gate. + go: + repository: "" + tag: "" + +admissionWebhooks: + # A production setup should use certManager to generate the certificate, without certmanager the certificate will be generated during the Helm install + certManager: + enabled: false + # The operator has validation and mutation hooks that need a certificate, with this we generate that automatically + autoGenerateCert: + enabled: true +``` +{% endcode %} + +Now install the collector, using the configuration file: + +```bash +helm upgrade --install opentelemetry-operator open-telemetry/opentelemetry-operator \ + --namespace open-telemetry \ + --values otel-operator.yaml +``` + +This only installs the operator. Continue to install the collector and enable auto-instrumentation. + +## The Open Telemetry collector + +The operator manages one or more collector deployments via a Kubernetes custom resource of kind `OpenTelemetryCollector`. We'll create one using the same configuration as used in the [Kubernetes getting started guide](./getting-started-k8s.md). + +It uses the secret created earlier in the guide. Make sure to replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert the name for your Kubernetes cluster instead of ``: + +{% code title="collector.yaml" lineNumbers="true" %} +```yaml +apiVersion: opentelemetry.io/v1beta1 +kind: OpenTelemetryCollector +metadata: + name: otel-collector +spec: + mode: deployment + envFrom: + - secretRef: + name: open-telemetry-collector + # optional service-account for pulling the collector image from a private registries + # serviceAccount: otel-collector + config: + receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + # Scrape the collectors own metrics + prometheus: + config: + scrape_configs: + - job_name: opentelemetry-collector + scrape_interval: 10s + static_configs: + - targets: + - ${env:MY_POD_IP}:8888 + extensions: + health_check: + endpoint: ${env:MY_POD_IP}:13133 + # Use the API key from the env for authentication + bearertokenauth: + scheme: SUSEObservability + token: "${env:API_KEY}" + exporters: + debug: {} + nop: {} + otlp/suse-observability: + auth: + authenticator: bearertokenauth + # Put in your own otlp endpoint, for example suse-observability.my.company.com:443 + endpoint: + compression: snappy + processors: + memory_limiter: + check_interval: 5s + limit_percentage: 80 + spike_limit_percentage: 25 + batch: {} + resource: + attributes: + - key: k8s.cluster.name + action: upsert + # Insert your own cluster name + value: + - key: service.instance.id + from_attribute: k8s.pod.uid + action: insert + # Use the k8s namespace also as the open telemetry namespace + - key: service.namespace + from_attribute: k8s.namespace.name + action: insert + connectors: + # Generate metrics for spans + spanmetrics: + metrics_expiration: 5m + namespace: otel_span + service: + extensions: [ health_check, bearertokenauth ] + pipelines: + traces: + receivers: [otlp] + processors: [memory_limiter, resource, batch] + exporters: [debug, spanmetrics, otlp/suse-observability] + metrics: + receivers: [otlp, spanmetrics, prometheus] + processors: [memory_limiter, resource, batch] + exporters: [debug, otlp/suse-observability] + logs: + receivers: [otlp] + processors: [] + exporters: [nop] + telemetry: + metrics: + address: ${env:MY_POD_IP}:8888 +``` +{% endcode %} + +{% hint style="warning" %} +**Use the same cluster name as used for installing the SUSE Observability agent** if you also use the SUSE Observability agent with the Kubernetes stackpack. Using a different cluster name will result in an empty traces perspective for Kubernetes components and will overall make correlating information much harder for SUSE Observability and your users. +{% endhint %} + +Now apply this `collector.yaml` in the `open-telemetry` namespace to deploy a collector: + +```bash +kubectl apply --namespace open-telemetry -f collector.yaml +``` + +The collector offers a lot more configuration receivers, processors and exporters, for more details see our [collector page](../collector.md). For production usage often large amounts of spans are generated and you will want to start setting up [sampling](../sampling.md). + +## Auto-instrumentation + +### Configure auto-instrumentation + +Now we need to tell the operator how to configure the auto instrumentation for the different languages using another custom resource, of kind `Instrumentation`. It is mainly used to configure the collector that was just deployed as the telemetry endpoint for the instrumented applications. + +It can be defined in a single place and used by all pods in the cluster, but it is also possible to have a different `Instrumentation` in each namespace. We'll be doing the former here. Note that if you used a different namespace or a different name for the otel collector the endpoint in this file needs to be updated accordingly. + +Create an `instrumentation.yaml`: + +{% code title="instrumentation.yaml" lineNumbers="true" %} +```yaml +apiVersion: opentelemetry.io/v1alpha1 +kind: Instrumentation +metadata: + name: otel-instrumentation +spec: + exporter: + # default endpoint for the instrumentation + endpoint: http://otel-collector-collector.open-telemetry.svc.cluster.local:4317 + propagators: + - tracecontext + - baggage + defaults: + # To use the standard app.kubernetes.io/ labels for the service name, version and namespace: + useLabelsForResourceAttributes: true + python: + env: + # Python autoinstrumentation uses http/proto by default, so data must be sent to 4318 instead of 4317. + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: http://otel-collector-collector.open-telemetry.svc.cluster.local:4318 + dotnet: + env: + # Dotnet autoinstrumentation uses http/proto by default, so data must be sent to 4318 instead of 4317. + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: http://otel-collector-collector.open-telemetry.svc.cluster.local:4318 + go: + env: + # Go autoinstrumentation uses http/proto by default, so data must be sent to 4318 instead of 4317. + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: http://otel-collector-collector.open-telemetry.svc.cluster.local:4318 +``` +{% endcode %} + +Now apply the `instrumentation.yaml` also in the `open-telemetry` namespace: + +```bash +kubectl apply --namespace open-telemetry -f instrumentation.yaml +``` + +### Enable auto-instrumentation for a pod + +To instruct the operator to auto-instrument your applicaction pods we need to add an annotation to the pod: +* Java: `instrumentation.opentelemetry.io/inject-java: open-telemetry/otel-instrumentation` +* NodeJS: `instrumentation.opentelemetry.io/inject-nodejs: open-telemetry/otel-instrumentation` +* Python: `instrumentation.opentelemetry.io/inject-python: open-telemetry/otel-instrumentation` +* Go: `instrumentation.opentelemetry.io/inject-go: open-telemetry/otel-instrumentation` + +Note that the value of the annotation refers to the namespace and name of the `Instrumentation` resource that we created. Other options are: + +* "true" - inject and `Instrumentation` custom resource from the namespace. +* "my-instrumentation" - name of `Instrumentation` custom resource in the current namespace. +* "my-other-namespace/my-instrumentation" - namespace and name of `Instrumentation` custom resource in another namespace. +* "false" - do not inject + +When a pod with one of the annotations is created the operator modifies the pod via a mutation hook: +* It adds an init container that provides the auto-instrumentation library +* It modifies the first container of the pod to load the instrumentation during start up and it adds environment variables to configure the instrumentation + +If you need to customize which containers should be instrumented use the [operator documentation](https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file#multi-container-pods-with-multiple-instrumentations). + +{% hint style="warning" %} +Go auto-instrumentation requires elevated permissions. These permissions are set automatically by the operator: + +```yaml +securityContext: + privileged: true + runAsUser: 0 +``` +{% endhint %} + +## View the results +Go to SUSE Observability and make sure the Open Telemetry Stackpack is installed (via the main menu -> Stackpacks). + +After a short while and if your pods are getting some traffic you should be able to find them under their service name in the Open Telemetry -> services and service instances overviews. Traces will appear in the [trace explorer](/use/traces/k8sTs-explore-traces.md) and in the [trace perspective](/use/views/k8s-traces-perspective.md) for the service and service instance components. Span metrics and language specific metrics (if available) will become available in the [metrics perspective](/use/views/k8s-metrics-perspective.md) for the components. + +If you also have the Kubernetes stackpack installed the instrumented pods will also have the traces available in the [trace perspective](/use/views/k8s-traces-perspective.md). + +## Next steps +You can add new charts to components, for example the service or service instance, for your application, by following [our guide](/use/metrics/k8s-add-charts.md). It is also possible to create [new monitors](/use/alerting/k8s-monitors.md) using the metrics and setup [notifications](/use/alerting/notifications/configure.md) to get notified when your application is not available or having performance issues. + +The operator, the `OpenTelemetryCollector`, and the `Instrumentation` custom resource, have more options that are documented in the [readme of the operator repository](https://github.com/open-telemetry/opentelemetry-operator). For example it is possible to install an optional [target allocator](https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file#target-allocator) via the `OpenTelemetryCollector` resource, it can be used to configure the Prometheus receiver of the collector. This is especially useful when you want to replace Prometheus operator and are using its `ServiceMonitor` and `PodMonitor` custom resources. + +# More info + +* [API keys](/use/security/k8s-ingestion-api-keys.md) +* [Open Telemetry API](../otlp-apis.md) +* [Customizing Open Telemetry Collector configuration](../collector.md) +* [Open Telemetry SDKs](../instrumentation/README.md) +* [Open Telemetry Operator](https://github.com/open-telemetry/opentelemetry-operator) \ No newline at end of file diff --git a/setup/otel/getting-started/getting-started-k8s.md b/setup/otel/getting-started/getting-started-k8s.md index 013635ceb..ae08bdff9 100644 --- a/setup/otel/getting-started/getting-started-k8s.md +++ b/setup/otel/getting-started/getting-started-k8s.md @@ -2,7 +2,7 @@ description: SUSE Observability --- -# Getting Started with Open Telemetry on Rancher / Kubernetes +# Getting Started with Open Telemetry on Kubernetes Here is the setup we'll be creating, for an application that needs to be monitored: @@ -10,12 +10,12 @@ Here is the setup we'll be creating, for an application that needs to be monitor * The Open Telemetry collector running near the observed application(s), so in cluster A, and sending the data to SUSE Observability * SUSE Observability running in cluster B, or SUSE Cloud Observability -![Container instrumentation with Opentelemetry via collector running as Kubernetes deployment](/.gitbook/assets/otel/open-telemetry-collector-kubernetes.png) +![Container instrumentation with Open Telemetry via collector running as Kubernetes deployment](/.gitbook/assets/otel/open-telemetry-collector-kubernetes.png) ## The Open Telemetry collector -{% hint type="info" %} +{% hint style="info" %} For a production setup it is strongly recommended to install the collector, since it allows your service to offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even sensitive data filtering. {% endhint %} @@ -47,7 +47,7 @@ We install the collector with a Helm chart provided by the Open Telemetry projec helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts ``` -Create a `otel-collector.yaml` values file for the Helm chart. Here is a good starting point for usage with SUSE Observability, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert the name for your Kubernetes cluster instead of ``: +Create a `otel-collector.yaml` values file for the Helm chart. Here is a good starting point for usage with SUSE Observability, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert the name for your Kubernetes cluster instead of ``: {% code title="otel-collector.yaml" lineNumbers="true" %} ```yaml @@ -86,8 +86,8 @@ config: otlp/suse-observability: auth: authenticator: bearertokenauth - # Put in your own otlp endpoint - endpoint: + # Put in your own otlp endpoint, for example suse-observability.my.company.com:443 + endpoint: compression: snappy processors: memory_limiter: @@ -131,7 +131,7 @@ config: ``` {% endcode %} -{% hint type="warning" %} +{% hint style="warning" %} **Use the same cluster name as used for installing the SUSE Observability agent** if you also use the SUSE Observability agent with the Kubernetes stackpack. Using a different cluster name will result in an empty traces perspective for Kubernetes components and will overall make correlating information much harder for SUSE Observability and your users. {% endhint %} diff --git a/setup/otel/getting-started/getting-started-lambda.md b/setup/otel/getting-started/getting-started-lambda.md index 509877e7c..7cc1f7fe2 100644 --- a/setup/otel/getting-started/getting-started-lambda.md +++ b/setup/otel/getting-started/getting-started-lambda.md @@ -9,11 +9,11 @@ We'll setup monitoring for one or more AWS Lambda functions: * The Open Telemetry collector * SUSE Observability or SUSE Cloud Observability -![AWS Lambda Instrumentation With Opentelemetry with Open Telemetry collector running in Kubernetes](/.gitbook/assets/otel/open-telemetry-collector-lambda.png) +![AWS Lambda Instrumentation With Open Telemetry with Open Telemetry collector running in Kubernetes](/.gitbook/assets/otel/open-telemetry-collector-lambda.png) ## The Open Telemetry collector -{% hint type="info" %} +{% hint style="info" %} For a production setup it is strongly recommended to install the collector, since it allows your service to offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even sensitive data filtering. {% endhint %} @@ -38,7 +38,7 @@ We install the collector with a Helm chart provided by the Open Telemetry projec helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts ``` -Create a `otel-collector.yaml` values file for the Helm chart. Here is a good starting point for usage with SUSE Observability, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert the name for your Kubernetes cluster instead of ``. When using the ingress configuration also make sure to insert your own domain name and the corresponding TLS certificate secret in the marked locations. +Create a `otel-collector.yaml` values file for the Helm chart. Here is a good starting point for usage with SUSE Observability, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert the name for your Kubernetes cluster instead of ``. When using the ingress configuration also make sure to insert your own domain name and the corresponding TLS certificate secret in the marked locations. {% code title="otel-collector.yaml" lineNumbers="true" %} ```yaml @@ -72,8 +72,8 @@ config: otlp: auth: authenticator: bearertokenauth - # Put in your own otlp endpoint - endpoint: + # Put in your own otlp endpoint, for example suse-observability.my.company.com:443 + endpoint: service: extensions: [health_check, bearertokenauth] diff --git a/setup/otel/getting-started/getting-started-linux.md b/setup/otel/getting-started/getting-started-linux.md index a10d5beed..8d3577985 100644 --- a/setup/otel/getting-started/getting-started-linux.md +++ b/setup/otel/getting-started/getting-started-linux.md @@ -14,7 +14,7 @@ Here is the setup we'll be creating, for an application that needs to be monitor ## Install the Open Telemetry collector -{% hint type="info" %} +{% hint style="info" %} For a production setup it is strongly recommended to install the collector, since it allows your service to offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even sensitive data filtering. {% endhint %} @@ -62,7 +62,7 @@ sudo rpm -iv1 otelcol-contrib_0.123.1_linux_arm64.rpm For other installation options use the [Open Telemetry instructions](https://opentelemetry.io/docs/collector/installation/#linux). -After installation modify the collector configuration by editing `/etc/otelcol-contrib/config.yaml`. Change the file such that it looks like the `config.yaml` example here, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert your receiver api key for `` (see [here](/use/security/k8s-ingestion-api-keys.md#api-keys) where to find it): +After installation modify the collector configuration by editing `/etc/otelcol-contrib/config.yaml`. Change the file such that it looks like the `config.yaml` example here, replace `` with your OTLP endpoint (see [OTLP API](../otlp-apis.md) for your endpoint) and insert your receiver api key for `` (see [here](/use/security/k8s-ingestion-api-keys.md#api-keys) where to find it): {% code title="config.yaml" lineNumbers="true" %} ```yaml @@ -99,8 +99,8 @@ exporters: compression: snappy auth: authenticator: bearertokenauth - # Put in your own otlp endpoint - endpoint: + # Put in your own otlp endpoint, for example suse-observability.my.company.com:443 + endpoint: processors: memory_limiter: check_interval: 5s diff --git a/setup/otel/instrumentation/sdk-exporter-config.md b/setup/otel/instrumentation/sdk-exporter-config.md index 83f9fa219..c7769ca9c 100644 --- a/setup/otel/instrumentation/sdk-exporter-config.md +++ b/setup/otel/instrumentation/sdk-exporter-config.md @@ -81,6 +81,6 @@ In small test setups it can be convenient to directly send data from your instru * For gRPC use the OTLP endpoint for SUSE Observability, see the [OTLP APIs page](../otlp-apis.md). * For HTTP use the OTLP over HTTP endpoint for SUSE Observability, see the [OTLP APIs page](../otlp-apis.md). -{% hint type="info" %} +{% hint style="info" %} Replace both the collector URL **and** the port with the SUSE Observability endpoints. Depending on your SUSE Observability installation the ports will be different. {% endhint %} diff --git a/setup/otel/otlp-apis.md b/setup/otel/otlp-apis.md index b460acb37..1fb6d96d2 100644 --- a/setup/otel/otlp-apis.md +++ b/setup/otel/otlp-apis.md @@ -10,8 +10,8 @@ SUSE Observability supports 2 versions of the OTLP protocol, the `grpc` version The endpoints for SUSE Cloud Observability are: -* OTLP: `https://otlp-.app.stackstate`.com:443 -* OTLP over HTTP: `https://otlp-http-`.app.stackstate.com +* OTLP: `https://otlp-.app.stackstate.io:443` +* OTLP over HTTP: `https://otlp-http-.app.stackstate.io` ## Self-hosted SUSE Observability