STAC-22297 Added documentation for auto-instrumentation of AWS Lambda…

SUMMARY.md

@@ -93,10 +93,12 @@
 ## 🔭 Open Telemetry
 * [Getting started](setup/otel/getting-started.md)
 * [Open telemetry collector](setup/otel/collector.md)
+  * [Collector as a proxy](setup/otel/proxy-collector.md)
 * [Languages](setup/otel/languages/README.md)
   * [Generic Exporter configuration](setup/otel/languages/sdk-exporter-config.md)
   * [Java](setup/otel/languages/java.md)
   * [Node.js](setup/otel/languages/node.js.md)
+    * [Auto-instrumentation of Lambdas](setup/otel/languages/node.js/auto-instrumentation-of-lambdas.md)
   * [.NET](setup/otel/languages/dot-net.md)
   * [Verify the results](setup/otel/languages/verify.md)
 * [Troubleshooting](setup/otel/troubleshooting.md)

setup/otel/languages/node.js/auto-instrumentation-of-lambdas.md

@@ -0,0 +1,167 @@
+---
+description: SUSE Observability
+---
+
+# Auto-Instrumenting a NodeJS Lambda
+
+## Introduction
+
+This document guides you through auto-instrumenting NodeJS Lambda functions using OpenTelemetry. Auto-instrumentation simplifies the process of adding observability to your Lambda functions by automatically capturing performance metrics and tracing information.
+
+## Prerequisites
+
+Before you begin, ensure you have the following:
+
+- **AWS Lambda function:** The function you want to instrument.
+- **OpenTelemetry SDK:** Installed in your Lambda function.
+- **OpenTelemetry Collector:** Deployed and configured.
+- **SUSE Observability:** An account with SUSE Observability where you'll send your telemetry data.
+- **Memory:** Enough memory to run the Lambda’s including the instrumentation.
+
+## Values supplied by the environment
+
+OpenTelemetry relies on various configuration values to function correctly. These values control aspects like data collection, exporting, and communication with backend systems. To make your OpenTelemetry deployment flexible and adaptable to different environments, you can provide these settings through environment variables. This approach offers several benefits:
+
+- **Dynamic Configuration:** Easily adjust settings without code changes.
+- **Environment-Specific Settings:** Configure OpenTelemetry differently for development, testing, and production.
+- **Secret Management:** Securely store sensitive information like API keys.
+
+For the OpenTelemetry setup described in this documentation, you'll need to define the following environment variables:
+
+- **`VERBOSITY`:** Controls the level of detail in OpenTelemetry logs.
+- **`OTLP_API_KEY`:** Authenticates your Lambda function to send data to SUSE Observability.
+- **`OTLP_ENDPOINT`:** Specifies the address of your SUSE Observability instance.
+- **`OPENTELEMETRY_COLLECTOR_CONFIG_FILE`:** Points to the configuration file for the OpenTelemetry Collector.
+- **`AWS_LAMBDA_EXEC_WRAPPER`:** Configures the Lambda execution environment to use the OpenTelemetry handler.
+- **`OTLP_INSTR_LAYER_ARN`:** Provides the ARN (Amazon Resource Name) of the OpenTelemetry instrumentation layer, which adds the necessary components for auto-instrumentation.
+- **`OTLP_COLLECTOR_LAYER_ARN`:** Provides the ARN of the OpenTelemetry collector layer, which is responsible for receiving, processing, and exporting telemetry data.
+
+**Important Considerations:**
+
+- **GRPC Endpoint:** The `OTLP_ENDPOINT` should specify the gRPC endpoint of your SUSE Observability instance without any `http` or `https` prefix. Use port 443 for secure communication.
+- **Region-Specific Layers:** Lambda layers are region-bound. Ensure that the ARNs you use for `OTLP_INSTR_LAYER_ARN` and `OTLP_COLLECTOR_LAYER_ARN` match the AWS region where your Lambda function is deployed.
+- **Architecture Matching:** The OpenTelemetry Collector layer is architecture-specific. Choose the correct ARN for your Lambda function's architecture (e.g., `amd64` or `arm64`).
+
+**A complete example: be aware you need to input your own values.**
+
+```yaml
+VERBOSITY: "normal"
+OTLP_API_KEY: "<your api key for sending data to SUSE Observability here>"
+OTLP_ENDPOINT: "<your-dns-name-for-suse-observability-here>:443"
+OPENTELEMETRY_COLLECTOR_CONFIG_FILE: "/var/task/collector.yaml"
+AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler"
+OTLP_INSTR_LAYER_ARN: "arn:aws:lambda:<aws-region>:184161586896:layer:opentelemetry-nodejs-0_11_0:1"
+OTLP_COLLECTOR_LAYER_ARN: "arn:aws:lambda:<aws-region>:184161586896:layer:opentelemetry-collector-<amd64|arm64>-0_12_0:1"
+```
+
+## The collector.yaml file
+
+OTEL collection configuration sets up how the data collected should be distributed.  This is done in the collector.yaml file placed in the src directory where the lambda files can be found.  Below is an example collector.yaml file.
+
+```yaml
+# collector.yaml in the root directory
+# Set an environemnt variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to
+# '/var/task/collector.yaml'
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+      http:
+
+exporters:
+  debug:
+    verbosity: "${env:VERBOSITY}"
+  otlp/stackstate:
+    headers:
+      Authorization: "SUSEObservability ${env:OTLP_API_KEY}"
+    endpoint: "${env:OTLP_ENDPOINT}"
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [debug, otlp/stackstate]
+      processors: []
+    metrics:
+      receivers: [otlp]
+      exporters: [debug, otlp/stackstate]
+      processors: []
+```
+
+Be aware this collector is used to send the data over to a next collector which then is used for tail sampling, metric aggregation, etc. before sending data over to SUSE Observability. This second collector also needs to run in the customer's environment. 
+
+Depending on the desired functionality, or based upon factors such as volumes of data being generated by lambdas instrumented in this way, collectors can be set up for batching, tail-sampling, and other pre-processing techniques to reduce the impact on SUSE Observability.
+
+See this page for [guidance and instruction](../../proxy-collector.md) on how to set up a batching collector that acts as a security proxy for SUSE Observability.
+See this page for [instructions](../../collector.md) on how to set up a collector that does tail-sampling as well.
+For more information about processor configuration on the opentelemetry collector, see the [official documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md).
+
+![AWS Lambda Instrumentation With Opentelemetry](/.gitbook/assets/otel/aws_nodejs_otel_auto_instrumentation.svg)
+
+## Package.json
+
+Make sure to add `"@opentelemetry/auto-instrumentations-node": "^0.55.2",` to `package.json` and execute `npm install` to add the auto-instrumentation client libraries to your NodeJS Lambda.
+
+## Troubleshooting
+
+### Timeouts
+
+If the addition of the OTEL Lambda layers results in lambdas that time out (checking the logs might indicate that the collector was asked to shut down while still busy, e.g. seeing the following log entry):
+
+```json
+{
+    "level": "info",
+    "ts": 1736867469.2312617,
+    "caller": "internal/retry_sender.go:126",
+    "msg": "Exporting failed. Will retry the request after interval.",
+    "kind": "exporter",
+    "data_type": "traces",
+    "name": "otlp/stackstate",
+    "error": "rpc error: code = Canceled desc = context canceled",
+    "interval": "5.125929689s"
+}
+```
+
+shortly after receiving the instruction to shut down:
+
+```json
+{
+    "level": "info",
+    "ts": 1736867468.4311068,
+    "logger": "lifecycle.manager",
+    "msg": "Received SHUTDOWN event"
+}
+```
+
+The above indicates that the allocated resources of the lambda are not sufficient to allow execution of the lambda and the additional strain added by the OTEL instrumentation.  To remedy this, the memory allocation and lambda timeout settings can be adjusted as necessary to allow the lambda to finish its work, while also allowing the telemetry collection to succeed.
+
+Try modifying the MemorySize and TimeOut properties of the lambdas that are failing:
+
+```yaml
+MemorySize: 256
+Timeout: 25
+```
+
+Note the default memory allocation is 128MB
+
+Note the memory increment is 128MB
+
+Note Timeout is an integer value denoting seconds.
+
+### Authentication and Source IP Filtering
+
+If you encounter `error 403 Unauthorized` when submitting collector data to your cluster, or to any pre-processing or proxy collector, double-check the source IP address of the VPC NAT gateway matches what is whitelisted by the collector ingress, 
+also double check that the chosen authentication mechanism matches source and destination, and also that credentials (secrets, etc.) are set up correctly.
+
+For more information about configuring authentication for the opentelemetry collector, please refer to the [official documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configauth/README.md).
+
+## References
+
+Auto-instrumentation docs → [https://opentelemetry.io/docs/faas/lambda-auto-instrument/](https://opentelemetry.io/docs/faas/lambda-auto-instrument/)
+
+Collector docs → [https://opentelemetry.io/docs/faas/lambda-collector/](https://opentelemetry.io/docs/faas/lambda-collector/)
+
+GitHub Releases Page for finding latest ARNs → [https://github.com/open-telemetry/opentelemetry-lambda/releases](https://github.com/open-telemetry/opentelemetry-lambda/releases)
+
+OTLP Exporter Configuration → [https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/)

setup/otel/proxy-collector.md

@@ -0,0 +1,95 @@
+---
+description: SUSE Observability
+---
+
+# Open Telemetry Collector as a proxy
+
+The normal configuration of the Opentelemetry Collector for tail-sampling traces can be found [here](collector.md)
+
+The below configuration describes a deployment that only does batching, and no further processing of traces, metrics, 
+or logs.  It is meant as a security proxy that exists outside the SUSE Observability cluster, but within trusted network
+infrastructure.  Security credentials for the proxy and SUSE Observability can be set up separately, adding a layer of
+authentication that does not reside with the caller, but with the host.
+
+![AWS Lambda Instrumentation With Opentelemetry via proxy collector](/.gitbook/assets/otel/aws_nodejs_otel_proxy_collector_configuration.svg)
+
+{% code title="otel-collector.yaml" lineNumbers="true" %}
+```yaml
+mode: deployment
+presets:
+   kubernetesAttributes:
+      enabled: true
+      # You can also configure the preset to add all the associated pod's labels and annotations to you telemetry.
+      # The label/annotation name will become the resource attribute's key.
+      extractAllPodLabels: true
+extraEnvsFrom:
+   - secretRef:
+        name: open-telemetry-collector
+image:
+   # Temporary override for image tag, the helm chart has not been released yet
+   tag: 0.97.0
+
+config:
+   receivers:
+      otlp:
+         protocols:
+            grpc:
+               endpoint: 0.0.0.0:4317
+            http:
+               endpoint: 0.0.0.0:4318
+
+   exporters:
+      # Exporter for traces to traffic mirror (used by the common config)
+      otlp:
+         endpoint: <url for opentelemetry ingestion by suse observability>
+         auth:
+            authenticator: bearertokenauth
+
+   extensions:
+      bearertokenauth:
+         scheme: SUSEObservability
+         token: "${env:API_KEY}"
+
+   service:
+      extensions: [health_check, bearertokenauth]
+      pipelines:
+         traces:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+         metrics:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+         logs:
+            receivers: [otlp]
+            processors: [batch]
+            exporters: [otlp]
+
+ingress:
+  enabled: true
+  annotations:
+    kubernetes.io/ingress.class: ingress-nginx-external
+    nginx.ingress.kubernetes.io/ingress.class: ingress-nginx-external
+    nginx.ingress.kubernetes.io/backend-protocol: GRPC
+    # "12.34.56.78/32" IP address of NatGateway in the VPC where the otel data is originating from
+    #  nginx.ingress.kubernetes.io/whitelist-source-range: "12.34.56.78/32"
+  hosts:
+    - host: "otlp-collector-proxy.${CLUSTER_NAME}"
+      paths:
+        - path: /
+          pathType: ImplementationSpecific
+          port: 4317
+  tls:
+    - secretName: ${CLUSTER_NODOT}-ecc-tls
+      hosts:
+        - "otlp-collector-proxy.${CLUSTER_NAME}"
+```
+{% endcode %}
+
+
+### Ingress Source Range Whitelisting
+
+To emphasize the role of the proxy collector as a security measure, it is recommended to use a source-range whitelist
+to filter out data from untrusted and/or unknown sources.  In contrast, the SUSE Observability ingestion collector may 
+have to accept data from multiple sources, maintaining a whitelist on that level does not scale well.