Add observability features to speech-to-speech workshop

speech-to-speech/workshops/README.md

@@ -1,6 +1,6 @@
 # Nova S2S workshop sample code
 
-> August 26, 2025 🆕🚀 A new Nova Sonic Multi-Agent Architecture lab using Amazon Bedrock AgentCore has been added. For more information and sample code refer to [./agent-core](./agent-core/) folder.
+> August 26, 2025 🆕🚀 A new Nova Sonic Multi-Agent Architecture lab using Amazon Bedrock AgentCore has been added. For more information and sample code refer to [./agent-core](./agent-core/README.md) folder.
 
 This project is for the [Amazon Nova Sonic speech-to-speech (S2S) workshop](https://catalog.workshops.aws/amazon-nova-sonic-s2s/en-US) and is intended for training purposes. It showcases a sample architecture for building applications that integrate with Nova Sonic, with features specifically designed to expose technical details for educational use.
 
@@ -106,7 +106,7 @@ cd nova-s2s-workshop
     npm start
     ```
 
-When using Chrome, if there’s no sound, please ensure the sound setting is set to Allow, as shown below.
+When using Chrome, if there's no sound, please ensure the sound setting is set to Allow, as shown below.
 ![chrome-sound](./static/chrome-sound-setting.png)
 
 ⚠️ **Warning:** Known issue: This UI is intended for demonstration purposes and may encounter state management issues after frequent conversation start/stop actions. Refreshing the page can help resolve the issue.
@@ -229,7 +229,79 @@ python server.py --agent strands
 ```
 - You can then try asking questions using the sample UI such as:
 ```
-What’s the weather like in Seattle today?
+What's the weather like in Seattle today?
 ```
 
 Refer to [the Strands Agent lab](https://catalog.workshops.aws/amazon-nova-sonic-s2s/en-US/200-labs/02-repeatable-pattern/03-strands) for more detailed instructions.
+
+## Enable Observability
+
+
+
+### Prerequisites
+
+1. Enable transaction search on Amazon CloudWatch. First-time users must enable CloudWatch Transaction Search to view Bedrock AgentCore spans and traces. To enable transaction search, please refer to the our documentation.
+
+<img src="agent-core/images/transactional_search.png" width="75%"/>
+
+
+2.  Create log group and log stream  in Amazon CloudWatch
+```bash
+import boto3
+cloudwatch_client = boto3.client("logs", region_name=region)
+response = cloudwatch_client.create_log_group(
+    logGroupName='bedrock-agentcore-observability'
+)
+```
+
+### Start backend with observability instrumentation
+
+Start the python server with the below shell command.
+
+```bash
+run_server_with_telemetry.sh
+```
+
+The shells script reads the environment variables from the local .env file with the following variables
+
+```bash
+# AWS OpenTelemetry Configuration 
+OTEL_PYTHON_DISTRO=aws_distro
+OTEL_PYTHON_CONFIGURATOR=aws_configurator
+
+# Service Identification
+OTEL_RESOURCE_ATTRIBUTES=service.name="s2s_agent"
+AGENT_OBSERVABILITY_ENABLED=true
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf 
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://xray.us-east-1.amazonaws.com/v1/traces"
+
+# CloudWatch Integration
+OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=bedrock-agentcore-observability,x-aws-log-stream=default,x-aws-metric-namespace=bedrock-agentcore
+
+# Instrumentation Exclusions - Extended to include all patterns
+OTEL_PYTHON_FASTAPI_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_REQUESTS_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_URLLIB3_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_HTTPX_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_AIOHTTP_CLIENT_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_BOTO3SQS_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+OTEL_PYTHON_BOTOCORE_EXCLUDED_URLS="/ws/.*|/health|/metrics|.*websocket.*|/api/.*|.*\.amazonaws\.com.*|.*bedrock-runtime\..*|.*dynamodb\..*|.*cognito-identity\..*|.*s3\..*"
+
+# Disable unwanted instrumentations
+OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="boto3sqs,botocore,requests,urllib3,httpx,aiohttp-client,asyncio,threading,logging,system_metrics,psutil,sqlite3,redis,pymongo,sqlalchemy,django,flask,tornado,pyramid,falcon,starlette,fastapi,websockets"
+
+# Propagation and Sampling
+OTEL_PROPAGATORS=tracecontext,baggage,xray
+OTEL_TRACES_SAMPLER=always_on
+OTEL_BSP_SCHEDULE_DELAY=1000
+OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
+OTEL_BSP_EXPORT_TIMEOUT=30000
+
+# AWS X-Ray specific
+AWS_XRAY_TRACING_NAME=s2s_agent
+AWS_XRAY_CONTEXT_MISSING=LOG_ERROR
+```
+
+This will create the following trace spans
+
+<img src="agent-core/images/s2s_observability.png" width="75%"/>

speech-to-speech/workshops/agent-core/banking_agent/.dockerignore

@@ -0,0 +1,68 @@
+# Build artifacts
+build/
+dist/
+*.egg-info/
+*.egg
+
+# Python cache
+__pycache__/
+__pycache__*
+*.py[cod]
+*$py.class
+*.so
+.Python
+
+# Virtual environments
+.venv/
+.env
+venv/
+env/
+ENV/
+
+# Testing
+.pytest_cache/
+.coverage
+.coverage*
+htmlcov/
+.tox/
+*.cover
+.hypothesis/
+.mypy_cache/
+.ruff_cache/
+
+# Development
+*.log
+*.bak
+*.swp
+*.swo
+*~
+.DS_Store
+
+# IDEs
+.vscode/
+.idea/
+
+# Version control
+.git/
+.gitignore
+.gitattributes
+
+# Documentation
+docs/
+*.md
+!README.md
+
+# CI/CD
+.github/
+.gitlab-ci.yml
+.travis.yml
+
+# Project specific
+tests/
+
+# Bedrock AgentCore specific - keep config but exclude runtime files
+.bedrock_agentcore.yaml
+.dockerignore
+
+# Keep wheelhouse for offline installations
+# wheelhouse/

speech-to-speech/workshops/agent-core/banking_agent/.env

@@ -0,0 +1,23 @@
+# AWS OpenTelemetry Configuration 
+OTEL_PYTHON_DISTRO=aws_distro
+OTEL_PYTHON_CONFIGURATOR=aws_configurator
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://xray.us-east-1.amazonaws.com/v1/traces"
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+
+# Service Identification
+OTEL_RESOURCE_ATTRIBUTES=service.name=banking_local_agent
+AGENT_OBSERVABILITY_ENABLED=true
+
+# CloudWatch Integration
+OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=bedrock-agentcore-observability,x-aws-log-stream=default,x-aws-metric-namespace=bedrock-agentcore
+
+# Propagation and Sampling - Enabled for custom exporter
+OTEL_PROPAGATORS=tracecontext,baggage,xray
+OTEL_TRACES_SAMPLER=always_on
+OTEL_BSP_SCHEDULE_DELAY=1000
+OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
+OTEL_BSP_EXPORT_TIMEOUT=30000
+
+# Disable unwanted instrumentations
+OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="bedrock-agentcore,strands-agents,strands-agents-tools,boto3sqs,botocore,requests,urllib3,httpx,aiohttp-client,asyncio,threading,logging,system_metrics,psutil,sqlite3,redis,pymongo,sqlalchemy,django,flask,tornado,pyramid,falcon,starlette,fastapi,websockets"

speech-to-speech/workshops/agent-core/banking_agent/Dockerfile

@@ -0,0 +1,49 @@
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
+WORKDIR /app
+
+# Configure UV for container environment
+ENV UV_SYSTEM_PYTHON=1 UV_COMPILE_BYTECODE=1
+
+
+
+COPY requirements.txt requirements.txt
+# Install from requirements file
+RUN uv pip install -r requirements.txt
+
+
+
+
+RUN uv pip install aws-opentelemetry-distro>=0.10.1
+
+
+# Set AWS region environment variable
+
+ENV AWS_REGION=us-east-1
+ENV AWS_DEFAULT_REGION=us-east-1
+
+
+# Signal that this is running in Docker for host binding logic
+ENV DOCKER_CONTAINER=1
+
+ENV OTEL_PYTHON_DISTRO=aws_distro
+ENV OTEL_PYTHON_CONFIGURATOR=aws_configurator
+ENV OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces
+ENV OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+ENV OTEL_RESOURCE_ATTRIBUTES=service.name={agent_name}
+ENV AGENT_OBSERVABILITY_ENABLED=true
+ENV OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=bedrock-agentcore-observability,x-aws-log-stream=default,x-aws-metric-namespace=bedrock-agentcore
+ENV OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=bedrock-agentcore,strands-agents,strands-agents-tools,boto3sqs,botocore,requests,urllib3,httpx,aiohttp-client,asyncio,threading,logging,system_metrics,psutil,sqlite3,redis,pymongo,sqlalchemy,django,flask,tornado,pyramid,falcon,starlette,fastapi,websockets
+
+# Create non-root user
+RUN useradd -m -u 1000 bedrock_agentcore
+USER bedrock_agentcore
+
+EXPOSE 8080
+EXPOSE 8000
+
+# Copy entire project (respecting .dockerignore)
+COPY . .
+
+# Use the full module path
+
+CMD ["opentelemetry-instrument", "python", "-m", "banking_agent"]

speech-to-speech/workshops/agent-core/banking_agent/banking_agent.py

@@ -3,6 +3,21 @@
 from bedrock_agentcore.runtime import BedrockAgentCoreApp
 from strands.models import BedrockModel
 import re, argparse
+import datetime
+import os
+
+# Import the tracer to create spans
+from strands.telemetry.tracer import get_tracer
+tracer = get_tracer()
+
+# # OpenTelemetry setup
+from opentelemetry import baggage, context
+
+def set_session_context(session_id):
+    """Set the session ID in OpenTelemetry baggage for trace correlation"""
+    ctx = baggage.set_baggage("session.id", session_id)
+    token = context.attach(ctx)
+    return token
 
 app = BedrockAgentCoreApp()
 
@@ -16,6 +31,7 @@ def get_account_balance(account_id) -> str:
 
     # In this sample, we use a mock response. 
     # The actual implementation will retrieve information from a database API or another backend service.
+
     result = {
       "account_id": "1234567890",
       "account_type": "Checking",
@@ -38,6 +54,9 @@ def get_statement(account_id: str, year_and_month: str) -> str:
     """
     # In this sample, we use a mock response. 
     # The actual implementation will retrieve information from a database API or another backend service.
+
+
+    # A sample bank statement for August 2025
     result = {
           "account_id": "1234567890",
           "account_type": "Checking",
@@ -125,13 +144,27 @@ def get_statement(account_id: str, year_and_month: str) -> str:
 
 @app.entrypoint
 def banking_agent(payload):
-    response = agent(json.dumps(payload))
-    output = response.message['content'][0]['text']
-    if "<response>" in output and "</response>" in output:
-        match = re.search(r"<response>(.*?)</response>", output, re.DOTALL)
-        if match:
-            output = match.group(1)
-    return output
+    try:
+      #generate a random session id to associate with this agent run
+      import uuid
+      session_id = str(uuid.uuid4())
+
+      context_token = set_session_context(session_id)
+
+      # Execute the agent
+      response = agent(json.dumps(payload))
+
+      # Process the response
+      output = response.message['content'][0]['text']
+      if "<response>" in output and "</response>" in output:
+          match = re.search(r"<response>(.*?)</response>", output, re.DOTALL)
+          if match:
+              output = match.group(1)
+      return output
+
+    finally:
+      context.detach(context_token)
+      pass
 
 if __name__ == "__main__":
-    app.run()
+    app.run()