4.2.0

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Upgrade Notes

• tracing
  • The Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.

New Features

• LLM Observability
  • Adds tracing of initialize requests and their responses on modelcontextprotocol/python-sdk servers.
  • This introduces automatic source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.
  • Changes format of MCP server tool call span input, output, and tags to include the full request and response objects.
  • Reasoning token counts are now captured from VertexAI responses.
  • Adds ability to automatically capture the intent and context of an MCP server tool call to modelcontextprotocol/python-sdk servers using the DD_MCP_CAPTURE_INTENT environment variable.
• profiling
  • Add support for asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.
  • Add support for asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.
  • Add support for asyncio.Semaphore lock type profiling in Python Lock Profiler.
  • the Profiler now properly stacks flame graphs for Tasks awaiting and awaited through asyncio.shield.
  • the Profiler now tracks Tasks created through asyncio.TaskGroup.
• AAP
  • Add business logic event detection for Stripe. This feature instruments the payment intent and checkout session creation actions as well as the payment_intent.succeeded, payment_intent.payment_failed and payment_intent.canceled events.
  • Proxy inferred spans now contain events when AppSec is enabled so that they are reported on the App and API Protection Endpoint Catalog.
• DBM
  • Adds container tags support
• DSM
  • Adds container tags support
• tracing
  • Proxy inferred spans now differentiate between API Gateway v1 and v2 apis by emitting the aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.
• vllm
  • Introduces tracing and LLM Observability support for vLLM V1 engine. Requires vLLM >= 0.10.2. See the docs for more information.

Bug Fixes

• CI Visibility
  • This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.
• crash tracking
  • Passing in all environment variables to the crashtracker receiver process caused conflicts with export location derivation. This change only passes in the DD_CRASHTRACKING_ERRORS_INTAKE_ENABLED environment variable.
• falcon
  • Fixes DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.
• tracing
  • Proxy inferred spans used to omit the https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.
  • psycopg
    • Fixes an OperationalError that occurred when patching a closed psycopg3 connection.
• dynamic instrumentation
  • fixed an issue that caused condition expressions containing isDefined to result in an evaluation error.
  • fixed an issue that prevented autocomplete information to be extracted from the running application.
• openfeature
  • This fix resolves an issue where Feature Flagging and Experimentation (FFE) was not receiving remote configuration in forking web server environments (gunicorn, uWSGI). This caused the OpenFeature provider to return default values instead of configured feature flags. FFE is now properly registered as a product during tracer initialization (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.
  • Fix exposure event deduplication to use (flag_key, subject_id) as cache key instead of (flag_key, variant_key, allocation_key). This ensures different users each receive their own exposure event while still deduplicating repeated evaluations for the same user. Also adds LRU eviction to prevent unbounded memory growth and respects the do_log flag from flag metadata.
• ray
• This fix resolves an issue where None metadata in Ray job submission caused a crash.
• openai_agents
  • This fix resolves an issue where a missing active span caused an AttributeError when calling tag_agent_manifest.
• LLM Observability
  • This fix resolves an issue where using multiple sequential annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.
  • Resolves an issue in activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.
• profiling
  • This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.
  • This fix resolves an issue where the Lock profiler would not call the necessary initialization function, which would sometimes result in crashes.
  • the Profiler now always uses the name of leaf tasks for the "Task name" label. Previously, one of the Stacks would be labelled with the parent task's name, which would lead to inconsistent behaviour across runs.
  • Fixes a bug where code that sub-classes our wrapped locks crashes with TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45
  • a bug causing crashes when using uvloop and forking has been resolved.
  • This improves the accuracy of stacks for on-CPU asyncio Tasks by reducing the odds of Frames for a Task polluting the stack of other Tasks.
  • the build now uses the correct location for the native extension module. Previously, linking would work correctly in tests, but published wheels failed to import the memory profiler extension.
• SSI
  • This fixes an issue where ddtrace fails to find _psutil_linux.abi3.so file in an injected environment.

4.2.0rc3

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Upgrade Notes

• tracing
  • The Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.

New Features

• LLM Observability
  • Adds tracing of initialize requests and their responses on modelcontextprotocol/python-sdk servers.
  • This introduces automatic source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.
  • Changes format of MCP server tool call span input, output, and tags to include the full request and response objects.
  • Reasoning token counts are now captured from VertexAI responses.
  • Adds ability to automatically capture the intent and context of an MCP server tool call to modelcontextprotocol/python-sdk servers using the DD_MCP_CAPTURE_INTENT environment variable.
• profiling
  • Add support for asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.
  • Add support for asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.
  • Add support for asyncio.Semaphore lock type profiling in Python Lock Profiler.
  • the Profiler now properly stacks flame graphs for Tasks awaiting and awaited through asyncio.shield.
  • the Profiler now tracks Tasks created through asyncio.TaskGroup.
• AAP
  • Add business logic event detection for Stripe. This feature instruments the payment intent and checkout session creation actions as well as the payment_intent.succeeded, payment_intent.payment_failed and payment_intent.canceled events.
  • Proxy inferred spans now contain events when AppSec is enabled so that they are reported on the App and API Protection Endpoint Catalog.
• DBM
  • Adds container tags support
• DSM
  • Adds container tags support
• tracing
  • Proxy inferred spans now differentiate between API Gateway v1 and v2 apis by emitting the aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.
• vllm
  • Introduces tracing and LLM Observability support for vLLM V1 engine. Requires vLLM >= 0.10.2. See the docs for more information.

Bug Fixes

• CI Visibility
  • This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.
• crash tracking
  • Passing in all environment variables to the crashtracker receiver process caused conflicts with export location derivation. This change only passes in the DD_CRASHTRACKING_ERRORS_INTAKE_ENABLED environment variable.
• falcon
  • Fixes DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.
• tracing
  • Proxy inferred spans used to omit the https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.
  • psycopg
    • Fixes an OperationalError that occurred when patching a closed psycopg3 connection.
• dynamic instrumentation
  • fixed an issue that caused condition expressions containing isDefined to result in an evaluation error.
  • fixed an issue that prevented autocomplete information to be extracted from the running application.
• openfeature
  • This fix resolves an issue where Feature Flagging and Experimentation (FFE) was not receiving remote configuration in forking web server environments (gunicorn, uWSGI). This caused the OpenFeature provider to return default values instead of configured feature flags. FFE is now properly registered as a product during tracer initialization (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.
  • Fix exposure event deduplication to use (flag_key, subject_id) as cache key instead of (flag_key, variant_key, allocation_key). This ensures different users each receive their own exposure event while still deduplicating repeated evaluations for the same user. Also adds LRU eviction to prevent unbounded memory growth and respects the do_log flag from flag metadata.
• ray
• This fix resolves an issue where None metadata in Ray job submission caused a crash.
• openai_agents
  • This fix resolves an issue where a missing active span caused an AttributeError when calling tag_agent_manifest.
• LLM Observability
  • This fix resolves an issue where using multiple sequential annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.
  • Resolves an issue in activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.
• profiling
  • This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.
  • This fix resolves an issue where the Lock profiler would not call the necessary initialization function, which would sometimes result in crashes.
  • the Profiler now always uses the name of leaf tasks for the "Task name" label. Previously, one of the Stacks would be labelled with the parent task's name, which would lead to inconsistent behaviour across runs.
  • Fixes a bug where code that sub-classes our wrapped locks crashes with TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45
  • a bug causing crashes when using uvloop and forking has been resolved.
  • This improves the accuracy of stacks for on-CPU asyncio Tasks by reducing the odds of Frames for a Task polluting the stack of other Tasks.
  • the build now uses the correct location for the native extension module. Previously, linking would work correctly in tests, but published wheels failed to import the memory profiler extension.
• SSI
  • This fixes an issue where ddtrace fails to find _psutil_linux.abi3.so file in an injected environment.

4.2.0rc2

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Upgrade Notes

• tracing
  • The Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.

New Features

• LLM Observability
  • Adds tracing of initialize requests and their responses on modelcontextprotocol/python-sdk servers.
  • This introduces automatic source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.
  • Changes format of MCP server tool call span input, output, and tags to include the full request and response objects.
  • Reasoning token counts are now captured from VertexAI responses.
  • Adds ability to automatically capture the intent and context of an MCP server tool call to modelcontextprotocol/python-sdk servers using the DD_MCP_CAPTURE_INTENT environment variable.
• profiling
  • Add support for asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.
  • Add support for asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.
  • Add support for asyncio.Semaphore lock type profiling in Python Lock Profiler.
  • the Profiler now properly stacks flame graphs for Tasks awaiting and awaited through asyncio.shield.
  • the Profiler now tracks Tasks created through asyncio.TaskGroup.
• AAP
  • Add business logic event detection for Stripe. This feature instruments the payment intent and checkout session creation actions as well as the payment_intent.succeeded, payment_intent.payment_failed and payment_intent.canceled events.
  • Proxy inferred spans now contain events when AppSec is enabled so that they are reported on the App and API Protection Endpoint Catalog.
• DBM
  • Adds container tags support
• DSM
  • Adds container tags support
• tracing
  • Proxy inferred spans now differentiate between API Gateway v1 and v2 apis by emitting the aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.
• vllm
  • Introduces tracing and LLM Observability support for vLLM V1 engine. Requires vLLM >= 0.10.2. See the docs for more information.

Bug Fixes

• CI Visibility
  • This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.
• crash tracking
  • Passing in all environment variables to the crashtracker receiver process caused conflicts with export location derivation. This change only passes in the DD_CRASHTRACKING_ERRORS_INTAKE_ENABLED environment variable.
• falcon
  • Fixes DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.
• tracing
  • Proxy inferred spans used to omit the https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.
  • psycopg
    • Fixes an OperationalError that occurred when patching a closed psycopg3 connection.
• dynamic instrumentation
  • fixed an issue that caused condition expressions containing isDefined to result in an evaluation error.
  • fixed an issue that prevented autocomplete information to be extracted from the running application.
• openfeature
  • This fix resolves an issue where Feature Flagging and Experimentation (FFE) was not receiving remote configuration in forking web server environments (gunicorn, uWSGI). This caused the OpenFeature provider to return default values instead of configured feature flags. FFE is now properly registered as a product during tracer initialization (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.
  • Fix exposure event deduplication to use (flag_key, subject_id) as cache key instead of (flag_key, variant_key, allocation_key). This ensures different users each receive their own exposure event while still deduplicating repeated evaluations for the same user. Also adds LRU eviction to prevent unbounded memory growth and respects the do_log flag from flag metadata.
• ray
• This fix resolves an issue where None metadata in Ray job submission caused a crash.
• openai_agents
  • This fix resolves an issue where a missing active span caused an AttributeError when calling tag_agent_manifest.
• LLM Observability
  • This fix resolves an issue where using multiple sequential annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.
  • Resolves an issue in activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.
• profiling
  • This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.
  • This fix resolves an issue where the Lock profiler would not call the necessary initialization function, which would sometimes result in crashes.
  • the Profiler now always uses the name of leaf tasks for the "Task name" label. Previously, one of the Stacks would be labelled with the parent task's name, which would lead to inconsistent behaviour across runs.
  • Fixes a bug where code that sub-classes our wrapped locks crashes with TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45
  • a bug causing crashes when using uvloop and forking has been resolved.
  • This improves the accuracy of stacks for on-CPU asyncio Tasks by reducing the odds of Frames for a Task polluting the stack of other Tasks.
  • the build now uses the correct location for the native extension module. Previously, linking would work correctly in tests, but published wheels failed to import the memory profiler extension.
• SSI
  • This fixes an issue where ddtrace fails to find _psutil_linux.abi3.so file in an injected environment.

4.1.3

Estimated end-of-life date, accurate to within three months: 07-2027
See the support level definitions for more information.

Bug Fixes

• SSI: This fixes an issue where ddtrace fails to find _psutil_linux.abi3.so file in an injected environment.

• LLM Observability: This fix resolves an issue where using multiple sequential annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.

• profiling: the Profiler now always uses the name of leaf tasks for the "Task name" label. Previously, one of the Stacks would be labelled with the parent task's name, which would lead to inconsistent behaviour across runs.

• profiling: a bug causing crashes when using uvloop and forking has been resolved.

• profiling: This improves the accuracy of stacks for on-CPU asyncio Tasks by reducing the odds of Frames for a Task polluting the stack of other Tasks.

• profiling: This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.

4.0.3

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.

• profiling: a bug causing crashes when using uvloop and forking has been resolved.

• profiling: This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.

• profiling: This fix ensures the profiler now correctly tracks dependencies between Tasks and Coroutines that are awaiting or being awaited via asyncio.wait.

4.2.0rc1

Estimated end-of-life date, accurate to within three months: 07-2027
See the support level definitions for more information.

Upgrade Notes

• tracing
  • The Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.

New Features

• LLM Observability
  • Adds tracing of initialize requests and their responses on modelcontextprotocol/python-sdk servers.
  • This introduces automatic source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.
  • Changes format of MCP server tool call span input, output, and tags to include the full request and response objects.
  • Reasoning token counts are now captured from VertexAI responses.
• profiling
  • Add support for asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.
  • Add support for asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.
  • Add support for asyncio.Semaphore lock type profiling in Python Lock Profiler.
• AAP
  • Add business logic event detection for Stripe. This feature instruments the payment intent and checkout session creation actions as well as the payment_intent.succeeded, payment_intent.payment_failed and payment_intent.canceled events.
  • Proxy inferred spans now contain events when AppSec is enabled so that they are reported on the App and API Protection Endpoint Catalog.
• DBM
  • Adds container tags support
• DSM
  • Adds container tags support
• tracing
  • Proxy inferred spans now differentiate between API Gateway v1 and v2 apis by emitting the aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.
• vllm
  • Introduces tracing and LLM Observability support for vLLM V1 engine. Requires vLLM >= 0.10.2. See the docs for more information.

Bug Fixes

• CI Visibility
  • This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.
• crash tracking
  • Passing in all environment variables to the crashtracker receiver process caused conflicts with export location derivation. This change only passes in the DD_CRASHTRACKING_ERRORS_INTAKE_ENABLED environment variable.
• falcon
  • Fixes DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.
• tracing
  • Proxy inferred spans used to omit the https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.
  • psycopg
    • Fixes an OperationalError that occurred when patching a closed psycopg3 connection.
• dynamic instrumentation
  • fixed an issue that caused condition expressions containing isDefined to result in an evaluation error.
  • fixed an issue that prevented autocomplete information to be extracted from the running application.
• openfeature
  • This fix resolves an issue where Feature Flagging and Experimentation (FFE) was not receiving remote configuration in forking web server environments (gunicorn, uWSGI). This caused the OpenFeature provider to return default values instead of configured feature flags. FFE is now properly registered as a product during tracer initialization (when DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.
  • Fix exposure event deduplication to use (flag_key, subject_id) as cache key instead of (flag_key, variant_key, allocation_key). This ensures different users each receive their own exposure event while still deduplicating repeated evaluations for the same user. Also adds LRU eviction to prevent unbounded memory growth and respects the do_log flag from flag metadata.
• ray
  • This fix resolves an issue where None metadata in Ray job submission caused a crash.
• openai_agents
  • This fix resolves an issue where a missing active span caused an AttributeError when calling tag_agent_manifest.
• profiling
  • This fix resolves an issue where the Lock profiler would not call the necessary initialization function, which would sometimes result in crashes.
  • the Profiler now always uses the name of leaf tasks for the "Task name" label. Previously, one of the Stacks would be labelled with the parent task's name, which would lead to inconsistent behaviour across runs.
  • Fixes a bug where code that sub-classes our wrapped locks crashes with TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45
  • a bug causing crashes when using uvloop and forking has been resolved.
  • This improves the accuracy of stacks for on-CPU asyncio Tasks by reducing the odds of Frames for a Task polluting the stack of other Tasks.
  • the build now uses the correct location for the native extension module. Previously, linking would work correctly in tests, b

3.19.4

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Deprecation Notes

• LLM Observability: Warning logs for incorrect usage of the LLM Observability SDK are deprecated and will be replaced with raised errors in ddtrace>=4.0.0.

Bug Fixes

• CI Visibility: This fix resolves an issue where code coverage instrumentation in Python 3.9 would raise an exception while handling line numbers in some corner cases.

• profiling: This fix resolves an issue where the Lock profiler would not call the necessary initialization function, which would sometimes result in crashes.

• profiling: This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.

• profiling: This fix ensures the profiler now correctly tracks dependencies between Tasks and Coroutines that are awaiting or being awaited via asyncio.wait.

4.1.2

Estimated end-of-life date, accurate to within three months: 07-2027
See the support level definitions for more information.

Bug Fixes

• ray
  • This fix resolves an issue where None metadata in Ray job submission caused a crash.
• profiling
  • the build now uses the correct location for the native extension module. Previously, linking would work correctly in tests, but published wheels failed to import the memory profiler extension.

4.1.1

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Bug Fixes

• Debugger
  • Reverts a change added in 4.1.0 that seems to be linked to "context already registered" errors encountered when DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true is set

4.1.0

⚠️ An issue was detected with memory profiling in this release. Please consider upgrading to v4.1.3 or newer

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Known Issues

• debugging: "context already registered" errors when DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true is set

Upgrade Notes

• LLM Observability
  • Experiments spans now contain metadata from the dataset record.
  • Experiments spans' input, output, expected_output fields are now emitted as is so that if data in any of the columns are objects, they can be searchable in Datadog.
  • Experiments spans and children spans are now tagged with human readable names to allow better analysis of experiments data. New tags added are: dataset_name, project_name, project_id, experiment_name.
• tornado
  • Updated minimum supported version to v6.1+.

Deprecation Notes

• tornado
  • Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
• LLM Observability
  • The ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.

New Features

• profiling
  • Add support for threading.BoundedSemaphore locking type profiling in Python. The implementation follows the same approach as threading.Semaphore, properly handling internal lock detection to prevent double-counting of the underlying threading.Lock object.
  • Add support for threading.Semaphore locking type profiling in Python. The Lock profiler now detects and marks "internal" Lock objects, i.e. those that are part of implementation of higher-level locking types. One example of such higher-level primitive is threading.Semaphore, which is implemented with threading.Condition, which itself uses threading.Lock internally. Marking internal lock as "internal" will prevent it from being sampled, ensuring that the high-level (e.g. Semaphore) sample is processed.
  • This adds support for Python 3.14 in the Continuous Profiler.
  • This adds the process_id tag to profiles. The value of this tag is the current process ID (PID).
  • The stack sampler supports async generators and asyncio.wait.
  • Shows fully qualified name of functions using codeobject.co_qualname in memory profiler and lock profiler flamegraphs for Python 3.11+. Stack profiler has already been using this. This aligns the user experience across different profile types.
  • This introduces tracking for the asyncio.as_completed util in the Profiler.
  • This introduces tracking for asyncio.wait in the Profiler. This makes it possible to track dependencies between Tasks/Coroutines that await/are awaited through asyncio.wait.
• AAP
  • attach Application and API Protection findings on API Gateway inferred spans to enable AppSec API Catalog coverage of lambda functions
  • This introduces proper support for API10 for redirected requests on urllib3
• anthropic
  • Adds support for the Anthropic Beta client API (client.beta.messages.create() and client.beta.messages.stream()). This feature requires Anthropic client version 0.37.0 or higher.
• aiokafka
  • Adds DSM instrumentation support.
  • Adds instrumentation support for aiokafka>=0.9.0. See the aiokafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#aiokafka> documentation for more information.
• Added support for uWSGI with gevent when threads are also patched. The use of the keyword argument thread=False is no longer required when performing monkey-patching with gevent via gevent.monkey.patch_all.
• LLM Observability
  • Reasoning token counts are now captured from Google GenAI responses.
  • The OpenAI integration now captures prompt metadata (id, version, variables, and chat template) for reusable prompts when using the responses endpoint (available in OpenAI SDK >= 1.87.0).
  • Experiments can now be run multiple times by using the optional runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Adds additional tags to MCP client session and tool call spans to power LLM Observability MCP tool call features.
  • Reasoning token counts are now captured from OpenAI and OpenAI Agents responses.
  • openai
    • This introduces support for capturing server-side MCP tool calls invoked via the OpenAI Responses API as a separate span.
• langchain
  • Adds support for tracing RunnableLambda instances.
• mcp
  • Marks client mcp tool call spans as errors when the corresponding server tool call errored
• Crashtracker
  • This introduces a fallback to capture runtime stack frames when Python's _Py_DumpTracebackThreads function is not available.
• ASGI
  • Enable context propagation between websocket message spans.

Bug Fixes

• avro
  • Fixes an issue where Avro instrumentation does not return method results when DSM is enabled.
• crashtracker
  • Fixes missing env variables inheritance for receiver process.
• dynamic instrumentation
  • uploading snapshots now retries on all HTTP error codes.
• exception replay
  • fixed the order in which frames are captured to ensure that the values of frames close to the point where the initial exception was thrown are always attached to the relevant spans.
  • fixed an infinite loop that could cause memory leaks when capturing exceptions, and improved overall speed and memory performance.
  • ensure exception information is captured when exceptions are raised by the GraphQL client library.
• Code Security
  • Fixes critical memory safety issue in IAST when used with forked worker processes (MCP servers with Gunicorn and Uvicorn). Workers previously crashed with segmentation faults due to stale PyObject pointers in native taint maps after fork.
• openai
  • Resolves an issue where instantiating an OpenAI client with a non-string API key resulted in parsing issues.
• tracing
  • Fixed a potential IndexError in partial flush when the finished span counter was out of sync with actual finished spans.
  • DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.
  • Resolves a potential deadlock when forking.
  • CI Visibility: Ensure the http connection is correctly reset in all error scenarios.
• ray
  • This fix resolves an issue where Ray jobs that did not explicitly call ray.init() at the top of their scripts were not properly instrumented, resulting in incomplete traces. To ensure full tracing capabilities, use ddtrace-run when starting your Ray cluster: DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head.
• AAP
  • This fix resolves an issue where the appsec layer was not compatible anymore with the lambda/serverless version of the tracer.
• lib-injection
  • do not inject into the gsutil tool
• LLM Observability
  • Fixes an issue where LLMObs.export_span() would raise when LLMObs is disabled.
  • Resolves an issue where self was being annotated as an input parameter using LLM Observability function decorators.
  • This fix resolves an issue where LLMObs.annotation_context() properties (tags, prompt, and name) were not applied to subsequent LLM operations within the same context block. This occurred when multiple sequential operations (such as Langchain batch calls with structured outputs) were performed, causing only the first operation to receive the annotations.
  • This fix resolves an issue where evaluation-metric labels containing dots could be interpreted as nested objects by adding validation that rejects such labels and provides a clear error message instructing users to use alternative naming conventions.
  • Fixes an issue where the Google ADK integration would throw an AttributeError when trying to access the name or description attributes of a tool.
• opentelemetry
  • Fixed spans going unsampled when using opentelemetry.trace.get_current_span() or NonRecordingSpan. Spans are now kept and appear in the UI unless explicitly dropped by the Agent or sampling rules.
• profiling
  • This fix resolves a critical issue where the Lock Profiler generated release samples for non-sampled lock acquires, resulting in inflated or negative (when integer overflows) lock hold times (e.g., "3.24k days per minute", "-970 days per minute"). This affected virtually all customers using sampling rates < 100% (which should be the majority).
  • This fix prevents a use-after-free crash from the memory profiler on Python version 3.10 and 3.11. The previous attempt to fix this bug itself had a bug, which this fix addresses.
  • improve reliability when parsing an empty span.
  • Fixes a segmentation fault caused by accessing frame.f_locals while trying to retrieve class name of a PyFrameObject.
  • This fix improves the detection of on-CPU asyncio Tasks. Previously, the Profiler would only consider a Task as running if its coroutine was running. The Profiler now recursively checks if any coroutine in the await chain of the Task's coroutine is running.
  • This fix makes stack sampling more accurate for on-CPU asyncio Tasks.
  • This fix resolves a race condition leading to incorrect stacks being reported for asyncio p...