Skip to content

Conversation

@staktrace
Copy link
Collaborator

In downstream consumers we sometimes see errors during tempest/dynamo startup but unclear root cause. This attempts to narrow down the source of the problem by adding some logging.

In downstream consumers we sometimes see errors during tempest/dynamo
startup but unclear root cause. This attempts to narrow down the source
of the problem by adding some logging.
@staktrace
Copy link
Collaborator Author

An example of the type of error we get:

10:29:06.900 [TestDynamoDbService STARTING] INFO app.cash.tempest2.testing.internal.TestDynamoDbService -- starting tempest2-jvm-dynamodb-local-32839
10:29:06.902 [ReadyService STARTING] INFO misk.ReadyService -- Starting ready service
10:29:06.905 [ReadyService STARTING] INFO misk.web.interceptors.RebalancingInterceptor -- sending connection: close to 0.00% of all responses
Initializing DynamoDB Local with the following configuration:
Port:	32839
InMemory:	true
Version:	2.6.1
DbPath:	null
SharedDb:	false
shouldDelayTransientStatuses:	false
CorsParams:	null

10:29:07.543 [JvmDynamoDbServer STARTING] INFO app.cash.tempest.testing.dynamodb.local.shaded.org.eclipse.jetty.server.Server -- jetty-12.0.27; built: 2025-09-10T23:47:49.595Z; git: 3569a3e83ad136ee44e26b370b74c1c5e9f33e61; jvm 21.0.9+10-LTS
10:29:07.559 [JvmDynamoDbServer STARTING] INFO app.cash.tempest.testing.dynamodb.local.shaded.org.eclipse.jetty.server.handler.ContextHandler -- Started acttdlsoejsh.ContextHandler@1492a954{ROOT,/,b=null,a=AVAILABLE,h=acttdlscasd2ls.LocalDynamoDBServerHandler@1b527132{STARTED}}
10:29:07.568 [JvmDynamoDbServer STARTING] INFO app.cash.tempest.testing.dynamodb.local.shaded.org.eclipse.jetty.server.AbstractConnector -- Started ServerConnector@14ae444d{HTTP/1.1, (http/1.1)}{0.0.0.0:32839}
10:29:07.570 [JvmDynamoDbServer STARTING] INFO app.cash.tempest.testing.dynamodb.local.shaded.org.eclipse.jetty.server.Server -- Started acttdlsoejs.Server@6abe7d26{STARTING}[12.0.27,sto=0] @10543ms
10:29:07.955 [TestDynamoDbService STARTING] ERROR misk.ServiceManagerModule -- Service misk.aws2.dynamodb.testing.TestDynamoDb@6cc53274 failed
java.lang.IllegalStateException: Expected the service DefaultTestDynamoDbClient [FAILED] to be RUNNING, but the service has FAILED
	at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:387)
	at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:307)
	at com.google.common.util.concurrent.AbstractIdleService.awaitRunning(AbstractIdleService.java:171)
	at app.cash.tempest2.testing.internal.TestDynamoDbService.startUp(TestDynamoDbService.kt:22)
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:65)
	at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Service returned HTTP status code 404 (Service: DynamoDb, Status Code: 404, Request ID: null) (SDK Attempt Count: 1)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:113)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:61)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.listTables(DefaultDynamoDbClient.java:8388)
	at software.amazon.awssdk.services.dynamodb.DynamoDbClient.listTables(DynamoDbClient.java:6989)
	at app.cash.tempest2.testing.internal.DefaultTestDynamoDbClient.reset(DefaultTestDynamoDbClient.kt:47)
	at app.cash.tempest2.testing.internal.DefaultTestDynamoDbClient.startUp(DefaultTestDynamoDbClient.kt:37)

@staktrace staktrace closed this Nov 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant