Skip to content

Fix 401 Unauthorized on log push after token refresh#269

Open
cchristous wants to merge 1 commit intosemaphoreci:masterfrom
cchristous:fix/retry-log-push-after-token-refresh
Open

Fix 401 Unauthorized on log push after token refresh#269
cchristous wants to merge 1 commit intosemaphoreci:masterfrom
cchristous:fix/retry-log-push-after-token-refresh

Conversation

@cchristous
Copy link

@cchristous cchristous commented Jan 31, 2026

Problem

When pushing logs to the Semaphore API, the agent periodically encounters 401 Unauthorized responses due to token expiration. The existing code correctly detects this and calls RefreshTokenFn() to obtain a new token, but then returns an error instead of retrying the request with the refreshed token.

This results in log push failures that look like:

Refreshing token for current job logs...
Successfully refreshed token for current job logs.
Error pushing logs: request to <url> failed: 401 Unauthorized

The token was refreshed successfully, but the log batch that triggered the refresh was never retried and is lost until the next push cycle.

Root Cause

In the original newRequest() function, the HTTP request was created inline. When a 401 was received:

  1. Token refresh was triggered and succeeded
  2. l.config.Token was updated with the new token
  3. An error was returned, abandoning the current request
  4. The next push cycle would use the new token, but the current batch's timing was disrupted

The request body (log buffer) was consumed by the first request attempt and couldn't be reused for a retry.

Solution

  1. Store buffer contents before the request: Capture buffer.Bytes() before creating the HTTP request so it can be reused
  2. Extract request logic to sendLogRequest(): Separate function that can be called recursively
  3. Retry once after token refresh: On 401, refresh the token and immediately retry with isRetry=true
  4. Prevent infinite loops: The isRetry flag ensures we only retry once - a second 401 returns an error

Why This Is Safe

  • Single retry limit: The isRetry boolean prevents infinite retry loops. If the refreshed token also fails, we return an error.
  • No data loss: Buffer contents are stored before the first request, so the same log batch is sent on retry.
  • Existing error handling preserved: All other status codes (200, 422, etc.) are handled identically to before.
  • Token refresh already tested: The RefreshTokenFn path was already exercised; we're just making use of the new token immediately.
  • Follows HTTP client conventions: Retrying with fresh credentials after a 401 is standard practice.

Test plan

  • Unit tests pass (go test ./...)
  • Integration test with token expiration scenario
  • Verify log push succeeds after token refresh in staging environment

Previously, when a log push request received a 401 Unauthorized response,
the agent would refresh the token but then return an error, causing a
spurious "Error pushing logs: 401 Unauthorized" message to be logged.
The next push would succeed with the new token, but the error log was
confusing and happened every hour when tokens expired.

This change:
- Extracts the HTTP request logic into a separate sendLogRequest function
- Stores the buffer contents before sending so they can be reused on retry
- After refreshing the token, immediately retries the request instead of
  returning an error
- Limits retry to once to avoid infinite loops

The result is cleaner logs: instead of seeing "Successfully refreshed
token" followed by "Error pushing logs: 401 Unauthorized", users will
see "Retrying log push with refreshed token..." and the request succeeds.
@cchristous cchristous marked this pull request as ready for review January 31, 2026 14:40
@cchristous
Copy link
Author

I cannot see the test results to debug the failure. I see a 404

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant