Worker: uncaught exception when logs failed

[josephjclark]

We shouldn't be seeing this. The log should fail and life does on. I think that exception might be messing things up inside the worker, and may even be causing a sort of zombie process where the worker just sits there owning capacity.

This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:

LightningTimeoutError: [run:log] timeout
    at Object.callback (file:///app/packages/ws-worker/dist/start.js:717:16)
    at file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:109:71
    at Array.forEach (<anonymous>)
    at Push.matchReceive (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:109:54)
    at Object.callback (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:140:12)
    at Channel.trigger (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:460:12)
    at Push.trigger (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:156:18)
    at Timeout.<anonymous> (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:143:12)
    at listOnTimeout (node:internal/timers:588:17)
    at process.processTimers (node:internal/timers:523:7)

GCP evidence 1

Note from the logs that this run seems to hang around for like 4 hours before it finally gives up, long after it was marked as lost.

[josephjclark]

5 runs lost yesterday with this pattern:

• Workflow finishes, everything normal
• A log times out: 62121d06-ac74-404c-86c1-6a604c08e974 :: run:log :: ERR: [run:log] timeout
• That uncaught exception pops up
• connection drops: Critical error in channel run:62121d06-ac74-404c-86c1-6a604c08e974 [null]
• Later, like 4 hours later, the run will try and fail to reconnect: REFUSED JOIN run:03efadf9-d72f-4e44-b2ef-c7e5fb516915 in 3ms

[josephjclark]

18 uncaught errors in the last 48 hours. Do we think each one results in a lost run?

It also strikes me that this event is occurring in the main thread (yes, of course, that's where the socket calls are). So it could cause a lot of problems there

[josephjclark]

The last worker release, which triggered all this, added the batch logging stuff. That can't be coincidence - it has to be something in that PR that's introduced this instability