[Bug]: 100% CPU usage and rapidly growing container size after an indeterminate time of normal operation

[whosmatt]

Bug Description

Running fredy via docker-compose with file shown below.
Docker version 29.1.2, build 890dcca,
Linux 6.1.57 #11 SMP Fri Apr 26 12:02:53 CST 2024 aarch64 GNU/Linux
Notice that I am running aarch64, this issue might be arm specific.

After some time (roughly once a week), fredy hogs 1 CPU core at 100% and the container size quickly grows (~70GB in a few hours). Restarting the container brings everything back to normal.
This is the second time this has happened.

I will try to inspect the container live the next time it happens.

Steps to Reproduce

1. Run fredy:

services:
  fredy:
    image: ghcr.io/orangecoding/fredy:master
    container_name: fredy
    restart: unless-stopped
    ports:
      - "9998:9998"
    volumes:
      - ./fredydb:/db
      - ./conf:/conf
    environment:
      CONFIG_JSON: |
        {
          "interval": "9",
          "port": 9998,
          "workingHours": { "from": "", "to": "" },
          "demoMode": false,
          "analyticsEnabled": true,
          "sqlitepath": "/db"
        }
    entrypoint: >
      sh -c 'echo "$$CONFIG_JSON" > /conf/config.json &&
             exec pm2-runtime index.js'

2. Wait (Took about a week in my case)

Expected Behavior

Fredy should recover from errors without leaving orphaned data and/or processes.

Actual Behavior

One Thread gets maxed out to a permanent 100% and the container size grows indefinitely.

Screenshots / Logs

fredy.log

Environment

Docker version 29.1.2, build 890dcca, Linux 6.1.57 #11 SMP Fri Apr 26 12:02:53 CST 2024 aarch64 GNU/Linux

Additional Context

No response

[orangecoding]

Hi @whosmatt ,

In the attached log I see Fredy getting into a Puppeteer failure spiral: repeated navigation/selector timeouts, then Chrome DevTools websocket resets (ECONNRESET), then Puppeteer can no longer bring up a new browser instance (Timed out after 60000 ms while waiting for the WS endpoint URL to appear in stdout!), and later hard resource exhaustion errors like pthread_create: Resource temporarily unavailable (11), fork: Resource temporarily unavailable (11), and spawn /usr/bin/chromium EAGAIN. After that, even PM2 starts failing to spawn simple processes (spawn ps EAGAIN).

That combination usually means the container is hitting a process/thread limit or running out of a kernel resource because Chrome processes are accumulating or not being cleaned up properly after failures. Once it is in that state, a tight retry loop will peg a CPU core, and Chrome can also spam crashpad or profile data on disk, which matches the “container grows to tens of GB quickly” symptom.

You have not mentioned which Fredy version you're on. Can you please add this info?

[whosmatt]

You have not mentioned which Fredy version you're on. Can you please add this info?

Right, sorry. The log is from V15.0.0, but I had the same behavior happen on the previous version.
I just updated to V16.3.0

[orangecoding]

ok. I added quite some improvements to the latest version. Can you please observe it for another week and ping me about the status please? I too have Fredy running here with almost 20 jobs in a docker container and I do not see any issues whatsoever.

I can only think of some issue with Puppeteer...

[orangecoding]

Can you please run node with the inspect flag so that you can provide me a heap dump when this happens again?

https://nodejs.org/en/learn/diagnostics/memory/using-heap-snapshot (Option 2)