The BlueAPI helm chart currently has some simple configuration for logging. BlueAPI configures the root logger, which is then automatically used by our dodal+bluesky+ophyd logs.
Our debug logs are too noisy to be sending to graylog, so we need a way to save our debug logs to a file when there's an exception.
Using k8's for logs also isn't suitable since it deletes them after 2 pod restarts.
Proposed short term solution:
- Run dodal's
set_up_DEBUG_memory_handler in our blueapi plan entry-points
- In ixx-values.yaml, mount the debug logging location to a PVC
This will effectively give us the same behaviour as Hyperion has
Longer-term solution:
See #1213
Acceptance Criteria
- All of our debug logs are retrievable when running BlueAPI