GCP: add testbed logging to cloud run

[aaronjae22]

This PR presents a new approach for logging in Cloud Run following best practices. It should (not sure since Cloud Run Logging has been a nightmare) ensure that all logs from the testbed package appear in Cloud Run Logs.

If this works, we can iterate over this implementation to make it more robust and use in our other projects.

Close #222

---

Currently we only have Cloud Run's infrastructure logs which are automatically generated for every HTTP request/response. Cloud Run creates these automatically without any inference from us (but we have been able to modify the looks of them and how they propagate).

Application logs seems to be missing due to StructuredLogHandler which writes logs to stdout but Cloud Run is not ingesting them.

Our new CloudLoggingHandler should sends logs directly to Cloud Logging API with explicit logName, NOT to stdout.

As a result, we'll have two types of logs:

1. Infrastructure Logs (already working):

{
  "logName": "projects/.../logs/run.googleapis.com%2Frequests",
  "httpRequest": {...},
  "trace": "projects/.../traces/751dff...",
  // Automatic from Cloud Run
}

2. Application Logs (NEW - what we'll add):

{
  "logName": "projects/.../logs/testbed",  // ← Custom name!
  "message": "User logged in successfully",
  "trace": "projects/.../traces/751dff...",  // ← Same trace!
  "jsonPayload": {
    "environment": "staging",
    "service": "activitypub-testbed-stg-run",
    ...
  }
}

This is an architecture overview describing the new implementation:

┌─────────────────────────────────────────────────────────────────┐
│                    Testbed Project                           │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │             Code: logger.info("User logged in")                │  │
│  │             logger.error("Database error")                │  │
│  └──────────────────────────────────────────────────────────┘  │
│                            ▼                                    │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │         Python Logging Framework (Django LOGGING)         │  │
│  │                                                            │  │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │  │
│  │  │   Logger:    │  │   Logger:    │  │   Logger:    │   │  │
│  │  │   "django"   │  │  "testbed"   │  │  "gunicorn"  │   │  │
│  │  └──────────────┘  └──────────────┘  └──────────────┘   │  │
│  │         │                 │                 │             │  │
│  │         └─────────────────┴─────────────────┘             │  │
│  │                            ▼                               │  │
│  │         CloudRunTraceFilter (adds metadata)                │  │
│  │                            ▼                               │  │
│  │       ┌──────────────────────────────────┐                │  │
│  │       │   CloudLoggingHandler            │                │  │
│  │       │   (name="testbed")               │                │  │
│  │       └──────────────────────────────────┘                │  │
│  └──────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                            ▼
        ┌──────────────────────────────────────┐
        │   Google Cloud Logging API           │
        │   logName: "projects/PROJECT/logs/testbed" │
        └──────────────────────────────────────┘

CloudLoggingHandlers (from google-cloud-logging library) sends logs records directly to Cloud Logging API

┌─────────────────────────────────────────────────────┐
│           Application Startup                        │
└─────────────────────────────────────────────────────┘
                    ▼
┌─────────────────────────────────────────────────────┐
│  Check: USE_GCLOUD_LOGGING environment variable?    │
└─────────────────────────────────────────────────────┘
         │                                 │
         │ YES (Cloud Run)                 │ NO (Local Dev)
         ▼                                 ▼
┌──────────────────────┐         ┌──────────────────────┐
│  Initialize:         │         │  Use Console Logger  │
│  - CloudLogging      │         │  - Rich formatting   │
│  - CloudLogging      │         │  - Structlog         │
│    Handler           │         │  - Human-readable    │
│  - TraceFilter       │         │    output            │
│                      │         │                      │
│  Attach to:          │         │                      │
│  - django logger     │         │                      │
│  - testbed logger    │         │                      │
│  - gunicorn logger   │         │                      │
└──────────────────────┘         └──────────────────────┘

I added a environment variable in our Cloud Run Staging Service called USE_CLOUD_LOGGING.

[lisad]

I guess this is a workaround for circular dependencies. But it's also an indicator that maybe this import, and the imports from google.cloud.logging, should be done elsewhere.

I also note that the Google Cloud settings are needed for production and staging, not for CI or dev settings.

I don't have a firm opinion what to do about this, but in looking around I did discover https://django-classy-settings.readthedocs.io/latest/ and https://github.com/dmitrikozlov/django-logging

[aaronjae22]

Yes, absolutely right about the circular dependencies and I completely missed that Google Cloud Settings were also being imported in other settings files. Thanks for it.

I've refactored the implementation and also took your suggestion into consideration and it's basically a trade off.

The django-classy-settings was something I thought about it when I was creating the settings structure a while ago but at the time I wanted to follow the approach we have had in our other projects. Also it would require refactoring all settings files and it's pretty much an approach shift. We can discuss whether we should implement it or keep it as it is.

And about the django-logging wrapper I think it has a couple of interesting things, I like it but google-cloud-logging already supports structured logging and it has a couple of custom methods like logger.notice() that are not part of Python's logging module but they work on Cloud Logging so I am not pretty confident about using it.

I decided to keep the implementation but used a factory function to eliminate the circular imports and a better environment isolation (Google Cloud related code was moved to production.py which will be inherited by staging.py)