pulp content upload --directory --use-temp-repository fails to call signing service

[josephtate]

Summary

When bulk-uploading a directory full of packages, all the packages get signed, but if I add --use-temp-repository, the packages all bypass signing.

Steps to reproduce

Create a repository with a signing service.
Gather some unsigned RPMs to test with, or strip signatures off existing ones.

Upload a package with --use-temp-repository and --directory:
pulp rpm content -t package upload --directory missing-pkgs --repository release-x86_64-os --chunk-size=8MB --use-temp-repository

Download a package from your published repository version and note that none of the uploaded packages are signed.

Expected behavior

Each package should be signed when it is added to the repository, whether or not it passes through a temporary repository.

Stacktrace/Error log

N/A

Pulp and pulp-cli version info

$ pulp --version
Pulp3 Command Line Interface, Version 0.32.0.dev
Plugin Versions:
  common: 0.32.0.dev
$ pulp status
{
  "versions": [
    {
      "component": "core",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.23.1",
      "package": "pulp-ansible",
      "module": "pulp_ansible.app",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.22.0",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": false
    },
    {
      "component": "deb",
      "version": "3.4.0",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "maven",
      "version": "0.8.1",
      "package": "pulp-maven",
      "module": "pulp_maven.app",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.4.4",
      "package": "pulp-ostree",
      "module": "pulp_ostree.app",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.12.5",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.27.3.dev",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    },
    {
      "component": "certguard",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/01961626-02e6-7906-a83e-4007fc4f0f0f/",
      "prn": "prn:core.worker:01961626-02e6-7906-a83e-4007fc4f0f0f",
      "pulp_created": "2025-04-08T16:05:12.040975Z",
      "pulp_last_updated": "2025-04-08T16:05:12.041042Z",
      "name": "1@pulp-worker-757fcbb6df-6c8jw",
      "last_heartbeat": "2025-05-22T04:57:44.073499Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/01961789-2643-7092-89ab-cb46f7609424/",
      "prn": "prn:core.worker:01961789-2643-7092-89ab-cb46f7609424",
      "pulp_created": "2025-04-08T22:33:06.373643Z",
      "pulp_last_updated": "2025-04-08T22:33:06.373661Z",
      "name": "1@pulp-worker-757fcbb6df-jzzqb",
      "last_heartbeat": "2025-05-22T04:57:47.139500Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    }
  ],
  "online_api_apps": [
    {
      "name": "9@pulp-api-5c7dbb4968-kr7wm",
      "last_heartbeat": "2025-05-22T04:57:51.651460Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "10@pulp-api-5c7dbb4968-kr7wm",
      "last_heartbeat": "2025-05-22T04:57:51.642939Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "online_content_apps": [
    {
      "name": "10@pulp-content-559495989b-gprv6",
      "last_heartbeat": "2025-05-22T04:57:48.425790Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "11@pulp-content-559495989b-gprv6",
      "last_heartbeat": "2025-05-22T04:57:45.453453Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": false
  },
  "storage": {
    "total": null,
    "used": 141642827633,
    "free": null
  },
  "content_settings": {
    "content_origin": "https://pulp.example.com",
    "content_path_prefix": "/pulp/content/"
  },
  "domain_enabled": false
}

Additonal context

N/A

[mdellweg]

@ggainey thoughts?

Is copy carrying along signatures? Should the temp-repository duplicate the signing config of the target repository?

[josephtate]

Signatures modify the files themselves, so having the temporary repository copy the signing config of the target would solve this. However, I think OTHER "content" operations might bypass signing too, e.g., copying from an existing non-temporary repository. Or copying from a repository with a different signing key. I don't necessarily need that since I'm using the same signing keys across projects that share RPMs, but it's an uncomfortable moment when you discover unsigned items where you thought you were covered.

[mdellweg]

So would you expect that the copy operation should handle signing? (No promises here. I actually expect a heated discussion to emerge from this.)

[josephtate]

I'd expect that content upload would perform the same set of actions/modifiers/checks, with or without --use-temp-repository, except with fewer repository versions. I'd expect it to be a way to simulate an atomic-style bulk upload; I.e., Roll back/discard if any part of the upload fails.

If that's done by copying the signing service from the primary to the temporary repository, I think that's probably cleaner. I DON'T think that normal copy API commands should result in package signing though (but I can think of use cases where I'd want that, just not good ones).

In my world, everything in Pulp should be signed. Whether that happens before it's uploaded to pulp or as it's uploaded doesn't particularly concern me, but it's definitely more convenient to have it during upload. I actually have 2 sets of repositories that I "collect" into my testing repo, one has signed packages, and is synced from a builder repository. The other comes from random builders, so it's the upload to pulp that triggers the signing.