SHIP-0042: Trusted certificates

[sayan-biswas]

Proposal to add a Shipwright feature to use trusted certificates in Build steps.

Changes

Submitter Checklist

• Includes tests if functionality changed/was added
• Includes docs if changes are user-facing
• Set a kind label on this PR
• Release notes block has been filled in, or marked NONE

See the contributor guide
for details on coding conventions, github and prow interactions, and the code review process.

Release Notes

[adambkaplan]

Initial comments since this proposal is in the draft state. I'd like to see the scope of this proposal more clearly defined before accepting it in the "provisional" state.

[adambkaplan]

I like that we are supporting multiple certificates - this enables more fine grained distribution of trust.

I am concerned that the naming "certificates" is too broad. Are we concerned with certificate authorities for TLS verficiation? X.509 certificates for workload identity? Mutual TLS certificates between services? These are all very different things that utilize similar underlying fundamental technologies.

[sayan-biswas]

I think we are only concerned about X.509 certificates here.

[adambkaplan]

I think the scope of this SHIP should be narrowed to TLS certificate authority trust bundles. The prerequisite is that a cluster administrator defines and distributes a trust bundle, which the containers in a Shipwright build consume.

Adding additional CAs to a Linux system is otherwise a huge pain - you need to save/mount the CA to the right location (which is distro-dependent), and likewise run the right "update-ca" command as root, which is likewise distro-dependent. Mounting the CA bundle avoids the requirement to run containers as root.

I used the following prompt in Google Gemini with its "Thinking" model to help research the general requirements:

I'm trying to understand how certificate authority trust bundles should be mounted into Linux
containers running on Kubernetes. I plan on using cert-manager's trust-manager component
to generate a CA bundle that is stored in a ConfigMap: https://cert-manager.io/docs/trust/. 
I would like to know the following: 

1. Where should this ca bundle in the ConfigMap be mounted in the container? 
2. Should this file be mounted in more than one place? 
3. Does mount location vary depending on the base image used for the container - ex is the
   mount point different for ubuntu, alpine, and Red Hat UBI images?
4. Do these files need to be mounted with a specific name?

Below is the response, which matches my knowledge and experience. Edit: I have called out a few areas where I think the LLM is wrong, or I at least disagree with the output:

---

When using cert-manager’s trust-manager, you are essentially automating the creation of a "source of truth" CA bundle. To make your containerized applications trust this bundle, you must mount it into the specific paths where those applications or the operating system's libraries look for root certificates.

1. Where should the CA bundle be mounted?

The mount point depends primarily on the base image of your container. Most Linux-based applications (like those written in Go, or tools like curl) look for a single file containing a concatenation of all trusted PEM-formatted certificates.

OS / Base Image	Primary Bundle Path
Ubuntu / Debian	/etc/ssl/certs/ca-certificates.crt
Alpine Linux	/etc/ssl/certs/ca-certificates.crt
Red Hat UBI / CentOS	/etc/pki/tls/certs/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Important: If you mount the ConfigMap directly to /etc/ssl/certs/, you will overwrite the entire directory. This removes all the default public CAs (like DigiCert, Let's Encrypt, etc.) that came with the image.

• Recommendation: Use subPath to mount only the specific file, or ensure your trust-manager Bundle includes useDefaultCAs: true so your generated bundle contains both your internal CAs and the public ones.

2. Should this file be mounted in more than one place?

Yes, in two specific scenarios:

• To support different libraries: Some legacy applications or specific languages look for certificates in different directories. For example, on RHEL/UBI, it is common to mount or link the bundle to both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem. Human note - this is wrong. On Fedora/RHEL /etc/pki/tls/certs/ca-bundle.crt is symlinked to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.
• Java Applications: Java does not read PEM files from the filesystem. It uses a binary "truststore" (usually called cacerts).
• Solution: trust-manager can generate a JKS or PKCS12 truststore automatically. You should mount the PEM bundle for OS tools and a separate JKS/PKCS12 file to $JAVA_HOME/lib/security/cacerts.

Human note: On Fedora/RHEL the Java cert bundle is stored in /etc/pki/ca-trust/extracted/java/cacerts.

3. Does the mount location vary by base image?

Yes. Linux distributions are split into two main families regarding certificate management:

• Debian-based (Ubuntu, Debian, Alpine): Use /etc/ssl/certs/ca-certificates.crt. Alpine also occasionally uses /etc/ssl/cert.pem as a symlink.
• Red Hat-based (UBI, RHEL, Fedora): Use /etc/pki/tls/certs/ca-bundle.crt. Red Hat systems also use a "trust anchors" system, but for a simple "mount and run" approach in a container, targeting the ca-bundle.crt path is standard. Human note: Is it, though?

4. Do these files need to be mounted with a specific name?

While many applications can be configured via environment variables to look for a custom filename (e.g., SSL_CERT_FILE for OpenSSL or NODE_EXTRA_CA_CERTS for Node.js), it is best practice to use the system-standard name to ensure all tools (curl, wget, etc.) work out of the box.

• For Ubuntu/Alpine: Mount the ConfigMap key as ca-certificates.crt.
• For Red Hat UBI: Mount the ConfigMap key as ca-bundle.crt.

Example Kubernetes Deployment Snippet

If you are using trust-manager to generate a bundle named my-org-bundle with a key root-ca.pem, here is how you would mount it for an Ubuntu container:

spec:
  containers:
  - name: my-app
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/ssl/certs/ca-certificates.crt
      subPath: ca-certificates.crt # Prevents overwriting the whole /etc/ssl/certs dir
  volumes:
  - name: trusted-ca
    configMap:
      name: my-org-bundle
      items:
      - key: root-ca.pem           # The key in your ConfigMap
        path: ca-certificates.crt  # The name it will take inside the container

Summary Checklist

1. Check the Base Image: Target /etc/ssl/certs/ for Ubuntu/Alpine and /etc/pki/tls/certs/ for UBI.
2. Use subPath: Unless you want to delete all existing system certificates, use subPath.
3. Environment Variables: For Node.js, additionally set NODE_EXTRA_CA_CERTS=/path/to/your/bundle.pem. For Python (Requests), set REQUESTS_CA_BUNDLE.
4. Java: Use the jks or pkcs12 target in your trust-manager Bundle spec and mount it to the Java cacerts path.

[adambkaplan]

The one thing I would add in light of a recent issue I became aware of (konflux-ci/build-definitions#3149), is that the recommended mount points should not overwrite a symlink.

[adambkaplan]

This is what I think the container mount points ultimately should look like for UBI containers:

spec:
  containers:
  - name: my-ubi-app
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      subPath: ca-bundle.pem # Prevents overwriting the whole directory, just the one file.
   - name: trusted-ca
     mountPath: /etc/pki/ca-trust/extracted/java/cacerts
     subPath: bundle.jks # Provided by the same ConfigMap
  volumes:
  - name: trusted-ca
    configMap:
      name: my-org-bundle

[sayan-biswas]

@adambkaplan Added the considerations regarding the mount point.

[adambkaplan]

/approve

This SHIP meets the criteria for a provisional SHIP, in that:

• There is a defined need/use case.
• There is a proposed solution that aligns with prior community discussions (see #265).

I have added comments with respect to getting this proposal to an "implementable" state - there are more details that need to be hashed out/refined before we broadcast this feature to the community.

[adambkaplan]

This implies that the build strategy also defines certificate mount points that are separate and distinct from the current build volumes API.

Since this SHIP is in the provisional phase, I'm not going to block merge on this point. However we should discuss if this API is necessary before merging this as "implementable"

[adambkaplan]

note - This actually breaks on the latest versions of Fedora CoreOS + OKD for very complex reasons. See konflux-ci/build-definitions#3149

[adambkaplan]

Feedback for getting this proposal to "implementable" - be specific about our "challenges" by linking to GitHub issues or other documented errors.

[adambkaplan]

Feedback for getting this proposal to "implementable" - articulate the non-goals. This is particularly relevant with respect to what we mean by certificates below:

• Are the referenced certificates intended to be additive certificate authorities to the system's trust store?
• Are the referenced certificates intended to be trust bundles for a system's trust store?

These two are very different use cases with different implementation requirements.

[adambkaplan]

This implies that the feature should be scoped to mounting trust bundles, not additive certificate authorities.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adambkaplan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [adambkaplan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment