Signing Container Images

Available for Advanced Code Signing, Open Source Code Signing.

Overview

There are multiple technologies available for signing container images and they all differ from classic code signing methods as used by most platforms. The different technologies follow their individual philosophies and have their specific advantages and shortcomings.

Why use SignPath for container signing?

SignPath provides the following advantages:

• Your signing keys are securely stored on a Hardware Security Module (HSM)
• You can use the full power of SignPath signing policies, including permissions, approval, and origin verification
• You can use all Pipeline integrity features of SignPath, ensuring that the image was built from approved source code on a trusted build system
• Configuration and policy management is aligned with other signing methods, such as Authenticode or Java signing
• SignPath maintains a full audit log of all signing activities including metadata such as the registry URL and signed image tag
• You can sign multiple images in a single signing request, making audits/reviews of multi-image releases a lot easier

For Notary (Notation) and Sigstore Cosign, there are additional specific advantages:

• You can sign your images before they are pushed, making sure that only signed images are available in your registry
• Signing tools can be centrally managed and updates/changes in technology have zero effect on your build pipelines

For Sigstore Cosign, there are additional specific advantages:

• You can authenticate automated build systems instead of individual developers and leverage origin verification for CI systems that do not support Cosign workload identities (currently only Github and Gitlab in their SaaS offerings)
• You can use your own key material and keep your signature data private without having to operate an own Fulcio certificate authority system

Different technologies

SignPath supports these technologies for signing container images:

• Notary (Notation): Sign containers using Notary - recommended by Microsoft (AKS) and Amazon (EKS)
• Sigstore Cosign: Sign containers using Cosign by Sigstore (a Linux foundation project)

The Code Signing Gateway additionally supports

• GPG: Sign containers using GPG keys for RedHat OpenShift

Docker Content Trust (DCT) is deprecated

SignPath also supports the deprecated Docker Content Trust (DCT) signing method.

Which technology to use

In practice, Notary (Notation) is recommended for Enterprises and Cosign for open source projects.

For a detailed comparison between all the different technologies, see the appendix.

Notary (Notation)

The Notary project (aka Notary v2) is the successor of Docker Content Trust. It is developed by the Cloud Native Computing Foundatio. Both Microsoft and Amazon recommend Notation for signing container images deployed in their respective Kubernetes offerings.

Notation uses standard PKI, is easy to set up and has a mature trust model. SignPath recommends Notation for signing container images in Enterprise environments.

Sigstore Cosign

Cosign is part of the Sigstore project. It is primarily targeted at the open source community, allowing individual developers to sign container images using OpenID user accounts from GitHub, Google or Microsoft. For those developers, Sigstore eliminates the need for certificates or locally stored private keys.

Sigstore architecture

Sigstore combines the Cosign tool, the Fulcio certificate authority and the Rekor transparency log as follows:

1. Cosign creates a metadata file for signing
2. Fulcio authenticates the user account using OpenID Connect (OIDC)
3. Fulcio creates a short-lived certificate for the OIDC identity using an ephemeral key and signs the metadata digest
4. Rekor logs the signature in its public transparency log
5. Cosign uploads signature and metadata to the image’s repository

No Fulcio support in SignPath

SignPath only supports cosign signatures based on public/private key pairs, not Fulcio-issued X.509 certificates. These certificates require specific additional attributes. The sigstore project is actively working on supporting custom certificates from traditional PKIs.

How to sign

With Advanced Code Signing

To sign container images with SignPath, the following steps are required:

1. Store the container image in an OCI-compliant layout

We recommend signing the image before it is pushed to a container registry. This requires the image to be stored in a .tar archive locally. You can use docker build with a build driver that supports building OCI tarballs, like docker-container.

docker buildx build -o type=oci,dest=image.tar ...

2. Optional: Strip the content layers

If you do not want to upload the entire container image to SignPath, we provide a utility tool to strip the content layers before signing and repack them afterwards. In this case, only the metadata is sent to SignPath.

On Linux

curl -s https://download.signpath.io/releases/sp-oci/1-latest/linux/x64/sp-oci.tar.gz | tar -xzf - sp-oci
./sp-oci strip image.tar --reference $imageReference --directory .tmp

Verifying the sp-oci signature

You can verify the signature of the sp-oci executable using our public GPG key by also extracting the detached signature file sp-oci.asc from the .tar.gz

curl -s ... | tar -C /tmp -xzf - sp-oci sp-oci.asc
gpg --verify sp-oci.asc sp-oci

On Windows

Invoke-WebRequest "https://download.signpath.io/releases/sp-oci/1-latest/windows/x64/sp-oci.zip" `
  -OutFile "${env:TEMP}\sp-oci.zip"
Expand-Archive -DestinationPath "$(Get-Location)\sp-oci" "${env:TEMP}\sp-oci.zip"

.\sp-oci\sp-oci.exe strip image.tar --reference $imageReference --directory .tmp

Verifying the sp-oci.exe signature

You can verify the signature of the sp-oci.exe executable by calling

Get-AuthenticodeSignature .\sp-oci\sp-oci.exe

No security degradation

All layer references are cryptographically secure and included in the metadata. There is no security degradation by not including them when signing.

3. Sign the tarball with SignPath

• Create an artifact configuration for signing containers with Notation or Cosign respectively.
• Submit the tarball for signing using any of the native build system integrations, the PowerShell module or the REST API.

4. If stripped, repack the content layers again

In case you stripped the content layers of the image in step 2, now it is time to repack them:

On Linux

./sp-oci repack image-signed.tar --reference $imageReference --directory .tmp

On Windows

.\sp-oci\sp-oci.exe repack image-signed.tar --reference $imageReference --directory .tmp

5. Publish the signed image

Use the oras tool to publish the container image including the signatures to your registry:

oras cp -r --from-oci-layout image-signed.tar:$imageTag registry.mycompany.com/$imageReference

Image references

An image reference consists of the following parts:

• Optionally, the registry host and port, e.g. docker.io or registry.mycompany.com:3000 - while the image is only available locally, this should be omitted
• The namespace and/or repository, e.g. jetbrains/teamcity-server
• The $tag identifying the version, e.g. latest

For images hosted on Docker Hub, the image reference is docker.io/$namespace/$repository:$tag, e.g. docker.io/jetbrains/teamcity-server:latest.

If you are using your own registry, specify the value you would use for Docker CLI commands, e.g. registry.mycompany.com/myrepo/myimage:latest.

With the Code Signing Gateway

Notary (Notation)

Notary signatures are created using the Notation CLI tool. SignPath provides a plugin for signing. See the SignPath Notation Plugin page for details.

Sigstore Cosign

The cosign CLI tool supports keys stored on HSMs via the PKCS11 protocol. You can use SignPath’s PKCS#11 module to sign your container image. See cosign’s official documentation for details.

GPG

Standard GPG signatures can be used to sign container images and verify them using podman. See Red Hat’s official documentation on how to perform the signing.

There is no standard way of distributing GPG-based container image signatures (e.g. in an OCI-compliant registry), they have to be transferred out-of-band to the target system.

To integrate SignPath, refer to our GPG CryptoProvider documentation.

Appendix: Detailed comparison between the different technologies