Ratify v2 Configuration Design

[binbin-li]

In Ratify v1, we defined four different Custom Resource Definitions (CRDs) to configure the following components:

• Verifier
• Store
• Policy Enforcer
• Key Management Provider (KMP)

To support multi-tenancy, each component has both cluster-scoped and namespaced variants. This results in a total of 8 CRDs. In a single-tenant setup, users (or Helm charts) must apply at least four separate Custom Resources (CRs) for Ratify to function correctly.

Motivation for Change

In Ratify v2, we are exploring the idea of consolidating all configurations into a single CR. The goal is to simplify setup and management, especially for common use cases.

User Experience

While the number of configuration parameters remains the same between the two approaches (split vs. single CR), user experience is notably different. For simple scenarios—such as verifying an image signed with a Notation signature—a single CR can be much more concise and user-friendly.

Example: Consolidated Configuration

{
    "scopes": [
        "ghcr.io",
        "test.azurecr.io"
    ],
    "verifiers": [
        {
            "name": "notation-1",
            "type": "notation",
            "parameters": {
                "certificates": [
                    {
                        "files": [
                            "${CERTIFICATE_PATH}"
                        ]
                    }
                ]
            }
        }
    ],
    "stores": {
        "ghcr.io": {
            "type": "registry-store",
            "parameters": {
                "credential": {
                    "username": "",
                    "password": ""
                }
            }
        }
    },
    "policyEnforcer": {
        "type": "threshold-policy",
        "parameters": {
            "policy": {
                "rules": [
                    {
                        "verifierName": "notation-1"
                    }
                ]
            }
        }
    }
}

Benefits of a Unified CR

• Simplicity: Especially beneficial for simple setups (e.g., single verifier + single store).
• Native Scope Support: Scoping is handled inherently in a single CR. In the split model, each resource would need an additional scope field, adding overhead and complexity.

However, a single "giant" CR can also introduce drawbacks:

• Managing individual resources (e.g., adding/removing verifiers or stores) requires editing and reapplying the entire configuration.
• Large configurations may exceed Kubernetes' etcd size limits.
• Error handling becomes less granular—one failure could block the entire configuration.

We'll focus on the user experience comparison in this section and ignore the implementation complexity. The below table shows the comparison between 2 approaches.

User Experience Comparison

Feature	Single Config (Unified CR)	Split Config (Multiple CRs)
Scalability	Risk of exceeding etcd object size limits	Each CR is small and manageable
Atomicity	All-or-nothing; failure in one area blocks the full config	Independent success/failure per CR
Error Reporting	Single CR cannot show all errors if too many	Each CR can show its error
Grouping by Scope	Natively supported	Each CR must include and manage a scope field
Management	Easier in single-scope use cases	Requires tracking multiple CRs
Update Resource	Requires editing the full config	Can update individual CRs
Add Resource	Must update and reapply the entire config	Simply create a new CR
Delete Resource	Must update and reapply the entire config	Simply delete the relevant CR

Recommendation

The choice between a single unified CR and multiple specialized CRs should be driven by the use case:

• Use a unified CR when:
  • The setup is simple (e.g., one verifier, one store)
  • You want native scoping support
  • Atomic configuration updates are desired
• Use split CRs when:
  • Configurations are large or complex
  • Fine-grained resource lifecycle management is important
  • You want better modularity and isolated error handling

[FeynmanZhou]

Thanks @binbin-li for proposing such detailed proposal.

I like the design of consolidating multiple CRDs to a single CRD since it aligns with the goal of simplifying setup and management for common use cases. In terms of the drawbacks listed above, here are considerations and potential trade-offs from the user's perspective:

Managing individual resources (e.g., adding/removing verifiers or stores) requires editing and reapplying the entire configuration.

In general, most of the configurations in these four CRDs are not frequently changed by users, it is likely an one-time configuration for each verifier/store/policy enforcer for users. A single unified CRD does not increase the maintenance complexity.

Large configurations may exceed Kubernetes' etcd size limits.

How large will the single unified CRD be? Is there a way to mitigate this risk in case Ratify unified CRD exceed Kubernetes' etcd size limits? I assume this is configurable for etcd https://etcd.io/docs/v3.4/dev-guide/limit/

Error handling becomes less granular—one failure could block the entire configuration.

As long as Ratify logs could locate where is the misconfiguration happended in the unified CRD, it looks no significant difference with multiple CRDs design since users can easily mitigate the failure via the error logs in case any misconfiguration.

---

In addition, I have two questions:

How easier of the migration path? Can users with existing multiple CRDs setups transition to a single CRD without service disruptions? I would expect transitioning a clear and well-documented migration path for users.

Are we able to support a hybrid approach and make two CRD models configurable? By default, it's a unified single CRD for common scenarios, Ratify allow users to change between single or multiple CRD configurations based on their specific requirements before or after setup. This provides flexibility and extensibility for advanced use cases.

[FeynmanZhou]

This is a primary change for Ratify users.

I would like to invite @lefteris @DahuK @fseldow @kingnarmer @kingnarmer to share your opinions or concerns if you have. Thanks :-)

[DahuK]

I agree the hybrid approach. We should add a more unified and simplified CRD. For most users, the current setup is overly complex, it requires cluster admins to understand certificate, KMS, IRSA/RRSA/workload identity, notation trust policies and complex Gatekeeper configurations...
Some users may only require basic compliance, they just need registry scope signing certificate configuration and a simple ORAS store to pull private registry signatures. We should offer a minimal, default-to-simplicity configuration approach for these cases.
For advanced users, maintaining loosely coupled configurations can still meet their fine-grained configuration needs.

[yizha1]

As shared during the community meeting, we could consider a flexible hybrid approach to balance usability, modularity and migration. The hybrid approach is that Ratify v2 introduces a new CRD as a higher-level CRD providing a simplified user interface for users. This higher-level CRD only inlucdes parameters for common scenarios. The existing multiple CRDs are still kept in v2 as lower-level CRDs, and users can also fine-tune resources using those lower-level CRDs if they want. With this, it's easy for v1 to migrate as all the parameters are still there. We will be recommending higher-level CRD for all users, watching any issues with it, and may deprecate lower-level CRDs when time comes. For v1 users in production, they have a period to figure out the path to transit to the higher-level CRD.

[binbin-li]

Thanks for reviewing the proposal and providing feedback! Before diving into the details, I’d like to clarify two points:

1. Ratify v2 is fundamentally different from v1 in terms of implementation. From a user experience standpoint, v2 may require either more or fewer inputs compared to v1. This means that even if we continue using separate CRDs, a component like the Notation Verifier could require different parameters in v2. That said, we’ll aim to minimize differences between v1 and v2 as much as possible.

2. A key advantage of a unified CRD is its ability to define a scope across a group of resources. Supporting this capability with split CRDs would require each CRD to define its own scope, which could become quite complex—especially when managing multiple scopes within the same tenant.

Overall, I like the idea of a hybrid approach—not just for easing the migration from v1 to v2, but also to offer users more flexibility in configuring their setup. However, as a long-term direction, if we choose to adopt a unified CRD, we would eventually deprecate the split CRDs to avoid the overhead of maintaining both. For the migration path, Ratify could offer tools to help users convert their CRDs from v1 to v2. In cases where CRs are too complex or incompatible, users may need to follow manual instructions to complete the migration.

And I'll provide some examples comparing v1 and v2 in different user scenarios to help explain the proposal.

[binbin-li]

Example Configurations Using the Unified CRD

⚠️ Note: These examples use placeholder values to illustrate the configuration format. Actual component configurations may vary.

Example: Validate a Docker Hub Image Signed with Notation

apiVersion: config.ratify.notaryproject.io/v2alpha1
kind: NamespacedRule
metadata:
  name: sample-1
spec:
  scopes:
    - docker.io/library
  verifiers:
    - name: notation-1
      type: notation
      parameters:
        certificates:
          - files:
              - ${CERTIFICATE_PATH}
  stores:
    type: registry-store
    parameters:
      credential:
        username: ""
        password: ""
  policyEnforcer:
    type: threshold-policy
    parameters:
      policy:
        rules:
          - verifierName: notation-1

Example: Validate ECR Images Signed with Cosign

apiVersion: config.ratify.notaryproject.io/v2alpha1
kind: NamespacedRule
metadata:
  name: sample-1
spec:
  scopes:
    - "*.ecr.<region>.amazonaws.com"
  verifiers:
    - name: cosign-1
      type: cosign
      parameters:
        keyless:
          certificateOIDCIssuer: 
          certificateidentity:
  stores:
    type: registry-store
    parameters:
      provider: awsEcrBasic
  policyEnforcer:
    type: threshold-policy
    parameters:
      policy:
        rules:
          - verifierName: cosign-1

Example: Validate Azure Container Registry Images with Notation

apiVersion: config.ratify.notaryproject.io/v2alpha1
kind: NamespacedRule
metadata:
  name: sample-1
spec:
  scopes:
    - *.azurecr.io
  verifiers:
    - name: notation-1
      type: notation
      parameters:
        certificates:
          - type: azurekeyvault
            vaultURI:
            tenantID:
            certificates:
              - name:
                version:
            keys:
              - name:
                version:
  stores:
    type: registry-store
    parameters:
      provider: azureContainerRegistry
      identityType: azureManagedIdentity
      clientID:
  policyEnforcer:
    type: threshold-policy
    parameters:
      policy:
        rules:
          - verifierName: notation-1

Example: Validate Alibaba Cloud Container Registry Images with Notation

apiVersion: config.ratify.notaryproject.io/v2alpha1
kind: NamespacedRule
metadata:
  name: sample-1
spec:
  scopes:
    - registry.<region>.aliyuncs.com
  verifiers:
    - name: notation-1
      type: notation
      parameters:
        certificates:
          - type: inline
            value: |
              -----BEGIN CERTIFICATE-----
              XXXXXX
              XXXXXX
              XXXXXX
              -----END CERTIFICATE-----
  stores:
    type: registry-store
    parameters:
      provider: alibabacloudAcrBasic
      defaultInstanceId:
      acrInstancesConfig:
  policyEnforcer:
    type: threshold-policy
    parameters:
      policy:
        rules:
          - verifierName: notation-1

[FeynmanZhou]

For the apiVersion, it's recommended to use config.ratify.dev/xxx or config.notaryproject.dev/xxx. There is no domain notaryproject.io

[binbin-li]

good call, the current apiVersion is config.ratify.dev/v2alpha1

[binbin-li]

Recording-20250627_103545.mp4

Pasting a demo for the new v2 CRD.

[DahuK]

I tested the basic capabilities of v2 using the templates in the deployments directory on an Alibaba Cloud cluster. The centralized configuration in the Executor is indeed much more convenient compared to V1. However, I have a few questions:

1. The default executor currently appears to be a thresholdPolicy-type policyEnforcer. However, the backend logs do not show actual image signature validation. How can I configure specific policies and specify the enforcer type?
2. I only tested the static provider in the stores.credential field. Does the current v2 architecture support the Alibaba Cloud ACR provider by default? Are additional developments required to enable compatibility with ACR?

[FeynmanZhou]

For the 2nd question, I believe each cloud provider requires additional development based on the Ratify v2.0 architecture, similar to @DahuK you did in Ratify v1.x.

[binbin-li]

1. We could add some logs around policy evaluation. And you could apply a new Executor CR with the new policy. However, currently ratify v2 only supports threshold policy which is fully redesigned. And we need to add documentation on the policy enforcer later.
2. We would need to add a credentialProvider implementation for alibaba under: https://github.com/notaryproject/ratify/tree/main/internal/store/credentialprovider, the main logics should be very similar to v1 auth provider.