Skip to main content
Version: v1.11

Customizing Resource Interpreter

Resource Interpreter Frameworkโ€‹

In the progress of propagating a resource from karmada-apiserver to member clusters, Karmada needs to know the resource definition. Take Propagating Deployment as an example, at the phase of building ResourceBinding, the karmada-controller-manager will parse the replicas from the deployment object.

For Kubernetes native resources, Karmada knows how to parse them, but for custom resources defined by CRD(or extended by something like aggregated-apiserver), as lack of the knowledge of the resource structure, they can only be treated as normal resources. Therefore, the advanced scheduling algorithms cannot be used for them.

The Resource Interpreter Framework is designed for interpreting resource structure. It consists of built-in and customized interpreters:

  • built-in interpreter: used for common Kubernetes native or well-known extended resources.
  • customized interpreter: interprets custom resources or overrides the built-in interpreters.

Note: The major difference between built-in and customized interpreters is that the built-in interpreter is implemented and maintained by Karmada community and will be built into Karmada components, such as karmada-controller-manager. On the contrary, the customized interpreter is implemented and maintained by users. It should be registered to Karmada as an Interpreter Webhook or declarative configuration (see Customized Interpreter for more details).

Interpreter Operationsโ€‹

When interpreting resources, we often get multiple pieces of information extracted. The Interpreter Operations defines the interpreter request type, and the Resource Interpreter Framework provides services for each operation type.

For all operations designed by Resource Interpreter Framework, please refer to Interpreter Operations.

Note: Not all the designed operations are supported (see below for supported operations).

Note: At most one interpreter will be consulted to when interpreting a resource with specific interpreter operation and the customized interpreter has higher priority than built-in interpreter if they are both interpreting the same resource. For example, the built-in interpreter serves InterpretReplica for Deployment with version apps/v1. If there is a customized interpreter registered to Karmada for interpreting the same resource, the customized interpreter wins and the built-in interpreter will be ignored.

Built-in Interpreterโ€‹

For the common Kubernetes native or well-known extended resources, the interpreter operations are built-in, which means the users usually don't need to implement customized interpreters. If you want more resources to be built-in, please feel free to file an issue to let us know your user case.

The built-in interpreter now supports following interpreter operations:

InterpretReplicaโ€‹

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • Job(batch/v1)
  • Pod(v1)

ReviseReplicaโ€‹

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • Job(batch/v1)

Retainโ€‹

Supported resources:

  • Pod(v1)
  • Service(v1)
  • ServiceAccount(v1)
  • PersistentVolumeClaim(v1)
  • PersistentVolume(V1)
  • Job(batch/v1)

AggregateStatusโ€‹

Supported resources:

  • Deployment(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • Job(batch/v1)
  • CronJob(batch/v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • Pod(v1)
  • PersistentVolume(V1)
  • PersistentVolumeClaim(v1)
  • PodDisruptionBudget(policy/v1)

InterpretStatusโ€‹

Supported resources:

  • Deployment(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • Job(batch/v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • PodDisruptionBudget(policy/v1)

InterpretDependencyโ€‹

Supported resources:

  • Deployment(apps/v1)
  • Job(batch/v1)
  • CronJob(batch/v1)
  • Pod(v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • Ingress(networking.k8s.io/v1)

InterpretHealthโ€‹

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • ReplicaSet(apps/v1)
  • DaemonSet(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • PersistentVolumeClaim(v1)
  • PodDisruptionBudget(policy/v1)
  • Pod(v1)

Customized Interpreterโ€‹

The customized interpreter is implemented and maintained by users, it can be extended in two ways, either by defining declarative configuration files or by running as webhook at runtime.

Note: Decalrative configuration has a higher priority than webhook.

Built-in Resource Declarative Configurationโ€‹

Karmada bundles some popular and open-sourced resources so that users can save the effort to customize them. The configurable interpreter now supports following interpreter operations:

InterpretReplicaโ€‹

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)

ReviseReplicaโ€‹

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)

Retainโ€‹

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

AggregateStatusโ€‹

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretStatusโ€‹

Supported resources:

  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretDependencyโ€‹

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretHealthโ€‹

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

Declarative Configurationโ€‹

What are interpreter declarative configuration?โ€‹

Users can quickly customize resource interpreters for both Kubernetes resources and CR resources by the rules declaraed in the ResourceInterpreterCustomization API specification.

Write with configurationโ€‹

You can configure resource interpretation rules by creating or updating ResourceInterpreterCustomization resource, the newest version supports the definition of lua scripts in the ResourceInterpreterCustomization. You can learn how to define the lua script in the API definition, take retention as an example.

Below we provide a yaml writing example of the ResourceInterpreterCustomization resource:

resource-interpreter-customization.yaml
apiVersion: config.karmada.io/v1alpha1
kind: ResourceInterpreterCustomization
metadata:
  name: declarative-configuration-example
spec:
  target:
    apiVersion: apps/v1
    kind: Deployment
  customizations:
    replicaResource:
      luaScript: >
        local kube = require("kube")
        function GetReplicas(obj)
          replica = obj.spec.replicas
          requirement = kube.accuratePodRequirements(obj.spec.template)
          return replica, requirement
        end
    replicaRevision:
      luaScript: >
        function ReviseReplica(obj, desiredReplica)
          obj.spec.replicas = desiredReplica
          return obj
        end
    retention:
      luaScript: >
        function Retain(desiredObj, observedObj)
          desiredObj.spec.paused = observedObj.spec.paused
          return desiredObj
        end
    statusAggregation:
      luaScript: >
        function AggregateStatus(desiredObj, statusItems)
          if statusItems == nil then
            return desiredObj
          end
          if desiredObj.status == nil then
            desiredObj.status = {}
          end
          replicas = 0
          for i = 1, #statusItems do
            if statusItems[i].status ~= nil and statusItems[i].status.replicas ~= nil then
              replicas = replicas + statusItems[i].status.replicas
            end
          end
          desiredObj.status.replicas = replicas
          return desiredObj
        end
    statusReflection:
      luaScript: >
        function ReflectStatus (observedObj)
          return observedObj.status
        end
    healthInterpretation:
      luaScript: >
        function InterpretHealth(observedObj)
          return observedObj.status.readyReplicas == observedObj.spec.replicas
        end
    dependencyInterpretation:
      luaScript: >
        function GetDependencies(desiredObj)
          dependentSas = {}
          refs = {}
          if desiredObj.spec.template.spec.serviceAccountName ~= nil and desiredObj.spec.template.spec.serviceAccountName ~= 'default' then
            dependentSas[desiredObj.spec.template.spec.serviceAccountName] = true
          end
          local idx = 1
          for key, value in pairs(dependentSas) do
            dependObj = {}
            dependObj.apiVersion = 'v1'
            dependObj.kind = 'ServiceAccount'
            dependObj.name = key
            dependObj.namespace = desiredObj.metadata.namespace
            refs[idx] = dependObj
            idx = idx + 1
          end
          return refs
        end

Verify the configurationโ€‹

Users can use the karmadactl interpret command to verify the ResourceInterpreterCustomization configuration before applying them to the system. Some examples are provided to help users better understand how this interpreter can be used, please refer to examples.

Webhookโ€‹

What are interpreter webhooks?โ€‹

Interpreter webhooks are HTTP callbacks that receive interpret requests and do something with them.

Write an interpreter webhook serverโ€‹

Please refer to the implementation of the Example of Customize Interpreter that is validated in Karmada E2E test. The webhook handles the ResourceInterpreterRequest request sent by the Karmada components (such as karmada-controller-manager), and sends back its decision as an ResourceInterpreterResponse.

Deploy the admission webhook serviceโ€‹

The Example of Customize Interpreter is deployed in the host cluster for E2E and exposed by a service as the front-end of the webhook server.

You may also deploy your webhooks outside the cluster. You will need to update your webhook configurations accordingly.

Configure webhook on the flyโ€‹

You can configure what resources and supported operations are subject to what interpreter webhook via ResourceInterpreterWebhookConfiguration.

The following is an example ResourceInterpreterWebhookConfiguration:

apiVersion: config.karmada.io/v1alpha1
kind: ResourceInterpreterWebhookConfiguration
metadata:
  name: examples
webhooks:
  - name: workloads.example.com
    rules:
      - operations: [ "InterpretReplica","ReviseReplica","Retain","AggregateStatus" ]
        apiGroups: [ "workload.example.io" ]
        apiVersions: [ "v1alpha1" ]
        kinds: [ "Workload" ]
    clientConfig:
      url: https://karmada-interpreter-webhook-example.karmada-system.svc:443/interpreter-workload
      caBundle: {{caBundle}}
    interpreterContextVersions: [ "v1alpha1" ]
    timeoutSeconds: 3

You can config more than one webhook in a ResourceInterpreterWebhookConfiguration, each webhook serves at least one operation.

Write the ResourceInterpreterCustomizationโ€‹

You can learn how to write the ResourceInterpreterCustomization to customize your resource.

First, we introduce the kube library functions. Then, we introduce how to write ResourceInterpreterCustomization using kyverno.io/v1/ClusterPolicy as an example.

build-in functions of luavmโ€‹

The rules declared in the ResourceInterpreterCustomization API specification define interpreter operations. These operations are written by lua and called via luavm. Users can use luavm's built-in functions when writing interpreter operations.

In kubeLibrary, there are two functions that are useful for writing interpreter operations. accuratePodRequirements is useful to write ReplicaResource operation and getPodDependenciesis useful to write DependencyInterpretation operation.

The accuratePodRequirements function accurates total resource requirements for pod. Its argument is PodTemplateSpec and its return value is ReplicaRequirements. PodTemplateSpec describes the data a pod should have when created from a template, and ReplicaRequirements represents the requirements required by each replica.

The getPodDependencies function gets total dependencies from podTemplate and namespace. Its arguments are PodTemplateSpec and namespace. Its return value is dependencies. PodTemplateSpec describes the data a pod should have when created from a template. namespace is the namespace of customized resource. And dependencies are the resources on which the customized resources depend.

ReplicaResourceโ€‹

ReplicaResource describes the rules for Karmada to discover the resource's replica as well as resource requirements. It would be useful for those CRD resources that declare workload types like Deployment.

A Kyverno ClusterPolicy is a collection of rules, which doesn't have fields like .spec.replicas or .spec.template.spec.nodeSelector. So there is no need to implement ReplicaResource for ClusterPolicy.

ReplicaRevisionโ€‹

ReplicaRevision describes the rules for Karmada to revise the resource's replica. It would be useful for those CRD resources that declare workload types like Deployment.

A Kyverno ClusterPolicy is a collection of rules, which doesn't have field like .spec.replicas. So there is no need to implement ReplicaRevision for ClusterPolicy.

Retentionโ€‹

Retention describes the desired behavior that Karmada should react on the changes made by member cluster components. This avoids system running into a meaningless loop that Karmada resource controller and the member cluster component continually applying opposite values of a field.

A Kyverno ClusterPolicy is a collection of rules, which is usually not changed by member cluster components. So there is no need to implement Retention for ClusterPolicy.

StatusAggregationโ€‹

StatusAggregation describes the rules for Karmada to aggregate status collected from member clusters to resource template.

A Kyverno ClusterPolicy is a collection of rules. Here we define the status aggregation rules for ClusterPolicy.

StatusAggregation-Defined-In-ResourceInterpreterCustomization
statusAggregation:
  luaScript: >
    function AggregateStatus(desiredObj, statusItems)
      if statusItems == nil then
        return desiredObj
      end
      desiredObj.status = {}
      desiredObj.status.conditions = {}
      rulecount = {}
      rulecount.validate = 0
      rulecount.generate = 0
      rulecount.mutate = 0
      rulecount.verifyimages = 0
      conditions = {}
      local conditionsIndex = 1
      for i = 1, #statusItems do
        if statusItems[i].status ~= nil and statusItems[i].status.autogen ~= nil then
          desiredObj.status.autogen = statusItems[i].status.autogen
        end
        if statusItems[i].status ~= nil and statusItems[i].status.ready ~= nil then
          desiredObj.status.ready = statusItems[i].status.ready
        end                        
        if statusItems[i].status ~= nil and statusItems[i].status.rulecount ~= nil then
          rulecount.validate = rulecount.validate + statusItems[i].status.rulecount.validate
          rulecount.generate = rulecount.generate + statusItems[i].status.rulecount.generate
          rulecount.mutate = rulecount.mutate + statusItems[i].status.rulecount.mutate
          rulecount.verifyimages = rulecount.verifyimages + statusItems[i].status.rulecount.verifyimages
        end
        if statusItems[i].status ~= nil and statusItems[i].status.conditions ~= nil then
          for conditionIndex = 1, #statusItems[i].status.conditions do
            statusItems[i].status.conditions[conditionIndex].message = statusItems[i].clusterName..'='..statusItems[i].status.conditions[conditionIndex].message
            hasCondition = false
            for index = 1, #conditions do
              if conditions[index].type == statusItems[i].status.conditions[conditionIndex].type and conditions[index].status == statusItems[i].status.conditions[conditionIndex].status and conditions[index].reason == statusItems[i].status.conditions[conditionIndex].reason then
                conditions[index].message = conditions[index].message..', '..statusItems[i].status.conditions[conditionIndex].message
                hasCondition = true
                break
              end
            end
            if not hasCondition then
              conditions[conditionsIndex] = statusItems[i].status.conditions[conditionIndex]
              conditionsIndex = conditionsIndex + 1                  
            end
          end
        end
      end
      desiredObj.status.rulecount = rulecount
      desiredObj.status.conditions = conditions
      return desiredObj
    end

StatusReflectionโ€‹

StatusReflection describes the rules for Karmada to pick the resource's status.

A Kyverno ClusterPolicy is a collection of rules, whose .status contains policy runtime data. StatusReflection determines which fields Karmada collect from the member clusters. Here we pick some fields from resource in member cluster.

StatusReflection-Defined-In-ResourceInterpreterCustomization
statusReflection:
  luaScript: >
    function ReflectStatus (observedObj)
      status = {}
      if observedObj == nil or observedObj.status == nil then 
        return status
      end
      status.ready = observedObj.status.ready
      status.conditions = observedObj.status.conditions
      status.autogen = observedObj.status.autogen
      status.rulecount = observedObj.status.rulecount
      return status
    end

HealthInterpretationโ€‹

HealthInterpretation describes the health assessment rules by which Karmada can assess the health state of the resource type.

A Kyverno ClusterPolicy is a collection of rules. We determine whether the ClusterPolicy in member cluster is healthy by defining the health assessment rules.

HealthInterpretation-Defined-In-ResourceInterpreterCustomization
healthInterpretation:
  luaScript: >
    function InterpretHealth(observedObj)
      if observedObj.status ~= nil and observedObj.status.ready ~= nil then
        return observedObj.status.ready
      end
      if observedObj.status ~= nil and observedObj.status.conditions ~= nil then
        for conditionIndex = 1, #observedObj.status.conditions do
          if observedObj.status.conditions[conditionIndex].type == 'Ready' and observedObj.status.conditions[conditionIndex].status == 'True' and observedObj.status.conditions[conditionIndex].reason == 'Succeeded' then
            return true
          end
        end
      end
      return false
    end

DependencyInterpretationโ€‹

DependencyInterpretation describes the rules for Karmada to analyze the dependent resources.

A Kyverno ClusterPolicy is a collection of rules, which doesn't depend on other resources. So there is no need to implement DependencyInterpretation for ClusterPolicy.

Important Notesโ€‹

Use the Retain Interpreter to Resolve Control Conflict between the Control Plane and Member Clustersโ€‹

Issue: Retain is a user-customizable interpreter in Karmada that resolves control conflicts when both the Karmada control plane and member clusters have control over member cluster resources. A typical scenario is when the replicas of a member cluster's Deployment is controlled by both the control plane's resource template and the member cluster's HPA. This leads to an abnormal state of the member cluster's Deployment due to repeated modifications of the replicas by both entities.

Solution:

  • Implement the corresponding Retain Interpreter for your workload type resources to decide when to respond to modifications from the control plane's resource template and when to respond to modifications from the member cluster's HPA. Currently, Karmada has only implemented the Retain interpreter for Deployment resources. The specific implementation is as follows: if the resource template has the label resource template.karmada.io/retain-replicas, it will be controlled by the member cluster's HPA; otherwise, it will be controlled by the control plane's resource template (if the hpaReplicasSyncer controller is explicitly enabled, Karmada can automatically label the Deployment enabled with HPA with this label). If you need to resolve this conflict for other resources or custom CRD resources, you can refer to the Retain solution for Deployments.
  • If you want a more elegant and comprehensive solution to the above problem, we recommend replacing HPA with FederatedHPA.