Migration From Kubefed

Karmada is developed in continuation of Kubernetes Federation v1 and Federation v2(aka Kubefed). Karmada inherited a lot of concepts from these two versions. For example:

• Resource template: Karmada uses Kubernetes Native API definition for federated resource template, to make it easy to integrate with existing tools that already adopt Kubernetes.
• Propagation Policy: Karmada offers a standalone Propagation(placement) Policy API to define multi-cluster scheduling and spreading requirements.
• Override Policy: Karmada provides a standalone Override Policy API for specializing cluster relevant configuration automation.

Most of the features in Kubefed have been reformed in Karmada, so Karmada would be the natural successor.

Generally speaking, migrating from Kubefed to Karmada would be pretty easy. This document outlines the basic migration path for Kubefed users. Note: This document is a work in progress, any feedback would be welcome.

Cluster Registration

Kubefed provides join and unjoin commands in kubefedctl command line tool, Karmada also implemented the two commands in karmadactl.

Refer to Kubefed Cluster Registration, and Karmada Cluster Registration for more details.

Joining Clusters

Assume you use the kubefedctl tool to join cluster as follows:

kubefedctl join cluster1 --cluster-context cluster1 --host-cluster-context cluster1

Now with Karmada, you can use karmadactl tool to do the same thing:

karmadactl join cluster1 --cluster-context cluster1 --karmada-context karmada

The behavior behind the join command is similar between Kubefed and Karmada. For Kubefed, it will create a KubeFedCluster, object and Karmada will create a Cluster object to describe the joined cluster.

Checking status of joined clusters

Assume you use the kubefedctl tool to check the status of the joined clusters as follows:

$ kubectl -n kube-federation-system get kubefedclusters

NAME       AGE   READY   KUBERNETES-VERSION
cluster1   1m    True    v1.21.2
cluster2   1m    True    v1.22.0

Now with Karmada, you can use karmadactl tool to do the same thing:

$ kubectl get clusters

NAME      VERSION   MODE   READY   AGE
member1   v1.20.7   Push   True    66s

Kubefed manages clusters with Push mode, however Karmada supports both Push and Pull modes. Refer to Overview of cluster mode for more details.

Unjoining clusters

Assume you use the kubefedctl tool to unjoin as follows:

kubefedctl unjoin cluster2 --cluster-context cluster2 --host-cluster-context cluster1

Now with Karmada, you can use karmadactl tool to do the same thing:

karmadactl unjoin cluster2 --cluster-context cluster2 --karmada-context karmada

The behavior behind the unjoin command is similar between Kubefed and Karmada, they both remove the cluster from the control plane by removing the cluster object.

Propagating workload to clusters

Assume you are going to propagate a workload (Deployment) to both clusters named cluster1 and cluster2, you might have to deploy following yaml to Kubefed:

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: test-deployment
  namespace: test-namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx
            name: nginx
  placement:
    clusters:
    - name: cluster2
    - name: cluster1
  overrides:
  - clusterName: cluster2
    clusterOverrides:
    - path: "/spec/replicas"
      value: 5
    - path: "/spec/template/spec/containers/0/image"
      value: "nginx:1.17.0-alpine"
    - path: "/metadata/annotations"
      op: "add"
      value:
        foo: bar
    - path: "/metadata/annotations/foo"
      op: "remove"

Now with Karmada, the yaml could be split into 3 yamls, one for each of the template, placement and overrides.

In Karmada, the template doesn't need to embed into Federated CRD, it just the same as Kubernetes native declaration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx
          name: nginx

For the placement part, Karmada provides PropagationPolicy API to hold the placement rules:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - cluster1
        - cluster2

The PropagationPolicy defines the rules of which resources(resourceSelectors) should be propagated to where (placement). See Resource Propagating for more details.

For the override part, Karmada provides OverridePolicy API to hold the rules for differentiation:

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example-override
  namespace: default
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  overrideRules:
    - targetCluster:
        clusterNames:
          - cluster2
      overriders:
        plaintext:
          - path: "/spec/replicas"
            operator: replace
            value: 5
          - path: "/metadata/annotations"
            operator: add
            value:
              foo: bar
          - path: "/metadata/annotations/foo"
            operator: remove
        imageOverrider:
          - component: Tag
            operator: replace
            value: 1.17.0-alpine

The OverridePolicy defines the rules of which resources(resourceSelectors) should be overwritten when propagating to where(targetCluster).

In addition to Kubefed, Karmada offers various alternatives to declare the override rules, see Overriders for more details.

FAQ

Will Karmada provide tools to smooth the migration?

We don't have the plan yet, as we reached some Kubefed users and found that they're usually not using vanilla Kubefed but the forked version, they extended Kubefed a lot to meet their requirements. So, it might be pretty hard to maintain a common tool to satisfy most users.

We are also looking forward to more feedback now, please feel free to reach us, and we are glad to support you finish the migration.