Multi-cluster Ingress

Users can use MultiClusterIngress API provided in Karmada to import external traffic to services in the member clusters.

Note: To use this feature, the Kubernetes version of the member cluster must be v1.21 or later.

Prerequisites

Karmada has been installed

We can install Karmada by referring to Quick Start, or directly run hack/local-up-karmada.sh script which is also used to run our E2E cases.

Cluster Network

Currently, we need to use the Multi-cluster Service feature to import external traffic.

So we need to ensure that the container networks between the host cluster and member clusters are connected. The host cluster indicates the cluster where the Karmada Control Plane is deployed.

• If you use the hack/local-up-karmada.sh script to deploy Karmada, Karmada will have three member clusters, and the container networks between the host cluster, member1 and member2 are connected.
• You can use Submariner or other related open source projects to connected networks between clusters.

Note: In order to prevent routing conflicts, Pod and Service CIDRs of clusters need non-overlapping.

Example

Step 1: Deploy ingress-nginx on the host cluster

We use multi-cluster-ingress-nginx as the demo for demonstration. We've made some changes based on the latest version(controller-v1.1.1) of ingress-nginx.

Download code

# for HTTPS
git clone https://github.com/karmada-io/multi-cluster-ingress-nginx.git
# for SSH
git clone git@github.com:karmada-io/multi-cluster-ingress-nginx.git

Build and deploy ingress-nginx

Using the existing karmada-host kind cluster to build and deploy the ingress controller.

export KUBECONFIG=~/.kube/karmada.config
export KIND_CLUSTER_NAME=karmada-host
kubectl config use-context karmada-host
cd multi-cluster-ingress-nginx
make dev-env

Apply kubeconfig secret

Create a secret that contains the karmada-apiserver authentication credential:

# get the 'karmada-apiserver' kubeconfig information and direct it to file /tmp/kubeconfig.yaml
kubectl -n karmada-system get secret kubeconfig --template={{.data.kubeconfig}} | base64 -d > /tmp/kubeconfig.yaml
# create secret with name 'kubeconfig' from file /tmp/kubeconfig.yaml
kubectl -n ingress-nginx create secret generic kubeconfig --from-file=kubeconfig=/tmp/kubeconfig.yaml

Edit ingress-nginx-controller deployment

We want nginx-ingress-controller to access karmada-apiserver to listen to changes in resources(such as multiclusteringress, endpointslices, and service). Therefore, we need to mount the authentication credential of karmada-apiserver to the nginx-ingress-controller.

kubectl -n ingress-nginx edit deployment ingress-nginx-controller

Edit as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  #...
  template:
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --karmada-kubeconfig=/etc/kubeconfig  # new line
        #...
        volumeMounts:
        #...
        - mountPath: /etc/kubeconfig            # new line
          name: kubeconfig                      # new line
          subPath: kubeconfig                   # new line
      volumes:
      #...
      - name: kubeconfig                        # new line
        secret:                                 # new line
          secretName: kubeconfig                # new line

Step 2: Use the MCS feature to discovery service

Install ServiceExport and ServiceImport CRDs

Refer to here.

Deploy web on member1 cluster

deploy.yaml:

unfold me to see the yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: hello-app
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
          protocol: TCP
---      
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  ports:
  - port: 81
    targetPort: 8080
  selector:
    app: web
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: mci-workload
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: web
    - apiVersion: v1
      kind: Service
      name: web
  placement:
    clusterAffinity:
      clusterNames:
        - member1

kubectl --context karmada-apiserver apply -f deploy.yaml

Export web service from member1 cluster

service_export.yaml:

unfold me to see the yaml

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: web
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: web-export-policy
spec:
  resourceSelectors:
    - apiVersion: multicluster.x-k8s.io/v1alpha1
      kind: ServiceExport
      name: web
  placement:
    clusterAffinity:
      clusterNames:
        - member1

kubectl --context karmada-apiserver apply -f service_export.yaml

Import web service to member2 cluster

service_import.yaml:

unfold me to see the yaml

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport
metadata:
  name: web
spec:
  type: ClusterSetIP
  ports:
  - port: 81
    protocol: TCP
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: web-import-policy
spec:
  resourceSelectors:
    - apiVersion: multicluster.x-k8s.io/v1alpha1
      kind: ServiceImport
      name: web
  placement:
    clusterAffinity:
      clusterNames:
        - member2

kubectl --context karmada-apiserver apply -f service_import.yaml

Step 3: Deploy multiclusteringress on karmada-controlplane

mci-web.yaml:

unfold me to see the yaml

apiVersion: networking.karmada.io/v1alpha1
kind: MultiClusterIngress
metadata:
  name: demo-localhost
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: demo.localdev.me
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 81
        path: /web
        pathType: Prefix

kubectl --context karmada-apiserver apply -f mci-web.yaml

Step 4: Local testing

Let's forward a local port to the ingress controller:

kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80

At this point, if you access http://demo.localdev.me:8080/web/, you should see an HTML page telling you:

Hello, world!
Version: 1.0.0
Hostname: web-xxx-xxx