Skip to main content
Version: Next

Installation by Binary

Step-by-step installation of binary high-availability karmada cluster.

Prerequisites​

Server​

3 servers required. E.g.

+---------------+-----------------+-----------------+
|   HostName    |      Host IP    |    Public IP    |
+---------------+-----------------+-----------------+
|  karmada-01   |  172.31.209.245 |  47.242.88.82   |
+---------------+-----------------+-----------------+
|  karmada-02   |  172.31.209.246 |                 |
+---------------+-----------------+-----------------+
|  karmada-03   |  172.31.209.247 |                 |
+---------------+-----------------+-----------------+

Public IP is not required. It is used to download some karmada dependent components from the public network and connect to karmada ApiServer through the public network

DNS Resolution​

Execute operations at karmada-01 karmada-02 karmada-03.

$ vi /etc/hosts
172.31.209.245  karmada-01
172.31.209.246  karmada-02
172.31.209.247  karmada-03

Alternatively, you can use "Linux Virtual Server" for load balancing, and don't change /etc/hosts file.

Environment​

karmada-01 requires the following environment.

Golang: Compile the karmada binary GCC: Compile nginx (ignore if using cloud load balancing)

Compile and Download Binaries​

Execute operations at karmada-01.

Kubernetes Binaries​

Download the kubernetes binary package.

Refer to this page to download binaries of different versions and architectures: https://kubernetes.io/releases/download/#binaries

wget https://dl.k8s.io/v1.23.3/kubernetes-server-linux-amd64.tar.gz
tar -zxvf kubernetes-server-linux-amd64.tar.gz --no-same-owner
cd kubernetes/server/bin
mv kube-apiserver kube-controller-manager kubectl /usr/local/sbin/

etcd Binaries​

Download the etcd binary package.

You may want to use a newer version of etcd, please refer to this page: https://etcd.io/docs/latest/install/

wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz --no-same-owner
cd etcd-v3.5.1-linux-amd64/
mv etcdctl etcd /usr/local/sbin/

Karmada Binaries​

Compile the karmada binaries from source.

git clone https://github.com/karmada-io/karmada
cd karmada
make karmada-aggregated-apiserver karmada-controller-manager karmada-scheduler karmada-webhook karmadactl kubectl-karmada
mv _output/bin/linux/amd64/* /usr/local/sbin/

Nginx Binaries​

Compile the nginx binary from source.

wget http://nginx.org/download/nginx-1.21.6.tar.gz
tar -zxvf nginx-1.21.6.tar.gz
cd nginx-1.21.6
./configure --with-stream --without-http --prefix=/usr/local/karmada-nginx --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
make && make install
mv /usr/local/karmada-nginx/sbin/nginx /usr/local/karmada-nginx/sbin/karmada-nginx

Distribute Binaries​

Upload the binary file to the karmada-02 karmada-03 server.

Generate Certificates​

Step 1: Create Bash Scripts and Configuration Files​

The scripts will generate certificates using the openssl command. Download this directory.

We separate CA & leaf certificates generation scripts, so when you need to change Subject Alternative Name of leaf certificates (aka Load Balancer IP), you can reuse CA certificates, and run generate_leaf.sh to generate only leaf certificates.

There are 3 CAs: front-proxy-ca, server-ca, etcd/ca. Why we need 3 CAs please see: PKI certificates and requirements, CA Reusage and Conflicts.

If you use etcd provided by others, you can ignore generate_etcd.sh and csr_config/etcd.

Step 2: Change <SERVER_IP>​

You need to change <SERVER_IP> in csr_config/**/*.conf file to your "Load Balancer IP" and "Server IP". If you only use Load Balancer to access your servers, you only need to fill in "Load Balancer IP".

You normally don't need to change *.sh files.

Step 3: Run Shell Scripts​

./generate_ca.sh
./generate_leaf.sh ca_cert/
./generate_etcd.sh

Step 4: Check the Certificates​

You can view the configuration of the certificate, take karmada.crt as an example.

openssl x509 -noout -text -in karmada.crt

Step 5: Create the Karmada Configuration Directory​

Copy the certificates to the /etc/karmada/pki directory.

mkdir -p /etc/karmada/pki

cd ca_cert
cp -r * /etc/karmada/pki

cd ../cert
cp -r * /etc/karmada/pki

Create the Karmada kubeconfig Files and etcd Encryption Key​

Execute operations at karmada-01.

Create kubeconfig Files​

Step 1: Download bash script

Download this file.

Step 2: execute bash script

172.31.209.245:5443 is the address of the nginx proxy for karmada-apiserver, we'll set it up later. You should replace it with your Load Balancer provided "host:port".

./create_kubeconfig_file.sh "https://172.31.209.245:5443"

Create etcd Encryption Key​

If you don't need to encrypt contents in etcd, ignore this section and corresponding kube-apiserver start parameter.

export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
cat > /etc/karmada/encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

Distribute Files​

Package the karmada configuration files and copy them to other nodes.

cd /etc
tar -cvf karmada.tar karmada
scp karmada.tar  karmada-02:/etc/
scp karmada.tar  karmada-03:/etc/

karmada-02 karmada-03 need to decompress archives.

cd /etc
tar -xvf karmada.tar

Health Check Script​

(1) You can use the following check_status.sh to check if all components are healthy.

Download this file.

Note: If you are using CentOS 7, you need to run yum update -y nss curl to update curl version, then curl may support tls 1.3. If curl still does not support tls 1.3, you need to update OS version which includes newer curl, or simply use a go program to do a health check.

(2) Usage: ./check_status.sh

Install etcd cluster​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

Create Systemd Service​

/usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/sbin/etcd \
  --advertise-client-urls https://172.31.209.245:2379 \
  --cert-file /etc/karmada/pki/etcd/server.crt \
  --client-cert-auth=true \
  --data-dir /var/lib/etcd \
  --initial-advertise-peer-urls https://172.31.209.245:2380 \
  --initial-cluster "karmada-01=https://172.31.209.245:2380,karmada-02=https://172.31.209.246:2380,karmada-03=https://172.31.209.247:2380" \
  --initial-cluster-state new \
  --initial-cluster-token etcd-cluster \
  --key-file /etc/karmada/pki/etcd/server.key \
  --listen-client-urls "https://172.31.209.245:2379,https://127.0.0.1:2379" \
  --listen-peer-urls "https://172.31.209.245:2380" \
  --name karmada-01 \
  --peer-cert-file /etc/karmada/pki/etcd/peer.crt \
  --peer-client-cert-auth=true \
  --peer-key-file /etc/karmada/pki/etcd/peer.key \
  --peer-trusted-ca-file /etc/karmada/pki/etcd/ca.crt \
  --snapshot-count 10000 \
  --trusted-ca-file /etc/karmada/pki/etcd/ca.crt \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Notice:

The parameters that karmada-02 karmada-03 need to change are:

--name

--initial-advertise-peer-urls

--listen-peer-urls

--listen-client-urls

--advertise-client-urls

You can use EnvironmentFile to separate mutable configs from immutable configs.

Start etcd cluster​

3 servers have to execute.

create etcd storage directory

mkdir /var/lib/etcd/
chmod 700 /var/lib/etcd

start etcd

systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
systemctl status etcd.service

Verify​

$ etcdctl --cacert /etc/karmada/pki/etcd/ca.crt \
    --cert /etc/karmada/pki/etcd/healthcheck-client.crt \
    --key /etc/karmada/pki/etcd/healthcheck-client.key \
    --endpoints "172.31.209.245:2379,172.31.209.246:2379,172.31.209.247:2379" \
    endpoint status --write-out="table"

+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 172.31.209.245:2379 | 689151f8cbf4ee95 |   3.5.1 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
| 172.31.209.246:2379 | 5db4dfb6ecc14de7 |   3.5.1 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
| 172.31.209.247:2379 | 7e59eef3c816aa57 |   3.5.1 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Install kube-apiserver​

Configure Nginx​

Execute operations at karmada-01.

configure load balancing for karmada apiserver

/usr/local/karmada-nginx/conf/nginx.conf

worker_processes 2;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash consistent;
        server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 172.31.209.245:5443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}

/lib/systemd/system/karmada-nginx.service

[Unit]
Description=The karmada karmada-apiserver nginx proxy server
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/usr/local/karmada-nginx/sbin/karmada-nginx -t
ExecStart=/usr/local/karmada-nginx/sbin/karmada-nginx
ExecReload=/usr/local/karmada-nginx/sbin/karmada-nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

start karmada nginx

systemctl daemon-reload
systemctl enable karmada-nginx.service
systemctl start karmada-nginx.service
systemctl status karmada-nginx.service

Create kube-apiserver Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/docs/home/
After=network.target

[Service]
# If you don't need to encrypt etcd, remove --encryption-provider-config
ExecStart=/usr/local/sbin/kube-apiserver \
  --allow-privileged=true \
  --anonymous-auth=false \
  --audit-webhook-batch-buffer-size 30000 \
  --audit-webhook-batch-max-size 800 \
  --authorization-mode "Node,RBAC" \
  --bind-address 0.0.0.0 \
  --client-ca-file /etc/karmada/pki/server-ca.crt \
  --default-watch-cache-size 200 \
  --delete-collection-workers 2 \
  --disable-admission-plugins "StorageObjectInUseProtection,ServiceAccount" \
  --enable-admission-plugins "NodeRestriction" \
  --enable-bootstrap-token-auth \
  --encryption-provider-config "/etc/karmada/encryption-config.yaml" \
  --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  --insecure-port 0 \
  --logtostderr=true \
  --max-mutating-requests-inflight 2000 \
  --max-requests-inflight 4000 \
  --proxy-client-cert-file /etc/karmada/pki/front-proxy-client.crt \
  --proxy-client-key-file /etc/karmada/pki/front-proxy-client.key \
  --requestheader-allowed-names "front-proxy-client" \
  --requestheader-client-ca-file /etc/karmada/pki/front-proxy-ca.crt \
  --requestheader-extra-headers-prefix "X-Remote-Extra-" \
  --requestheader-group-headers "X-Remote-Group" \
  --requestheader-username-headers "X-Remote-User" \
  --runtime-config "api/all=true" \
  --secure-port 6443 \
  --service-account-issuer "https://kubernetes.default.svc.cluster.local" \
  --service-account-key-file /etc/karmada/pki/sa.pub \
  --service-account-signing-key-file /etc/karmada/pki/sa.key \
  --service-cluster-ip-range "10.254.0.0/16" \
  --tls-cert-file /etc/karmada/pki/kube-apiserver.crt \
  --tls-private-key-file /etc/karmada/pki/kube-apiserver.key \

Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start kube-apiserver​

3 servers have to execute.

systemctl daemon-reload
systemctl enable kube-apiserver.service
systemctl start kube-apiserver.service
systemctl status kube-apiserver.service

Verify​

$ ./check_status.sh
###### Start check kube-apiserver
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
livez check passed

###### kube-apiserver check success

Install karmada-aggregated-apiserver​

Create namespace and bind the cluster admin role. Execute operations at karmada-01.

kubectl create ns karmada-system
kubectl create clusterrolebinding cluster-admin:karmada --clusterrole=cluster-admin --user system:karmada

Then, like karmada-webhook, use nginx for high availability.

modify the nginx configuration and add the following configuration,Execute operations at karmada-01.

$ cat /usr/local/karmada-nginx/conf/nginx.conf
worker_processes 2;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash  consistent;
        server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
    }

    upstream webhook {
        hash  consistent;
        server 172.31.209.245:8443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:8443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:8443        max_fails=3 fail_timeout=30s;
    }

    upstream aa {
        hash  consistent;
        server 172.31.209.245:7443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:7443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:7443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 172.31.209.245:5443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }

    server {
        listen 172.31.209.245:4443;
        proxy_connect_timeout 1s;
        proxy_pass webhook;
    }

    server {
        listen 172.31.209.245:443;
        proxy_connect_timeout 1s;
        proxy_pass aa;
    }
}

Reload nginx configuration

systemctl restart karmada-nginx

Create Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-aggregated-apiserver.service

[Unit]
Description=Karmada Aggregated ApiServer
Documentation=https://github.com/karmada-io/karmada

[Service]
ExecStart=/usr/local/sbin/karmada-aggregated-apiserver \
  --audit-log-maxage 0  \
  --audit-log-maxbackup 0 \
  --audit-log-path -  \
  --authentication-kubeconfig /etc/karmada/karmada.kubeconfig  \
  --authorization-kubeconfig /etc/karmada/karmada.kubeconfig  \
  --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  --feature-gates "APIPriorityAndFairness=false"  \
  --kubeconfig /etc/karmada/karmada.kubeconfig \
  --logtostderr=true \
  --secure-port 7443 \
  --tls-cert-file /etc/karmada/pki/karmada.crt \
  --tls-private-key-file /etc/karmada/pki/karmada.key \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start karmada-aggregated-apiserver​

systemctl daemon-reload
systemctl enable karmada-aggregated-apiserver.service
systemctl start karmada-aggregated-apiserver.service
systemctl status karmada-aggregated-apiserver.service

Create APIService​

externalName is the host name of the node where nginx is located (karmada-01).

(1) create file: karmada-aggregated-apiserver-apiservice.yaml

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1alpha1.cluster.karmada.io
  labels:
    app: karmada-aggregated-apiserver
    apiserver: "true"
spec:
  insecureSkipTLSVerify: true
  group: cluster.karmada.io
  groupPriorityMinimum: 2000
  service:
    name: karmada-aggregated-apiserver
    namespace: karmada-system
    port: 443
  version: v1alpha1
  versionPriority: 10
---
apiVersion: v1
kind: Service
metadata:
  name: karmada-aggregated-apiserver
  namespace: karmada-system
spec:
  type: ExternalName
  externalName: karmada-01

(2) kubectl create -f karmada-aggregated-apiserver-apiservice.yaml

Verify​

$ ./check_status.sh
###### Start check karmada-aggregated-apiserver
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/start-aggregated-server-informers ok
livez check passed

###### karmada-aggregated-apiserver check success

Install kube-controller-manager​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

Create Systemd Service​

/usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://kubernetes.io/docs/home/
After=network.target

[Service]
ExecStart=/usr/local/sbin/kube-controller-manager \
  --authentication-kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  --authorization-kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  --bind-address "0.0.0.0" \
  --client-ca-file /etc/karmada/pki/server-ca.crt \
  --cluster-name karmada \
  --cluster-signing-cert-file /etc/karmada/pki/server-ca.crt \
  --cluster-signing-key-file /etc/karmada/pki/server-ca.key \
  --concurrent-deployment-syncs 10 \
  --concurrent-gc-syncs 30 \
  --concurrent-service-syncs 1 \
  --controllers "namespace,garbagecollector,serviceaccount-token" \
  --feature-gates "RotateKubeletServerCertificate=true" \
  --horizontal-pod-autoscaler-sync-period 10s \
  --kube-api-burst 2000 \
  --kube-api-qps 1000 \
  --kubeconfig /etc/karmada/kube-controller-manager.kubeconfig \
  --leader-elect \
  --logtostderr=true \
  --node-cidr-mask-size 24 \
  --pod-eviction-timeout 5m \
  --requestheader-allowed-names "front-proxy-client" \
  --requestheader-client-ca-file /etc/karmada/pki/front-proxy-ca.crt \
  --requestheader-extra-headers-prefix "X-Remote-Extra-" \
  --requestheader-group-headers "X-Remote-Group" \
  --requestheader-username-headers "X-Remote-User" \
  --root-ca-file /etc/karmada/pki/server-ca.crt \
  --service-account-private-key-file /etc/karmada/pki/sa.key \
  --service-cluster-ip-range "10.254.0.0/16" \
  --terminated-pod-gc-threshold 10000 \
  --tls-cert-file /etc/karmada/pki/kube-controller-manager.crt \
  --tls-private-key-file /etc/karmada/pki/kube-controller-manager.key \
  --use-service-account-credentials \
  --v 4 \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start kube-controller-manager​

systemctl daemon-reload
systemctl enable kube-controller-manager.service
systemctl start kube-controller-manager.service
systemctl status kube-controller-manager.service

Verify​

$ ./check_status.sh
###### Start check kube-controller-manager
[+]leaderElection ok
healthz check passed

###### kube-controller-manager check success

Install karmada-controller-manager​

Create Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-controller-manager.service

[Unit]
Description=Karmada Controller Manager
Documentation=https://github.com/karmada-io/karmada

[Service]
ExecStart=/usr/local/sbin/karmada-controller-manager \
  --bind-address 0.0.0.0 \
  --cluster-status-update-frequency 10s \
  --kubeconfig /etc/karmada/karmada.kubeconfig \
  --logtostderr=true \
  --metrics-bind-address ":10358" \
  --secure-port 10357 \
  --v=4 \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start karmada-controller-manager​

systemctl daemon-reload
systemctl enable karmada-controller-manager.service
systemctl start karmada-controller-manager.service
systemctl status karmada-controller-manager.service

Verify​

$ ./check_status.sh
###### Start check karmada-controller-manager
[+]ping ok
healthz check passed

###### karmada-controller-manager check success

Install karmada-scheduler​

Create Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-scheduler.service

[Unit]
Description=Karmada Scheduler
Documentation=https://github.com/karmada-io/karmada

[Service]
ExecStart=/usr/local/sbin/karmada-scheduler \
  --bind-address 0.0.0.0 \
  --enable-scheduler-estimator=true \
  --kubeconfig /etc/karmada/karmada.kubeconfig \
  --logtostderr=true \
  --scheduler-estimator-port 10352 \
  --secure-port 10511 \
  --v=4 \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start karmada-scheduler​

systemctl daemon-reload
systemctl enable karmada-scheduler.service
systemctl start karmada-scheduler.service
systemctl status karmada-scheduler.service

Verify​

$ ./check_status.sh
###### Start check karmada-scheduler
ok
###### karmada-scheduler check success

Install karmada-webhook​

karmada-webhook is different from scheduler and controller-manager, and its high availability needs to be implemented with nginx.

modify the nginx configuration and add the following configuration,Execute operations at karmada-01.

$ cat /usr/local/karmada-nginx/conf/nginx.conf
worker_processes 2;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash  consistent;
        server 172.31.209.245:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:6443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:6443        max_fails=3 fail_timeout=30s;
    }

    upstream webhook {
        hash  consistent;
        server 172.31.209.245:8443        max_fails=3 fail_timeout=30s;
        server 172.31.209.246:8443        max_fails=3 fail_timeout=30s;
        server 172.31.209.247:8443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 172.31.209.245:5443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }

    server {
        listen 172.31.209.245:4443;
        proxy_connect_timeout 1s;
        proxy_pass webhook;
    }
}

Reload nginx configuration

systemctl restart karmada-nginx

Create Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-webhook.service

[Unit]
Description=Karmada Webhook
Documentation=https://github.com/karmada-io/karmada

[Service]
ExecStart=/usr/local/sbin/karmada-webhook \
  --bind-address 0.0.0.0 \
  --cert-dir /etc/karmada/pki \
  --health-probe-bind-address ":8444" \
  --kubeconfig /etc/karmada/karmada.kubeconfig \
  --logtostderr=true \
  --metrics-bind-address ":8445" \
  --secure-port 8443 \
  --tls-cert-file-name "karmada.crt" \
  --tls-private-key-file-name "karmada.key" \
  --v=4 \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start karmada-webook​

systemctl daemon-reload
systemctl enable karmada-webhook.service
systemctl start karmada-webhook.service
systemctl status karmada-webhook.service

Configurate karmada-webhook​

Download the webhook-configuration.yaml file: https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/webhook-configuration.yaml

ca_string=$(cat /etc/karmada/pki/server-ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i "s/{{caBundle}}/${ca_string}/g"  webhook-configuration.yaml
# You need to change 172.31.209.245:4443 to your Load Balancer host:port.
sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g'  webhook-configuration.yaml

kubectl create -f  webhook-configuration.yaml

Verify​

$ ./check_status.sh
###### Start check karmada-webhook
ok
###### karmada-webhook check success

Initialize Karmada​

Execute operations at karmada-01.

git clone https://github.com/karmada-io/karmada
cd karmada/charts/karmada/_crds/bases

kubectl apply -f .

cd ../patches/
ca_string=$(cat /etc/karmada/pki/server-ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i "s/{{caBundle}}/${ca_string}/g" webhook_in_resourcebindings.yaml
sed -i "s/{{caBundle}}/${ca_string}/g"  webhook_in_clusterresourcebindings.yaml
# You need to change 172.31.209.245:4443 to your Load Balancer host:port.
sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g' webhook_in_resourcebindings.yaml
sed -i 's/karmada-webhook.karmada-system.svc:443/172.31.209.245:4443/g' webhook_in_clusterresourcebindings.yaml

kubectl patch CustomResourceDefinition resourcebindings.work.karmada.io --patch-file webhook_in_resourcebindings.yaml
kubectl patch CustomResourceDefinition clusterresourcebindings.work.karmada.io --patch-file webhook_in_clusterresourcebindings.yaml

Now, all the required components have been installed, and the member clusters could join Karmada control plane. If you want to use karmadactl to query, please run following command:

cat <<EOF | kubectl --kubeconfig=/etc/karmada/admin.kubeconfig  apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-proxy-clusterrole
rules:
- apiGroups:
  - 'cluster.karmada.io'
  resources:
  - clusters/proxy
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-proxy-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-proxy-clusterrole
subjects:
  - kind: User
    name: admin  # The name should be the user of client certificate
  # The token generated by the serviceaccount can parse the group information. Therefore, you need to specify the group information below.
  - kind: Group
    name: "system:masters" # The name should be the gourp of client certificate
EOF

Install karmada-scheduler-estimator (Optional)​

karmada-scheduler uses gRPC to access karmada-scheduler-estimator. You can use "Linux Virtual Server" as the Load Balancer.

You need to deploy above components and join member cluster into Karmada Control Plane first. See: Install Karmada on your own cluster

Create Systemd Service​

In the example below, "/etc/karmada/physical-machine-karmada-member-1.kubeconfig" is kubeconfig file of the joined member cluster.

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-scheduler-estimator.service

[Unit]
Description=Karmada Scheduler Estimator
Documentation=https://github.com/karmada-io/karmada

[Service]
# You need to change `--cluster-name` `--kubeconfig`
ExecStart=/usr/local/sbin/karmada-scheduler-estimator \
  --cluster-name "physical-machine-karmada-member-1" \
  --kubeconfig "/etc/karmada/physical-machine-karmada-member-1.kubeconfig" \
  --logtostderr=true \
  --server-port 10352 \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start karmada-scheduler-estimator​

systemctl daemon-reload
systemctl enable karmada-scheduler-estimator.service
systemctl start karmada-scheduler-estimator.service
systemctl status karmada-scheduler-estimator.service

Create Service​

(1) Create karmada-scheduler-estimator.yaml 

apiVersion: v1
kind: Service
metadata:
  name: karmada-scheduler-estimator-{{MEMBER_CLUSTER_NAME}}
  namespace: karmada-system
  labels:
    cluster: {{MEMBER_CLUSTER_NAME}}
spec:
  ports:
    - protocol: TCP
      port: {{PORT}}
      targetPort: {{TARGET_PORT}}
  type: ExternalName
  externalName: {{EXTERNAL_NAME}}

{{PORT}}: "--scheduler-estimator-port" parameter of "karmada-scheduler".

{{TARGET_PORT}}: LoadBalancer IP.

{{EXTERNAL_NAME}}: LoadBalancer Host.

{{MEMBER_CLUSTER_NAME}}: Member Cluster Name

(2) Create Service

kubectl create --kubeconfig "/etc/karmada/admin.kubeconfig" -f karmada-scheduler-estimator.yaml

Verify​

$ ./check_status.sh
###### Start check karmada-scheduler-estimator
ok
###### karmada-scheduler-estimator check success

Trouble Shooting​

1. karmada-scheduler takes some time to find the new karmada-scheduler-estimator. If you can't wait, you can just restart karmada-scheduler.

systemctl restart karmada-scheduler.service

Install karmada-search (Optional)​

Create Systemd Service​

Execute operations at karmada-01 karmada-02 karmada-03. Take karmada-01 as an example.

/usr/lib/systemd/system/karmada-search.service

[Unit]
Description=Karmada Search
Documentation=https://github.com/karmada-io/karmada

[Service]
ExecStart=/usr/local/sbin/karmada-search \
  --audit-log-maxage 0  \
  --audit-log-maxbackup 0 \
  --audit-log-path -  \
  --authentication-kubeconfig /etc/karmada/karmada.kubeconfig  \
  --authorization-kubeconfig /etc/karmada/karmada.kubeconfig  \
  --etcd-cafile /etc/karmada/pki/etcd/ca.crt \
  --etcd-certfile /etc/karmada/pki/etcd/apiserver-etcd-client.crt \
  --etcd-keyfile /etc/karmada/pki/etcd/apiserver-etcd-client.key \
  --etcd-servers "https://172.31.209.245:2379,https://172.31.209.246:2379,https://172.31.209.247:2379" \
  --feature-gates "APIPriorityAndFairness=false"  \
  --kubeconfig /etc/karmada/karmada.kubeconfig \
  --logtostderr=true \
  --secure-port 9443 \
  --tls-cert-file /etc/karmada/pki/karmada.crt \
  --tls-private-key-file /etc/karmada/pki/karmada.key \

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable karmada-search.service
systemctl start karmada-search.service
systemctl status karmada-search.service

Verify​

$ ./check_status.sh
###### Start check karmada-search
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/max-in-flight-filter ok
[+]poststarthook/start-karmada-search-informers ok
[+]poststarthook/start-karmada-informers ok
[+]poststarthook/start-karmada-search-controller ok
[+]poststarthook/start-karmada-proxy-controller ok
livez check passed

###### karmada-search check success

Configure​

Proxy Global Resources