Skip to main content
Version: v1.10

Deploy apiserver-network-proxy (ANP) For Pull mode

Purpose​

For a member cluster that joins Karmada in the pull mode, you need to provide a method to connect the network between the Karmada control plane and the member cluster, so that karmada-aggregated-apiserver can access this member cluster.

Deploying ANP to achieve this is one of the methods. This document describes how to deploy ANP for Karmada.

Environment​

Karmada can be deployed using the kind tool.

You can directly use hack/local-up-karmada.sh to deploy Karmada.

Actions​

Step 1: Download code​

To facilitate demonstration, the code is modified based on ANP v0.0.24 to support access to the front server through HTTP. Here is the code repository address: https://github.com/mrlihanbo/apiserver-network-proxy/tree/v0.0.24/dev.

git clone -b v0.0.24/dev https://github.com/mrlihanbo/apiserver-network-proxy.git
cd apiserver-network-proxy/

Step 2: Build images​

Build the proxy-server and proxy-agent images.

docker build . --build-arg ARCH=amd64 -f artifacts/images/agent-build.Dockerfile -t swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24

docker build . --build-arg ARCH=amd64 -f artifacts/images/server-build.Dockerfile -t swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24

Step 3: Generate certificates​

Run the command to check the IP address of karmada-host:

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane

Run the make certs command to generate certificates and specify PROXY_SERVER_IP as the IP address obtained in the preceding command.

make certs PROXY_SERVER_IP=x.x.x.x

The certificates are generated in the certs folder.

Step 4: Deploy proxy-server​

Save the proxy-server.yaml file in the root directory of the ANP code repository.

unfold me to see the yaml
# proxy-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: proxy-server
namespace: karmada-system
spec:
replicas: 1
selector:
matchLabels:
app: proxy-server
template:
metadata:
labels:
app: proxy-server
spec:
containers:
- command:
- /proxy-server
args:
- --health-port=8092
- --cluster-ca-cert=/var/certs/server/cluster-ca-cert.crt
- --cluster-cert=/var/certs/server/cluster-cert.crt
- --cluster-key=/var/certs/server/cluster-key.key
- --mode=http-connect
- --proxy-strategies=destHost
- --server-ca-cert=/var/certs/server/server-ca-cert.crt
- --server-cert=/var/certs/server/server-cert.crt
- --server-key=/var/certs/server/server-key.key
image: swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8092
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 60
name: proxy-server
volumeMounts:
- mountPath: /var/certs/server
name: cert
restartPolicy: Always
hostNetwork: true
volumes:
- name: cert
secret:
secretName: proxy-server-cert
---
apiVersion: v1
kind: Secret
metadata:
name: proxy-server-cert
namespace: karmada-system
type: Opaque
data:
server-ca-cert.crt: |
{{server_ca_cert}}
server-cert.crt: |
{{server_cert}}
server-key.key: |
{{server_key}}
cluster-ca-cert.crt: |
{{cluster_ca_cert}}
cluster-cert.crt: |
{{cluster_cert}}
cluster-key.key: |
{{cluster_key}}

Save the replace-proxy-server.sh file in the root directory of the ANP code repository.

unfold me to see the shell
#!/bin/bash

cert_yaml=proxy-server.yaml

SERVER_CA_CERT=$(cat certs/frontend/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{server_ca_cert}}/${SERVER_CA_CERT}/g" ${cert_yaml}

SERVER_CERT=$(cat certs/frontend/issued/proxy-frontend.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{server_cert}}/${SERVER_CERT}/g" ${cert_yaml}

SERVER_KEY=$(cat certs/frontend/private/proxy-frontend.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{server_key}}/${SERVER_KEY}/g" ${cert_yaml}

CLUSTER_CA_CERT=$(cat certs/agent/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{cluster_ca_cert}}/${CLUSTER_CA_CERT}/g" ${cert_yaml}

CLUSTER_CERT=$(cat certs/agent/issued/proxy-frontend.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{cluster_cert}}/${CLUSTER_CERT}/g" ${cert_yaml}


CLUSTER_KEY=$(cat certs/agent/private/proxy-frontend.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{cluster_key}}/${CLUSTER_KEY}/g" ${cert_yaml}

Run the following commands to run the script:

chmod +x replace-proxy-server.sh
bash replace-proxy-server.sh

Deploy the proxy-server on the karmada-host:

kind load docker-image swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-server:0.0.24 --name karmada-host
export KUBECONFIG=/root/.kube/karmada.config
kubectl --context=karmada-host apply -f proxy-server.yaml

Step 5: Deploy proxy-agent​

Save the proxy-agent.yaml file in the root directory of the ANP code repository.

unfold me to see the yaml
# proxy-agent.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: proxy-agent
name: proxy-agent
namespace: karmada-system
spec:
replicas: 1
selector:
matchLabels:
app: proxy-agent
template:
metadata:
labels:
app: proxy-agent
spec:
containers:
- command:
- /proxy-agent
args:
- '--ca-cert=/var/certs/agent/ca.crt'
- '--agent-cert=/var/certs/agent/proxy-agent.crt'
- '--agent-key=/var/certs/agent/proxy-agent.key'
- '--proxy-server-host={{proxy_server_addr}}'
- '--proxy-server-port=8091'
- '--agent-identifiers=host={{identifiers}}'
image: swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24
imagePullPolicy: IfNotPresent
name: proxy-agent
livenessProbe:
httpGet:
scheme: HTTP
port: 8093
path: /healthz
initialDelaySeconds: 15
timeoutSeconds: 60
volumeMounts:
- mountPath: /var/certs/agent
name: cert
volumes:
- name: cert
secret:
secretName: proxy-agent-cert
---
apiVersion: v1
kind: Secret
metadata:
name: proxy-agent-cert
namespace: karmada-system
type: Opaque
data:
ca.crt: |
{{proxy_agent_ca_crt}}
proxy-agent.crt: |
{{proxy_agent_crt}}
proxy-agent.key: |
{{proxy_agent_key}}

Save the replace-proxy-agent.sh file in the root directory of the ANP code repository.

unfold me to see the shell
#!/bin/bash

cert_yaml=proxy-agent.yaml

karmada_control_plane_addr=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane)
member3_cluster_addr=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' member3-control-plane)
sed -i'' -e "s/{{proxy_server_addr}}/${karmada_control_plane_addr}/g" ${cert_yaml}
sed -i'' -e "s/{{identifiers}}/${member3_cluster_addr}/g" ${cert_yaml}

PROXY_AGENT_CA_CRT=$(cat certs/agent/issued/ca.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{proxy_agent_ca_crt}}/${PROXY_AGENT_CA_CRT}/g" ${cert_yaml}

PROXY_AGENT_CRT=$(cat certs/agent/issued/proxy-agent.crt | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{proxy_agent_crt}}/${PROXY_AGENT_CRT}/g" ${cert_yaml}

PROXY_AGENT_KEY=$(cat certs/agent/private/proxy-agent.key | base64 | tr "\n" " "|sed s/[[:space:]]//g)
sed -i'' -e "s/{{proxy_agent_key}}/${PROXY_AGENT_KEY}/g" ${cert_yaml}

Run the following commands to run the script:

chmod +x replace-proxy-agent.sh
bash replace-proxy-agent.sh

Deploy the proxy-agent in the pull mode for a member cluster (in this example, the member3 cluster is in the pull mode.):

kind load docker-image swr.ap-southeast-1.myhuaweicloud.com/karmada/proxy-agent:0.0.24 --name member3
kubectl --kubeconfig=/root/.kube/members.config --context=member3 apply -f proxy-agent.yaml

The ANP deployment is completed now.

Step 6: Add command flags for the karmada-agent deployment​

After deploying the ANP deployment, you need to add extra command flags --cluster-api-endpoint and --proxy-server-address for the karmada-agent deployment in the member3 cluster.

Where --cluster-api-endpoint is the APIEndpoint of the cluster. You can obtain it from the KubeConfig file of the member3 cluster.

Where --proxy-server-address is the address of the proxy server that is used to proxy the cluster. In current case, you can set --proxy-server-address to http://<karmada_control_plane_addr>:8088. Get karmada_control_plane_addr value through the following command:

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' karmada-host-control-plane

Set port 8088 by modifying the code in ANP: https://github.com/mrlihanbo/apiserver-network-proxy/blob/v0.0.24/dev/cmd/server/app/server.go#L267. You can also modify it to a different value.