Performance Test Setup for Karmada
Abstract
As Karmada is being implemented in more and more enterprises and organizations, scalability and scale of Karmada is gradually becoming new concerns for the community. In this article, we will introduce how to conduct large-scale testing for Karmada and how to monitor metrics from Karmada control plane.
Build large scale environment
Create member clusters using kind
Why kind
Kind is a tool for running local Kubernetes clusters using Docker containers. Kind was primarily designed for testing Kubernetes itself, so it play a good role in simulating member clusters.
Usage
Follow the kind installation guide.
Create 10 member clusters:
for ((i=1; i<=10; i ++)); do
kind create cluster --name member$i
done;
Simulate a large number of fake nodes using fake-kubelet
Why fake-kubelet
Compare to Kubemark
Kubemark is directly implemented with the code of kubelet, replacing the runtime part, except that it does not actually start the container, other behaviors are exactly the same as kubelet, mainly used for Kubernetes own e2e test, simulating a large number of nodes and pods will occupy the same memory as the real scene.
Fake-kubelet is a tool used to simulate any number of nodes and maintain pods on those nodes. It only does the minimum work of maintaining nodes and pods, so that it is very suitable for simulating a large number of nodes and pods for pressure testing on the control plane.
Usage
Deploy the fake-kubelet:
Note: Set container ENV
GENERATE_REPLICASin fake-kubelet deployment to set node replicas you want to create
export GENERATE_REPLICAS=your_replicas
curl https://raw.githubusercontent.com/wzshiming/fake-kubelet/master/deploy.yaml > fakekubelet.yml
# GENERATE_REPLICAS default value is 5
sed -i "s/5/$GENERATE_REPLICAS/g" fakekubelet.yml
kubectl apply -f fakekubelet.yml
kubectl get node You will find fake nodes.
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
fake-0 Ready agent 10s fake 10.88.0.136 <none> <unknown> <unknown> <unknown>
fake-1 Ready agent 10s fake 10.88.0.136 <none> <unknown> <unknown> <unknown>
fake-2 Ready agent 10s fake 10.88.0.136 <none> <unknown> <unknown> <unknown>
fake-3 Ready agent 10s fake 10.88.0.136 <none> <unknown> <unknown> <unknown>
fake-4 Ready agent 10s fake 10.88.0.136 <none> <unknown> <unknown> <unknown>
Deploy an sample deployment to test:
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: fake-pod
namespace: default
spec:
replicas: 4
selector:
matchLabels:
app: fake-pod
template:
metadata:
labels:
app: fake-pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: In
values:
- fake-kubelet
tolerations: # A taints was added to an automatically created Node. You can remove taints of Node or add this tolerations
- key: "fake-kubelet/provider"
operator: "Exists"
effect: "NoSchedule"
# nodeName: fake-0 # Or direct scheduling to a fake node
containers:
- name: fake-pod
image: fake
EOF
kubectl get pod You will find that it has been started, although the image does not exist.
$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fake-pod-78884479b7-52qcx 1/1 Running 0 6s 10.0.0.23 fake-4 <none> <none>
fake-pod-78884479b7-bd6nk 1/1 Running 0 6s 10.0.0.13 fake-2 <none> <none>
fake-pod-78884479b7-dqjtn 1/1 Running 0 6s 10.0.0.15 fake-2 <none> <none>
fake-pod-78884479b7-h2fv6 1/1 Running 0 6s 10.0.0.31 fake-0 <none> <none>
Distribute resources using ClusterLoader2
ClusterLoader2
ClusterLoader2 is an open source Kubernetes cluster testing tool. It tests against Kubernetes-defined SLIs/SLOs metrics to verify that clusters meet various quality of service standards. ClusterLoader2 is a tool oriented single cluster, it is complex to test karmada control plane meanwhile distribute resources to member clusters. Therefore, we just use the ClusterLoader2 to distribute resources to clusters managed by karmada.
Prepare a simple config
Let's prepare our config (config.yaml) to distribute resources. This config will:
Create 10 namespace
Create 20 deployments with 1000 pods inside that namespace
We will create file config.yaml that describes this test. First we need to start with defining test name:
name: test
ClusterLoader2 will create namespaces automatically, but we need to specify how many namespaces we want and whether delete the namespaces after distributing resources:
namespace:
number: 10
deleteAutomanagedNamespaces: false
Next, we need to specify TuningSets. TuningSet describes how actions are executed. The qps means 1/qps s per action interval. In order to distribute resources slowly to relieve the pressure on the apiserver, the qps of Uniformtinyqps is set to 0.1, which means that after distributing a deployment, we wait 10s before continuing to distribute the next deployment.
tuningSets:
- name: Uniformtinyqps
qpsLoad:
qps: 0.1
- name: Uniform1qps
qpsLoad:
qps: 1
Finally, we will create a phase that creates deployment and propagation policy. We need to specify in which namespaces we want the deployment and propagation policy to be created, how many of these deployments per namespace. Also, we will need to specify template for our deployment and propagation policy , which we will do later. For now, let's assume that this template allows us to specify numbers of replicas in deployment and propagation policy.
steps:
- name: Create deployment
phases:
- namespaceRange:
min: 1
max: 10
replicasPerNamespace: 20
tuningSet: Uniformtinyqps
objectBundle:
- basename: test-deployment
objectTemplatePath: "deployment.yaml"
templateFillMap:
Replicas: 1000
- namespaceRange:
min: 1
max: 10
replicasPerNamespace: 1
tuningSet: Uniform1qps
objectBundle:
- basename: test-policy
objectTemplatePath: "policy.yaml"
templateFillMap:
Replicas: 1
The whole config.yaml will look like this:
name: test
namespace:
number: 10
deleteAutomanagedNamespaces: false
tuningSets:
- name: Uniformtinyqps
qpsLoad:
qps: 0.1
- name: Uniform1qps
qpsLoad:
qps: 1
steps:
- name: Create deployment
phases:
- namespaceRange:
min: 1
max: 10
replicasPerNamespace: 20
tuningSet: Uniformtinyqps
objectBundle:
- basename: test-deployment
objectTemplatePath: "deployment.yaml"
templateFillMap:
Replicas: 1000
- namespaceRange:
min: 1
max: 10
replicasPerNamespace: 1
tuningSet: Uniform1qps
objectBundle:
- basename: test-policy
objectTemplatePath: "policy.yaml"
Now, we need to specify deployment and propagation template. ClusterLoader2 by default adds parameter Name that you can use in your template. In our config, we also passed Replicas parameter. So our template for deployment and propagation policy will look like following:
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{.Name}}
labels:
group: test-deployment
spec:
replicas: {{.Replicas}}
selector:
matchLabels:
app: fake-pod
template:
metadata:
labels:
app: fake-pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: In
values:
- fake-kubelet
tolerations: # A taints was added to an automatically created Node. You can remove taints of Node or add this tolerations
- key: "fake-kubelet/provider"
operator: "Exists"
effect: "NoSchedule"
containers:
- image: fake-pod
name: {{.Name}}
# policy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: test
spec:
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
placement:
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
Start Distributing
To distributing resources, run:
export KARMADA_APISERVERCONFIG=your_config
export KARMADA_APISERVERIP=your_ip
cd clusterloader2/
go run cmd/clusterloader.go --testconfig=config.yaml --provider=local --kubeconfig=$KARMADA_APISERVERCONFIG --v=2 --k8s-clients-number=1 --skip-cluster-verification=true --masterip=$KARMADA_APISERVERIP --enable-exec-service=false
The meaning of args above shows as following:
- k8s-clients-number: the number of karmada apiserver client number.
- skip-cluster-verification: whether to skip the cluster verification, which expects at least one schedulable node in the cluster.
- enable-exec-service: whether to enable exec service that allows executing arbitrary commands from a pod running in the cluster.
Since the resources of member cluster cannot be accessed in karmada control plane, we have to turn off enable-exec-service and cluster-verification.
Note: If the
deleteAutomanagedNamespacesparameter in config file is set to true, when the whole distribution of resources is complete, the resources will be immediately deleted.
Monitor Karmada control plane using Prometheus and Grafana
Deploy Prometheus and Grafana
Follow the Prometheus and Grafana Deploy Guide
Create Grafana DashBoards to observe Karmada control plane metrics
Here's an example to monitor the mutating api call latency for works and resourcebindings of the karmada apiserver through grafana. Monitor the metrics you want by modifying the Query statement.
Create a dashboard
Follow the Grafana support For Prometheus document.
Modify Query Statement
Enter the following Prometheus expression into the Query field.
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{verb!="WATCH|GET|LIST", resource~="works|resourcebindings"}[5m])) by (resource, verb, le))
The gragh will show as follow:
