Abstract​

Cloud native implementations, growing in scale and complexity, are challenging organizations on how to efficiently, reliably manage large-scale resource pools to meet growing demands. Players in the cloud field attempted to scale out single clusters by customizing native Kubernetes components, which complicated single-cluster operations and maintenance, beclouded cluster upgrade paths, let alone many other problems. This is where multi-cluster technologies come into play. They can scale resource pools horizontally without invasively modifying each single cluster, while reducing O&M costs.

The popularity of Karmada is now drawing users' attention to Karmada's scalability and deployment at scale. Therefore, we launched a large-scale test on Karmada to obtain baseline performance metrics for Karmada managing multiple Kubernetes clusters. For multi-cluster systems represented by Karmada, the size of a single cluster is not a limiting factor restricting the scalability. On that account, we referred to the standard configurations of Kubernetes large-scale clusters and real-world implementations, and tested Karmada on managing 100 Kubernetes clusters (each cluster containing 5k nodes and 20k pods) at the same time. Limited by the environment and tooling, this test is not designed for stress testing Karmada, but for using Karmada in typical multi-cluster scenarios in production. The test results show that Karmada can stably support 100 large-scale clusters with 500,000 nodes connected at the same time, running more than 2 million pods.

This article will introduce the metrics used in the test, how to conduct large-scale testing, and how we realize massive connection of nodes and clusters.

Background​

Cloud computing is entering a new stage featuring multicloud and distributed clouds. As surveyed by Flexera, a well-known analyst company, more than 93% of enterprises are using services from multiple cloud vendors at the same time. Single Kubernetes clusters, limited by their capacity and fault recovery capabilities, cannot support services to run as distributed as wanted, especially if one's organization wants to go globalization. A hybrid cloud or multi-public cloud architecture helps avoid vendor lock-in or optimize costs. Karmada users are also demanding large-scale node and application management in their multi-cluster deployments.

Karmada是开放的多云多集群容器编排引擎，旨在帮助用户在多云环境下部署和运维业务应用。凭借兼容Kubernetes原生API的能力，Karmada可以平滑迁移单集群工作负载，并且仍可保持与Kubernetes周边生态工具链协同。

在最新发布的1.3版本中，Karmada重新设计了应用跨集群故障迁移功能，实现了基于污点的故障驱逐机制，并提供平滑的故障迁移过程，可以有效保障服务迁移过程的连续性（不断服）。

本版本新增加的特性：

• 增加了面向多集群的资源代理新特性，通过该代理平台业务方可以在不感知多集群的情况下，以单集群访问姿势直接操纵部署在多集群的工作负载；
• 提供针对集群资源建模能力，通过自定义的集群资源模型，调度器可以更精准地进行资源调度；
• 提供基于Bootstrap令牌来注册Pull模式集群的能力，不仅可以简化集群注册过程，还可以方便地进行权限控制；

此外，基于生产环境的用户反馈，本版本还进行了诸多性能优化，系统运行过程中CPU和内存资源需求大大降低，详细的性能测试报告稍后发布。

与之前版本一样，v1.3与前面的版本仍然保持兼容，前面版本的用户仍可以平滑升级。

In terms of multi-cluster management, Industrial and Commercial Bank of China (ICBC) found a new way to do it efficiently, that is, using Karmada. At KubeCon 2021, Kevin Wang from Huawei Cloud and Shen Yifan from ICBC shared how they managed it.