跳转到文档内容
版本:v1.3

Working with Kyverno

Kyverno, a Cloud Native Computing Foundation project, is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans. Kyverno policies are Kubernetes resources and do not require learning a new language. Kyverno is designed to work nicely with tools you already use like kubectl, kustomize, and Git.

This document gives an example to demonstrate how to use the Kyverno to manage policy.

Prerequisites

Setup Karmada

You just need to clone Karmada repo, and run the following script in Karmada directory.

hack/local-up-karmada.sh

Kyverno Installations

In this case, we will use Kyverno v1.6.3. Related deployment files are from here.

Install Kyverno APIs on Karmada

  1. Create resource objects of Kyverno in Karmada controller plane, the content is as follows.

    kubectl config use-context karmada-apiserver

    Deploy namespace: https://github.com/kyverno/kyverno/blob/release-1.6/config/install.yaml#L1-L12

    Deploy configmap: https://github.com/kyverno/kyverno/blob/release-1.6/config/install.yaml#L7751-L7783

    Deploy Kyverno CRDs: https://github.com/kyverno/kyverno/blob/release-1.6/config/install.yaml#L12-L7291

Install Kyverno components on host cluster

  1. Create resource objects of Kyverno in karmada-host context, the content is as follows.

    kubectl config use-context karmada-host  

    Deploy namespace: https://github.com/kyverno/kyverno/blob/release-1.6/config/install.yaml#L1-L12

    Deploy RBAC resources: https://github.com/kyverno/kyverno/blob/release-1.6/config/install.yaml#L7292-L7750

    Deploy Kyverno controllers and service:

    apiVersion: v1
    kind: Service
    metadata:
    labels:
    app: kyverno
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
    app.kubernetes.io/part-of: kyverno
    app.kubernetes.io/version: v1.6.3
    name: kyverno-svc
    namespace: kyverno
    spec:
    type: NodePort
    ports:
    - name: https
    port: 443
    targetPort: https
    nodePort: {{nodePort}}
    selector:
    app: kyverno
    app.kubernetes.io/name: kyverno
    ---
    apiVersion: v1
    kind: Service
    metadata:
    labels:
    app: kyverno
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
    app.kubernetes.io/part-of: kyverno
    app.kubernetes.io/version: v1.6.3
    name: kyverno-svc-metrics
    namespace: kyverno
    spec:
    ports:
    - name: metrics-port
    port: 8000
    targetPort: metrics-port
    selector:
    app: kyverno
    app.kubernetes.io/name: kyverno
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    labels:
    app: kyverno
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
    app.kubernetes.io/part-of: kyverno
    app.kubernetes.io/version: v1.6.3
    name: kyverno
    namespace: kyverno
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: kyverno
    app.kubernetes.io/name: kyverno
    strategy:
    rollingUpdate:
    maxSurge: 1
    maxUnavailable: 40%
    type: RollingUpdate
    template:
    metadata:
    labels:
    app: kyverno
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
    app.kubernetes.io/part-of: kyverno
    app.kubernetes.io/version: v1.6.3
    spec:
    affinity:
    podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - podAffinityTerm:
    labelSelector:
    matchExpressions:
    - key: app.kubernetes.io/name
    operator: In
    values:
    - kyverno
    topologyKey: kubernetes.io/hostname
    weight: 1
    containers:
    - args:
    - --filterK8sResources=[Event,*,*][*,kube-system,*][*,kube-public,*][*,kube-node-lease,*][Node,*,*][APIService,*,*][TokenReview,*,*][SubjectAccessReview,*,*][*,kyverno,kyverno*][Binding,*,*][ReplicaSet,*,*][ReportChangeRequest,*,*][ClusterReportChangeRequest,*,*][PolicyReport,*,*][ClusterPolicyReport,*,*]
    - -v=2
    - --kubeconfig=/etc/kubeconfig
    - --serverIP={{nodeIP}}:{{nodePort}}
    env:
    - name: INIT_CONFIG
    value: kyverno
    - name: METRICS_CONFIG
    value: kyverno-metrics
    - name: KYVERNO_NAMESPACE
    valueFrom:
    fieldRef:
    fieldPath: metadata.namespace
    - name: KYVERNO_SVC
    value: kyverno-svc
    - name: TUF_ROOT
    value: /.sigstore
    image: ghcr.io/kyverno/kyverno:v1.6.3
    imagePullPolicy: Always
    livenessProbe:
    failureThreshold: 2
    httpGet:
    path: /health/liveness
    port: 9443
    scheme: HTTPS
    initialDelaySeconds: 15
    periodSeconds: 30
    successThreshold: 1
    timeoutSeconds: 5
    name: kyverno
    ports:
    - containerPort: 9443
    name: https
    protocol: TCP
    - containerPort: 8000
    name: metrics-port
    protocol: TCP
    readinessProbe:
    failureThreshold: 4
    httpGet:
    path: /health/readiness
    port: 9443
    scheme: HTTPS
    initialDelaySeconds: 5
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
    resources:
    limits:
    memory: 384Mi
    requests:
    cpu: 100m
    memory: 128Mi
    securityContext:
    allowPrivilegeEscalation: false
    capabilities:
    drop:
    - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    volumeMounts:
    - mountPath: /.sigstore
    name: sigstore
    - mountPath: /etc/kubeconfig
    name: kubeconfig
    subPath: kubeconfig
    initContainers:
    - env:
    - name: METRICS_CONFIG
    value: kyverno-metrics
    - name: KYVERNO_NAMESPACE
    valueFrom:
    fieldRef:
    fieldPath: metadata.namespace
    image: ghcr.io/kyverno/kyvernopre:v1.6.3
    imagePullPolicy: Always
    name: kyverno-pre
    resources:
    limits:
    cpu: 100m
    memory: 256Mi
    requests:
    cpu: 10m
    memory: 64Mi
    securityContext:
    allowPrivilegeEscalation: false
    capabilities:
    drop:
    - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    securityContext:
    runAsNonRoot: true
    serviceAccountName: kyverno-service-account
    volumes:
    - emptyDir: {}
    name: sigstore
    - name: kubeconfig
    secret:
    defaultMode: 420
    secretName: kubeconfig
    ---
    apiVersion: v1
    stringData:
    kubeconfig: |-
    apiVersion: v1
    clusters:
    - cluster:
    certificate-authority-data: {{ca_crt}}
    server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
    name: kind-karmada
    contexts:
    - context:
    cluster: kind-karmada
    user: kind-karmada
    name: karmada
    current-context: karmada
    kind: Config
    preferences: {}
    users:
    - name: kind-karmada
    user:
    client-certificate-data: {{client_cer}}
    client-key-data: {{client_key}}
    kind: Secret
    metadata:
    name: kubeconfig
    namespace: kyverno

    For multi-cluster deployment, we need to add the config of --serverIP which is the address of the webhook server. So you need to ensure that the network from node in Karmada control plane to those in karmada-host cluster is connected and expose Kyverno controller pods to control plane, for example, using nodePort above. Then, fill in the secret which represents kubeconfig pointing to karmada-apiserver, such as ca_crt, client_cer and client_key above.

Run demo

Create require-labels ClusterPolicy

ClusterPolicy is a CRD which Kyverno offers to support different kinds of rules. Here is an example ClusterPolicy which means that you must create pod with app.kubernetes.io/name label.

kubectl config use-context karmada-apiserver  
kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-labels
spec:
validationFailureAction: enforce
rules:
- name: check-for-labels
match:
any:
- resources:
kinds:
- Pod
validate:
message: "label 'app.kubernetes.io/name' is required"
pattern:
metadata:
labels:
app.kubernetes.io/name: "?*"
EOF

Create a bad deployment without labels

kubectl create deployment nginx --image=nginx
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request

Reference