跳转到文档内容
版本:v1.8

Working with Gatekeeper(OPA)

Gatekeeper , is a customizable admission webhook for Kubernetes that enforces policies executed by the Open Policy Agent (OPA), a policy engine for Cloud Native environments hosted by Cloud Native Computing Foundation.

This document demonstrates how to use the Gatekeeper to manage OPA policies.

Prerequisites

Start up Karmada clusters

You just need to clone Karmada repo, and run the following script in the Karmada directory.

hack/local-up-karmada.sh

Gatekeeper Installations

In this case, you will use Gatekeeper v3.7.2. Related deployment files are from here.

Install Gatekeeper APIs on Karmada

  1. Create resource objects for Gatekeeper in karmada controller plane, the content is as follows.

    kubectl config use-context karmada-apiserver

    Deploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9

    Deploy Gatekeeper CRDs: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L27-L1999

    Deploy Gatekeeper secrets: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L2261-L2267

    Deploy webhook config:

    apiVersion: admissionregistration.k8s.io/v1
    kind: MutatingWebhookConfiguration
    metadata:
    labels:
    gatekeeper.sh/system: "yes"
    name: gatekeeper-mutating-webhook-configuration
    webhooks:
    - admissionReviewVersions:
    - v1
    - v1beta1
    clientConfig:
    #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate
    failurePolicy: Ignore
    matchPolicy: Exact
    name: mutation.gatekeeper.sh
    namespaceSelector:
    matchExpressions:
    - key: admission.gatekeeper.sh/ignore
    operator: DoesNotExist
    rules:
    - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    sideEffects: None
    timeoutSeconds: 1
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
    labels:
    gatekeeper.sh/system: "yes"
    name: gatekeeper-validating-webhook-configuration
    webhooks:
    - admissionReviewVersions:
    - v1
    - v1beta1
    clientConfig:
    #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit
    failurePolicy: Ignore
    matchPolicy: Exact
    name: validation.gatekeeper.sh
    namespaceSelector:
    matchExpressions:
    - key: admission.gatekeeper.sh/ignore
    operator: DoesNotExist
    rules:
    - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    sideEffects: None
    timeoutSeconds: 3
    - admissionReviewVersions:
    - v1
    - v1beta1
    clientConfig:
    #Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
    url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel
    failurePolicy: Fail
    matchPolicy: Exact
    name: check-ignore-label.gatekeeper.sh
    rules:
    - apiGroups:
    - ""
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - namespaces
    sideEffects: None
    timeoutSeconds: 3

    You need to change the clientconfig from service type to url type for multi-cluster deployment.

    Also, you need to deploy a dummy pod in gatekeeper-system namespace in karmada-apiserver context, because when Gatekeeper generates a policy template CRD, a status object is generated to monitor the status of the policy template, and the status object is bound by the controller Pod through the OwnerReference. Therefore, when the CRD and the controller are not in the same cluster, a dummy Pod needs to be used instead of the controller. The Pod enables the status object to be successfully generated.

    For example:

    apiVersion: v1
    kind: Pod
    metadata:
    name: dummy-pod
    namespace: gatekeeper-system
    spec:
    containers:
    - name: dummy-pod
    image: nginx:latest
    imagePullPolicy: Always

Install GateKeeper components on host cluster

kubectl config use-context karmada-host  

Deploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9

Deploy RBAC resources for deployment: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1999-L2375

Deploy Gatekeeper controllers and secret as kubeconfig:

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
name: gatekeeper-audit
namespace: gatekeeper-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
template:
metadata:
annotations:
container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
labels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
spec:
automountServiceAccountToken: true
containers:
- args:
- --operation=audit
- --operation=status
- --operation=mutation-status
- --logtostderr
- --disable-opa-builtin={http.send}
- --kubeconfig=/etc/kubeconfig
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
value: {{POD_NAME}}
image: openpolicyagent/gatekeeper:v3.7.2
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
name: manager
ports:
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: 9090
resources:
limits:
cpu: 1000m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp/audit
name: tmp-volume
- mountPath: /etc/kubeconfig
name: kubeconfig
subPath: kubeconfig
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: tmp-volume
- name: kubeconfig
secret:
defaultMode: 420
secretName: kubeconfig
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
name: gatekeeper-controller-manager
namespace: gatekeeper-system
spec:
replicas: 3
selector:
matchLabels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
template:
metadata:
annotations:
container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
labels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: gatekeeper.sh/operation
operator: In
values:
- webhook
topologyKey: kubernetes.io/hostname
weight: 100
automountServiceAccountToken: true
containers:
- args:
- --port=8443
- --logtostderr
- --exempt-namespace=gatekeeper-system
- --operation=webhook
- --operation=mutation-webhook
- --disable-opa-builtin={http.send}
- --kubeconfig=/etc/kubeconfig
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
value: {{POD_NAME}}
image: openpolicyagent/gatekeeper:v3.7.2
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
name: manager
ports:
- containerPort: 8443
name: webhook-server
protocol: TCP
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: 9090
resources:
limits:
cpu: 1000m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /certs
name: cert
readOnly: true
- mountPath: /etc/kubeconfig
name: kubeconfig
subPath: kubeconfig
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- name: cert
secret:
defaultMode: 420
secretName: gatekeeper-webhook-server-cert
- name: kubeconfig
secret:
defaultMode: 420
secretName: kubeconfig
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
labels:
gatekeeper.sh/system: "yes"
name: gatekeeper-controller-manager
namespace: gatekeeper-system
spec:
minAvailable: 1
selector:
matchLabels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
---
apiVersion: v1
stringData:
kubeconfig: |-
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: {{ca_crt}}
server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
name: kind-karmada
contexts:
- context:
cluster: kind-karmada
user: kind-karmada
name: karmada
current-context: karmada
kind: Config
preferences: {}
users:
- name: kind-karmada
user:
client-certificate-data: {{client_cer}}
client-key-data: {{client_key}}
kind: Secret
metadata:
name: kubeconfig
namespace: gatekeeper-system

You need to fill in the dummy pod created in step 1 to {{ POD_NAME }} and fill in the secret which represents kubeconfig pointing to karmada-apiserver.

Deploy ResourceQuota: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L10-L26

Extra steps

After all, we need to copy the secret gatekeeper-webhook-server-cert in karmada-apiserver context to that in karmada-host context to keep secrets stored in etcd and volumes mounted in controller the same.

Run demo

Create k8srequiredlabels template

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
description: Describe K8sRequiredLabels crd parameters
properties:
labels:
type: array
items:
type: string
description: A label string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels

violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("you must provide labels: %v", [missing])
}

Create k8srequiredlabels constraint

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: ns-must-have-gk
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
parameters:
labels: ["gatekeepers"]

Create a bad namespace

kubectl create ns test
Error from server ([ns-must-have-gk] you must provide labels: {"gatekeepers"}): admission webhook "validation.gatekeeper.sh" denied the request: [ns-must-have-gk] you must provide labels: {"gatekeepers"}

Reference