Working with Gatekeeper(OPA)
Gatekeeper , is a customizable admission webhook for Kubernetes that enforces policies executed by the Open Policy Agent (OPA), a policy engine for Cloud Native environments hosted by Cloud Native Computing Foundation.
This document demonstrates how to use the Gatekeeper to manage OPA policies.
Prerequisites
Start up Karmada clusters
You just need to clone Karmada repo, and run the following script in the Karmada directory.
hack/local-up-karmada.sh
Gatekeeper Installations
In this case, you will use Gatekeeper v3.7.2. Related deployment files are from here.
Install Gatekeeper APIs on Karmada
Create resource objects for Gatekeeper in karmada controller plane, the content is as follows.
kubectl config use-context karmada-apiserverDeploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9
Deploy Gatekeeper CRDs: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L27-L1999
Deploy Gatekeeper secrets: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L2261-L2267
Deploy webhook config:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
labels:
gatekeeper.sh/system: "yes"
name: gatekeeper-mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
#Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate
failurePolicy: Ignore
matchPolicy: Exact
name: mutation.gatekeeper.sh
namespaceSelector:
matchExpressions:
- key: admission.gatekeeper.sh/ignore
operator: DoesNotExist
rules:
- apiGroups:
- '*'
apiVersions:
- '*'
operations:
- CREATE
- UPDATE
resources:
- '*'
sideEffects: None
timeoutSeconds: 1
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
gatekeeper.sh/system: "yes"
name: gatekeeper-validating-webhook-configuration
webhooks:
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
#Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit
failurePolicy: Ignore
matchPolicy: Exact
name: validation.gatekeeper.sh
namespaceSelector:
matchExpressions:
- key: admission.gatekeeper.sh/ignore
operator: DoesNotExist
rules:
- apiGroups:
- '*'
apiVersions:
- '*'
operations:
- CREATE
- UPDATE
resources:
- '*'
sideEffects: None
timeoutSeconds: 3
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
#Change the clientconfig from service type to url type cause webhook config and service are not in the same cluster.
url: https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel
failurePolicy: Fail
matchPolicy: Exact
name: check-ignore-label.gatekeeper.sh
rules:
- apiGroups:
- ""
apiVersions:
- '*'
operations:
- CREATE
- UPDATE
resources:
- namespaces
sideEffects: None
timeoutSeconds: 3You need to change the clientconfig from service type to url type for multi-cluster deployment.
Also, you need to deploy a dummy pod in gatekeeper-system namespace in karmada-apiserver context, because when Gatekeeper generates a policy template CRD, a status object is generated to monitor the status of the policy template, and the status object is bound by the controller Pod through the OwnerReference. Therefore, when the CRD and the controller are not in the same cluster, a dummy Pod needs to be used instead of the controller. The Pod enables the status object to be successfully generated.
For example:
apiVersion: v1
kind: Pod
metadata:
name: dummy-pod
namespace: gatekeeper-system
spec:
containers:
- name: dummy-pod
image: nginx:latest
imagePullPolicy: Always
Install GateKeeper components on host cluster
kubectl config use-context karmada-host
Deploy namespace: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1-L9
Deploy RBAC resources for deployment: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L1999-L2375
Deploy Gatekeeper controllers and secret as kubeconfig:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
name: gatekeeper-audit
namespace: gatekeeper-system
spec:
replicas: 1
selector:
matchLabels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
template:
metadata:
annotations:
container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
labels:
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
spec:
automountServiceAccountToken: true
containers:
- args:
- --operation=audit
- --operation=status
- --operation=mutation-status
- --logtostderr
- --disable-opa-builtin={http.send}
- --kubeconfig=/etc/kubeconfig
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
value: {{POD_NAME}}
image: openpolicyagent/gatekeeper:v3.7.2
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
name: manager
ports:
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: 9090
resources:
limits:
cpu: 1000m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp/audit
name: tmp-volume
- mountPath: /etc/kubeconfig
name: kubeconfig
subPath: kubeconfig
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: tmp-volume
- name: kubeconfig
secret:
defaultMode: 420
secretName: kubeconfig
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
name: gatekeeper-controller-manager
namespace: gatekeeper-system
spec:
replicas: 3
selector:
matchLabels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
template:
metadata:
annotations:
container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
labels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: gatekeeper.sh/operation
operator: In
values:
- webhook
topologyKey: kubernetes.io/hostname
weight: 100
automountServiceAccountToken: true
containers:
- args:
- --port=8443
- --logtostderr
- --exempt-namespace=gatekeeper-system
- --operation=webhook
- --operation=mutation-webhook
- --disable-opa-builtin={http.send}
- --kubeconfig=/etc/kubeconfig
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
value: {{POD_NAME}}
image: openpolicyagent/gatekeeper:v3.7.2
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 9090
name: manager
ports:
- containerPort: 8443
name: webhook-server
protocol: TCP
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: 9090
resources:
limits:
cpu: 1000m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /certs
name: cert
readOnly: true
- mountPath: /etc/kubeconfig
name: kubeconfig
subPath: kubeconfig
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- name: cert
secret:
defaultMode: 420
secretName: gatekeeper-webhook-server-cert
- name: kubeconfig
secret:
defaultMode: 420
secretName: kubeconfig
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
labels:
gatekeeper.sh/system: "yes"
name: gatekeeper-controller-manager
namespace: gatekeeper-system
spec:
minAvailable: 1
selector:
matchLabels:
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
---
apiVersion: v1
stringData:
kubeconfig: |-
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: {{ca_crt}}
server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
name: kind-karmada
contexts:
- context:
cluster: kind-karmada
user: kind-karmada
name: karmada
current-context: karmada
kind: Config
preferences: {}
users:
- name: kind-karmada
user:
client-certificate-data: {{client_cer}}
client-key-data: {{client_key}}
kind: Secret
metadata:
name: kubeconfig
namespace: gatekeeper-system
You need to fill in the dummy pod created in step 1 to {{ POD_NAME }} and fill in the secret which represents kubeconfig pointing to karmada-apiserver.
Deploy ResourceQuota: https://github.com/open-policy-agent/gatekeeper/blob/release-3.7/deploy/gatekeeper.yaml#L10-L26
Extra steps
After all, we need to copy the secret gatekeeper-webhook-server-cert in karmada-apiserver context to that in karmada-host context to keep secrets stored in etcd and volumes mounted in controller the same.
Run demo
Create k8srequiredlabels template
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
description: Describe K8sRequiredLabels crd parameters
properties:
labels:
type: array
items:
type: string
description: A label string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("you must provide labels: %v", [missing])
}
Create k8srequiredlabels constraint
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: ns-must-have-gk
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
parameters:
labels: ["gatekeepers"]
Create a bad namespace
kubectl create ns test
Error from server ([ns-must-have-gk] you must provide labels: {"gatekeepers"}): admission webhook "validation.gatekeeper.sh" denied the request: [ns-must-have-gk] you must provide labels: {"gatekeepers"}