Add Security Policy

Kyverno provides a very versatile rules engine that can:

  1. Mutate the resources applied in the cluster.
  2. Validate the resources applied in the cluster.

By default, EKF comes with Kyverno validation polices that are enabled in all user namespaces. Namely, these are

  • disallow-host-namespaces
  • disallow-privileged-containers

This guide will walk you through extending the list of Kyverno policies that are applied in all user namespaces.

We will demonstrate how you can add a new security policy in all user namespaces using the Disallow hostPath Policy as an example. With this policy, users will not be allowed to create Pods that use host directories and volumes.

What You’ll Need

Procedure

  1. Go to your GitOps repository, inside your rok-tools management environment:

    root@rok-tools:~# cd ~/ops/deployments
  2. Add the policy manifests under the deploy overlay of the skel-resources kustomization, that is, under path kubeflow/manifests/common/skel-resources/overlays/deploy/. For example, create file kubeflow/manifests/common/skel-resources/overlays/deploy/disallow-host-path.yaml:

    disallow-host-path.yaml
    1apiVersion: kyverno.io/v1
    2kind: Policy
    3metadata:
    4-29
    4 name: disallow-host-path
    5 annotations:
    6 policies.kyverno.io/title: Disallow hostPath
    7 policies.kyverno.io/category: Pod Security Standards (Baseline)
    8 policies.kyverno.io/severity: medium
    9 policies.kyverno.io/subject: Pod,Volume
    10 kyverno.io/kyverno-version: 1.6.0
    11 kyverno.io/kubernetes-version: "1.22-1.23"
    12 policies.kyverno.io/description: >-
    13 HostPath volumes let Pods use host directories and volumes in containers.
    14 Using host resources can be used to access shared data or escalate privileges
    15 and should not be allowed. This policy ensures no hostPath volumes are in use.
    16spec:
    17 validationFailureAction: enforce
    18 background: true
    19 rules:
    20 - name: host-path
    21 match:
    22 any:
    23 - resources:
    24 kinds:
    25 - Pod
    26 validate:
    27 message: >-
    28 HostPath volumes are forbidden. The field spec.volumes[*].hostPath must be unset.
    29 pattern:
    30 spec:
    31 =(volumes):
    32 - X(hostPath): "null"

    Important

    Make sure that the policy is kind: Policy and not kind: ClusterPolicy. Deploying ClusterPolicy resources may impact critical cluster services.

  3. Edit kubeflow/manifests/common/skel-resources/overlays/deploy/kustomization.yaml and include the newly added manifest in the kustomization resources:

    resources: - ../../base - disallow-host-path.yaml
  4. Stage changes for commit:

    root@rok-tools:~/ops/deployments# git add .
  5. Commit your changes:

    root@rok-tools:~/ops/deployments# git commit -m "Add new Kyverno Policy"
  6. Apply the kustomization:

    root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/skel-resources/overlays/deploy

Verify

  1. Verify that there are no ClusterPolicy resources:

    root@rok-tools:~/ops/deployments# kubectl get clusterpolicies No resources found
  2. Verify that the new policy is created in the kubeflow-skel namespace and it is in the READY state:

    root@rok-tools:~/ops/deployments# kubectl get policies -n kubeflow-skel NAME BACKGROUND ACTION READY disallow-host-namespaces true enforce true disallow-host-path true enforce true disallow-privileged-containers true enforce true
  3. Specify a user namespace:

    root@rok-tools:~/ops/deployments# export NAMESPACE=<NAMESPACE>

    Replace <NAMESPACE> with the namespace of a user. For example:

    root@rok-tools:~/ops/deployments# export NAMESPACE=kubeflow-user
  4. Verify that the new policy is created in the user namespace and shows up as READY:

    root@rok-tools:~/ops/deployments# kubectl get policies -n ${NAMESPACE?} NAME BACKGROUND ACTION READY disallow-host-namespaces true enforce true disallow-host-path true enforce true disallow-privileged-containers true enforce true
  5. Verify that you cannot create Pods that mount host paths:

    1. Save the Pod manifest provided below in pod.yaml:

      host-path-pod.yaml
      1apiVersion: v1
      2kind: Pod
      3metadata:
      4-13
      4 name: test-kyverno-policy
      5spec:
      6 containers:
      7 - name: busybox
      8 image: busybox:1.32.1
      9 command: ["/bin/sh", "-c", "cat /target/os-release && sleep infinity"]
      10 volumeMounts:
      11 - mountPath: /target
      12 name: etc-volume
      13 volumes:
      14 - name: etc-volume
      15 hostPath:
      16 path: /etc
    2. Verify that you cannot apply it in a user namespace:

      root@rok-tools:~/ops/deployments# kubectl apply -f pod.yaml -n ${NAMESPACE?} Error from server: error when creating "pod.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/kubeflow-user/test-kyverno-policy was blocked due to the following policies disallow-host-path: host-path: 'validation error: HostPath volumes are forbidden. The field spec.volumes[*].hostPath must be unset. Rule host-path failed at path /spec/volumes/5/hostPath/'
    3. Clean up the Pod manifest you previously created:

      root@rok-tools:~/ops/deployments# rm pod.yaml

Summary

You have successfully added a new Kyverno security policy in all user namespaces.

What’s Next

Check out the rest of the operations guides related to Kyverno security.