My experiments with Pod Security Admission in kubernetes cluster
The importance of pod security is increasing in k8s clusters. We need controls to define them using policies.
Starting from K8s version v1.22 Pod Security Policy(PSP) is replaced with a new enhancement called Pod Security Admission (PSA). PSPs allow administrators to control the security of pods that are created in the k8s cluster. PSA provides a way to validate that a pod meets certain security requirements before it is allowed to run. Capabilities like which user can access a pod, which mount volume a pod can access, what network policies can be applied etc. PSA improves flexibility and security of PSP.
PSA works by intercepting requests to create or update pods and sending them to an admission controller. The admission controller then evaluates the security policies defined and either approves or denies the pod creation request based on those policies. By enforcing security policies at the pod level, PSA can help ensure that containers are running in a secure environment and that they are not vulnerable to attacks such as privilege escalation, network sniffing, and code injection.
Why is Pod Security Admission Important?
Pod Security Admission is important because it helps prevent security risks associated with running containers. By enforcing security policies at the pod level, PSA can help ensure that containers are running in a secure environment and that they are not vulnerable to attacks such as privilege escalation, network sniffing, and code injection.
Some of the security policies that can be enforced with PSA include:
- Preventing privileged containers: Containers that run with root privileges are a security risk, and PSA can prevent them from being deployed.
- Enforcing container image policies: PSA can ensure that only trusted container images are used in pods.
- Limiting host access: PSA can limit a pod’s access to the host operating system to prevent unauthorized access.
- Restricting network access: PSA can restrict a pod’s network access to prevent unauthorized access or data exfiltration.
PSA Levels
There are 3 different policy standards that define the level of restriction from permissive to restrictive. The latest k8s documentation refers to them as Levels [Refer: PSA]. For example, for a pod definition these levels restrict hostPath for volumes, ports for containers, securityContext etc. Below are the three different levels in PSA
- Privileged
- Baseline
- Restricted
Privileged
This is aimed for workloads managed by privileged, trusted users. It is an open policy, which means there are no restrictions.
Baseline
Aimed for ease of adoption for workloads and covers known security and escalations. This covers most basic security escalations.
Restricted
This policy is targeted for security critical workloads and includes best practices at the expense of some compatibility. This is highly restricted and covers best practices.
PSA Modes
While the levels define the standards that are permitted, we can use them in different modes. The modes can be enforce, audit or warn . This is usually defined in namespace and the admission control takes the action based on these modes. The three modes are explained below;
warn
A user facing warning will be provided if there is a policy violation and will permit the creation of pod in spite of violations.
audit
An audit annotation is added to the event record during creation of pod, if there is any violation
enforce
If there is any policy violation for a pod, then the admission controller will deny the creation of the pod.
These levels and modes can be applied either at namespace level or as an AdmissionConfiguration resource. So basically, if we need per namespace granularity, we can add labels as below to a namespace.
# MODE must be one of `enforce`, `audit`, or `warn`.
# LEVEL must be one of `privileged`, `baseline`, or `restricted`.
pod-security.kubernetes.io/<MODE>: <LEVEL>We also have option to pin to a kubernetes version as below
pod-security.kubernetes.io/warn-version=v1.26If we need cluster level, we can use AdmissionConfiguration which allows cluster level defaulting along with custom exceptions.
Let’s get some hands-on with PSA. I am going to use a simple Kind k8s cluster for this demo, with k8s version 1.26.
➜ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-sample-control-plane Ready control-plane 12d v1.26.3 172.18.0.6 <none> Ubuntu 22.04.2 LTS 5.10.104-linuxkit containerd://1.6.19-46-g941215f49
k8s-sample-worker Ready <none> 12d v1.26.3 172.18.0.8 <none> Ubuntu 22.04.2 LTS 5.10.104-linuxkit containerd://1.6.19-46-g941215f49
k8s-sample-worker2 NotReady <none> 12d v1.26.3 172.18.0.7 <none> Ubuntu 22.04.2 LTS 5.10.104-linuxkit containerd://1.6.19-46-g941215f49
k8s-sample-worker3 Ready <none> 12d v1.26.3 172.18.0.9 <none> Ubuntu 22.04.2 LTS 5.10.104-linuxkit containerd://1.6.19-46-g941215f49Below is the kind config used to create this cluster
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
PodSecurity: true
nodes:
- role: control-plane
- role: worker
- role: worker
- role: workerRefer my previous article on setting up kind clusters for reference — KIND
To use Pod Security Admission, cluster administrators must first enable it in their Kubernetes environment. Once enabled, administrators can define security policies using Kubernetes native policy language, NetworkPolicy.
In my kind cluster, I used the below command to check if pod security is enabled in the api server. Verify if “enable-admission-plugins” is included.
➜ kubectl get pod -n kube-system -l component=kube-apiserver -o custom-columns=\
"COMMAND:.spec.containers[*].command[*]" | tr "," "\n" | grep "enable-admission-plugins"
--enable-admission-plugins=NodeRestrictionI am creating a namespace “psa-test” and deploying nginx pods in that namespace.
➜ kubectl create ns psa-test
namespace/psa-test created
➜ kubectl create deploy nginx --image=nginx -n psa-test --replicas=3
deployment.apps/nginx created
➜ kubectl get pods -n psa-test
NAME READY STATUS RESTARTS AGE
nginx-748c667d99-c4gx2 1/1 Running 0 7s
nginx-748c667d99-kzzkg 1/1 Running 0 7s
nginx-748c667d99-mp5nk 1/1 Running 0 7sWe will try different modes now.
Warn
I am going to start with namespace level policy implementation with “warn” mode. We can add a namespace label with privileged level (Mode= warn, Level= privileged) as below;
➜ kubectl label --overwrite ns psa-test pod-security.kubernetes.io/warn=privileged
namespace/psa-test labeledLet’s add some policy violations. I am going to modify security context to allow the pod to run as root for the nginx deployment. This is a security violation.
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: falseI can see the existing pods are running fine and there are no events or warnings. Thich means “Priviledged” level is not covering the security violation of running as root user. Let’s increase the level to “Baseline” level.
➜ kubectl label --overwrite ns psa-test pod-security.kubernetes.io/warn=baseline
namespace/psa-test labeledThen I do a rollout restart of the nginx pod.
➜ kubectl rollout restart deploy nginx -n psa-test
deployment.apps/nginx restarted
➜ kubectl get pods -n psa-test
NAME READY STATUS RESTARTS AGE
nginx-6cf94cffd4-ksv2q 1/1 Running 0 20s
nginx-6cf94cffd4-qstbk 1/1 Running 0 17s
nginx-6cf94cffd4-xxjvs 1/1 Running 0 14s
➜ kubectl get events -n psa-test | grep -i security
➜ So “baseline” is also not covering the security violation which we made. Lets increase to “restricted” — the highest security level.
➜ kubectl label --overwrite ns psa-test pod-security.kubernetes.io/warn=restricted
namespace/psa-test labeled
➜ kubectl rollout restart deploy nginx -n psa-test
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx restarted
➜ kubectl get pods -n psa-test
NAME READY STATUS RESTARTS AGE
nginx-6cf94cffd4-ksv2q 1/1 Running 0 20s
nginx-6cf94cffd4-qstbk 1/1 Running 0 17s
nginx-6cf94cffd4-xxjvs 1/1 Running 0 14s
➜ kubectl get events -n psa-test | grep -i security
➜ We got a warning for the policy violation, but permitted the pod restart and there were no events.
We observed that this policy violation is covered by “restricted” level. Lets check “audit” and “enforce” mode.
Audit
➜ kubectl label --overwrite ns psa-test pod-security.kubernetes.io/audit=restricted
namespace/psa-test not labeled
➜ kubectl rollout restart deploy nginx -n psa-test
'deployment.apps/nginx restarted
➜ kubectl get pods -n psa-test
NAME READY STATUS RESTARTS AGE
nginx-6b6c74877c-jz5vp 1/1 Running 0 7s
nginx-6b6c74877c-vvnl5 1/1 Running 0 10s
nginx-6b6c74877c-w4s26 1/1 Running 0 5s
➜ kind k get events -n psa-test | grep -i security
➜Enforce
➜ kubectl label --overwrite ns psa-test pod-security.kubernetes.io/enforce=restricted
Warning: existing pods in namespace "psa-test" violate the new PodSecurity enforce level "restricted:latest"
Warning: nginx-6b6c74877c-jz5vp (and 2 other pods): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/psa-test labeled
➜ kind k rollout restart deploy nginx -n psa-test
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx restarted
➜ kind kubectl get pods -n psa-test
NAME READY STATUS RESTARTS AGE
nginx-6b6c74877c-jz5vp 1/1 Running 0 4m25s
nginx-6b6c74877c-vvnl5 1/1 Running 0 4m28s
nginx-6b6c74877c-w4s26 1/1 Running 0 4m23s
➜ kind k get events -n psa-test | grep -i security
39s Warning FailedCreate replicaset/nginx-7ff8fb9bf4 Error creating: pods "nginx-7ff8fb9bf4-6hrb7" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
39s Warning FailedCreate replicaset/nginx-7ff8fb9bf4 Error creating: pods "nginx-7ff8fb9bf4-8j772" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")As you can see replicaset was not allowed to create new pods. The deployment failed. We got both warning message during deployment and the violation is visible in events as well.
We will now be thinking how can we see the list of policies covered in each level. This is covered by the different security standards mentioned in this page — https://kubernetes.io/docs/concepts/security/pod-security-standards/
Naturally, the next requirement will be to have fine grained controls for these policies. For example, we have a namespace where an application needs root level access and another pod require a specific hostpath to be mounted. Let’s say, this is a business requirement and we need exceptions defined in our policies. Unfortunately, PSA does not give these controls, though its predecessor PSP had these options. PSA helps us implement some standards — (Pod Security Standards) and it dictates to implement these standards quickly. We can mix different levels in a namespace and use.
Summary
To summarize, we need to remember the 3 levels and 3 modes of increasing restriction.
Levels: Privileged, Baseline, Restricted
Modes: warn, audit, enforce
We can use them in multiple combinations at namespace or cluster level. PSA helps us implement Pod Security Standards.
Conclusion
Pod Security Admission is a powerful tool that can help ensure the security of containerized workloads in Kubernetes environments. By enforcing security policies at the pod level, PSA can help prevent common security risks associated with running containers. It is easy to set up and built within k8s (v1.23+). PSS defines the standards, we do not need to write policies. We can have audits and warnings along with version pinning.
Some of the major drawbacks I see is,
- It lacks mutation ability, i.e to change a resource to a default policy if it violates something.
- It lacks capability to restrict the controllers. We saw earlier, it allowed the deployment to create replicasets and then silently not permitting pods to be created when there is violations.
- There is no way we can see high level reports of these violations.
- There are no fine grained controls or exceptions we can include in the policies, as the policy is defined in PSS. Use it or not use it.
Kubernetes might come with improvements to these issues, in future releases. In my future articles, I will explore other k8s native Pod security policy solutions (eg: Kyverno) and share my experience.
I hope this article was helpful and adds value to your journey to implement security policies. If you liked my article, you can follow my publication for future articles, which give me the motivation to write more. — https://devopsforyou.com/
