The Essential Guide to Pod Eviction On Kubernetes

• Pod preemption
• Out of resource eviction

• Quality of Service
• Pod Priority
  

Quality of Service

Guaranteed

• Every container in pod has CPU request and limit with request == limit
• Every container in pod has memory request and limit with request == limit

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-nginx
  namespace: demo
spec:
  containers:
  - name: guaranteed-nginx
    image: nginx
    resources:
      limits:
        memory: "512Mi"
        cpu: "1024m"
      requests:
        memory: "512Mi"
        cpu: "1024m"

Burstable

• At least one container has memory and CPU request
• Pod should not meet the criteria of Guaranteed as mentioned above

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-nginx
  namespace: demo
spec:
  containers:
  - name: guaranteed-nginx
    image: nginx
    resources:
      limits:
        memory: "1024Mi"
      requests:
        memory: "512Mi"

Best Effort

• None of the containers have any memory or CPU request or limit

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-nginx
  namespace: demo
spec:
  containers:
  - name: guaranteed-nginx
    image: nginx

Pod Priority

PriorityClass config with preemption disabled

apiVersion: scheduling.k8s.io/v
kind: PriorityClass
metadata:
  name: high-priority
preemptionPolicy: Never
value: 1000000
globalDefault: false
description: "This priority class will not preempt other pods"

Pod with priority class

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: demo
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: high-priority

How Quality of Service (QoS) and Pod Priority Relate to the Stability of Pods

Pod Preemption
When pods are created, they are placed into the scheduling queue based on their priority. Scheduler picks up a pod for scheduling and filters nodes based on the requirements specified by the pod. If the scheduler is unable to find any suitable node for the pod, then preemption logic is invoked for the pending pod provided preemptionPolicy for pending pod is not set to Never.

Preemption logic tries to find nodes which have lower priority pods than the pending pod so that pending pod can be scheduled on this node after
removal of low priority pods.

Limitations of Pod Preemption

1. Setting up Priority for First Time on Existing Cluster
When you are setting up pod priority for the first time, you must start with the pods with highest priority or keep preemptionPolicy as Never. Because he default priority of the pod is 0, if you set priority for low priority pod first then it may preempt critical pod which may not have priority set and may result in outage.

2. PodDisruptionBudget (PDB) is Not Guaranteed
PDB is only on a best effort basis. It will try to find a node such that eviction of lower priority pods will not result in violation of PDB. If it is unable to find any such node, then it will evict low priority pods from node to schedule high priority pod even if eviction results in violation of PDB and may result in outage.

3. Affinity with Low Priority Pod
In case high priority pod (H) has inter pod affinity with lower priority pod (L), it is possible that scheduler may end up evicting L from the node in order to make space for H. If it happens then inter pod affinity will no longer be satisfied and H will not be scheduled on this node. This loop can continue and can have a negative impact on availability of services.

4. Preemption May Not Follow Strict Priority Order

Best Practices for Pod Preemption

• Always use PodClassPriority and not priority directly
• Don’t create too many levels of priorities
• Have preemptPolicy PreemptLowerPriority only for critical services and use
  system-cluster-critical or system-node-critical as priorityClassName

Out of Resource Eviction
In over committed nodes, pods will be killed if the system runs out of resources. Kubelet proactively monitors compute resources for eviction. It supports eviction decisions based on incompressible resources, namely

• memory.available
• nodefs.available
• nodefs.inodesFree
• imagefs.available
• imagefs.inodesFree

Soft Eviction Threshold 

Hard Eviction Threshold

Eviction Policy

In case of memory pressure, pods are sorted first based on whether their
memory usage exceeds their request or not, then by pod priority and then by consumption of memory relative to memory requests. Pods which don’t exceed memory requests are not evicted. A lower priority pod which doesn’t exceed memory request will not be evicted. Whereas, a higher priority pod which exceeds memory request will be evicted.

Best Practices for Out of Resource Eviction

• Always define memory request and limit for pods.
• For critical pods over provision resource request and limit with request equal to limit so that pods have guaranteed QoS and are not evicted in case of memory pressure.
• For non-critical pods, keep resource request 80-90% of the limit, this allows Kubernetes to oversubscribe nodes and will provide a good trade-off between cost and QoS.

Node Out of Memory (OOM) Kill
In case a node experiences OOM behaviour prior to Kubelet being able to
reclaim memory, the node depends on oom_killer to respond.

Previously published on: //devtron.ai/blog/ultimate-guide-of-pod-eviction-on-kubernetes/