Kubernetes PriorityClass preemption can evict security pods despite PDBs

How preemption targets low-priority pods

Kubernetes' PriorityClass API maps a name to an integer, and that integer governs scheduling order, not trust. According to a dev.to article, when the cluster runs short of capacity, the scheduler may evict pods with lower values to admit a pod carrying a higher one. The mechanism exists for good reasons — workloads that must start quickly, such as front-line services or critical jobs — but nothing in it distinguishes expendable workloads from protective ones. A logging collector, admission controller, policy agent, or incident-response tool holding a modest priority value is just as eligible for eviction as any batch job.

The article cites a warning from the Kubernetes documentation itself: in clusters where not every user is trusted, anyone able to create high-priority pods can disrupt other workloads.

A common assumption is that a PodDisruptionBudget shields critical pods from this fate. It does not. As the dev.to piece explains, the scheduler consults PDBs during preemption only on a best-effort basis, and if it cannot find victims that satisfy the budget it will preempt protected pods anyway.

Why namespace guardrails do not help

PriorityClass objects live at cluster scope, so the blast radius extends well beyond any single namespace. A default class applies when a workload omits priorityClassName, and an explicit reference takes that class's value. Victim selection in the scheduler follows priority, and kubelet node-pressure eviction weighs priority too, alongside resource usage and Quality of Service class.

The practical consequence, per the article: a namespace can look tightly constrained while a workload inside it selects a risky cluster-wide class. A service account may spawn jobs that inherit a high-priority template. A ResourceQuota caps how many pods run but does nothing to stop one of them from carrying a preempting class.

Testing the failure path safely

The article recommends a staged rehearsal rather than waiting for a production incident.

Stand up a non-production cluster or namespace that mirrors production admission policies and node pool configuration. Inventory the priority values assigned to security agents, DNS, ingress, logging, and each application tier. Submit a test pod configured with preemptionPolicy: Never and confirm that it simply waits in the queue without evicting anything. Only then, inside the test environment, run a pod with a genuinely preempting class and observe what follows: scheduler events, PDB behavior, restart policies, and whether telemetry goes dark.

Alongside the exercise, the author advises reviewing ResourceQuota, RBAC, admission policy, and the default PriorityClass as one combined set rather than in isolation; listing classes with kubectl get priorityclass; and scanning workload templates for explicit priority references. Teams should also alert on preemption events and on dips in the expected replica counts of security services.

What a safer configuration looks like

The hardening steps the article proposes: restrict which identities may create or reference privileged classes; require production workloads to name their class explicitly instead of inheriting a default; and run security infrastructure on nodes or workloads whose failure capacity is budgeted separately. Operators should also verify that the control plane can continue gathering evidence while a high-priority workload triggers repeated evictions.

The framing matters as much as the mechanics. A priority value should express an availability requirement with an accountable owner — it is not a stand-in for high availability, scheduling constraints, or protected network paths. Anyone able to raise a workload's priority is, in effect, deciding which protective controls get sacrificed first when capacity runs out.

Why it matters

Preemption is quiet by design: the scheduler resolves a resource shortfall and emits an event, but the workloads it removes are often the ones operators need most during an incident. Losing logging, policy enforcement, or monitoring at the precise moment of node pressure eliminates both the guardrails and the evidence required to respond. Because PriorityClass is cluster-scoped and invisible to namespace-level quotas, this failure path can persist in clusters that otherwise pass review. A short audit — enumerate the classes, tighten RBAC over their creation and use, and rehearse the eviction path with preemptionPolicy: Never — closes a gap that conventional policy checks routinely miss.