Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOpsSchool!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Kubernetes Troubleshooting: Pods in Pending – Causes & Fixes

Below is a complete, production-grade list of all common reasons why a Kubernetes pod stays in Pending state, along with detailed solutions, commands, and how to verify and fix each issue.



🔍 How to Start Investigating

kubectl describe pod <pod-name>
Code language: HTML, XML (xml)

Focus on the Events: section — it will reveal why the pod is stuck.


🔁 Common Reasons and Solutions

#ReasonError Message / SymptomCommand to DiagnoseHow to Fix
1No available nodes (unschedulable)0/2 nodes are available: Not schedulablekubectl get nodesEnsure at least one node is Ready and schedulable. Use: kubectl uncordon <node>
2Node Taints (control-plane nodes tainted)pod didn't tolerate node taint`kubectl describe nodegrep Taint`
3Node Selectors / Affinity don’t match0/2 nodes match node selector`kubectl get pod -o yamlgrep -A5 nodeSelector`
4Tolerations missing for tainted nodesNo matching tolerations for taintskubectl describe node <node> Check taints:Add toleration in pod spec:yaml<br>tolerations:<br> - key: "example-key"<br> operator: "Exists"<br>
5Insufficient CPU or Memoryinsufficient memory, insufficient cpukubectl describe pod <pod> kubectl describe node <node>Reduce pod resources.requests in YAML:yaml<br>resources:<br> requests:<br> cpu: "100m"<br> memory: "256Mi"<br>
6Too many pods on node (maxPods limit reached)Too many pods`kubectl describe nodegrep pods`
7PersistentVolumeClaim (PVC) pendingpod has unbound PersistentVolumeClaimskubectl get pvcCreate or bind the PVC:kubectl get pvEnsure storage class and capacity match
8ImagePullBackOff (incorrect image or no access)Appears first as Pending, then ContainerCreating, then ImagePullBackOffkubectl describe pod <pod>Check image name and registry authFix typo or use imagePullSecrets
9Missing CNI plugin (pod networking not ready)network plugin is not readykubectl get pods -n kube-systemEnsure CNI is deployed:kubectl apply -f <cni-yaml> (e.g., Calico, Flannel)
10DNS issues inside clusterPods remain stuck in Pending or ContainerCreatingkubectl logs <pod> or kubectl exec -it <pod> -- nslookup kubernetesEnsure kube-dns or CoreDNS is running:kubectl get pods -n kube-system
11Pod Disruption Budgets (PDBs)Not enough available pods to meet the PDBkubectl get pdbAdjust minAvailable or maxUnavailable in your PDB
12InitContainers stuck or failingPod hangs in Init:kubectl describe pod <pod> Check Init: sectionFix issues in the InitContainer: volume mounts, scripts, dependencies
13Pod Quotas / LimitRanges hitLimitRange violated, ResourceQuota exceededkubectl describe quota kubectl describe limitrangeAdjust resource quotas / limits:kubectl edit quota <name>
14Custom Scheduler misconfigurationNo default-scheduler eventskubectl describe pod <pod> check .spec.schedulerNameUse correct scheduler, or omit schedulerName to default to default-scheduler
15No available IPs (CNI limit)Not shown in event, but pod stuckCheck kubelet logs or CNI plugin logsEnsure node’s CNI plugin can allocate more IPs (esp. AWS, Azure)
16Container runtime errors (e.g., containerd/dockerd)Pod stuck in Pending or ContainerCreatingjournalctl -u containerd or docker infoRestart the runtime:sudo systemctl restart containerd
17Cluster Autoscaler delay (in autoscaled clusters)waiting for node scale upkubectl describe pod and look for scaling delay messagesWait or trigger autoscaler node scaling
18Security context or PodSecurityPolicy blocksviolates PodSecurityPolicykubectl describe pod <pod>Ensure pod adheres to allowed securityContext / capabilities
19ServiceAccount or RBAC missingForbidden: ServiceAccount ...kubectl describe podCreate or bind proper ServiceAccount with correct RBAC
20Wrong Namespace usedPod is Pending, PVC not foundkubectl get pods -A kubectl get pvc -AEnsure objects are created in the same namespace or use -n flag

🧪 Bonus: Best Commands for Troubleshooting

# Check events on the pod
kubectl describe pod <pod-name>

# Check node pod limits
kubectl describe node <node-name> | grep -A10 Allocatable

# List all pods on a node
kubectl get pods --all-namespaces -o wide | grep <node-name>

# PVC status
kubectl get pvc

# CNI status
kubectl get pods -n kube-system | grep -E 'cni|calico|flannel'

# Resource quotas
kubectl describe resourcequota
kubectl describe limitrange

# Get pod spec with scheduler/affinity/tolerations
kubectl get pod <pod-name> -o yaml
Code language: PHP (php)

🧭 How to Fix Pending in General

  1. Start with kubectl describe pod
  2. If it says:
    • Too many pods → Increase maxPods or add nodes
    • Insufficient cpu/memory → Reduce resource requests or free up node
    • Taint → Add toleration or remove taint
    • PVC pending → Fix volume
    • No message → Check CNI/DNS

Here is a Bash script that checks for the 20 most common reasons why Kubernetes pods are stuck in Pending state.

✅ It loops over all Pending pods and checks taints, resources, PVCs, tolerations, affinity, maxPods, etc.

📦 You can copy, save as check-pending-pods.sh, and run on your control-plane/master node.


check-pending-pods.sh

#!/bin/bash

echo "=========================="
echo "🔍 Checking Pending Pods..."
echo "=========================="

PENDING_PODS=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}')

if [ -z "$PENDING_PODS" ]; then
  echo "✅ No pending pods found."
  exit 0
fi

echo "$PENDING_PODS" | while read namespace pod; do
  echo ""
  echo "🔍 Analyzing pod: $pod in namespace: $namespace"
  echo "------------------------------------------------"

  # Describe pod
  kubectl describe pod $pod -n $namespace > /tmp/pod_desc.txt

  # 1. Check for failed scheduling
  grep -i "FailedScheduling" /tmp/pod_desc.txt

  # 2. Check node selectors
  echo "🧪 NodeSelector:"
  grep -A2 "Node-Selectors" /tmp/pod_desc.txt

  # 3. Check tolerations
  echo "🧪 Tolerations:"
  grep -A5 "Tolerations:" /tmp/pod_desc.txt

  # 4. Check affinity
  echo "🧪 Affinity:"
  kubectl get pod $pod -n $namespace -o jsonpath='{.spec.affinity}' || echo "None"

  # 5. Check resource requests
  echo "🧪 Resource Requests:"
  kubectl get pod $pod -n $namespace -o jsonpath='{range .spec.containers[*]}{.name}{" => CPU: "}{.resources.requests.cpu}{" | MEM: "}{.resources.requests.memory}{"\n"}{end}'

  # 6. Check PVCs
  echo "🧪 PVCs:"
  PVCs=$(kubectl get pod $pod -n $namespace -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
  for pvc in $PVCs; do
    echo "  🔄 PVC: $pvc => Status: $(kubectl get pvc $pvc -n $namespace -o jsonpath='{.status.phase}')"
  done

  # 7. Check scheduler
  echo "🧪 Scheduler:"
  kubectl get pod $pod -n $namespace -o jsonpath='{.spec.schedulerName}'; echo

  echo ""
done

echo "==============================="
echo "🔍 Checking Node Conditions..."
echo "==============================="

for node in $(kubectl get nodes -o name); do
  echo ""
  echo "Node: $node"
  echo "-----------"

  echo "🧪 Taints:"
  kubectl describe $node | grep Taint || echo "No taints"

  echo "🧪 Allocatable Resources:"
  kubectl describe $node | grep -A10 "Allocatable"

  echo "🧪 Max Pods Limit:"
  kubectl describe $node | grep -A10 Allocatable | grep "pods"
  
  echo "🧪 Running Pods Count:"
  nodeName=$(basename $node)
  kubectl get pods --all-namespaces -o wide | grep $nodeName | wc -l
done

echo ""
echo "✅ Done checking all pending pod conditions!"
Code language: PHP (php)

🧪 How to Use

  1. Save the script:
nano check-pending-pods.sh
# Paste the code
chmod +x check-pending-pods.sh
Code language: CSS (css)
  1. Run the script:
./check-pending-pods.sh

✅ What it Checks

  • Pod scheduling failures
  • Node selectors
  • Tolerations
  • Affinity/anti-affinity
  • CPU/Memory resource requests
  • PVC binding status
  • Scheduler used
  • Taints on nodes
  • Allocatable and used pod count
  • Max pod limits

Subscribe
Notify of
guest
0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments

Certification Courses

DevOpsSchool has introduced a series of professional certification courses designed to enhance your skills and expertise in cutting-edge technologies and methodologies. Whether you are aiming to excel in development, security, or operations, these certifications provide a comprehensive learning experience. Explore the following programs:

DevOps Certification, SRE Certification, and DevSecOps Certification by DevOpsSchool

Explore our DevOps Certification, SRE Certification, and DevSecOps Certification programs at DevOpsSchool. Gain the expertise needed to excel in your career with hands-on training and globally recognized certifications.

0
Would love your thoughts, please comment.x
()
x