Prometheus is one of the most popular open-source monitoring systems for Kubernetes. It collects metrics from Kubernetes nodes, pods, services, and applications, and gives you a powerful query language called PromQL to analyze them.
In this tutorial, we will install Prometheus in a Kubernetes cluster using the prometheus-community/prometheus Helm chart. We will also fix the common PVC pending issue by creating static hostPath PersistentVolumes manually.
This guide is useful for lab, demo, practice, and small Kubernetes clusters.
Architecture
We will install the following Prometheus components:
Prometheus Server
Alertmanager
Kube State Metrics
Node Exporter
Pushgateway
Prometheus Server will store data under:
/mnt/prometheus-server
Alertmanager will store data under:
/mnt/alertmanager
These directories will be created on the Kubernetes worker node using hostPath.
Important Note About hostPath
hostPath storage is node-specific.
That means data is stored directly on one Kubernetes node. If the pod moves to another node, the same data may not be available there.
For production, use a proper storage solution such as:
AWS EBS CSI Driver
NFS
Longhorn
Ceph
OpenEBS
Cloud provider storage class
For learning and testing, hostPath is fine.
Step 1: Check Kubernetes Nodes
First, check that your Kubernetes cluster is running.
kubectl get nodes
Example output:
NAME STATUS ROLES AGE VERSION
ip-172-31-5-107 Ready control-plane ...
ip-172-31-0-191 Ready worker ...
In this example, the Prometheus pod later runs on:
ip-172-31-0-191
So the hostPath directories must be created on that node.
Step 2: Install Helm
Check if Helm is already installed:
helm
If Helm is not installed, install it.
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh
Verify Helm:
helm version
Step 3: Add Prometheus Helm Repository
Add the Prometheus Community Helm repository:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Update Helm repositories:
helm repo update
Verify the repository:
helm repo list
Step 4: Install Prometheus Using Helm
Install Prometheus in the default namespace:
helm install my-prometheus prometheus-community/prometheus --version 29.13.0
Check all pods:
kubectl get pods
You may initially see something like this:
NAME READY STATUS AGE
my-prometheus-alertmanager-0 0/1 Pending ...
my-prometheus-server-67b68bd4c6-wv79t 0/2 Pending ...
my-prometheus-kube-state-metrics-7c4457f7d6-j7c8f 1/1 Running ...
my-prometheus-prometheus-node-exporter-5gs22 1/1 Running ...
my-prometheus-prometheus-node-exporter-kzqz9 1/1 Running ...
my-prometheus-prometheus-pushgateway-78fc6d6f89-khpfd 1/1 Running ...
If Prometheus Server and Alertmanager are pending, it is usually because their PVCs are waiting for storage.
Step 5: Check PVC Status
Run:
kubectl get pvc
Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-prometheus-server Pending <unset> 6m
storage-my-prometheus-alertmanager-0 Pending <unset> 6m
This means Kubernetes created PVCs, but there are no matching PVs available.
Step 6: Describe the Prometheus Pod
Check the Prometheus Server pod:
kubectl describe pod my-prometheus-server-67b68bd4c6-wv79t
You may see this error:
Warning FailedScheduling default-scheduler 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims.
This means Prometheus cannot start because its PVC is not bound.
Step 7: Create PersistentVolume for Prometheus Server
Create a file:
cat > prometheus-server-pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus-server-pv
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
claimRef:
namespace: default
name: my-prometheus-server
hostPath:
path: /mnt/prometheus-server
type: DirectoryOrCreate
EOF
Apply it:
kubectl apply -f prometheus-server-pv.yaml
Step 8: Create PersistentVolume for Alertmanager
Create another PV:
cat > alertmanager-pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: alertmanager-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
claimRef:
namespace: default
name: storage-my-prometheus-alertmanager-0
hostPath:
path: /mnt/alertmanager
type: DirectoryOrCreate
EOF
Apply it:
kubectl apply -f alertmanager-pv.yaml
Step 9: Verify PV and PVC Binding
Check PVs:
kubectl get pv
Expected output:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
prometheus-server-pv 8Gi RWO Retain Bound default/my-prometheus-server
alertmanager-pv 2Gi RWO Retain Bound default/storage-my-prometheus-alertmanager-0
Check PVCs:
kubectl get pvc
Expected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES
my-prometheus-server Bound prometheus-server-pv 8Gi RWO
storage-my-prometheus-alertmanager-0 Bound alertmanager-pv 2Gi RWO
At this point, the storage issue is fixed.
Step 10: Create hostPath Directories on Worker Node
Now create the required directories on the node where Prometheus is scheduled.
First, check where the pod is running:
kubectl get pod -o wide
Example:
my-prometheus-server-67b68bd4c6-wv79t 1/2 Running ip-172-31-0-191
SSH into that worker node:
ssh ubuntu@ip-172-31-0-191
Create directories:
sudo mkdir -p /mnt/prometheus-server
sudo mkdir -p /mnt/alertmanager
sudo mkdir -p /mnt/data
Set ownership:
sudo chown -R 65534:65534 /mnt/prometheus-server
sudo chown -R 65534:65534 /mnt/alertmanager
sudo chown -R 65534:65534 /mnt/data
Set permissions:
sudo chmod -R 775 /mnt/prometheus-server
sudo chmod -R 775 /mnt/alertmanager
sudo chmod -R 775 /mnt/data
Why 65534:65534?
Prometheus commonly runs as a non-root user inside the container. If the hostPath directory is owned by root and not writable, Prometheus may crash with a permission error.
Step 11: Check Prometheus Pods Again
Run:
kubectl get pods
Example output:
NAME READY STATUS RESTARTS AGE
my-prometheus-alertmanager-0 1/1 Running 0 13m
my-prometheus-kube-state-metrics-7c4457f7d6-j7c8f 1/1 Running 0 13m
my-prometheus-prometheus-node-exporter-5gs22 1/1 Running 0 13m
my-prometheus-prometheus-node-exporter-kzqz9 1/1 Running 0 13m
my-prometheus-prometheus-pushgateway-78fc6d6f89-khpfd 1/1 Running 0 13m
my-prometheus-server-67b68bd4c6-b7qn2 2/2 Running 0 49s
This means Prometheus is now successfully running.
Step 12: Troubleshoot CrashLoopBackOff
If Prometheus Server shows:
CrashLoopBackOff
Check the pod:
kubectl describe pod my-prometheus-server-67b68bd4c6-wv79t
Check logs of the correct container:
kubectl logs my-prometheus-server-67b68bd4c6-wv79t -c prometheus-server
If the pod restarted already, check previous logs:
kubectl logs my-prometheus-server-67b68bd4c6-wv79t -c prometheus-server --previous
A common issue is permission denied on /data.
Fix it on the worker node:
sudo mkdir -p /mnt/prometheus-server
sudo chown -R 65534:65534 /mnt/prometheus-server
sudo chmod -R 775 /mnt/prometheus-server
Then delete the pod so Kubernetes recreates it:
kubectl delete pod my-prometheus-server-67b68bd4c6-wv79t
Check again:
kubectl get pods
Step 13: Access Prometheus Using Port Forward
You can expose Prometheus locally using kubectl port-forward.
First, check the Prometheus Server pod name:
kubectl get pods
Example pod name:
my-prometheus-server-67b68bd4c6-b7qn2
Run port-forward:
kubectl --namespace default port-forward pod/my-prometheus-server-67b68bd4c6-b7qn2 9090:9090
Now open:
http://localhost:9090
Step 14: Access Prometheus from Anywhere Using 0.0.0.0
If you want Prometheus to be accessible using your server public IP, use:
kubectl --namespace default port-forward --address 0.0.0.0 pod/my-prometheus-server-67b68bd4c6-b7qn2 9090:9090
Now open:
http://<server-public-ip>:9090
Example:
http://13.XX.XX.XX:9090
If you are running on AWS EC2, make sure port 9090 is allowed in the Security Group.
Allow inbound:
Type: Custom TCP
Port: 9090
Source: Your IP address
For testing, you may use 0.0.0.0/0, but it is not recommended because Prometheus does not have authentication by default.
Step 15: Access Prometheus Using Service Port Forward
Instead of forwarding directly to a pod, it is better to forward to the service.
Check services:
kubectl get svc
You should see something like:
my-prometheus-server ClusterIP ... 80/TCP
Forward service port 80 to local port 9090:
kubectl --namespace default port-forward --address 0.0.0.0 svc/my-prometheus-server 9090:80
Now access:
http://<server-public-ip>:9090
This method is better than pod port-forwarding because pod names change when pods restart.
Step 16: Verify Prometheus UI
Open Prometheus in your browser:
http://<server-public-ip>:9090
Go to:
Status -> Targets
You should see targets such as:
kubernetes-apiservers
kubernetes-nodes
kubernetes-nodes-cadvisor
kubernetes-service-endpoints
prometheus
node-exporter
kube-state-metrics
pushgateway
Healthy targets should show:
UP
Step 17: Run Basic Prometheus Queries
In the Prometheus UI, go to the query box and try these queries.
Check all up targets:
up
Check Prometheus server health:
prometheus_build_info
Check Kubernetes node CPU metrics:
node_cpu_seconds_total
Check memory available:
node_memory_MemAvailable_bytes
Check filesystem size:
node_filesystem_size_bytes
Check pod metrics:
kube_pod_info
Check deployment replicas:
kube_deployment_status_replicas
Step 18: Useful Kubernetes Verification Commands
Check pods:
kubectl get pods
Check pods with node details:
kubectl get pods -o wide
Check services:
kubectl get svc
Check PV:
kubectl get pv
Check PVC:
kubectl get pvc
Describe Prometheus Server pod:
kubectl describe pod <prometheus-server-pod-name>
Check Prometheus Server logs:
kubectl logs <prometheus-server-pod-name> -c prometheus-server
Check Config Reloader logs:
kubectl logs <prometheus-server-pod-name> -c prometheus-server-configmap-reload
Check Helm release:
helm list
Check Helm status:
helm status my-prometheus
Step 19: Optional — Create Node Exporter Dashboard in Grafana
Prometheus collects metrics, but for dashboards, you usually install Grafana.
You can later install Grafana using Helm:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install my-grafana grafana/grafana
Then connect Grafana to Prometheus using this URL inside Kubernetes:
http://my-prometheus-server.default.svc.cluster.local
Step 20: Cleanup Commands
If you want to remove Prometheus:
helm uninstall my-prometheus
Delete PVs:
kubectl delete pv prometheus-server-pv
kubectl delete pv alertmanager-pv
Delete local data from worker node carefully:
sudo rm -rf /mnt/prometheus-server
sudo rm -rf /mnt/alertmanager
Only run the above commands if you really want to delete Prometheus data.
Common Issues and Fixes
Issue 1: PVC Pending
Error:
pod has unbound immediate PersistentVolumeClaims
Check PVC:
kubectl get pvc
Fix:
Create matching PVs for the PVCs:
kubectl apply -f prometheus-server-pv.yaml
kubectl apply -f alertmanager-pv.yaml
Issue 2: Prometheus Server CrashLoopBackOff
Check logs:
kubectl logs <prometheus-server-pod-name> -c prometheus-server
Possible reason:
permission denied
Fix permissions on worker node:
sudo chown -R 65534:65534 /mnt/prometheus-server
sudo chmod -R 775 /mnt/prometheus-server
Restart pod:
kubectl delete pod <prometheus-server-pod-name>
Issue 3: Port Forward Works Locally but Not from Browser
Use:
kubectl --namespace default port-forward --address 0.0.0.0 svc/my-prometheus-server 9090:80
Then open:
http://<server-public-ip>:9090
Also check cloud firewall or AWS Security Group.
Issue 4: Wrong Container Logs
If you run:
kubectl logs my-prometheus-server-67b68bd4c6-wv79t
Kubernetes may show logs from the default container:
prometheus-server-configmap-reload
To check actual Prometheus logs, run:
kubectl logs my-prometheus-server-67b68bd4c6-wv79t -c prometheus-server
Final Working Commands Summary
kubectl get nodes
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install my-prometheus prometheus-community/prometheus --version 29.13.0
kubectl get pods
kubectl get pvc
kubectl get pv
Create Prometheus PV:
cat > prometheus-server-pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus-server-pv
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
claimRef:
namespace: default
name: my-prometheus-server
hostPath:
path: /mnt/prometheus-server
type: DirectoryOrCreate
EOF
kubectl apply -f prometheus-server-pv.yaml
Create Alertmanager PV:
cat > alertmanager-pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: alertmanager-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
claimRef:
namespace: default
name: storage-my-prometheus-alertmanager-0
hostPath:
path: /mnt/alertmanager
type: DirectoryOrCreate
EOF
kubectl apply -f alertmanager-pv.yaml
Create directories on worker node:
sudo mkdir -p /mnt/prometheus-server /mnt/alertmanager /mnt/data
sudo chown -R 65534:65534 /mnt/prometheus-server /mnt/alertmanager /mnt/data
sudo chmod -R 775 /mnt/prometheus-server /mnt/alertmanager /mnt/data
Restart Prometheus pod if needed:
kubectl delete pod <prometheus-server-pod-name>
Verify:
kubectl get pods
kubectl get pv,pvc
Expose Prometheus:
kubectl --namespace default port-forward --address 0.0.0.0 svc/my-prometheus-server 9090:80
Access:
http://<server-public-ip>:9090
Conclusion
In this tutorial, we installed Prometheus on Kubernetes using Helm. We also fixed the most common storage issue where Prometheus pods remain pending because PVCs are not bound.
We manually created hostPath PersistentVolumes for Prometheus Server and Alertmanager, created the required host directories on the worker node, fixed permissions, verified all pods, and finally accessed Prometheus from the browser.
This setup is good for learning and practice. For production, replace hostPath with a proper dynamic storage provisioner such as AWS EBS CSI Driver or another production-grade storage backend.
