When you delete a Kubernetes object (CRD, Namespace, Pod, etc.), Kubernetes does not delete it immediately<\/strong>.<\/p>\n\n\n\n Instead it:<\/p>\n\n\n\n A finalizer<\/strong> is just a string tag like:<\/p>\n\n\n\n It means:<\/p>\n\n\n\n \u201cBefore you remove this object from etcd, call the controller that owns this finalizer so it can clean stuff up (external state, dependent resources, DNS, volumes, etc). Once it\u2019s done, it will remove its finalizer, and then K8s can truly delete the object.\u201d<\/p>\n<\/blockquote>\n\n\n\n So if the responsible controller never does its job<\/strong>, or is gone, or is misconfigured \u2192 finalizer never gets removed \u2192 resource stays Terminating forever<\/strong>.<\/p>\n\n\n\n That\u2019s exactly what you saw with:<\/p>\n\n\n\n Common reasons:<\/p>\n\n\n\n So the long waits \/ hangs are by design<\/strong>: Kubernetes is saying There\u2019s no magic global flag like \u201cignore all finalizers\u201d, but you can<\/em> make this much less painful by following some practices:<\/p>\n\n\n\n For things like KEDA, Prometheus, cert-manager, etc:<\/p>\n\n\n\n Deleting CRDs or namespaces first and operators later is the most common way to get into trouble.<\/p>\n\n\n\n What you did (patching For dev\/sandbox clusters<\/strong> \u2192 totally fine. When something is Terminating forever, my standard steps are:<\/p>\n\n\n\n For your use case (EKS + addons like KEDA, Datadog, etc.):<\/p>\n\n\n\n 1. What are finalizers, really? When you delete a Kubernetes object (CRD, Namespace, Pod, etc.), Kubernetes does not delete it immediately. Instead it: A finalizer is just a string tag like: It means: \u201cBefore you remove this object from etcd, call the controller that owns this finalizer so it can clean stuff up (external state,…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_joinchat":[],"footnotes":""},"categories":[11138],"tags":[],"class_list":["post-54339","post","type-post","status-publish","format-standard","hentry","category-best-tools"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/54339","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=54339"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/54339\/revisions"}],"predecessor-version":[{"id":54340,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/54339\/revisions\/54340"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=54339"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=54339"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=54339"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
metadata.deletionTimestamp<\/code><\/li>\n\n\n\nmetadata.finalizers<\/code><\/li>\n<\/ol>\n\n\n\n\n
\"kubernetes\"<\/code> (for namespaces)<\/li>\n\n\n\n\"customresourcecleanup.apiextensions.k8s.io\"<\/code> (for CRDs)<\/li>\n\n\n\n\"finalizer.keda.sh\"<\/code> (for KEDA)<\/li>\n\n\n\n\"foregroundDeletion\"<\/code> (for some resources)<\/li>\n<\/ul>\n\n\n\n\n
\n
customresourcecleanup.apiextensions.k8s.io<\/code> on your CRDs<\/li>\n\n\n\nkubernetes<\/code> finalizer on the keda<\/code> namespace<\/li>\n<\/ul>\n\n\n\n
\n\n\n\n2. Why does it sometimes take SO long (or never complete)?<\/h2>\n\n\n\n
\n
\n
\n
kubernetes<\/code>)<\/strong>\n\n
Terminating<\/code> forever.<\/li>\n<\/ul>\n<\/li>\n\n\n\n\n
\u201cBefore I forget this object, I must give controllers a chance to clean up external stuff.\u201d<\/p>\n\n\n\n
\n\n\n\n3. Is there a \u201cbetter\u201d way? (In practice)<\/h2>\n\n\n\n
a) Always uninstall the app\/operator correctly<\/h3>\n\n\n\n
\n
\n
\n\n\n\nb) Only patch finalizers as a last resort<\/strong><\/h3>\n\n\n\n
finalizers: []<\/code>) is the right last step<\/em>, but it comes with trade-offs:<\/p>\n\n\n\n\n
For prod<\/strong> \u2192 should be done carefully, knowing what might be left behind.<\/p>\n\n\n\n
\n\n\n\nc) How to quickly diagnose \u201cwhy is this stuck?\u201d<\/h3>\n\n\n\n
\n
kubectl get <kind> <name> -n <ns> -o jsonpath='{.metadata.finalizers}'<\/code> That tells you who is holding the deletion<\/strong>.<\/li>\n\n\n\nkubectl describe <kind> <name> -n <ns><\/code> Sometimes you\u2019ll see helpful errors like:\n\n
\n
kubectl logs -n keda deploy\/keda-operator<\/code><\/li>\n\n\n\nkube-apiserver<\/code> \/ apiextensions-apiserver<\/code> logs (harder on managed clusters)<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
\n\n\n\nd) How to avoid this pain in future?<\/h3>\n\n\n\n
\n
\n
helm uninstall <release><\/code> first, then<\/em> delete CRDs if needed.<\/li>\n<\/ul>\n<\/li>\n\n\n\n\n
\n
helm uninstall keda -n keda<\/code><\/li>\n\n\n\n\n
\n
kubectl patch ... finalizers: []<\/code>) is a perfectly OK escape hatch.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
\n\n\n\n4. TL;DR in human language<\/h2>\n\n\n\n
\n
\n