{"id":79714,"date":"2026-11-02T04:55:41","date_gmt":"2026-11-02T04:55:41","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=79714"},"modified":"2026-11-02T04:55:42","modified_gmt":"2026-11-02T04:55:42","slug":"debugging-terraform-helm-drift-on-eks-failure-behaviours-and-fixes","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/debugging-terraform-helm-drift-on-eks-failure-behaviours-and-fixes\/","title":{"rendered":"Debugging Terraform + Helm Drift on EKS: Failure Behaviours and Fixes"},"content":{"rendered":"\n

A field tutorial based on a real incident removing a stale CriticalAddonsOnly<\/code> toleration from metrics-server<\/code> and KEDA<\/code> on EKS Auto Mode clusters managed by Terraform Cloud (TFC) + the eks-blueprints-addons<\/code> module.<\/em><\/p>\n\n\n\n

\n\n\n\n

TL;DR<\/h2>\n\n\n\n

We tried to remove a Kubernetes toleration<\/strong> that was set through a Terraform helm_release<\/code>. The code change looked trivial, but it took several apply cycles before the cluster actually changed. Along the way we hit five distinct failure behaviours<\/strong>, none of which threw an error \u2014 they all silently did nothing<\/em>. This post documents each one, the symptom, the root cause, and the fix, plus the mental models and commands that make these problems obvious.<\/p>\n\n\n\n

The single most useful signal throughout: the pod’s AGE<\/code><\/strong>. If a Helm value truly changed the rendered manifest, the pod rolls and AGE<\/code> resets. If AGE<\/code> doesn’t reset, nothing actually changed<\/em> \u2014 no matter what Terraform or Helm said.<\/p>\n\n\n\n

\n\n\n\n

Background: the change we wanted<\/h2>\n\n\n\n

EKS Auto Mode clusters here run a single built-in general-purpose<\/code><\/strong> NodePool. There is no system<\/code> NodePool<\/strong>, so nothing in the cluster carries the CriticalAddonsOnly:NoSchedule<\/code> taint.<\/p>\n\n\n\n

metrics-server<\/code> (and KEDA) had this toleration set via Terraform:<\/p>\n\n\n

metrics_server = {\n  values = [yamlencode({\n    tolerations = [{\n      key      = \"CriticalAddonsOnly\"<\/span>\n      operator = \"Exists\"<\/span>\n      effect   = \"NoSchedule\"<\/span>\n    }]\n  })]\n}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Mental model #1 \u2014 a toleration is permissive, not attractive<\/h3>\n\n\n\n
A toleration does not<\/strong> force, attract, or pin a pod to a tainted node. It only permits<\/em> a pod to land on a node that carries the matching taint. With no system<\/code> NodePool, there is no matching taint, so the toleration matches nothing and has zero effect<\/strong>. It is dead config.<\/p>\n\n\n\n
\n
Removing it is pure cleanup \u2014 it never changed where the pod ran. The pod was always on general-purpose<\/code>, with or without the toleration.<\/p>\n<\/blockquote>\n\n\n\n
That made it a perfect low-risk change… and a perfect way to surface every Terraform\/Helm drift trap, because the expected<\/em> runtime effect was nil and we had to rely on tooling signals to know whether anything happened.<\/p>\n\n\n\n
\n\n\n\n
Failure Behaviour #1 \u2014 Local edits never reached the pipeline<\/h2>\n\n\n\n
Symptom:<\/strong> Edited the .tf<\/code> file, “applied,” but the cluster was unchanged. helm get values<\/code> still showed the toleration; the pod AGE<\/code> was unchanged.<\/p>\n\n\n\n
Root cause:<\/strong> TFC builds from the VCS-connected branch<\/strong>, not from local working-tree files. Editing a file on the laptop (or even committing it to a feature branch<\/em>) does nothing until that commit is on the branch the TFC workspace tracks (here, main<\/code>).<\/p>\n\n\n\n
How we detected it:<\/strong><\/p>\n\n\n
git status -sb                 <em<\/span>><\/span># showed work was on a feature branch, not main<\/em<\/span>><\/span>\ngit show origin\/main:<file<\/span>><\/span>    <em<\/span>><\/span># main still contained the toleration<\/em<\/span>><\/span>\ngit log origin\/main..HEAD      <em<\/span>><\/span># the fix commit was not in main<\/em<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Fix:<\/strong> Merge the change into the branch TFC tracks, then apply.<\/p>\n\n\n\n
\n
Lesson:<\/strong> “I applied it” means the pipeline applied the merged code<\/em>, not “I saved the file.” Always confirm which commit the run used.<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\n
Failure Behaviour #2 \u2014 Stale origin\/main<\/code> gave a wrong conclusion<\/h2>\n\n\n\n
Symptom:<\/strong> We concluded “main doesn’t have the fix” \u2014 but it actually did. The PR had already been merged.<\/p>\n\n\n\n
Root cause:<\/strong> Local remote-tracking refs are a snapshot from the last fetch<\/strong>. git show origin\/main:...<\/code> reads that stale snapshot, not the live remote.<\/p>\n\n\n\n
Fix:<\/strong><\/p>\n\n\n
git fetch origin --quiet\ngit show origin\/main:<file<\/span>><\/span>                       <em<\/span>><\/span># now reflects reality<\/em<\/span>><\/span>\ngit merge-base --is-ancestor <sha<\/span>><\/span> origin\/main && echo \"in main\"\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
\n
Lesson:<\/strong> Always git fetch<\/code> before reasoning about remote branch state. Most “the merge didn’t happen” panics are stale refs.<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\n
Failure Behaviour #3 \u2014 Helm reported “updated”, but nothing rolled<\/h2>\n\n\n\n
Symptom:<\/strong> The TFC apply log said:<\/p>\n\n\n
Plan: 0 to add, 2 to change, 0 to destroy.\n... module....metrics_server.helm_release.this[0]: Plan to update\nupdated  id=metrics-server\n<\/code><\/span><\/pre>\n\n\n
A new Helm revision<\/strong> was created. Yet:<\/p>\n\n\n
helm -n kube-system get values metrics-server   <em<\/span>><\/span># toleration STILL present<\/em<\/span>><\/span>\nkubectl get pods -l ... -o wide                  <em<\/span>><\/span># pod AGE unchanged (47h)<\/em<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Root cause:<\/strong> Two things were true at once:<\/p>\n\n\n\n
    \n
  1. The run executed against a commit that\u00a0still had the toleration<\/em>\u00a0(timing \/ pinned-commit issue \u2014 see #1\/#2). So the values Helm received were unchanged in the part that matters.<\/li>\n\n\n\n
  2. A Helm\u00a0upgrade always bumps the revision number<\/strong>, even when the rendered manifest is identical. A new revision is\u00a0not<\/em>\u00a0proof of a real change.<\/li>\n<\/ol>\n\n\n\n
    Mental model #2 \u2014 revision number is not a diff.<\/strong> The only reliable evidence that the manifest changed is the rolled pod<\/strong> (fresh AGE<\/code>) and the actual rendered output (helm get manifest<\/code>).<\/p>\n\n\n\n
    How we detected it:<\/strong><\/p>\n\n\n
    helm -n kube-system history metrics-server          <em<\/span>><\/span># revision incremented...<\/em<\/span>><\/span>\nkubectl get pods -l app.kubernetes.io\/name=metrics-server -o wide  <em<\/span>><\/span># ...but AGE did NOT reset<\/em<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
    \n
    Lesson:<\/strong> Verify outcomes by observed cluster state<\/strong>, not by Terraform’s “updated” or Helm’s revision counter. Pod AGE<\/code> is the truth.<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    Failure Behaviour #4 \u2014 The Terraform helm_release<\/code> drift blind spot<\/h2>\n\n\n\n
    This was the core, non-obvious one.<\/p>\n\n\n\n
    Symptom:<\/strong> After the fix was genuinely in main<\/code>, a fresh TFC run reported “No changes”<\/strong> \u2014 while the live release still<\/em> carried the toleration.<\/p>\n\n\n
    helm -n kube-system get values metrics-server\n<em># USER-SUPPLIED VALUES still shows the toleration<\/em><\/span>\n\n<em># ...but TFC plan: No changes. Your infrastructure matches the configuration.<\/em><\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    Root cause:<\/strong> The Terraform Helm provider compares your config<\/strong> against Terraform state<\/strong> \u2014 it does not<\/strong> re-read and diff the live release’s<\/em> actual values on every plan. Once a prior run wrote “no toleration” into state<\/strong>, config == state<\/code>, so Terraform sees nothing to do \u2014 even though the live cluster release had drifted<\/strong> and still had the toleration.<\/p>\n\n\n\n
       config (no toleration)  ==  state (no toleration)   -> \"No changes\"\n                                       \u2260\n                          live release (HAS toleration)   <-- invisible to plan\n<\/code><\/pre>\n\n\n\n
    Mental model #3 \u2014 helm_release<\/code> trusts state, not the cluster.<\/strong> Out-of-band drift in a release’s rendered values is frequently not<\/em> detected by a normal plan. “No changes” means “config matches state,” not “cluster matches config.”<\/p>\n\n\n\n
    How we confirmed it:<\/strong><\/p>\n\n\n
    <em<\/span>><\/span># What the live release actually has:<\/em<\/span>><\/span>\nhelm -n kube-system get values metrics-server | grep -i -A3 toleration   <em<\/span>><\/span># present<\/em<\/span>><\/span>\n\n<em<\/span>><\/span># What Terraform state thinks it has:<\/em<\/span>><\/span>\nterraform state show '...metrics_server.helm_release.this[0]' | grep -i -A3 toleration  <em<\/span>><\/span># absent<\/em<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Present in one, absent in the other = confirmed drift the plan won’t catch.<\/p>\n\n\n\n
    Fixes (any one works)<\/h3>\n\n\n\n
    A. Force a real value change (VCS-native, no special access).<\/strong> Make the values<\/code> genuinely differ from state so Terraform must<\/em> run helm upgrade<\/code>, which re-renders from current config and drops the toleration:<\/p>\n\n\n
    metrics_server = {\n  values = [yamlencode({\n    podLabels = { \"k8s-app\"<\/span> = \"metrics-server\"<\/span> }   # benign, also matches other envs<\/span>\n  })]\n}\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    B. Force-replace the release (the literal “force apply”).<\/strong> TFC has no UI button for targeted replace; use the API replace-addrs<\/code> or a CLI-driven run:<\/p>\n\n\n
    <em># CLI-driven (only if the workspace permits CLI applies):<\/em><\/span>\nterraform apply -replace='module....metrics_server.helm_release.this[0]'<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n
    <em># TFC API run with replace-addrs (works for VCS workspaces):<\/em><\/span>\ncurl -s \\\n  --header \"Authorization: Bearer $TFC_TOKEN\"<\/span> \\\n  --header \"Content-Type: application\/vnd.api+json\"<\/span> \\\n  --request POST \\\n  --data '{\n    \"data\": {\n      \"type\": \"runs\",\n      \"attributes\": {\n        \"message\": \"force replace metrics-server (toleration drift)\",\n        \"replace-addrs\": [\"module....metrics_server.helm_release.this[0]\"]\n      },\n      \"relationships\": { \"workspace\": { \"data\": { \"type\": \"workspaces\", \"id\": \"ws-XXXX\" } } }\n    }\n  }'<\/span> \\\n  https:\/\/app.terraform.io\/api\/v2\/runs<\/span>\n<em># then Confirm & Apply the run in the UI<\/em><\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    We used Option A<\/strong> \u2014 it goes through the normal PR \u2192 apply gate and is auditable.<\/p>\n\n\n\n
    \n
    Lesson:<\/strong> When plan<\/code> says “no changes” but the cluster disagrees, you have state\u2194cluster drift. Don’t keep re-running plan \u2014 force the resource to reconcile (value change or -replace<\/code>).<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    Failure Behaviour #5 \u2014 Branch hygiene \/ “fix is in a feature branch”<\/h2>\n\n\n\n
    Symptom:<\/strong> The fix existed and was committed… on an unmerged feature branch, while the workspace deployed from main<\/code>.<\/p>\n\n\n\n
    Fix:<\/strong> Standard hygiene \u2014 confirm the fix is on the deployed branch:<\/p>\n\n\n
    git fetch origin\ngit merge-base --is-ancestor <fix-sha> origin\/main && echo<\/span> \"deployable\"<\/span> || echo<\/span> \"still in a branch\"<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    \n
    Lesson:<\/strong> “Committed” \u2260 “deployed.” Confirm the SHA the workspace will use.<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    What success finally looked like<\/h2>\n\n\n
    $ helm -n kube-system history metrics-server\nREVISION  UPDATED                   STATUS       ...\n1<\/span>         Wed Mar  4<\/span> 09<\/span>:53<\/span>:46<\/span> 2026<\/span>  superseded\n2<\/span>         Fri Jun 19<\/span> 04<\/span>:25<\/span>:37<\/span> 2026<\/span>  superseded\n3<\/span>         Fri Jun 19<\/span> 05<\/span>:21<\/span>:07<\/span> 2026<\/span>  deployed     <em># <-- fresh<\/em><\/span>\n\n$ helm -n kube-system get values metrics-server\nUSER-SUPPLIED VALUES:\npodLabels:\n  k8s-app: metrics-server          <em># <-- no tolerations<\/em><\/span>\n\n$ kubectl -n kube-system get pods -l app.kubernetes.io\/name=metrics-server -o wide\nNAME                              READY  STATUS   RESTARTS  AGE     ...\nmetrics-server-64585697<\/span>f6-nbgsg   1<\/span>\/1<\/span>    Running  0<\/span>         2<\/span>m39s   <em># <-- AGE reset = rolled<\/em><\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    New revision +<\/strong> value gone +<\/strong> pod AGE<\/code> reset = genuinely done.<\/p>\n\n\n\n
    \n\n\n\n
    Diagnostic cheat sheet<\/h2>\n\n\n\n
    Question<\/th>Command<\/th><\/tr><\/thead>
    Which commit\/branch did the run use?<\/td>TFC run detail \u2192 “Configuration \/ Commit”; locally git fetch && git log origin\/main..HEAD<\/code><\/td><\/tr>
    Is my fix actually in the deployed branch?<\/td>git merge-base --is-ancestor <sha> origin\/main<\/code><\/td><\/tr>
    What did Helm actually deploy?<\/td>helm -n <ns> get manifest <release><\/code><\/td><\/tr>
    What user values are live?<\/td>helm -n <ns> get values <release><\/code><\/td><\/tr>
    Did a new revision deploy?<\/td>helm -n <ns> history <release><\/code><\/td><\/tr>
    Did the pod actually roll?<\/strong><\/td>kubectl -n <ns> get pods -l <selector> -o wide<\/code> \u2192 check AGE<\/code><\/td><\/tr>
    What does TF state think?<\/td>terraform state show '<addr>'<\/code><\/td><\/tr>
    Are there NodePools\/taints that matter?<\/td>kubectl get nodepools.karpenter.sh<\/code>; kubectl get nodepool <n> -o jsonpath='{.spec.template.spec.taints}'<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    Key takeaways<\/h2>\n\n\n\n
      \n
    1. Toleration \u2260 placement.<\/strong>\u00a0A toleration only permits scheduling onto a matching taint; with no such taint it is inert. Removing inert config is cleanup, not a behaviour change.<\/li>\n\n\n\n
    2. Pod\u00a0AGE<\/code>\u00a0is the source of truth.<\/strong>\u00a0Terraform “updated” and Helm revision bumps can both happen with\u00a0no real change<\/em>. A rolled pod proves it.<\/li>\n\n\n\n
    3. helm_release<\/code>\u00a0compares config to state, not to the cluster.<\/strong>\u00a0“No changes” can hide live drift. Reconcile with a forced value change or\u00a0-replace<\/code>.<\/li>\n\n\n\n
    4. “Applied” means the pipeline applied the merged code.<\/strong>\u00a0Always verify the commit\/branch the run used;\u00a0git fetch<\/code>\u00a0before trusting remote refs.<\/li>\n\n\n\n
    5. Verify on the cluster, not in the plan.<\/strong>\u00a0helm get values<\/code>\u00a0\/\u00a0get manifest<\/code>\u00a0\/ pod\u00a0AGE<\/code>\u00a0beat any amount of staring at Terraform output.<\/li>\n<\/ol>\n\n\n\n
      \n\n\n\n
      Generated from an incident review. Commands shown for EKS Auto Mode + Terraform Cloud + aws-ia\/eks-blueprints-addons<\/code>; the principles apply to any Terraform + Helm provider setup.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
      A field tutorial based on a real incident removing a stale CriticalAddonsOnly toleration from metrics-server and KEDA on EKS Auto Mode clusters managed by Terraform Cloud (TFC) + the eks-blueprints-addons module. TL;DR We tried to… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[11138],"tags":[],"class_list":["post-79714","post","type-post","status-publish","format-standard","hentry","category-best-tools"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/79714","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=79714"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/79714\/revisions"}],"predecessor-version":[{"id":79715,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/79714\/revisions\/79715"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=79714"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=79714"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=79714"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}