{"id":262,"date":"2026-10-13T04:51:20","date_gmt":"2026-10-13T04:51:20","guid":{"rendered":"http:\/\/www.scmgalaxy.com\/tutorials\/2010\/06\/21\/how-we-reduced-build-time-from-8-hours-to-1-hour\/"},"modified":"2026-10-13T04:51:21","modified_gmt":"2026-10-13T04:51:21","slug":"how-we-reduced-build-time-from-8-hours-to-1-hour","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/how-we-reduced-build-time-from-8-hours-to-1-hour\/","title":{"rendered":"From 8-Hour Builds to 1-Hour Feedback: A Practical CI Optimization Pattern"},"content":{"rendered":"\n

This case study shows how a team reduced build feedback time from 8 hours to 1 hour<\/strong> by restructuring build and test execution. It is most useful for DevOps engineers and tech leads who need faster feedback, more stable integration cycles, and better use of existing build hardware.<\/p>\n\n\n\n

Originally published on 2010-06-21; fully revised and updated.<\/em><\/p>\n\n\n\n

Source article: How we reduced build time from 8 hours to 1 hour<\/a>.<\/p>\n\n\n\n

\"hudson<\/p>\n\n\n\n

The starting point: slow builds and low confidence<\/h2>\n\n\n\n

The original situation described a common CI pain pattern: long-running nightly pipelines, frequent failures, and delayed feedback. That combination usually reduces developer trust in the build signal.<\/p>\n\n\n\n

  • Build duration was around 8 hours, with frequent nightly failures.<\/li>
  • Test execution consumed more time than compilation.<\/li>
  • Developers had low confidence in nightly build outcomes during integration cycles.<\/li><\/ul>\n\n\n\n

    Workflow that reduced feedback time<\/h2>\n\n\n\n

    The key shift was to separate fast per-commit validation<\/strong> from full daily validation<\/strong>. Fast tests ran on every commit, while the heavier test suite moved to a scheduled run on a remote agent.<\/p>\n\n\n\n

    \nflowchart LR\n    A[Developer commit] --> B[CI build trigger]\n    B --> C[Parallel module compile]\n    C --> D[Run fast tests]\n    D --> E{Build pass}\n    E -- no --> F[Fix and recommit]\n    E -- yes --> G[Fast feedback to team]\n    B --> H[Nightly job on remote agent]\n    H --> I[Run full regression suite]\n    I --> J[Integration and release confidence]\n<\/pre>\n\n\n\n
    Changes implemented and why they helped<\/h2>\n\n\n\n
    Implemented change<\/th>Problem addressed<\/th>Why it helped<\/th><\/tr><\/thead>
    Refactored build scripts for parallel module compilation (where possible)<\/td>Compilation bottlenecks<\/td>Reduced total build wall-clock time by using available concurrency<\/td><\/tr>
    Selected tests with no external-service dependency for build-stage execution<\/td>Slow, fragile test phase<\/td>Kept per-commit pipeline fast and deterministic<\/td><\/tr>
    Established CI (originally using open-source CruiseControl) for per-commit builds<\/td>Late defect discovery<\/td>Provided earlier quality feedback after each change<\/td><\/tr>
    Added a low-cost remote agent for the remaining test suite once per day<\/td>Resource contention in the main build path<\/td>Offloaded expensive tests without blocking developer feedback<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Commit-stage vs nightly-stage testing model<\/h2>\n\n\n\n
    Stage<\/th>Main goal<\/th>Typical test scope<\/th>Dependencies<\/th>Frequency<\/th><\/tr><\/thead>
    Per-commit CI stage<\/td>Fast signal for developers<\/td>Compilation plus fast tests<\/td>Avoid external services when possible<\/td>Every commit<\/td><\/tr>
    Nightly remote-agent stage<\/td>Broader validation and integration confidence<\/td>Remaining full suite<\/td>Can include external systems and heavier setup<\/td>Once per day<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Example implementation patterns (illustrative)<\/h2>\n\n\n\n
    The original post does not publish exact scripts, so the snippets below are implementation patterns<\/strong> that mirror the approach: parallel build plus fast tests on commit, full suite on a scheduled remote agent.<\/p>\n\n\n\n
    Per-commit job commands<\/h3>\n\n\n
    #!\/usr\/bin\/env bash<\/span>\nset<\/span> -euo pipefail\n\n.\/build.sh --parallel\n.\/run-tests.sh --suite fast<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    Nightly job on remote agent<\/h3>\n\n\n
    #!\/usr\/bin\/env bash<\/span>\nset<\/span> -euo pipefail\n\n.\/run-tests.sh --suite full --agent remote-1 --report-dir reports\/nightly<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    Scheduled CI workflow example<\/h3>\n\n\n
    name: nightly-full-suite\non:\n  schedule:\n    - cron: '0 2 * * *'<\/span>\njobs:\n  full-regression:\n    runs-on: [self<\/span>-hosted, remote-agent]\n    steps:\n      - run: .\/run-tests.sh --suite full<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    Reported outcomes from the original case<\/h2>\n\n\n\n
    • Build duration was reduced from 8 hours to about 1 hour.<\/li>
    • Faster feedback improved developer confidence in CI results.<\/li>
    • The team was able to adopt additional engineering best practices.<\/li>
    • Existing hardware was used more effectively, avoiding additional infrastructure cost.<\/li>
    • Release management became smoother.<\/li><\/ul>\n\n\n\n
      Best practices and common pitfalls<\/h2>\n\n\n\n
      • Tier tests intentionally:<\/strong> keep per-commit checks fast; run heavier suites on schedule or on-demand.<\/li>
      • Parallelize safely:<\/strong> only parallelize modules with clean dependency boundaries.<\/li>
      • Protect fast feedback:<\/strong> avoid adding flaky external-service tests to the commit path.<\/li>
      • Use remote agents for expensive jobs:<\/strong> isolate heavy workloads from developer-facing CI.<\/li>
      • Track the right metrics:<\/strong> build duration, queue time, test flakiness, failure cause, and feedback latency.<\/li>
      • Avoid over-optimization:<\/strong> speed improvements should not hide critical regressions; keep full-suite coverage in place.<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
        This case-study rewrite breaks down how build feedback was reduced from 8 hours to 1 hour by combining parallel compilation, test-suite tiering, and a remote nightly agent. It also shows how to apply the same pattern in modern CI pipelines.<\/p>\n","protected":false},"author":1,"featured_media":4425,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[2],"tags":[4255,4252,4254,4257,1122,3113,4253,1806,4256,4240,593,4251,4258,104,1452],"class_list":["post-262","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorised","tag-build-process-time","tag-build-time","tag-build-time-optimization","tag-decrease-build-time","tag-guide","tag-how-to","tag-how-to-reduced-build-time","tag-instruction","tag-lessen-build-time","tag-method","tag-process","tag-reduce","tag-save-build-time","tag-tutorials","tag-ways"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/262","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=262"}],"version-history":[{"count":4,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/262\/revisions"}],"predecessor-version":[{"id":79090,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/262\/revisions\/79090"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4425"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=262"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}