Build Server Performance Optimization: Practical Checklist for Faster CI Pipelines

Build servers slow down when they rebuild too much code, keep too much data locally, or run on hosts with competing workloads. This guide helps DevOps and CI engineers optimize build throughput with a structured checklist covering infrastructure, CI configuration, application server tuning, and system-level hygiene.

Originally published on 2013-02-04; fully revised and updated.

Start with a baseline before tuning

Before changing anything, capture a baseline so you can verify whether each optimization helps. Focus on build duration, queue time, host resource pressure, and CI data growth.

• Average and percentile build time per pipeline
• Queue wait time and executor utilization
• CPU, memory, and disk I/O on build hosts
• Growth of workspaces, logs, and build artifacts

date

time mvn -DskipTests package

du -sh /var/lib/jenkins

df -hCode language: JavaScript (javascript)

Build server optimization workflow

flowchart LR
    A[Collect baseline metrics] --> B{Biggest bottleneck}
    B --> C[Too much code rebuilt]
    B --> D[Host resource contention]
    B --> E[CI data growth logs artifacts]
    C --> F[Use selective or module builds]
    D --> G[Stop noisy background services]
    E --> H[Apply retention and offload artifacts]
    F --> I[Measure again]
    G --> I
    H --> I
    I --> B

Build infrastructure improvements

Build only what changed

The original post correctly emphasizes avoiding full rebuilds when only part of the code changed. In practice, this means using tool features for selective/module builds and dependency-aware compilation.

# Make: parallelize and rebuild only out-of-date targets
make -j$(nproc)

# Maven: build selected modules and required dependencies
mvn -pl service-a,service-b -am test

# Gradle: target modules and enable cache/parallelism
./gradlew :service-a:build :service-b:build --parallel --build-cacheCode language: PHP (php)

Split projects into independently buildable modules

• Separate components so each module can build and test independently.
• Define clear dependency boundaries to avoid triggering unrelated builds.
• Integrate modules only where needed in downstream packaging or release stages.

Plan for disk and artifact growth early

Workspace size and retained artifacts grow quickly. Plan retention and cleanup from day one instead of waiting for emergency disk incidents.

du -sh /var/lib/jenkins/jobs
find /var/lib/jenkins/workspace -maxdepth 1 -mindepth 1 -type d | wc -lCode language: JavaScript (javascript)

Application server and CI service tuning

If your CI UI is fronted by Apache or another web server, keep it lean: reduce unnecessary modules, tune logging, and follow hardened configuration practices for your environment.

• Set practical log levels and rotate logs regularly.
• Disable unused modules and integrations.
• Review CI plugins/extensions periodically and remove unused ones.
• Set build history retention so old records and artifacts are pruned.

The original article referenced a Jenkins optimization paper: 7WaysToOptimizeJenkins.pdf.

System-level tuning for dedicated build hosts

Build hosts should be treated as dedicated infrastructure. Background services and ad hoc usage often create noisy-neighbor effects and unstable build times.

• Stop unnecessary background tasks that consume CPU or memory.
• Keep OS and runtime packages updated under controlled change management.
• Avoid using build servers for unrelated interactive work.
• Use storage that matches your I/O profile for builds and test workloads.

ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head
free -h
iostat -xz 1 5

Compare common optimization strategies

Example Jenkins guardrails for retention and stability

In Jenkins, distributed workers (commonly called agents; older material may use legacy terms) and retention defaults can significantly reduce controller pressure.

pipeline {
  agent { label 'linux' }
  options {
    disableConcurrentBuilds()
    timeout(time: 45, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '10'))
  }
  stages {
    stage('Checkout') {
      steps { checkout scm }
    }
    stage('Build') {
      steps { sh 'mvn -pl service-a,service-b -am test' }
    }
  }
}Code language: JavaScript (javascript)

# Example artifact offload step (repository URL/path will vary)
curl -u "$REPO_USER:$REPO_PASS" --upload-file target/app.jar \
  https://repo.example.com/repository/releases/com/example/app/1.0.0/app-1.0.0.jarCode language: PHP (php)

Common pitfalls

• Optimizing without a baseline, so gains cannot be verified.
• Keeping unlimited build logs and artifacts on the CI host.
• Running unrelated services or manual workloads on build machines.
• Using module-selective builds without maintaining dependency accuracy.
• Scaling agents without monitoring queue behavior and executor saturation.

Rajesh Kumar

I’m Rajesh Kumar, a DevOps, SRE, DevSecOps, Cloud, and Platform Engineering expert passionate about sharing practical knowledge, real-world experiences, and industry best practices. I have worked at Cotocus and regularly write about technology, travel, investing, health, product reviews, and digital marketing through my various platforms.

I publish technical articles at DevOps School, travel stories at Holiday Landmark, stock market insights at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow, and SEO and digital marketing strategies at Wizbrand.