Luiz Ozorio's Blog
Speeding Up Azure Pipelines

Speeding Up Azure Pipelines

A Practical Guide to Faster Builds

Luiz Ozorio's photo
Luiz Ozorio
·

11 min read

Table of contents

Introduction

In today's fast-paced development environment, waiting for sluggish CI/CD pipelines is a challenge no team wants to face. Slow pipelines can bottleneck your workflow, delay releases, and frustrate everyone involved. This guide aims to help you optimize your Azure Pipelines for speed and efficiency, ensuring that your team can focus on delivering high-quality software without unnecessary delays.

Why Pipeline Efficiency Matters

Efficient pipelines lead to faster feedback loops, enabling developers to identify and fix issues promptly. They also improve productivity by reducing downtime spent waiting for builds and tests to complete. Ultimately, speeding up your pipelines can lead to cost savings and a happier, more productive team.

What You'll Learn

We'll explore actionable strategies to accelerate your Azure Pipelines, from parallelization techniques to caching mechanisms. Whether you're a tech lead, developer, or DevOps professional, this guide offers insights that you can apply immediately to enhance your CI/CD processes.

1. Understanding Pipeline Performance Bottlenecks

Before optimizing, it's crucial to identify what's slowing down your pipelines.

Common Issues

  1. Sequential Execution: Tasks running one after another increase total build time.

  2. Resource Limitations: Insufficient agents lead to queued jobs and delays.

  3. Inefficient Job Design: Large, monolithic jobs that could be broken down for parallel execution.

  4. Unnecessary Processes: Running redundant tasks or tests that don't impact current changes.

Impact on Development Cycles

  • Delayed Feedback: Slow pipelines mean developers wait longer for results, hindering rapid iteration.

  • Reduced Productivity: Time spent waiting is time not spent coding or solving problems.

  • Higher Costs: Inefficient pipelines consume more resources, potentially increasing operational expenses.

2. Parallelization and Job Segmentation

Optimizing your pipeline starts with making the most of parallel execution and structuring your jobs effectively.

Benefits of Parallelization

  • Faster Pipeline Execution: Running tasks simultaneously reduces total build time.

  • Efficient Resource Utilization: Maximizes the use of available agents.

  • Quick Feedback: Faster results enable quicker iterations and more responsive development cycles.

Techniques for Parallel Execution

Define Independent Jobs

Split your pipeline into jobs that can run concurrently, reducing overall execution time.

jobs:
  - job: Build
    steps:
      - script: npm ci
      - script: npm run build
  - job: Lint
    steps:
      - script: npm ci
      - script: npm run lint
  - job: Test
    steps:
      - script: npm ci
      - script: npm run test
  - job: Deploy
    dependsOn:
      - Build
      - Lint
      - Test
    steps:
      - script: npm run deploy

Common Libraries Used:

  • Build: Using webpack or tsc for building JavaScript or TypeScript applications.

  • Lint: Utilizing ESLint for code linting.

  • Test: Running tests with frameworks like Jest, Mocha, or Jasmine.

Job Segmentation for Enhanced Parallelism

Breaking Down Different Types of Tests

Separate unit tests, integration tests, and end-to-end (E2E) tests into different jobs to run them in parallel.

jobs:
  - job: UnitTests
    steps:
      - script: npm ci
      - script: npm run test:unit
  - job: IntegrationTests
    steps:
      - script: npm ci
      - script: npm run test:integration
  - job: E2ETests
    steps:
      - script: npm ci
      - script: npm run test:e2e

Common Libraries Used:

  • Unit Tests: Jest, Mocha, Chai for testing individual units of code.

  • Integration Tests: SuperTest, Jest for testing how different parts of the application work together.

  • E2E Tests: Cypress, Selenium WebDriver, Playwright for end-to-end testing.

Parallel Testing and Test Sharding

Test sharding involves splitting your test suite into multiple parts (shards) and running them in parallel to reduce total test execution time.

Implementing Test Sharding in TypeScript

While Jest (commonly used with TypeScript) doesn't natively support test sharding across multiple machines, you can achieve it through custom scripts or by using third-party tools.

Method 1: Manual Test Splitting

Divide your test files into separate groups and run each group in parallel jobs.

Example:

jobs:
  - job: UnitTestsGroup1
    steps:
      - script: |
          npm ci
          npm run test -- tests/unit/group1/**/*.test.ts
  - job: UnitTestsGroup2
    steps:
      - script: |
          npm ci
          npm run test -- tests/unit/group2/**/*.test.ts
Method 2: Dynamic Test Sharding with Custom Scripts

Create a script to split test files dynamically based on the number of shards.

shard-tests.js

// shard-tests.js
const glob = require('glob');

const allTests = glob.sync('tests/**/*.test.ts');
const shardIndex = parseInt(process.env.SHARD_INDEX, 10);
const totalShards = parseInt(process.env.TOTAL_SHARDS, 10);

const testsPerShard = Math.ceil(allTests.length / totalShards);
const shardTests = allTests.slice(
  shardIndex * testsPerShard,
  (shardIndex + 1) * testsPerShard
);

console.log(shardTests.join(' '));

Pipeline Configuration:

jobs:
  - job: UnitTestsShard0
    variables:
      SHARD_INDEX: 0
      TOTAL_SHARDS: 2
    steps:
      - script: |
          npm ci
          TEST_FILES=$(node shard-tests.js)
          npm run test -- $TEST_FILES
  - job: UnitTestsShard1
    variables:
      SHARD_INDEX: 1
      TOTAL_SHARDS: 2
    steps:
      - script: |
          npm ci
          TEST_FILES=$(node shard-tests.js)
          npm run test -- $TEST_FILES
Method 3: Using Third-Party Tools

Use tools like jest-runner-groups or jest-split to automate test splitting.

Example with jest-split:

  1. Install jest-split:

     npm install jest-split --save-dev

  2. Update your test script in package.json:

     {
   "scripts": {
     "test": "jest-split --splits=2 --group=$JEST_GROUP"
   }
 }

  3. Set the JEST_GROUP environment variable in your pipeline:

     jobs:
   - job: UnitTestsGroup1
     variables:
       JEST_GROUP: 1
     steps:
       - script: |
           npm ci
           npm run test
   - job: UnitTestsGroup2
     variables:
       JEST_GROUP: 2
     steps:
       - script: |
           npm ci
           npm run test

Implementing Test Sharding with Other Frameworks

Cypress for E2E Tests

Cypress supports parallel test execution with its Dashboard Service.

Example:

  1. Set up Cypress Dashboard:

    • Sign up for a Cypress Dashboard account.

    • Obtain your project ID and record key.

  2. Modify your pipeline to run tests in parallel:

     steps:
   - script: |
       npm ci
       npx cypress run --record --parallel --key YOUR_RECORD_KEY
Azure DevOps Test Plan

If you're using Azure DevOps Test Plan with Visual Studio Test tasks, you can distribute tests across agents.

Example:

steps:
  - task: VisualStudioTestPlatformInstaller@1
  - task: VsTest@2
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        **\*test*.dll
        !**\*TestAdapter.dll
        !**\obj\**
      searchFolder: '$(System.DefaultWorkingDirectory)'
      runInParallel: true

This configuration will distribute tests across available agents.

Common Pitfalls in Parallelization and How to Avoid Them

Over-Paralleling Leading to Overhead

Excessive parallel jobs can increase overhead, such as redundant dependency installations.

  • Solution: Use caching or pre-install dependencies on agents to reduce overhead.

Resource Contention

Parallel jobs might compete for shared resources, causing conflicts.

  • Solution: Use isolated environments or mock services during testing.

Unmanaged Dependencies

Jobs may fail due to missing prerequisites.

  • Solution: Define dependencies using dependsOn to ensure correct execution order.

Inconsistent Environments

Variations in agent configurations can cause inconsistent results.

  • Solution: Standardize agent environments and specify software versions. Docker can help.

3. Managing Resource Congestion and Scaling Runners

Efficient resource management ensures that your pipeline can handle workloads without unnecessary delays.

Recognizing Signs of Congestion

  • Job Queues: Long waiting times before jobs start executing.

  • Pipeline Timeouts: Pipelines exceed time limits due to resource shortages.

  • Agent Shortages: Frequent alerts about unavailable agents or over-utilization.

Strategies for Optimizing Limited Runners

Prioritize Critical Pipelines

Allocate resources to essential builds to ensure they complete promptly.

Avoid Over-Scheduling Non-Critical Jobs

Be cautious not to over-schedule, as this can lead to resource congestion. Remember that testing is often critical and should not be delayed.

Optimize Job Durations

Streamline tasks to reduce execution time, freeing up agents more quickly.

Avoiding Over-Paralleling

Installing dependencies or other repetitive steps in multiple parallel jobs can create significant overhead.

  • Solution: Use caching mechanisms or shared workspaces to avoid redundant installations.

Leveraging Self-Hosted Agents for Increased Capacity

Self-hosted agents provide greater control and can alleviate resource constraints.

Installing NPM Tools with Binaries on Self-Hosted Agents

Pre-install tools that require binaries (like Puppeteer or Playwright) on self-hosted agents to save time during builds.

# On self-hosted agent
npm install -g puppeteer playwright

Benefits:

  • Reduced Build Times: Avoid time-consuming installations during pipeline execution.

  • Customized Environment: Tailor the agent to your specific needs, ensuring consistency and efficiency.

4. Implementing Fail-Fast Strategies

Failing fast allows you to catch issues early, saving time and resources.

Executing Fast Smoke Tests First

Run quick smoke tests before longer-running tests to identify critical failures early.

jobs:
  - job: SmokeTests
    steps:
      - script: npm ci
      - script: npm run test:smoke
    continueOnError: false
  - job: FullTests
    dependsOn: SmokeTests
    condition: succeeded()
    steps:
      - script: npm run test:full

Common Libraries Used:

  • Smoke Tests: Quick tests using frameworks like Jest or Mocha to verify basic functionality.

  • Full Tests: Comprehensive test suites covering unit, integration, and E2E tests.

Configuring Tasks to Fail Fast

Set your pipeline to stop execution immediately upon encountering a failure.

Use dependencies to halt execution of long running tasks

When faced with long-running tasks, it may be preferable to run them only if fast tasks succeed, allowing pipelines to fail faster.

jobs:
- job: A
  steps:
  - script: sleep 30
- job: B
  dependsOn: A 
  steps:
    - script: echo step 2.1
      condition: eq(variables['Build.SourceBranch'], 'refs/heads/main', succeeded())

The downside of this approach is it really only benefits teams that have a higher number of pipelines in parallel and have the resources to constantly trigger them, as this can be a highly manual process without further development.

Using Conditions to Prevent Unnecessary Execution

Employ conditions to skip tasks when previous steps have failed.

condition: succeeded()

Benefits of Fail-Fast Approaches

  • Resource Efficiency: Saves time and computing resources by not executing doomed tasks.

  • Faster Feedback: Developers can address issues sooner, improving productivity.

  • Streamlined Pipelines: Keeps the focus on passing builds and successful deployments.

5. Caching, Incremental Builds, and Optimizing Docker Builds

Effective use of caching and incremental builds can significantly reduce build times.

Understanding Caching in Azure Pipelines

Caching allows you to reuse previous outputs, reducing redundant work.

  • What to Cache: Common items include npm packages, build artifacts, and Docker layers.

Implementing Caching (with npm and TypeScript Examples)

Cache npm Packages

Caching npm dependencies can save time on installation during each build.

steps:
  - task: Cache@2
    inputs:
      key: 'npm | "$(Agent.OS)" | package-lock.json'
      path: $(Pipeline.Workspace)/.npm
  - script: npm ci --cache $(Pipeline.Workspace)/.npm

Cache TypeScript Build Artifacts

Caching build artifacts accelerates subsequent builds by reusing compiled outputs.

steps:
  - task: Cache@2
    inputs:
      key: 'tsc | "$(Agent.OS)" | tsconfig.json'
      path: |
        .tsbuildinfo
        dist/
  - script: npm run build

Configuring Incremental Builds

Enable incremental compilation in TypeScript to avoid recompiling unchanged code.

tsconfig.json

{
  "compilerOptions": {
    "incremental": true,
    "tsBuildInfoFile": "./.tsbuildinfo"
  }
}

Optimizing Docker Builds for Speed

Leverage Layer Caching

Order Dockerfile instructions to maximize cache efficiency.

FROM node:14

WORKDIR /app

COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

CMD ["node", "dist/index.js"]

Common Libraries Used:

  • Build Tools: webpack, babel for bundling and transpiling code.

Use Docker BuildKit

Enable BuildKit for improved build performance.

variables:
  DOCKER_BUILDKIT: 1

Best Practices to Maximize Build Efficiency

  • Monitor Cache Effectiveness: Regularly check if caching improves build times.

  • Avoid Cache Busting: Be cautious with changes that invalidate caches unnecessarily.

  • Optimize Dockerfile Instructions: Place frequently changing commands later in the Dockerfile to leverage layer caching.

6. Optimizing Tests in Monorepos

Monorepos can complicate CI/CD processes due to their size and complexity.

Challenges of Monorepos in CI/CD

  • Increased Build Times: Larger codebases take longer to build and test.

  • Interdependencies: Changes in one module may affect others, complicating testing.

Techniques for Targeted Testing

Change Detection

Identify which parts of the codebase have changed to focus testing efforts.

CHANGED_FILES=$(git diff --name-only HEAD~1)

Selective Testing

Run tests only for the modules or packages that have been modified.

steps:
  - script: |
      CHANGED_FILES=$(git diff --name-only HEAD~1)
      if echo "$CHANGED_FILES" | grep -q "packages/package-a/"; then
        npm run test --workspace=packages/package-a
      fi

Using Lerna Filtering for Efficient Builds

Lerna's filtering capabilities allow you to run scripts only on changed packages.

steps:
  - script: npx lerna run build --since HEAD~1
  - script: npx lerna run test --since HEAD~1

Common Libraries Used:

  • Monorepo Management: Lerna, Nx for managing multiple packages within a single repository.

Tools and Practices for Aligning Tests with Code Changes

  • Automated Scripts: Use scripts to automate the detection of changes and execution of relevant tests.

  • Dependency Graphs: Utilize tools like Nx to map out module dependencies, helping to understand the impact of changes.

Conclusion

Optimizing your Azure Pipelines for speed doesn't have to be a daunting task. By applying the strategies outlined in this guide, you can significantly reduce build times, provide faster feedback to developers, and improve overall productivity.

Recap of Key Strategies

  • Parallelization: Run tasks concurrently to expedite pipeline execution.

  • Resource Management: Optimize agent utilization and manage resource constraints.

  • Fail-Fast Approaches: Catch issues early to save time and resources.

  • Caching and Incremental Builds: Reuse previous work to speed up builds.

  • Monorepo Optimization: Focus testing efforts where they're needed most.

Embracing a Team-Based Approach

Breaking down complex projects like pipeline optimization into manageable chunks and executing them in parallel as a team can greatly enhance efficiency. This collaborative, long-term to mid-term strategy allows team members to focus on specific areas, making the overall project more manageable and effective. By leveraging the diverse skills within your team, you can accelerate improvements and foster a culture of continuous optimization.

Continuous Improvement

Remember that pipeline optimization is an ongoing process. Regularly review your pipelines, gather feedback, and make adjustments as necessary to maintain optimal performance.

Take Action

Implement these strategies in your Azure Pipelines to accelerate your development cycles. Your team will benefit from the improved efficiency, and you'll be better equipped to meet the demands of today's fast-paced development environments.

By focusing on practical, actionable steps and fostering a collaborative team approach, you can transform your Azure Pipelines into a streamlined, efficient component of your development process.

ci/ciazure-devopsAzureTypeScript