Optimising CI/CD Processes - Identifying Common Steps
ℹ️ This is a multipart series exploring CI/CD optimisations ℹ️
- Intro!
- Identifying Common Steps
- Caching + Docker (and layer caching)
- Test Parallelization
- Hopper Configuration Upgrade (Deliveroo-specific)
Let’s get on the same page!
A few things before we get started (I know you may have already read this. If so, skip this section).
First, I’m going to talk very specifically here about JS/TS projects, but these ideas can be applied to all CI/CD processes.
Second, I’m going to use CircleCI as the CI/CD platform when talking through examples. These concepts can likely apply to other CI/CD platforms.
Lastly, some definitions so we’re all on the same page:
- Step: A step is a single unit of work in a CI/CD process
- For example, installing dependencies, setting up environment variables, initiating commands, etc.
- Job: A collection of steps
- For example, running tests, linting code, building a Docker image, etc.
- Build Pipeline: A collection of jobs that represents all work
Identifying common steps
Ask yourself, “are there any steps that are common across multiple jobs?” or “does my build pipeline have multiple jobs with similar steps?“.
If you answered, “Yes,” you can have a go at extracting these similar steps into a single job, which can then be used by other jobs.
Let’s look at a concrete example.
We have two jobs, test and build. Both of these jobs need the same dependencies to be installed, thus they both have an “Install Dependencies” step.
The “Install Dependencies” step is common to both. It’s a prime candidate to be extracted to its own job!
In the newly extracted job, we can install the dependencies and save them as an artifact, allowing for other jobs in the Build Pipeline to use them! I’ll talk more about this in my post about caching.
Once we’ve extracted the job our Build Pipeline would look something like this:
This helps in many ways.
First, we cut down on wasted time, and likely money, by not doing the exact same thing twice in two different places.
Second, it introduces consistency. While we should be installing the exact same dependencies everywhere (based on our lockfile), sometimes this may not be the case.
By installing dependencies in one place and reusing them, we can ensure the exact same dependencies are used for building, testing, linting, and running our application.
Be sure to have a look at the other posts (links at the top) in this series about optimising CI/CD processes!
Until next time...