Understanding Docker Layer Caching

What this article covers

• How the Docker build cache works.
• What causes layer invalidation.
• How to use cache mounts.
• Practical tips for faster builds.
• Caching in CI/CD and locally.

Introduction: Understanding Docker Layer Caching

Every line in a Dockerfile creates a layer. Docker stores these layers in cache and reuses them when nothing has changed. Organizing your commands correctly can drastically reduce build times. However, a single changed file early in your Dockerfile can invalidate all subsequent layers. Understanding layer caching is therefore essential for building Docker images efficiently.

This article explains how the cache works and how to use it effectively.

Key terms

• Layer: A level within a Docker image.
• Build cache: Storage for layers between builds.
• Cache invalidation: When a layer becomes unusable.
• BuildKit: Modern Docker builder with advanced caching.
• Cache mount: Temporary storage persisted within a build.
• Pull-through cache: Storage for downloaded images.
• Registry cache: Caching within a remote registry.
• Snapshot: Intermediate state of a layer.

How the cache works

When building an image, Docker checks whether an identical layer already exists for each instruction. If it does, that layer is reused from the cache. If not, the layer is built fresh, and all subsequent layers must be rebuilt.

A layer is considered unchanged when:

• The command is identical.
• The input files are identical.
• The previous layer is identical.

Order matters

Copy files that rarely change before those that do:

# Good: Dependencies before source code
COPY package*.json ./
RUN npm ci
COPY . .

# Bad: Every code change invalidates npm ci
COPY . .
RUN npm ci

Cache invalidation

Any change to a file copied before a COPY instruction invalidates the cache. This is particularly important with multi-stage builds and build dependencies.

Example:

COPY package.json /app
RUN npm install

If package.json changes, npm install must run again.

BuildKit cache mounts

BuildKit enables cache mounts that persist within a build:

# syntax=docker/dockerfile:1.7
FROM python:3.11-slim
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

This preserves the pip cache between builds.

npm cache

# syntax=docker/dockerfile:1.7
FROM node:20-slim
RUN --mount=type=cache,target=/root/.npm \
    npm ci

Maven cache

# syntax=docker/dockerfile:1.7
FROM maven:3.9-eclipse-temurin
RUN --mount=type=cache,target=/root/.m2 \
    mvn package -DskipTests

Cache in CI/CD

In continuous integration environments, the cache is often not persistent. Solutions include:

• docker buildx with cache-from and cache-to.
• External cache backends such as S3 or a registry.
• Cache layers stored in a Docker registry.

docker buildx build \
  --cache-from=type=registry,ref=user/image:cache \
  --cache-to=type=registry,ref=user/image:cache,mode=max \
  -t user/image:latest .

Checking the local cache

docker system df

Removing the cache:

docker builder prune

Cache statistics

BuildKit shows which layers came from the cache:

DOCKER_BUILDKIT=1 docker build --progress=plain .

Output marked CACHED indicates reused layers.

Tips for faster builds

• Copy files that change infrequently early in the Dockerfile.
• Use .dockerignore.
• Combine RUN commands only when they logically belong together.
• Take advantage of cache mounts.
• Avoid running RUN apt-get update standalone multiple times.
• Store CI cache externally.
• Leverage multi-stage builds.

Common pitfalls

• Cache invalidated: COPY . . appears too early.
• Too many layers: Each RUN creates its own layer.
• Cache bloat: Never running docker builder prune.
• CI without cache: Every build starts from scratch.
• Outdated base images: No caching of the latest security updates.
• Missing .dockerignore: Changes to irrelevant files invalidate layers.

Further reading and resources

• BotServ.de Writing Dockerfiles
• BotServ.de Docker Multi-Stage Builds
• BotServ.de Docker Image Optimization
• BotServ.de Docker Commands

FAQ: Docker Layer Caching

How long does the cache persist? Locally until you manually delete it; in CI it typically persists between builds.

Why is my cache being invalidated? Because files, commands, or base images have changed.

Should I always use BuildKit? Yes, it provides better caching options.

What is a cache mount? Persistent storage during a build, such as for package manager caches.

How do I verify that layers are cached? Use --progress=plain during the build.

Sources and further reading

• Docker Cache: https://docs.docker.com/build/cache/
• BuildKit: https://docs.docker.com/build/buildkit/
• Cache backends: https://docs.docker.com/build/cache/backends/

Summary: Understanding Docker Layer Caching

Docker layer caching significantly speeds up builds when you order your instructions correctly and copy less frequently changed files first. BuildKit extends caching with cache mounts for package managers and external cache backends. A proper .dockerignore, clean multi-stage builds, and regular cache maintenance are essential tools for achieving fast and reproducible builds. Understanding the cache mechanism saves time both locally and in your CI/CD pipeline.