GitLab Runner job dies with exit code 137 because the container exceeds the CI job memory limit

2026-11-14 · 4 min read

GitLab Runner job dies with exit code 137 because the container exceeds the CI job memory limit

Your pipeline gets 18 minutes into a test suite, the job log stops mid-line, and the job fails with ERROR: Job failed: exit code 137. The retry fails at roughly the same place. Nothing in the log names a failing test, because nothing failed a test — the kernel killed the process tree. If the job runs on a shared runner, each of those retries also burns 18 minutes of quota before dying the same way.

What gitlab runner exit code 137 actually means

137 is 128 + 9: the process received signal 9 (SIGKILL). Nothing in the container chose to die, so nothing in the container can report why. The signal comes from outside, and in CI there are two realistic sources:

• The container's own memory cgroup limit. The Docker executor sets a limit if memory is configured in config.toml; the Kubernetes executor sets one from KUBERNETES_MEMORY_LIMIT. When the cgroup hits its ceiling, the kernel kills a process inside it and the container exits 137.
• A host-level shortage. The runner host itself ran out of RAM (or swap), and the kernel picked the heaviest process. This shows up as 137 too, but the fix is different.

Both are OOM kills. Neither is a compiler bug, a flaky test, or a corrupt cache, so don't start debugging the application until you've ruled them out.

Confirm which limit fired before changing anything

1. Run docker inspect <container> --format '{{.State.OOMKilled}}' on the runner host, or check the pod's lastState.terminated.reason for OOMKilled under kubectl describe pod.

2. Run dmesg -T | grep -i 'killed process' on the host. The timestamp should line up with the job's failure, and the process named should belong to your job.

3. Read the runner's config.toml. Look for memory, memory_swap, or KUBERNETES_MEMORY_LIMIT on the relevant [[runners]] section.

4. Compare against the job's actual peak: run the same workload locally or on a scratch host under docker stats, or scrape cgroup memory.peak if your kernel exposes it.

5. Check whether the shape of the job changed recently — a new dependency, a bigger fixture set, a parallel worker count bump.

If step 1 or 2 shows an OOM kill, you have your answer. If neither does, you're looking at something else that returns 137.

Fix the limit, then fix the job

Raising the ceiling is the fast fix and the one most people stop at. With the Docker executor, set a memory value in config.toml — but note that memory applies to every job that runner picks up, so a per-job allowance means either a dedicated runner or the Kubernetes executor with KUBERNETES_MEMORY_LIMIT set from a CI variable. Bump it in steps and re-measure; a number chosen to stop the current failure is a guess.

The slower fix is reducing peak usage, and it's the durable one:

• Cap parallelism — Jest --maxWorkers, pytest -n, make -j — to roughly the core count rather than unbounded.
• Split the suite across jobs so each holds less in memory at once.
• Stream large artifacts (tar to a file) instead of buffering them in a variable or in shell memory.
• Run the heavy stage in a resource_group-free, single-purpose job rather than inside a job that also builds and packages.

When NOT to raise the memory limit

Don't raise it if the job is dying on a shared, hosted runner you don't control — the limit is fixed and config.toml isn't yours to edit; reduce usage instead. Don't raise it if dmesg shows the whole host under pressure from several concurrent jobs: you'll just move the OOM kill to a neighbour. And don't raise it blind for a job that only fails *sometimes* — intermittent OOM usually means a leak or an unbounded loop, and more RAM only delays the failure.

Checklist

• [ ] Exit code reads exactly 137, not 137 in a shell-wrapped message like 137: command not found
• [ ] OOMKilled is true, or dmesg shows the kill at the failure timestamp
• [ ] memory / KUBERNETES_MEMORY_LIMIT located and its current value recorded
• [ ] Peak usage measured, not estimated
• [ ] Limit raised in a step, or parallelism capped, or suite split
• [ ] Re-run confirms the job survives; limit revisited if peak is still within 20% of the ceiling

If you've now seen gitlab runner exit code 137 twice on the same job and both OOMKilled and dmesg are silent, check for a wrapper script that converts signals — [link needed] — before touching memory settings at all.