Prometheus alert emails flap every hour because for: is shorter than the scrape_interval

2026-11-13 · 3 min read

Prometheus alert emails flap every hour because for: is shorter than the scrape_interval

At 10:00, 11:00, and 12:00, the same alert sends two emails: firing, then resolved. The service had one failed scrape each hour. The rule used for: 30s, but the target’s scrape_interval was 60s. That is enough for prometheus alert flapping: the pending period completes before the next scrape can clear the condition, so every brief scrape failure becomes a page.

The cost is not just inbox noise. On-call learns to ignore the alert, and a real outage that lasts several scrapes looks like the same hourly blip.

How a short for causes prometheus alert flapping

Prometheus evaluates alerting rules on evaluation_interval. A rule with for: 30s and an evaluation interval of 15s can become firing after two evaluations. If the target is scraped every 60s, a single failed scrape at t=0 is enough: the alert goes pending, reaches for, and fires before the next scrape at t=60. If that scrape succeeds, the alert resolves. The next failed scrape repeats the cycle.

The for field is the pending duration before firing [link needed]. scrape_interval controls how often samples arrive [link needed]. When for is shorter than scrape_interval, the rule can decide based on a sample that is already stale by the time the next scrape arrives. Add a target that fails intermittently and you get regular firing/resolved pairs, even though nothing changed in the service.

Confirm the mismatch before changing rules

1. In Alertmanager, find firing/resolved pairs for the same alertname and instance. Note the timestamps. If the resolved notification arrives roughly one scrape interval after firing, suspect this pattern.

2. Open the rule file and record the for value. Check the global scrape_interval and any per-job override in scrape_configs.

3. Check the loaded configuration and active rules: promtool check rules <file>, Prometheus /api/v1/status/config, and /api/v1/rules [link needed].

4. Compare the intervals. If for is less than or equal to the target’s scrape_interval, you have a plausible cause.

5. Confirm the source is intermittent, not continuously down. Query up{job="..."} over the flap window. Continuous zeros mean a different problem.

Fix it by making for longer than scrape_interval

The direct fix for prometheus alert flapping is to set for comfortably longer than the scrape interval. For a 60s scrape interval, use at least 90s to ignore one missed scrape. To require two consecutive failed scrapes, use more than 120s; add one evaluation interval as margin. With a 15s evaluation interval, 2m30s is a practical starting point.

Example:


- alert: TargetDown
  expr: up{job="api"} == 0
  for: 2m30s
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.instance }} is down"

If only the resolution flaps, consider keep_firing_for to hold the alert firing briefly after the expression clears [link needed]. Alertmanager group_interval and repeat_interval can reduce email volume, but they do not fix the underlying firing/resolved state changes.

When NOT to do this

Do not lengthen for blindly on latency-critical alerts where a 30s delay matters more than a noisy email. If the scrape interval is too long for the metric’s change rate, lower the scrape interval first and accept the extra scrape cost. If the alert comes from a recording rule with a different evaluation cadence, compare against that cadence, not only the target scrape interval. And if a single failed scrape should page, keep the short for and tune grouping instead.

Checklist