p24-infra Docs

promtail-not-sending-logs

4 min read

Playbook: PromtailNotSendingLogs

Alert: PromtailNotSendingLogs Severity: Warning Fires when: rate(promtail_sent_entries_total{job="promtail"}[10m]) == 0 for 15 minutes Server: vps-h1 (72.60.32.61)

What triggers this

Promtail is up and reachable (Prometheus can scrape :9080/metrics) but the promtail_sent_entries_total counter has not incremented in 15 minutes.

Most common cause (confirmed 2026-06-23): Loki on vps-i1 is unreachable. Promtail successfully scrapes Docker container logs but cannot push to https://loki.vps-i1.infra.zintegrowana.online/loki/api/v1/push. After the Promtail client hits its maximum retry budget it stops logging retry attempts — the container appears silent but is actually buffering logs internally.

Less common causes:

  • Docker socket not mounted (/var/run/docker.sock missing inside container)
  • Promtail config error (wrong Loki URL, bad auth)
  • Positions file corruption (Promtail skips all already-read positions)

How to confirm

Step 1 — Check Promtail logs for root cause

ssh root@72.60.32.61
docker logs root-promtail-1 --tail=50 2>&1

If you see only context deadline exceeded / net/http: request canceled errors pointing to loki.vps-i1.infra.zintegrowana.online: → Root cause is Loki being unreachable. This is self-healing. See Self-healing path.

If you see docker.sock or permission errors: → Root cause is Docker socket issue. See Fix: Docker socket issue.

If you see config parsing errors: → Root cause is config corruption. See Fix: Config error.

Step 2 — Verify Docker socket is mounted

docker exec root-promtail-1 ls -la /var/run/docker.sock

Expected: srw-rw---- 1 root <gid> 0 <date> /var/run/docker.sock

Step 3 — Check Loki reachability

# From vps-h1
curl -s -o /dev/null -w "%{http_code}" https://loki.vps-i1.infra.zintegrowana.online/ready
# Expected when healthy: 200
# Expected when down: 000 (connection refused/timeout)

Step 4 — Check entries sent counter

curl -s http://localhost:9080/metrics | grep promtail_sent_entries_total
# If value > 0 and matches last known value → counter frozen (Loki down, no new pushes)
# If value = 0 → Promtail never sent anything (config/socket issue)

Self-healing path: Loki was down

No action required. Promtail uses an exponential backoff retry strategy. When Loki recovers, Promtail automatically resumes pushing the buffered log entries.

To verify recovery after Loki comes back:

# Watch the counter increment (run on vps-h1)
watch -n 10 'curl -s http://localhost:9080/metrics | grep promtail_sent_entries_total'

If Promtail still shows 0 entries 30+ minutes after Loki is confirmed healthy, proceed to Fix: Positions file corruption.

Fix: Docker socket issue

If /var/run/docker.sock is missing or has wrong permissions inside the container:

ssh root@72.60.32.61
cd /root  # or wherever docker-compose.yml is deployed
 
# Verify compose file mounts the socket
grep -A5 promtail docker-compose.yml
 
# If socket is missing from volumes, update hostinger/docker-compose.yml in p24-infra
# and redeploy via secrets-sync.yml or manually:
docker compose up -d --no-deps promtail

The correct volume mount in hostinger/docker-compose.yml:

volumes:
  - ./promtail/config-vps-h1.yml:/etc/promtail/config.yml:ro
  - /var/run/docker.sock:/var/run/docker.sock:ro

Fix: Config error

ssh root@72.60.32.61
# Check the config file deployed on server
cat /root/promtail/config-vps-h1.yml  # or wherever it is mounted
 
# Restart to pick up any config fix
docker compose restart promtail
 
# Watch logs for startup errors
docker logs root-promtail-1 --tail=20

Fix: Positions file corruption

Positions file (/tmp/positions.yaml inside container) tracks the last read offset for each log file. If corrupted, Promtail may skip or refuse to read logs.

ssh root@72.60.32.61
 
# Stop promtail
docker compose stop promtail
 
# Remove the positions file (it lives inside the container — reset on restart)
docker run --rm -v /tmp:/tmp alpine rm -f /tmp/positions.yaml 2>/dev/null || true
 
# Restart
docker compose start promtail
docker logs root-promtail-1 --tail=20

Note: Promtail will re-read recent container logs from their current tail (not from the beginning) — some log lines from the downtime window may not be shipped to Loki. This is acceptable.

Escalation path

If none of the above fixes resolve the alert within 30 minutes:

  1. Check whether the vps-h1 Docker daemon itself has issues: systemctl status docker
  2. Verify the Promtail image version is not pulling a broken tag: docker inspect root-promtail-1 --format '{{.Config.Image}}'
  3. Check for disk full on vps-h1: df -h — Promtail writes to /tmp/positions.yaml
  4. Create a human-action issue in radieu/p24-infra with full output of docker logs root-promtail-1 --tail=100

Prevention

  • The PromtailNotSendingLogs alert fires whenever Loki (on vps-i1) is down. This is expected behaviour during vps-i1 restarts/maintenance windows.
  • If vps-i1 maintenance is planned, silence this alert in Alertmanager for the duration to avoid alert noise.
  • The alert annotation now includes the correct triage guidance (updated 2026-06-23, issue #1124).

Written 2026-06-23 — triggered by vps-i1 outage causing Loki to be unreachable. Investigation confirmed: Promtail healthy, Docker socket mounted, issue was Loki-side.

Graph View

Backlinks