admin-actions-investigation

Playbook: Admin Actions Investigation

When to use: You need to determine who ran a command, changed a file, installed a package, or modified credentials on any server. Also use for post-incident forensics, compliance audits, or suspicious activity triage.

---

Overview — log sources

Source	What it covers	Retention	Access
Mezmo	All servers · all logs (syslog, auth, containers) — incl. BMS hosts	~few days	mezmo.com web UI or CLI
Loki	Docker container logs + syslog/auth.log for vps-i1, vps-h1 (Docker promtail) and bms-2/3/4 (#1392 — see note below).	14 days	Grafana → Explore
auditd	sudo/su, file writes (passwd/shadow/sudoers/sshd_config), package installs, docker, user/group mgmt — plus every command when auditd_log_all_execve=true	Local only (search with ausearch)	ausearch / journalctl on server
/var/log/auth.log	SSH logins, sudo, su, passwd	Local ~4 weeks (logrotate)	grep on server or via Mezmo

Where each host’s syslog lands: vps-i1 and vps-h1 run Promtail as a Docker container that ships /var/log/{auth.log,syslog} to the central Loki on vps-i1 (configs: monitoring/promtail/config-vps-i1.yml, hostinger/promtail/config-vps-h1.yml). The BMS hosts (bms-2/3/4) get auditd (Ansible bms-servers.yml, audit-daemon role) and ship syslog to central Loki (#1392), so {job="syslog", host="bms-N"} works for them too:

• bms-3 / bms-4 — the promtail-syslog systemd role (Ansible bms-servers.yml, group syslog_promtail_hosts, listens on :9080).
• bms-2 — already runs a Docker Promtail on :9080 (unmanaged secondary monitoring stack), so the systemd role is not deployed there; instead the job_name: syslog block from monitoring/promtail/config-bms-2.yml is merged into that Docker Promtail’s config by hand.

Deploying these requires running bms-servers.yml (human-gated — bms agents can’t SSH bms-2/3, see #1335). Until applied, fall back to Mezmo + on-box ausearch for BMS investigations.

To enable full command (execve) logging on a specific host, set auditd_log_all_execve: true in its host_vars/ file — not on MongoDB primaries without sizing Loki/Mezmo rate limits first (high log volume).

---

Step 1 — Start with Mezmo (fastest, all servers)

Mezmo has all logs from all servers in one place. Start here if the event is recent (< few days).

Search patterns

# Who used sudo on any server in last 24h?
sudo

# Package install/uninstall
apt install
apt remove
dpkg -i

# SSH logins (successful)
Accepted publickey
Accepted password

# Failed SSH attempts
Failed password
Invalid user

# Password changes
passwd
chpasswd

# Docker operations
docker run
docker compose up
docker compose down

# Specific user activity (e.g. root on bms-3)
host:bms-3 root

# Time-boxed investigation (use Mezmo date filter in UI)
# Set range: e.g. 2026-06-25 14:00 — 15:30

Mezmo filters

• Filter by host field to narrow to a single server
• Filter by app field (e.g. sudo, sshd, apt) for specific programs
• Use Views → New View to save a query for recurring checks

---

Step 2 — Loki / Grafana (longer history, structured queries)

Use when: event is older than Mezmo retention, or you want correlated Docker + syslog view.

URL: https://grafana.vps-i1.infra.zintegrowana.online → Explore → Loki

LogQL patterns

# All sudo events on vps-i1
{job="syslog", host="vps-i1"} |= "sudo"
 
# SSH logins, all VPS hosts
{job="syslog"} |= "Accepted"
 
# Root command execution via auditd (after audit-daemon is deployed)
{job="syslog", host="vps-i1"} |= "type=EXECVE"
 
# Package installs
{job="syslog"} |= "apt" |= "install"
 
# Failed sudo (wrong password / not in sudoers)
{job="syslog"} |= "sudo" |= "authentication failure"
 
# Auth events on a specific server in a time window
{job="syslog", host="bms-3"} | logfmt | program="sshd"

Grafana time range

Set in top-right corner. Narrow to ±1h around the suspected event time to reduce noise.

---

Step 3 — auditd on the server (most granular, after audit-daemon deployed)

SSH to the relevant server. auditd logs every execve syscall — you can reconstruct exactly what was run.

# All admin events in the last hour
ausearch -ts recent -i | tail -200
 
# Specific user's commands (e.g. root)
ausearch -ua root -ts today -i
 
# Sudo events only
ausearch -k privilege_escalation -ts today -i
 
# Package installs
ausearch -k package_mgmt -ts today -i
 
# File modifications in /etc
ausearch -k identity -ts today -i
ausearch -k sudoers -ts today -i
ausearch -k sshd_config -ts today -i
 
# Commands run in a specific time window (format: HH:MM:SS)
ausearch -ts 14:00:00 -te 15:30:00 -i
 
# All events for a specific time range (full date)
ausearch -ts 06/25/2026 14:00:00 -te 06/25/2026 15:30:00 -i
 
# Follow live (like tail -f)
aureport -ts recent -i --summary
 
# Summary report — what happened today
aureport --summary -ts today
 
# Failed authentications today
aureport --auth --failed -ts today -i

Decode auditd output

# ausearch -i flag converts UIDs to usernames and syscall names to readable form
# Example output line:
# type=EXECVE msg=audit(1719316800.123:456): argc=3 a0="sudo" a1="apt" a2="install vim"
# → user root ran: sudo apt install vim
 
# Correlate with session (AUID = original login user, even after sudo)
ausearch -ts today -i | grep -A5 "type=USER_LOGIN"

journalctl fallback (if ausearch not available)

# Auth log (sudo, ssh, passwd)
journalctl -u ssh --since "2026-06-25 14:00" --until "2026-06-25 15:30"
 
# All sudo events
journalctl | grep sudo | grep -v "pam_unix"
 
# Last 100 authentication events
journalctl _COMM=sudo --since today
 
# Kernel audit events
journalctl _TRANSPORT=audit --since today | head -100

---

Step 4 — Shell history (last resort, not tamper-proof)

Not reliable — root can clear history, it’s not real-time, and it’s per-session. Use only to corroborate other evidence.

# Root history on the server
cat /root/.bash_history
 
# History with timestamps (if HISTTIMEFORMAT was set)
HISTTIMEFORMAT="%F %T " history
 
# Last login times
last -n 20
 
# Currently logged-in users and their activity
w
who

---

Investigation template

Copy this when starting an investigation:

## Admin Action Investigation — [DATE] [SERVER]

**Trigger:** [What raised the alarm — alert, user report, anomaly]
**Time range:** [From] → [To]
**Servers in scope:** [vps-i1 / vps-h1 / bms-1 / bms-2 / bms-3 / bms-4]

### Step 1 — Mezmo search
Query: ___
Findings:
- [ ] 

### Step 2 — Loki query
LogQL: ___
Findings:
- [ ]

### Step 3 — auditd on server
Command: ausearch -ts ___ -te ___ -i
Findings:
- [ ]

### Conclusion
- Who: ___
- What: ___
- When: ___
- Why (if known): ___
- Impact: ___

### Actions taken
- [ ]

### Prevention
- [ ]

---

Common investigation scenarios

”Did someone install a package on bms-3 last night?”

1. Mezmo: search apt install + filter host:bms-3, set time to last 24h
2. If in Loki: {job="syslog", host="bms-3"} |= "apt" |= "install"
3. On server: ausearch -k package_mgmt -ts yesterday -i

”Someone’s SSH key may have been added to authorized_keys”

1. auditd: ausearch -k identity -ts today -i | grep authorized_keys
2. Mezmo: search authorized_keys
3. Direct: stat /root/.ssh/authorized_keys + git log if the key is in a managed dotfile

”Who restarted the Docker stack at 3am?”

1. Mezmo: docker compose + filter by time
2. Loki: {job="syslog"} |= "docker" |= "compose" |= "up" in the time window
3. auditd: ausearch -k docker_mgmt -ts 03:00:00 -te 03:30:00 -i

”Who changed the Grafana admin password?”

1. Mezmo: search grafana + password in the relevant time window
2. Check /opt/p24-infra/monitoring/.env modification time: stat /opt/p24-infra/monitoring/.env
3. auditd: ausearch -k identity -ts today -i
4. git log on secrets/monitoring.env.sops: git log --oneline -- secrets/monitoring.env.sops

”An alert fired but we can’t tell which server changed”

1. Mezmo: search the alert condition string across all hosts — Mezmo aggregates everything
2. Loki: remove host filter, query across all labels
3. Check Prometheus alert annotations for server label

---

After the investigation

1. Document findings in the GitHub issue that triggered the investigation (or create one)
2. If a credential was exposed or used unexpectedly → follow credential-rotation playbook
3. If an unauthorized change was made → revert it and create a GitHub issue with label bug + security
4. If this reveals a gap in monitoring → create a GitHub issue with label enhancement