Logs and Detection

Log locations and recorded details vary by distribution, systemd, auditd, EDR, and container configuration. In writeups, distinguish places where events could be recorded from places actually checked.

If you changed the target’s state during an investigation, use the record template on this page. For privilege boundaries, follow the pages linked from the Linux overview.

Possible checks

Additional check examples:

journalctl -u <service> --no-pager
journalctl _COMM=sudo --no-pager
grep -RniE 'sudo|session opened|authentication failure' /var/log/auth.log /var/log/secure 2>/dev/null
grep -RniE '<suspicious-path>|<payload-marker>' /var/log /var/www 2>/dev/null
ausearch -x sudo
ausearch -f <path>

ausearch is an example check when auditd is present. It produces no results in environments without the relevant audit configuration.

Places worth recording as General principles:

• sudo execution often appears in /var/log/auth.log, /var/log/secure, or journald.
• For systemd services, journalctl -u <service> may show standard output, standard error, and restarts.
• If the required auditd rules are enabled, they may record chmod, chown, execve, file writes, and network connections.

What to look for

• Web access logs: uploads, ../, .pickle.gz, .pt, .php, and POST requests to an admin page.
• sudo and authentication logs: sudo -l, execution under NOPASSWD, su root, and SSH logons.
• systemd journal: service restarts, oneshot units and timers, and scripts run as root.
• auditd: process execution, file creation, chmod, chown, and network connections.
• EDR: reverse shells, /dev/tcp, python -c, bash -c, and unusual child processes.
• Unix domain sockets: connections, permission changes, and unexpected clients.
• File changes: SUID bits, new symlinks, writable plugins or checkpoints, and temporary files owned by root.
• Containers: Docker socket access, privileged containers, capabilities, and permission changes on bind mounts.

Activities that can look suspicious by mistake:

• Legitimate backup, monitoring, or training jobs may create many logs, models, or checkpoints.
• Development servers and HMR may serve many /@hmr paths or vendor assets.
• An administrative UI may call systemctl status during normal operation.
• PDF, OCR, and ML processing can normally produce high CPU usage and temporary files.

Traces likely to remain

• Installing auditd is not the same as observing the relevant event. Use ausearch and journalctl to collect chmod, execution, and connection events.
• Web access logs may retain requests to admin pages or localhost UIs, paths containing ../, and uploads. Application logs or file state may also confirm creation of model files.
• If a daemon creates a quarantine archive after detecting a keyword, its log entry and the archive record that side effect.
• A SUID root binary created in a world-writable directory remains visible as a file-permission change.

For files created through shared mounts and uploads, record the stat result even after cleanup. If packets were captured, record the executing user and network namespace too.

Record state changes

Write down how to reverse an action before carrying it out.

Target:
Change:
Original state:
Executing user:
Recovery command:
Recovery verification:
Unverified:

Defensive measures

• Configure services running as root not to read directories writable by lower-privilege users. If necessary, recheck owner, permissions, and inode immediately before use.
• Separate public Web directories from inputs to analysis workers.
• Do not accept file formats requiring unsafe deserialization from outside the trust boundary.
• Add authentication and authorization to localhost services as well.
• Restrict which users can connect to Unix domain sockets and verify peer credentials on connection.
• Scripts run through sudo should use absolute paths, limit allowed input, and control environment variables and paths read by root.
• Configure systemd units with NoNewPrivileges, PrivateTmp, ProtectSystem, ReadWritePaths, and a dedicated User.
• Minimize bind mounts and prevent users inside containers from weakening host-side permissions.
• Use auditd or EDR to monitor added SUID bits, unexpected .pth or .pt files, symlinks, /dev/tcp, python -c, sudo execution, and Unix domain socket connections.