When I think about defense‑in‑depth I keep coming back to the idea that if you can’t see what’s happening you can’t stop it. Logging is the eyes; alerting is the alarm. Put those together and you get a layer that shines a light on even the sneakiest moves.

What good logging actually looks like

A log file that only contains a handful of generic messages is like a blurry security camera – you see something, but not enough to know if it’s a cat or a burglar. A proper log should record the what, when, where and who of every important event. In Linux the go‑to daemon is rsyslog. The default config lives in /etc/rsyslog.conf and looks something like this:

# /etc/rsyslog.conf
module(load="imuxsock")    # provides support for local system logging
module(load="imklog")      # reads kernel messages

# Store auth messages separately for easier review
auth,authpriv.*    /var/log/auth.log

# Preserve all other messages in a central file
*.*    -/var/log/syslog

Notice the dash before the file name for syslog. That tells rsyslog to use asynchronous writing – it reduces the chance that a sudden burst of logs will stall the system.

In my view the next step is to rotate logs so they don’t fill the disk. logrotate handles that. A minimal /etc/logrotate.d/syslog could be:

/var/log/syslog /var/log/auth.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    create 0640 syslog adm
}

Four weeks of history is usually enough to spot trends without hoarding gigabytes. If you need longer for compliance you can bump the number.

Turning raw logs into actionable alerts

Logs are great, but you don’t want to stare at them all day. That’s where a tool like Fail2Ban shines. It watches a log, matches a regex, and blocks the offending IP with a firewall rule. A typical /etc/fail2ban/jail.local fragment for SSH brute‑force attempts looks like this:

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600

What happens is: Fail2Ban reads auth.log, sees five failed password lines from the same address, then runs iptables -I INPUT -s $IP -j DROP. The rule lives for an hour and then disappears. It’s simple, it’s effective, and it gives you an early warning that someone is trying to guess passwords.

For more complex environments I sometimes layer a log‑aggregation service like the ELK stack (Elasticsearch, Logstash, Kibana). That lets you query across machines and set up visual dashboards. A tiny Logstash config that pulls syslog from local files could be:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:host} %{DATA:program}: %{GREEDYDATA:msg}" }
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

The grok pattern pulls out timestamp, host and program fields so you can slice and dice later. Once the data is in Elasticsearch a simple Kibana query for repeated 404 responses from the same IP can flag a scanning tool.

How attackers hide in noisy logs

Knowing what to look for is half the battle. Attackers often spray low‑volume requests to stay below thresholds. Think of a slow‑loris style HTTP flood that sends just enough packets to keep a connection alive without tripping a rate‑limit. If you only alert on >100 requests per minute you’ll miss that. That’s why I think you need a baseline of normal traffic and watch for deviations.

One technique is to enable auditd for privileged commands. The audit daemon records every execve call that runs as root. A snippet from /etc/audit/rules.d/audit.rules could be:

-a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k privileged_cmd

When a non‑root user escalates, you’ll see an entry like:

type=EXECVE msg=audit(1625077643.123:456): argc=2 a0="/usr/bin/sudo" a1="/bin/bash"

If you pair that with a watch on /var/log/audit/audit.log you can trigger fail2ban or a custom script that sends an email. That catches privilege escalation attempts before the attacker can move laterally.

Putting alerts in the right place

Not all alerts deserve a pager. For low‑severity noise I prefer to ship them to a Slack channel or a Teams webhook. A tiny script that reads stdin and posts to Slack might be:

#!/usr/bin/env python3
import sys, json, urllib.request
url = "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXX"
payload = {"text": sys.stdin.read()}
req = urllib.request.Request(url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
urllib.request.urlopen(req)

Just pipe the log line into it and you get a quick heads‑up. For critical alerts – say a sudden spike in SSH failures or a detection of a known malicious hash – I route to email with high priority header:

Subject: [ALERT] SSH brute force detected
X-Priority: 1

An IP 203.0.113.45 has failed to login 15 times in the last 5 minutes.

The idea is to match the channel to the severity. If you drown in noise you start ignoring everything, which defeats the purpose.

Testing your logging and alerting chain

After you set everything up you need to prove it works. A quick way is to trigger a fake failure. For SSH you can run:

ssh -o PreferredAuthentications=keyboard-interactive -o PubkeyAuthentication=no user@localhost

Enter a wrong password a few times. Check /var/log/auth.log for the failed attempts and confirm fail2ban added the IP. Then look at your Slack channel or email. If the alert didn’t fire, double‑check the regex in the fail2ban filter or the Fail2Ban status with fail2ban-client status sshd.

Another test is to generate a custom audit event. Run:

sudo -u nobody /bin/true

Even though nothing serious happened, auditd will log the execve and you can see if your rule caught it.

Keeping the system healthy

All this logging does add overhead. Disk usage can blow up if you keep every debug line. That’s why rotating logs and setting size limits matters. In rsyslog you can add a line like:

$MaxMessageSize 2k

It caps each message at 2 KB, which stops a single noisy process from filling the log in seconds.

Also, regular review of the alert thresholds is a habit I try to keep. Attackers change tactics, so your baselines need to evolve. A monthly check of the top ten most common alerts in your monitoring dashboard helps you prune false positives and tighten the rules that actually catch abuse.

Final thoughts

Logging and alerting are not a single tool you install and forget. They are a feedback loop: collect data, filter for the weird, raise a signal, verify the signal, adjust the filter. If you treat that loop as a living part of your security stack you get a defensive layer that catches things before they become a breach. The effort is modest – a few config files, a couple of cron jobs, and a Slack webhook – but the payoff is seeing an intrusion attempt and stopping it while it’s still a handful of packets.

Photo by panumas nikhomkhai on Pexels