Caddy + GeoIP + Fail2Ban (Pushover) — Setup Notes

by
Tags: caddy fail2ban pushover
Categories: webserver linux firewall

This document describes:

  • Building a custom Caddy container image with the GeoIP plugin (so Caddy can enrich access logs with country code/name).
  • Configuring Caddy JSON access logs to include GeoIP fields.
  • Setting up Fail2Ban to parse Caddy logs and send Pushover notifications with GeoIP info via mmdblookup.
  • Optional “SOC dashboard” style fields (severity, jail type, ban time, until).

Assumptions: Debian-based host, Docker + Compose, Caddy running in a container, Fail2Ban running on the host.

1) Why a custom Caddy build?

Stock Caddy does not include third‑party modules. If your Caddyfile contains a directive from a plugin (e.g., geoip), Caddy will fail to start with:

unrecognized directive: geoip

So we build Caddy with the GeoIP module compiled in.

2) Build a custom Caddy image with GeoIP support

We use xcaddy to compile Caddy with the plugin:

  • github.com/IT-Hock/caddy-geoip
     

Example Dockerfile

# ---- builder stage ----
FROM caddy:builder AS builder
RUN xcaddy build --with github.com/IT-Hock/caddy-geoip

# ---- runtime stage ----
FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
 

Example docker-compose.yam file

services:
  webserver:
    image: nginx:alpine
    container_name: webserver
    command: >
      sh -c "printf 'Hello, world!\n' > /usr/share/nginx/html/index.html
      && nginx -g 'daemon off;'"
    expose:
      - "80"
    restart: unless-stopped

  caddy:
    build:
      context: .
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - caddy_data:/data
      - caddy_config:/config
      - /var/log/caddy:/var/log/caddy
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./GeoLite2-Country.mmdb:/data/GeoLite2-Country.mmdb:ro

volumes:
  caddy_data:
  caddy_config:
 

Build and run with Docker Compose

docker build .
docker compose up -d
 

Verify modules are present

Inside the container:

docker exec -it caddy caddy list-modules | grep -i geo

Expected (or similar):

  • http.handlers.geoip
  • caddy.logging.encoders.filter.geoip

If you see those, the plugin is compiled in.

3) Configure Caddy to enrich access logs with GeoIP

Once the plugin exists, Caddy can add GeoIP information into access logs (JSON logs).

 

Add GeoIP fields to JSON access log

In your log format / encoder, ensure you output these fields:

  • geoip_country_code
  • geoip_country_name

Example: (conceptual; actual stanza depends on your Caddyfile layout)

log {
  output file /var/log/caddy/my-caddy-access.log
  format json
}
 

Caddyfile (final)

This is the Caddyfile we ended up with (GeoIP runs first, adds country code/name into the JSON access log, drops common scanner noise early, and reverse-proxies the rest to our webserver):

{
  order geoip first
}

hostname.example.com {

  route {

    geoip * /data/GeoLite2-Country.mmdb

    # Drop common web-scanner noise early (adjust as you like)
    @scanners {
      path_regexp bad ^/(?:\.env|.*\.env|wp-|wp/|wordpress|actuator|phpmyadmin|\.vscode|cgi-bin|vendor/|src/|config/|\.git|\.DS_Store)
    }

    respond @scanners 404

    # OPTIONAL: add GeoIP fields into the access log entries
    # (these placeholders are provided by the GeoIP plugin)
    log_append geoip_country_code {geoip_country_code}
    log_append geoip_country_name {geoip_country_name}

    # Everything else goes to webserver
    reverse_proxy http://webserver:80 {
      header_up Host {host}
      header_up X-Forwarded-Proto {scheme}
      header_up X-Forwarded-Host {host}
      header_up X-Forwarded-For {remote}
    }
  }

  log {
    output file /var/log/caddy/my-caddy-access.log
    format json
  }
}

Notes

  • order geoip first ensures the GeoIP handler runs early enough that placeholders like {geoip_country_code} / {geoip_country_name} are available for later handlers (like log_append).
  • The @scanners matcher + respond 404 is purely “noise reduction” so common mass-scans don’t hit our Webserver at all.
  • log_append adds extra top-level fields into each JSON access log entry, which is why Fail2Ban can later enrich notifications without doing GeoIP itself (but we chose mmdblookup in the notification path instead).
  • The header_up lines are optional: Caddy’s reverse_proxy already forwards most of these by default; keeping Host is sometimes useful when upstream apps care about it.
 

Validate Caddyfile

You can validate without starting Caddy:

docker exec -it caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile

 
Then ensure the GeoIP handler/filter is enabled in the relevant request path so those fields get added.

 

Confirm GeoIP fields appear

sudo tail -n 1 /var/log/caddy/my-caddy-access.log | jq .

Example output includes:

"geoip_country_name": "Denmark",
"geoip_country_code": "DK"

4) Fail2Ban setup for Caddy logs

Fail2Ban watches the Caddy access log and bans abusive IPs according to your jail/filter rules.

Jail (example idea)

Your jail points at:

  • /var/log/caddy/my-caddy-access.log

and uses your chosen filter and ban time settings.

5) Pushover notifications + GeoIP via mmdblookup (Option A)

Instead of scraping GeoIP from Caddy logs, we can resolve GeoIP directly from a local MaxMind DB using mmdblookup.

Install mmdblookup

On Debian:

sudo apt-get update
sudo apt-get install -y mmdb-bin
 

Place the MaxMind DB

Example path used below:

  • /data/GeoLite2-Country.mmdb

(Adjust to your real location, and ensure Fail2Ban can read it.)

6) Pushover sender script

File: /usr/local/bin/fail2ban-pushover

Sends a message to Pushover using env vars.

#!/usr/bin/env bash
set -euo pipefail

TITLE="${1:-Fail2Ban}"
MESSAGE="${2:-}"
PRIORITY="${3:-0}"

: "${PUSHOVER_USER_KEY:?missing PUSHOVER_USER_KEY}"
: "${PUSHOVER_APP_TOKEN:?missing PUSHOVER_APP_TOKEN}"

DEVICE="${PUSHOVER_DEVICE:-}"
SOUND="${PUSHOVER_SOUND:-}"
URL="${PUSHOVER_URL:-}"
URL_TITLE="${PUSHOVER_URL_TITLE:-}"

args=(
  -fsS
  --retry 2
  --retry-delay 1
  -X POST
  -d "token=${PUSHOVER_APP_TOKEN}"
  -d "user=${PUSHOVER_USER_KEY}"
  -d "title=${TITLE}"
  --data-urlencode "message=${MESSAGE}"
  -d "priority=${PRIORITY}"
  https://api.pushover.net/1/messages.json
)

[[ -n "$DEVICE" ]] && args+=( -d "device=${DEVICE}" )
[[ -n "$SOUND"  ]] && args+=( -d "sound=${SOUND}" )
[[ -n "$URL"    ]] && args+=( -d "url=${URL}" )
[[ -n "$URL_TITLE" ]] && args+=( -d "url_title=${URL_TITLE}" )

timeout 10s curl --connect-timeout 3 --max-time 8 "${args[@]}" >/dev/null || true

 
Make executable:

sudo chmod +x /usr/local/bin/fail2ban-pushover
 

Provide secrets to Fail2Ban environment

Fail2Ban runs as a service; it may not inherit your shell env. Provide secrets in a root‑only env file and source it in the script or service.

One pattern:

  • /etc/fail2ban/pushover.env (permissions 600)
PUSHOVER_USER_KEY="user-token"
PUSHOVER_APP_TOKEN="app-token"

 
Then in /usr/local/bin/fail2ban-pushover, add near the top:

# Optional: source env for systemd services
if [[ -r /etc/fail2ban/pushover.env ]]; then
  # shellcheck disable=SC1091
  source /etc/fail2ban/pushover.env
fi

7) SOC-style wrapper: Fail2Ban -> Pushover with GeoIP, severity, jail type

File: /usr/local/bin/fail2ban-pushover-soc

This wrapper:

  • looks up GeoIP via mmdblookup
  • formats a clean message
  • optionally adds “SOC dashboard” fields:
    • Severity emoji based on attempts
    • Jail type (auth vs scan vs other)
    • Ban time + until (if passed by Fail2Ban action)

Wrapper script (example)

#!/usr/bin/env bash
set -euo pipefail

# Optional debugging:
# exec 2>>/var/log/fail2ban-pushover-soc.err
# set -x

mmdb_str() {
  sed -n 's/^[[:space:]]*"\([^"]*\)".*/\1/p' | head -n1
}

JAIL="${1:?jail}"
IP="${2:?ip}"
FAILURES_RAW="${3:-}"
LOGPATH="${4:-}"
# ignore provided hostname (can be "<hostname>"); compute reliable one:
HOST="$(hostname -f 2>/dev/null || hostname)"
PRIO="${6:-0}"

# Optional extra args if you decide to pass them from the action:
EVENT="${7:-ban}"           # ban | unban
BAN_EPOCH="${8:-}"
TIMEOUT_S="${9:-}"

# Normalize failures to integer
FAILURES=0
if [[ "${FAILURES_RAW}" =~ ^[0-9]+$ ]]; then
  FAILURES="${FAILURES_RAW}"
fi

# --- GeoIP from MaxMind DB (mmdblookup) ---
DB="/data/GeoLite2-Country.mmdb"
GEO=""
if command -v mmdblookup >/dev/null 2>&1 && [[ -r "${DB}" ]]; then
  CC="$(mmdblookup --file "${DB}" --ip "${IP}" country iso_code 2>/dev/null | mmdb_str || true)"
  CN="$(mmdblookup --file "${DB}" --ip "${IP}" country names en 2>/dev/null | mmdb_str || true)"
  GEO="${CC}${CC:+ - }${CN}"
fi

# Jail type (simple mapping; extend as you like)
JAIL_TYPE="other"
case "${JAIL,,}" in
  *scan*) JAIL_TYPE="scan" ;;
  *auth*|*login*) JAIL_TYPE="auth" ;;
esac

# Severity based on attempts (tune thresholds)
SEV="🟢"
if (( FAILURES >= 25 )); then SEV="🔴"
elif (( FAILURES >= 10 )); then SEV="🟠"
elif (( FAILURES >= 1 )); then SEV="🟡"
fi

# Cosmetic ban time / until (requires passing banEpoch/timeout from the action)
BANTIME_LINE=""
UNTIL_LINE=""
if [[ "${EVENT}" == "ban" && "${BAN_EPOCH}" =~ ^[0-9]+$ && "${TIMEOUT_S}" =~ ^[0-9]+$ ]]; then
  if (( TIMEOUT_S >= 86400 )); then
    days=$(( TIMEOUT_S / 86400 ))
    BANTIME_LINE="Ban time: ${days}d"
  elif (( TIMEOUT_S >= 3600 )); then
    hrs=$(( TIMEOUT_S / 3600 ))
    BANTIME_LINE="Ban time: ${hrs}h"
  elif (( TIMEOUT_S >= 60 )); then
    mins=$(( TIMEOUT_S / 60 ))
    BANTIME_LINE="Ban time: ${mins}m"
  else
    BANTIME_LINE="Ban time: ${TIMEOUT_S}s"
  fi

  until_epoch=$(( BAN_EPOCH + TIMEOUT_S ))
  UNTIL_LINE="Until: $(date -d "@${until_epoch}" '+%Y-%m-%d %H:%M:%S %Z' 2>/dev/null || true)"
fi

# Title + message
if [[ "${EVENT}" == "unban" ]]; then
  TITLE="✅ Fail2Ban — Unbanned (${JAIL})"
  ACTION_LINE="Action: IP unbanned"
else
  TITLE="${SEV} Fail2Ban — Banned (${JAIL_TYPE})"
  ACTION_LINE="Action: IP banned"
fi

MSG=$(
  printf "%s\n" \
    "Host: ${HOST}" \
    "Service: ${JAIL}" \
    "${ACTION_LINE}" \
    "Source IP: ${IP}" \
    "${GEO:+GeoIP: ${GEO}}" \
    "Attempts: ${FAILURES}" \
    "${BANTIME_LINE}" \
    "${UNTIL_LINE}" \
    "${LOGPATH:+Log file: ${LOGPATH}}"
)

# IMPORTANT: never block Fail2Ban; swallow failures
/usr/local/bin/fail2ban-pushover "${TITLE}" "${MSG}" "${PRIO}" || true
exit 0

 
Make executable:

sudo chmod +x /usr/local/bin/fail2ban-pushover-soc

8) Fail2Ban action definition (pushover)

File: /etc/fail2ban/action.d/pushover.conf

Basic version (6 args):

[Definition]
actionstart =
actionstop =

actionban  = /usr/local/bin/fail2ban-pushover-soc "<name>" "<ip>" "<failures>" "<logpath>" "<hostname>" "0"
actionunban= /usr/local/bin/fail2ban-pushover-soc "<name>" "<ip>" "" "<logpath>" "<hostname>" "-1"
 

Optional: pass banEpoch/timeout for cosmetic ban time + until

If your Fail2Ban version supports these action properties (common), you can pass:

  • <banEpoch>
  • <timeout>
[Definition]
actionstart =
actionstop =

actionban  = /usr/local/bin/fail2ban-pushover-soc "<name>" "<ip>" "<failures>" "<logpath>" "<hostname>" "0"  "ban"   "<banEpoch>" "<timeout>"
actionunban= /usr/local/bin/fail2ban-pushover-soc "<name>" "<ip>" "<failures>" "<logpath>" "<hostname>" "-1" "unban" "<banEpoch>" "<timeout>"

Reload Fail2Ban after changes:

sudo fail2ban-client reload

9) Testing & troubleshooting

Trigger a manual ban

sudo fail2ban-client set caddy banip 1.2.3.4
 

Check Fail2Ban log

sudo tail -n 100 /var/log/fail2ban.log
 

If your action times out

Fail2Ban kills actions that run too long (commonly 60s). Causes include:

  • missing env vars causing scripts to block/hang
  • network issues to Pushover API
  • script attempting slow external commands

Fix by:

  • making Pushover sender robust (curl --retry, -fsS)
  • ensuring secrets are available to the service
  • ensuring the wrapper never blocks Fail2Ban (|| true + exit 0)

Why did “Host: ” show up?

Fail2Ban sometimes passes the literal placeholder text "<hostname>" depending on config/version.
The wrapper script computes a reliable hostname using:

HOST="$(hostname -f 2>/dev/null || hostname)"

So the “Host:” line shows a real FQDN like hostname.example.com.

10) Summary

  • Caddy: custom-built with GeoIP module using xcaddy, verified via caddy list-modules.
  • Logs: JSON access log enriched with geoip_country_code and geoip_country_name.
  • Fail2Ban: watches Caddy access log, bans offenders.
  • Notifications: a custom wrapper uses mmdblookup to generate GeoIP and sends clean messages to Pushover.
  • SOC polish: severity, jail type, ban time/until can be included without breaking Fail2Ban.