The Log Never Lies
Every request to the server is a decision. Most are legitimate — a visitor loading a page, a crawler indexing content, a CDN fetching assets. But a significant fraction are not. Today's GoAccess log for July 14 told a story: 8,517 valid requests, 392 unique visitors, 42.6 MB of bandwidth. And one IP address responsible for 60.3% of all valid traffic.
The Numbers
After three analysis passes (Jul 8, Jul 11, Jul 14), the canonical threat database now tracks 137 bad actors across 94 original entries and 43 new ones from today's log. The ip_topology.csv file classifies 144 IPs and domains as enemy. Here's what today's log revealed:
- 5,137 hits from
185.177.72.23— a single IP generating 60% of all valid traffic. Same /24 subnet as185.177.72.24, which was already tracked from July 8 with 307 hits. The scanning operation escalated dramatically. - 421 hits across 3 IPs in the
103.4.25xrange —103.4.251.169(144),103.4.250.194(142),103.4.250.175(135). Nearly identical hit counts. Coordinated scanning from the same /23 as103.4.251.67, which was already tracked with 987 hits from July 11. - 8 new IPs with 100+ hits each —
104.164.126.93,103.196.9.241,171.22.217.67,154.28.229.76,93.123.109.52,163.7.11.71,157.143.3.35, and a known returner65.21.124.77(down from 705 hits on July 11). - 5,756 4xx responses (67.6% of all status codes) — the server is spending most of its time telling scanners to go away.
The Subnet Pattern
The most important pattern in today's log is subnet escalation. Individual IP blocking is a whack-a-mole game. When you block 185.177.72.24, the scanner moves to 185.177.72.23. When you block 103.4.251.67, three new IPs appear in 103.4.250.x. The solution is subnet-level blocking: 185.177.72.0/24 and 103.4.250.0/23 are now in the threat database as critical subnet blocks.
This is the evolution of the threat model. The first analysis pass (Jul 8) tracked individual IPs. The second pass (Jul 11) identified coordinated subnets with 8-10 IPs sharing identical hit counts. The third pass (today) confirmed the pattern: when you block individual IPs in a subnet, the operation shifts to other IPs in the same range. Block the range.
What They're Looking For
The 404 patterns tell the story of what these scanners want:
- Secrets and environment files (574 hits) —
/.env,/.git/config,/.ssh/id_rsa,/.aws/credentials. The classic credential harvesting playbook. - Configuration files (275 hits) —
/wp-config.php,/config/database.yml,/application.properties. Looking for database credentials and API keys left by careless deployments. - API and development endpoints (182 hits) —
/api/v1/,/debug/pprof,/actuator/health,/.well-known/openid-configuration. Probing for exposed debug interfaces and misconfigured API gateways. - Git and Docker (142 hits) —
/.git/HEAD,/docker-compose.yml,/Dockerfile. Trying to clone the repository or find container configuration that reveals internal architecture. - Admin panels (40 hits) —
/admin,/phpmyadmin,/manager/html. The low-volume but high-value probes — if any of these return 200, the scanner has a foothold.
None of these exist on the server. Every probe returns 404 or 403. But the scanners don't care — they're automated, they're cheap to run, and they only need one hit across a thousand targets to be worthwhile.
Defensive Domain Registration
Five referrer domains appeared in today's log: catgameresearch.shop, catgameresearch.store, catgameresearch.info, catgameresearch.com, catgameresearch.us. These are our own defensive domain registrations — purchased to prevent typosquatters from grabbing lookalike TLDs. They're classified as friend in the topology database. The referrer traffic is expected: DNS records point them at the server, and crawlers following those domains generate log entries.
The Defense Stack
The server's defense-in-depth architecture now includes:
- UFW firewall — default deny incoming, 200+ IPs blocked, SSH rate-limited. Subnet blocks for
185.177.72.0/24and103.4.250.0/23pending deployment. - Dynamic firewall — a script that tails Apache logs in real-time, scores IPs by 404/403/scan patterns, and automatically blocks high-score offenders. This is how the 94 original bad actors got blocked — not manually, but by an automated system that's faster than any human analyst.
- GET-only method enforcement — mod_rewrite blocks HEAD, POST, PUT, DELETE, PROPFIND, PATCH, TRACE, and CONNECT across all subdomains. Only GET requests are allowed. This eliminates entire classes of attacks that rely on POST bodies or PUT file uploads.
- ModSecurity query string filtering — SQL injection patterns, path traversal sequences, and known attack signatures are blocked at the WAF level before reaching the application.
- GoAccess log analysis — batch-mode every 5 minutes, 403/404 filtered, crawler policy tuned to show legitimate crawlers while filtering exploit paths. The JSON stats log is archived for historical analysis.
- Crypt keeper — all server credentials are stored in an HMAC-validated encrypted vault with mlock'd memory and Unix socket IPC. No passwords in plaintext files, no credentials in environment variables visible to
/proc.
The Workflow
The analysis process itself has been refined across three sessions:
- Download the GoAccess JSON stats log from the server.
- Parse with Python — extract all hosts with hits ≥ 10, cross-reference against existing
bad_actors.csvandip_topology.csvto identify new threats. - Classify by threat score: Critical (100, massive volume or subnet), High (70-90, 100+ hits), Medium (40, 20-99 hits), Low (20, 10-19 hits).
- Append new entries to canonical CSV files with run ID and timestamp.
- Archive the GoAccess JSON snapshot to
docs/sec/data/goaccess_YYYYMMDD.jsonfor historical reference. - Deploy UFW blocks for critical and high-threat IPs/subnets via
./scripts/network.sh sudo www1. - Update GoAccess exclude list to prevent future logs from being dominated by known scanners.
The whole process takes about 30 minutes. The last step — deploying UFW blocks — requires a crypt session for sudo access, which is a deliberate friction point. Automated blocking is for the dynamic firewall. Manual blocking is for the analyst who looked at the data and made a judgment call.
What 137 Bad Actors Looks Like
The threat database is a historical record. The oldest entries are from July 8 — individual IPs with 4-9 hits that barely registered. The July 11 analysis added 27 entries including two coordinated /24 subnets and a Tencent Cloud /8 block. Today adds 43 more, including two subnet blocks and five typosquatting domains.
The growth pattern is linear, not exponential. The dynamic firewall is working — blocked IPs don't appear in subsequent logs. But new scanners appear every day. The internet is hostile, and a public-facing server is a target by default. The question isn't whether you'll be scanned — you will — but whether you'll notice, and whether you'll have a system in place to respond.
The GoAccess log is the server's heartbeat. Every request is a pulse. Most are healthy. Some are not. The job of the security analyst is to read the heartbeat and tell the difference — before the arrhythmia becomes a heart attack.
Next Steps
- Deploy UFW blocks for
185.177.72.0/24,103.4.250.0/23, and the 7 high-threat IPs (100+ hits each). - Add subnet exclusions to GoAccess config so future logs aren't dominated by these scanners.
- Monitor for subnet rotation — if
185.177.72.xstops working for the scanner, watch for185.177.73.xor adjacent ranges. - Automate the analysis pipeline — the current manual process works but should be scripted for daily runs with alerting on critical threats.
The logs will keep coming. The scanners will keep scanning. The database will keep growing. And the server will keep saying 404, one request at a time.
