How to Block Unwanted User Agents and Referrers in Apache, Nginx, and WordPress

Direct answer: identify the exact traffic pattern in your logs, then apply the narrowest rule at the highest practical layer. Apache can use SetEnvIfNoCase with Require for a small User-Agent deny list, or mod_rewrite when you must combine headers with IPs or rewrite decisions. Nginx is best handled with map for User-Agent classification and valid_referers with $invalid_referer for hotlink policies. WordPress can use .htaccess for targeted compatibility rules, but sustained attacks belong behind rate limiting or an edge WAF.

User-Agent and Referer are request headers supplied by the client. They help reduce repetitive nuisance traffic, but they do not prove identity. Apache warns that User-Agent matching “can be trivially circumvented,” and Nginx notes that fabricating a Referer is easy. Treat these rules as filters, not authentication.

Before you block anything

Inspect logs. Confirm the path, method, response code, User-Agent, Referer, source IP as seen by your proxy, and request rate. Look for a distinctive token such as a bot name, not a broad word like bot or Mozilla.
Define the blast radius. Decide whether the rule applies to one directory, one virtual host, a login endpoint, images, or the entire site. Start with the smallest path that solves the problem.
List required clients. Preserve search crawlers, uptime monitors, accessibility tools, payment callbacks, partner integrations, and internal automation that your application needs.
Choose a response. Use 403 for a deliberate denial. Use 429 when the behavior is excessive and rate limiting is more appropriate. Do not redirect abusive requests into application routes.
Roll out gradually. Log or stage the rule, review false positives, then enforce it. Recheck after a CDN or reverse-proxy change because the origin may receive different IPs or headers.

Header filters make an early allow or deny decision before the request reaches the application.

Apache: block User-Agents with SetEnvIfNoCase

For a small deny list and a known protected path, Apache’s SetEnvIfNoCase plus authorization rules is easy to audit. The pattern below follows Apache’s documented approach.

SetEnvIfNoCase User-Agent "^NameOfBadRobot" goaway

<Location "/secret/files">
    <RequireAll>
        Require all granted
        Require not env goaway
    </RequireAll>
</Location>

Replace the anchored token with the exact string found in your logs. Keep the expression narrow. If several names are related, use separate SetEnvIfNoCase lines or a carefully reviewed expression. The SetEnvIfNoCase documentation explains how request headers set environment variables, while Apache access control guidance covers authorization composition.

Apache with mod_rewrite and an IP condition

Use mod_rewrite when a header match must be combined with an address range or when the result needs a rewrite or redirect. This example denies a named robot only from a narrow address range:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "^NameOfBadRobot" [NC]
RewriteCond %{REMOTE_ADDR} "=123\.45\.67\.[8-9]"
RewriteRule "^/secret/files/" "-" [F]

[F] returns a forbidden response. Keep every condition restrictive; a broad User-Agent condition can block legitimate traffic. See Apache’s mod_rewrite documentation for context and rule processing.

Blocking hotlinking by Referer

A simple Referer policy can be expressed with SetEnvIf and Require, or with rewrite conditions when you need path-specific behavior. A rewrite example for image files is:

RewriteEngine On
RewriteCond %{REQUEST_URI} \.(?:png|jpe?g|gif|webp)$ [NC]
RewriteCond %{HTTP_REFERER} !^https://(www\.)?example\.com/ [NC]
RewriteCond %{HTTP_REFERER} !^$ [NC]
RewriteRule "^" "-" [F]

This allows direct requests with an empty Referer while denying requests that identify another site. Decide deliberately whether missing Referer values should be accepted; privacy tools and browsers can omit the header.

.htaccess context gotcha

In per-directory .htaccess context, Apache removes the directory prefix before matching. A rule copied from a virtual-host configuration may therefore need its leading path removed. Put custom rules outside the block between WordPress rewrite markers so WordPress updates do not replace them.

Nginx: maintainable User-Agent and Referer rules

Classify User-Agents with map

Keep the list in a map so it is centralized and reviewable. Place the map in the http block, then use the resulting variable in the relevant server or location block:

http {
    map $http_user_agent $bad_user_agent {
        default 0;
        ~*^(badbot|scraper-name) 1;
    }

    server {
        location /secret/files/ {
            if ($bad_user_agent) { return 403; }
        }
    }
}

Anchor expressions where practical and avoid matching generic browser fragments. Validate before reload:

sudo nginx -t
sudo systemctl reload nginx

The Nginx map module documents variable classification and the Referer module documents referer validation.

Control hotlinking with valid_referers

Nginx provides an explicit allow list and the $invalid_referer variable:

server {
    location ~* \.(png|jpe?g|gif|webp)$ {
        valid_referers none blocked server_names *.example.com example.*;
        if ($invalid_referer) { return 403; }
    }
}

none permits a missing Referer. blocked permits a header altered by a proxy or firewall. Remove either token if your policy requires stricter behavior, but expect more false positives. A forged Referer can still bypass this control, so combine it with signed URLs, authentication, rate limits, or edge rules when the content is valuable.

WordPress: .htaccess, plugins, and the edge

WordPress can express targeted header rules in Apache’s .htaccess. The Codex describes checking HTTP_REFERER and HTTP_USER_AGENT for direct spam-bot requests. Use that technique for a narrow compatibility fix and keep your custom directives outside the managed rewrite block.

# Place before or after the WordPress marker block, not inside it.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "^NameOfBadRobot" [NC,OR]
RewriteCond %{HTTP_REFERER} "spam-example\.tld" [NC]
RewriteRule ^ - [F]

For current WordPress operations, defense in depth is safer than a long plugin deny list. WordPress administration guidance recommends CAPTCHA or Turnstile on login, restricting or disabling XML-RPC when it is unnecessary, rate limiting exposed endpoints, and using an edge or host-provided WAF such as Cloudflare or Sucuri so abusive requests are stopped before they consume origin resources. See the WordPress hardening handbook.

When a plugin helps

A security plugin can provide log review, IP reputation, login challenges, and rate limits without editing server files. Check where it enforces a rule: a plugin usually runs after the request reaches PHP, while a CDN or WAF can reject it before origin bandwidth and CPU are used. Avoid installing overlapping plugins that each rewrite headers or challenge the same endpoint.

Choosing the right enforcement layer

Layer	Good for	Limitations
Apache or Nginx	Small, stable lists and path-specific rules	Consumes an origin connection; headers and IPs can be spoofed
WordPress/plugin	Application-aware login and XML-RPC controls	PHP and database work may happen before rejection
Edge WAF/CDN	Distributed abuse, rate limits, managed reputation	Policy complexity and provider cost; proxy headers must be understood
Firewall/network	Persistent attacks from known networks	Large blast radius and maintenance burden

Use headers to reduce repetitive nuisance traffic. Escalate spoofed, distributed, or sustained abuse to rate limiting, IP reputation, authentication, firewall controls, or an edge WAF.

Safe rollout checklist

Save the current configuration and know how to roll back.
Test a matching request and a known-good request.
Review access logs for false positives after deployment.
Monitor 403 and 429 rates by path and client type.
Confirm the real client IP setting after proxy changes.
Document why each token, domain, IP range, or exception exists.
Revisit the list; abandoned bot names become dead configuration.

Troubleshooting common failures

Everything returns 403

Cause: an expression is too broad, an allow list is missing, or an OR condition groups incorrectly. Fix: test each condition independently, anchor the User-Agent token, and temporarily log the value before enforcing.

The rule never matches

Cause: the proxy rewrites headers, the expression is case-sensitive, or an Apache rule includes a directory prefix that is stripped in .htaccess. Fix: inspect the origin log, use [NC] or a case-insensitive Nginx expression, and adjust the path for per-directory context.

Legitimate crawlers are blocked

Cause: relying on a claimed User-Agent rather than verification. Fix: allow required integrations explicitly and verify important crawlers with provider-documented reverse and forward DNS checks or an edge reputation product.

Hotlink protection breaks images in privacy-focused browsers

Cause: those clients omit or trim Referer. Fix: permit none, use signed image URLs, or protect only high-value assets.

Nginx refuses to reload

Cause: a map is outside the http block, a regex is malformed, or an if is placed in an invalid context. Fix: run nginx -t, correct the reported file and line, then reload.

WordPress rules disappear

Cause: they were placed inside the generated rewrite marker block. Fix: move custom rules outside that block and keep a deployment-managed copy.

Performance, reliability, and cost

Header comparisons are cheap, but a rule that runs after PHP, a database lookup, or a remote reputation query costs more than a decision at the edge. Keep regular expressions short, avoid dozens of overlapping rewrite rules, and prefer a single Nginx map or a small Apache environment-variable list. Logging every denied request can become expensive during a flood; sample or aggregate logs while retaining enough data to audit decisions.

Reliability depends on exceptions. A missing Referer is normal for direct navigation, mobile privacy features, and some corporate proxies. A claimed User-Agent is not proof of a crawler. Return deterministic 403 or 429 responses, monitor changes in status-code volume, and rehearse rollback before a major campaign or migration.

Origin rules have no separate service charge, but they still consume connections, bandwidth, CPU, and operations time. Edge WAF pricing and capabilities vary by provider; compare enforcement location, rate-limit granularity, logging, and false-positive controls rather than choosing only by feature count.

Or skip the browser setup

If your goal is to capture pages for QA, documentation, or monitoring after filtering unwanted traffic, ScreenshotNeo provides a website screenshot API and MCP server. Its capture flow accepts consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before the shot; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result. An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.

A clean capture removes common overlays before the screenshot is produced.

One request returns PNG, JPEG, WebP, or PDF:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python:

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

See the ScreenshotNeo API documentation for request options. You can capture full pages with lazy images, select an element by CSS, set dark mode and any viewport, use device presets and retina scale, create PDFs with paper size, margins, landscape, and page ranges, render HTML/CSS, run custom CSS or JavaScript, click before capture, hide selectors, wait for a selector, delay, or network idle, block ads, trackers, requests, or resource types, send headers, cookies, user agents, and Authorization, set timezone and geolocation, use transparent backgrounds, resize images, cache with your chosen TTL, create signed links, run asynchronous jobs with signed webhooks, capture up to 100 URLs per bulk call, and query usage.

Pricing: 1,000 screenshots per month are free with no card. Paid plans start at $5 for 3,000 shots; yearly billing gives two months free, and every feature is available on every plan. Create a free ScreenshotNeo account.

FAQ

Can a User-Agent rule stop a determined scraper?

No. The client can change the header. Combine header filtering with rate limits, authentication, IP reputation, or an edge WAF.

Should I block an empty Referer?

Usually not for a public site. Direct visits, privacy tools, and some proxies omit it. Block missing values only when your application has a clear, tested requirement.

Is 403 or 429 better?

Use 403 for a policy denial and 429 for excessive request frequency. Make the choice consistent with monitoring and client expectations.

Where should I store a growing deny list?

Move it to a centralized Nginx map, Apache-managed configuration, edge rule set, or WAF policy with review and change history. Avoid an unmaintainable collection of ad hoc WordPress snippets.