How to monitor a website redesign across hundreds of URLs

Monitor a redesign at the URL level: map every old URL to its intended new destination, test every redirect before and after launch, and track indexing, crawl activity, and user traffic as separate signals. Search Console shows Google search and indexing behavior; server logs show requests and responses; analytics shows measured user activity. No one source answers all three questions.

Keep an inventory that records each old URL, its disposition, destination, redirect result, new-page status, indexability, and follow-up evidence. Compare the same inventory repeatedly after launch. Google says Googlebot must visit every old and new URL at least once for it to consider a move complete, and crawl frequency has no fixed schedule. Larger sites can take longer, so do not use a universal recovery deadline.

1. Build the URL inventory and migration map

Export the existing URL set from your CMS, sitemap, analytics, server logs, and a crawl of the live site. Combine and deduplicate those sources: each can reveal URLs that another misses, such as pages with traffic but no internal links or URLs absent from the current sitemap.

For each URL, assign one disposition:

Stay: The URL remains unchanged and should continue to serve the same page.
Move: Map it to the equivalent new URL.
Merge: Map it to the most relevant surviving page, if there is a genuine replacement.
Remove: Decide deliberately whether it should return a not-found or gone response, or whether a relevant replacement exists. Do not redirect every removed page to the homepage.

A migration map should have at least these columns:

Field	Purpose
old_url	Source URL to check and redirect
expected_action	Redirect, remain, or intentionally removed
expected_destination	Final relevant new URL, when applicable
observed_status	HTTP response observed during a check
observed_destination	Final URL reached after redirects
new_status	Whether the destination loads successfully
indexability	Robots, meta robots, and canonical checks
last_checked	Time of check, useful for repeated monitoring
notes	Fault, exception, owner, or resolution

Make destinations specific and relevant. A redirect to the wrong page is not a successful migration just because it returns a redirect status. Google recommends mapping old URLs to their new counterparts, using permanent server-side redirects where possible, avoiding chains, and retaining redirects long term. See Google’s site move guidance.

2. Check the new site before launch

Test the staging or preproduction site using the same representative URL set you will monitor after launch. Check more than page appearance:

Important pages and templates load, including images, downloads, forms, and other essential assets.
Production pages are crawlable and do not retain staging-only authentication or blocks.
Migration-only noindex directives and robots.txt disallows will be removed at launch.
Canonical links point to the intended production URLs.
Internal links point directly to new URLs, rather than relying on redirects.
The new sitemap lists canonical, live URLs and is ready to submit at launch.
Redirect rules have been tested against the full mapping, including URL casing, trailing slashes, query strings, encoded characters, and paths with special characters.

For a large site, consider moving sections in stages if that makes issues easier to isolate and fix. Google describes staged moves as an option for large sites; smaller and medium sites are generally easier to move together. Choose a rollout that your team can monitor and support.

3. Test every redirect in bulk

Run the check against the full source URL inventory, not only a sample. A sample can find rule-level mistakes; only a complete run can identify individual URLs that were omitted or mapped incorrectly. For each row, record the redirect status, final URL, number of hops, and whether the final page loads.

Python: bulk redirect and destination checker

Save the source URLs in a UTF-8 text file named old-urls.txt, one absolute URL per line. Install the dependency with python -m pip install requests, then save and run this script. It follows redirects, reports the final destination and hop count, and checks whether the destination responds successfully. It does not decide whether a destination is semantically relevant; compare it with your migration map.

import csv
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

TIMEOUT = 20
WORKERS = 8

with open("old-urls.txt", encoding="utf-8") as f:
    urls = [line.strip() for line in f if line.strip() and not line.lstrip().startswith("#")]

def check(source):
    row = {"source": source, "status": "", "final_url": "", "hops": "", "destination_status": "", "error": ""}
    try:
        response = requests.get(
            source,
            allow_redirects=True,
            timeout=TIMEOUT,
            headers={"User-Agent": "RedesignRedirectAudit/1.0"},
            stream=True,
        )
        row["status"] = response.history[0].status_code if response.history else response.status_code
        row["final_url"] = response.url
        row["hops"] = len(response.history)
        row["destination_status"] = response.status_code
        response.close()
    except requests.RequestException as exc:
        row["error"] = f"{type(exc).__name__}: {exc}"
    return row

with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    results = list(pool.map(check, urls))

with open("redirect-results.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["source", "status", "final_url", "hops", "destination_status", "error"])
    writer.writeheader()
    writer.writerows(results)

print(f"Checked {len(results)} URLs; wrote redirect-results.csv")

Keep concurrency modest and increase it only if the destination site can handle the requests. This script makes GET requests to live pages. For a strict production environment, run from an approved network, coordinate the crawl rate, and avoid repeatedly downloading large page bodies. Its streamed response minimizes content transfer, but the server still processes each request.

cURL: inspect one redirect chain

Use this for a focused check. -I sends a HEAD request; some servers handle HEAD differently from normal page requests, so confirm suspicious results with GET.

curl -sS -I -L --max-redirs 10 -o /dev/null \
  -w 'final_status=%{http_code}\nfinal_url=%{url_effective}\nredirects=%{num_redirects}\n' \
  'https://old.example.com/old-path'

To observe individual responses and Location headers, use:

curl -sS -D - -o /dev/null -L --max-redirs 10 'https://old.example.com/old-path'

Node.js: bulk redirect checker

Save one URL per line in old-urls.txt. This runnable Node.js script uses the built-in fetch, follows redirects, and writes a JSON Lines report.

import { readFile, writeFile } from 'node:fs/promises';

const urls = (await readFile('old-urls.txt', 'utf8'))
  .split(/\r?\n/)
  .map((line) => line.trim())
  .filter((line) => line && !line.startsWith('#'));

async function check(source) {
  try {
    const response = await fetch(source, {
      redirect: 'follow',
      signal: AbortSignal.timeout(20_000),
      headers: { 'user-agent': 'RedesignRedirectAudit/1.0' },
    });
    const result = {
      source,
      destinationStatus: response.status,
      finalUrl: response.url,
      redirected: response.redirected,
    };
    await response.body?.cancel();
    return result;
  } catch (error) {
    return { source, error: `${error.name}: ${error.message}` };
  }
}

const results = await Promise.all(urls.map(check));
await writeFile('redirect-results.jsonl', results.map((r) => JSON.stringify(r)).join('\n') + '\n');
console.log(`Checked ${results.length} URLs; wrote redirect-results.jsonl`);

Built-in fetch does not expose every redirect hop in a followed chain. Use cURL with response headers, a crawler, or a custom redirect loop if you need to enumerate every hop. Also compare finalUrl against the expected destination in your map.

4. Launch-day checks

Immediately after deployment, rerun the complete redirect and destination check against production. Verify that:

Each moved old URL redirects to its mapped final destination.
Redirects are permanent server-side redirects where appropriate, and do not form chains or loops.
New destination pages return success and render their essential content and assets.
Intentionally removed pages have the planned response, rather than an accidental redirect or server error.
Robots.txt, meta robots, and canonical directives allow the intended new pages to be indexed.
Internal links and the new sitemap use the new URLs directly.
Old and new server logs are retained and accessible for comparison.

Submit the new sitemap in Search Console. A sitemap helps Google discover URLs; it does not prove that Google indexed them. Keep the old URL set available as a reference while migration progress is being assessed.

5. Monitor indexing, crawl activity, and traffic

Use three evidence sources together because they answer different questions:

Source	What it tells you	What it cannot establish alone
Search Console	Google search performance, sitemap processing, indexing signals, and reported crawl or indexing issues	That every redirect in your map works or that visitors can use every page
Server logs	Requests received, response codes served, and observed Googlebot activity	That a URL was indexed or that a visit came from organic search demand
Analytics	Measured user sessions and conversions, when correctly installed	Google crawl or indexing status
Crawler or script	URL-by-URL response, redirect destination, chains, and technical page checks	Whether Google indexed the pages or whether demand changed

In Search Console, compare the old and new URL groups and sitemap/indexing trends. The expected broad direction is for old URLs to decline while new URLs appear and become indexed. Review search queries, impressions, and clicks for the new URLs as they emerge. At the URL level, use URL Inspection to investigate representative and affected pages; at inventory scale, use a crawler or script to verify the full set.

In logs, look for Googlebot requests to old and new paths, repeated requests to URLs returning errors, and unexpected increases in 5xx responses. Validate that apparent Googlebot traffic is genuine if your operational process requires it. In analytics, annotate the launch date and compare equivalent page groups and time ranges, accounting for tracking changes introduced by the redesign.

Keep checks running until the URL inventory has been revisited and the remaining issues are understood. Google does not define a universal polling frequency or completion duration. Its guidance says many small to medium site moves take a few weeks to process, while larger sites take longer; this is qualitative guidance, not a recovery guarantee. Google’s migration documentation explains that crawl timing depends on the site and available serving capacity.

6. Diagnose a traffic or indexing drop

Start with URL-level evidence rather than assuming the redesign itself is the only cause. Compare affected pages with their mapped destinations and inspect the following:

Wrong or missing redirects: Compare each old URL’s actual final URL with the migration map. Add or correct mappings and remove avoidable chains.
New 404s, 5xx responses, or timeouts: Check the destination response and logs. Fix routing, application errors, capacity limits, or transient infrastructure problems.
Indexing blocks: Check robots.txt, page-level noindex, authentication, and canonical tags. Remove staging restrictions that were accidentally deployed.
Stale discovery signals: Submit the corrected sitemap and update internal links to point directly to the new canonical URLs.
Rendering or resource failures: Check whether essential scripts, styles, images, and content load for users and crawlers. Review blocked or failing resources.
Capacity problems: Review server load and crawl or application errors around launch. A site that cannot serve requests reliably can delay crawling and harm user access.
Changed content or intent: Compare the old page and its replacement. A technically correct redirect can still point to a page that no longer satisfies the old page’s purpose.
Demand and seasonality: Compare the affected queries and pages over equivalent periods. Search interest and seasonal demand can change independently of migration faults.

A short-term fluctuation while Google recrawls and reindexes is possible. An early dip alone does not prove permanent damage. Google’s traffic-drop guide recommends checking technical causes and considering changes in demand and seasonality; it does not promise a fixed recovery date. See Debug Google Search traffic drops and troubleshoot crawling errors.

7. Make monitoring repeatable

Store the migration map and check results in version control or another system with history. Run a full check before launch, after deployment, and on a repeat schedule suited to the launch risk and issue rate. Keep results as dated snapshots so that a newly broken destination or rising error count is visible.

Useful alert conditions include:

A moved URL stops redirecting or reaches a different destination than expected.
A final destination returns a non-success status, times out, or loses a required asset.
A redirect chain or loop appears.
A production page becomes blocked or marked noindex.
Unexpected 404 or 5xx rates rise in logs.
New sitemap URLs fail or indexing issues increase.
Search impressions or clicks fall for a group of mapped URLs, prompting investigation rather than an automatic assumption about cause.

Set crawl concurrency and frequency with server capacity in mind. A large audit can add load, particularly if it fetches full pages and assets. Use modest concurrency, timeouts, and a clear user agent; avoid overlapping runs. A screenshot can help a person inspect the rendered appearance of a sample page, but it does not replace HTTP checks, redirect verification, Search Console, logs, or analytics.

Or skip the browser setup

For visual spot checks of a new page, ScreenshotNeo returns an image or PDF from one API request. See the API documentation. This does not replace bulk redirect tests or indexing monitoring; use it to inspect rendered pages in your migration sample.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners, newsletter popups, and chat widgets are removed before the shot; each cleanup step can be turned off.
Bot checks, blank pages, failed loads, timeouts, and cache hits are never billed. Response headers report the page verdict and billing status.
An MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf.
1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000 screenshots. Every feature is on every plan.

Common errors and fixes

Symptom	Likely cause	What to do
Old URL returns 404	Missing mapping, rule mismatch, or URL normalization difference	Compare the exact path and host with the migration map; add or correct the redirect if a replacement exists.
Redirect reaches the wrong page	Broad rule, bad mapping, or fallback to homepage	Set an explicit relevant destination and verify the final URL against the expected target.
Too many redirects or a loop	Conflicting old/new rules or destination redirects back to source	Trace each `Location` header and fix the rule sequence so the source reaches the final URL directly.
Destination returns 5xx	Application, infrastructure, or capacity fault	Check server logs and health; resolve the error and rerun the affected URL set.
New page is not indexed	It may not have been crawled, or it may be blocked, canonicalized elsewhere, or failing to render	Inspect the URL in Search Console, check robots and `noindex`, confirm the canonical, and inspect logs for requests and errors.
Search traffic falls but technical checks pass	Changed query demand, seasonality, content relevance, or measurement	Compare affected queries/pages and equivalent periods; verify analytics tagging before attributing the drop to redirects.
Bulk script reports timeout or connection errors	Network instability, rate limiting, overloaded origin, or an intentionally slow page	Lower concurrency, use a longer per-request timeout where justified, retry failed rows later, and check origin logs.
HEAD check disagrees with browser or GET	Server treats HEAD differently	Confirm with a GET request and inspect the actual redirect response headers.

FAQ

How do I monitor a site migration?

Keep a complete old-to-new URL map, verify redirects and destinations, submit the new sitemap, and compare Search Console, server logs, and analytics over time. Investigate at URL level when an aggregate signal changes.

How do I check redirects in bulk?

Export old URLs to a file, run a script or crawler across the entire list, and compare each final destination and response with the expected mapping. A spot check alone cannot prove coverage.

Why did my organic traffic drop after a redesign?

Possible causes include incorrect redirects, indexing blocks, server errors, changed content, or broken rendering. Demand and seasonality can also affect traffic. Compare affected pages and queries, then use crawl, log, Search Console, and analytics evidence to narrow the cause.

How long should I monitor the migration?

There is no universal completion date. Keep monitoring while old and new URLs are still being recrawled, issues remain, or the inventory has not yet been revisited. Larger sites may take longer to process.