How to Automate SEO Data Testing

Automate SEO data testing by scheduling Search Console API queries for search performance and sitemap data, inspecting a selected set of important URLs, and running Lighthouse audits against representative pages. Store dated results and alert on changes that need investigation.

Keep the evidence separate: Search Analytics reports search performance, URL Inspection reports Google’s indexed information, a live URL test examines a current fetch, and Lighthouse audits a page. A successful check does not guarantee that a URL will appear in Google Search.

1. Decide what each automated check should answer

Start by naming the question and data source for each check. Search Console’s API includes Search Analytics, Sitemaps, Sites, and URL Inspection services. Lighthouse is a separate page-audit tool.

Question	Source	What it tells you	What it does not establish
Did clicks, impressions, or another search-performance measure change?	Search Analytics API	Performance data for a property over a date range, optionally filtered or grouped by dimensions such as country or device.	Why a change happened, or whether a page is currently indexed.
What sitemap information is available for the property?	Sitemaps API	Sitemap records and their reported state in Search Console.	That every submitted URL is indexed or will appear in search results.
What does Google report about this selected URL in its index?	URL Inspection API	URL-level information for a property the account can access in Search Console.	A complete live crawl of every URL or a promise of search appearance.
What happens when the URL is fetched now?	URL Inspection live test in Search Console	A real-time fetch and examination of the current page, which can differ from indexed information.	Every possible indexing issue. The live test does not determine whether a URL is a duplicate or alternate page.
Did a page audit regress?	Lighthouse or Lighthouse CI	Repeatable automated page-audit results; Lighthouse CI can help prevent regressions.	Search performance or Google’s index status.

Google describes the URL Inspection tool as providing information about Google’s indexed version of a specific page and allowing users to test whether a URL might be indexable. Keep those two views labeled separately in dashboards and alerts. Google Search Console Help: URL Inspection tool.

2. Set up access and choose a scope

Choose the Search Console property to monitor. Confirm the account or service identity you use has access to it. URL Inspection API checks apply to URLs in a property managed in Search Console.
Enable the Search Console API for the project used by your application and configure Google authentication for the environment where the job runs. Store credentials using your deployment platform’s secret or identity mechanism; do not commit credential files or tokens.
Choose a small, representative set of important URLs for URL Inspection. Define how URLs map to the property and keep the exact inspected URL in each result.
Choose the Search Analytics date range, dimensions, and filters that answer a specific monitoring question. Avoid treating every possible query combination as a useful alert.
Choose representative pages for Lighthouse runs, including important templates. Run audits in CI for regressions, on a schedule for ongoing checks, or both.

The Search Console API reference documents the available services and request methods. See the Search Console API reference for current endpoint details and access requirements.

3. Query Search Analytics on a schedule

Search Analytics queries accept a date range and can use dimensions and filters. The example below requests daily clicks, impressions, click-through rate, and average position for a property. It is a runnable Python example when the Google API client is installed and Application Default Credentials are configured with Search Console access.

pip install google-api-python-client google-auth

import json
from googleapiclient.discovery import build
from google.auth import default

PROPERTY = "sc-domain:example.com"  # Use your verified property identifier.
credentials, _ = default(scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
service = build("searchconsole", "v1", credentials=credentials)

request = {
    "startDate": "2026-09-01",
    "endDate": "2026-09-07",
    "dimensions": ["date"],
    "rowLimit": 25000,
}
response = service.searchanalytics().query(siteUrl=PROPERTY, body=request).execute()
print(json.dumps(response.get("rows", []), indent=2))

Use dates that have fully settled for your reporting purpose; recent Search Console data can be incomplete. The precise date range, cadence, and alert thresholds are implementation choices, not guarantees provided by the API.

Use filters and dimensions deliberately

Dimensions such as date, query, page, country, and device help isolate a segment. Filters can narrow the requested data. For example, a page-level query can focus on a landing page:

request = {
    "startDate": "2026-09-01",
    "endDate": "2026-09-07",
    "dimensions": ["date", "page"],
    "dimensionFilterGroups": [{
        "filters": [{
            "dimension": "page",
            "operator": "equals",
            "expression": "https://example.com/docs/"
        }]
    }],
    "rowLimit": 25000,
}
response = service.searchanalytics().query(siteUrl=PROPERTY, body=request).execute()

For a query-segment check, change the filter dimension to query; for a country or device segment, use that dimension. Check the API reference for supported operators and request fields before expanding a production query. Decide whether you need aggregated totals or grouped rows: adding dimensions changes the shape and granularity of the result.

4. Inspect selected URLs and sitemap state

URL Inspection is useful for a curated set of important URLs, such as newly published pages, key templates, or URLs involved in an incident. It is not a substitute for crawling every URL on a site. The following Python example requests indexed information for one URL. The URL must belong to a Search Console property accessible to the authenticated account.

import json
from googleapiclient.discovery import build
from google.auth import default

PROPERTY = "sc-domain:example.com"
URL = "https://example.com/docs/"
credentials, _ = default(scopes=["https://www.googleapis.com/auth/webmasters"])
service = build("searchconsole", "v1", credentials=credentials)

result = service.urlInspection().index().inspect(body={
    "inspectionUrl": URL,
    "siteUrl": PROPERTY,
    "languageCode": "en-US",
}).execute()
print(json.dumps(result, indent=2))

Record the inspection timestamp, URL, property, and response fields your team relies on. Treat this as Google’s indexed view. The separate live URL test is available through the Search Console URL Inspection tool; a live fetch may differ from indexed information and does not check every possible indexing issue. Google Search Central’s URL Inspection API announcement.

For sitemap monitoring, use the Sitemaps methods documented in the API reference to list sitemap records for the property and, where appropriate, retrieve a sitemap record. Store the response and timestamp so a change in reported state can be compared with prior observations. Do not infer that sitemap submission means all contained URLs are indexed.

5. Run Lighthouse audits in development or CI

Lighthouse audits a page automatically. Lighthouse CI can be used to help prevent regressions. Run it against representative URLs and templates, then retain the reports or the metrics your team has chosen to track.

npm install --save-dev @lhci/cli
npx lhci autorun --collect.url=https://example.com/ --collect.url=https://example.com/docs/

Configure Lighthouse CI for your repository and CI environment according to the Lighthouse documentation and your chosen CI provider. Set audit assertions that fit the pages and workflow. Lighthouse findings are page-audit signals; pair them with Search Console data when the question concerns clicks, impressions, sitemap information, or Google’s index.

6. Save history and make alerts actionable

A recurring check becomes useful when its result can be compared with earlier results and leads to a clear investigation. Store dated records rather than overwriting the last response.

Record the source, property, URL or query segment, request date range, collection timestamp, and relevant response values.
Keep indexed URL Inspection results and current live-test results in distinct fields or records.
Define a baseline and threshold for each alert. Prefer changes that matter to a named page group or business question over alerts for every small fluctuation.
Include the observed value, expected condition, affected URL or segment, data source, and a link to the relevant Search Console or audit report.
Use a review step before changing pages in response to an anomaly. A data change can flag a question; it does not by itself explain the cause.

For example, a useful alert says that impressions for a named page segment fell below the team’s chosen threshold over a completed date window, gives the comparison window, and identifies Search Analytics as the source. A separate URL inspection alert can report that a selected URL’s indexed information changed. Do not merge these into a single “SEO passed” status.

7. cURL, Python, and Node.js request examples

For teams calling the Search Console REST API directly, the following cURL example shows the Search Analytics request shape. It assumes TOKEN is an OAuth access token with suitable Search Console access and PROPERTY is URL-encoded. Use a secure credential flow to obtain and refresh tokens; do not paste a long-lived credential into source control.

curl -X POST \
  "https://www.googleapis.com/webmasters/v3/sites/${PROPERTY}/searchAnalytics/query" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"startDate":"2026-09-01","endDate":"2026-09-07","dimensions":["date"],"rowLimit":25000}'

The Google API client examples above provide runnable Python for Search Analytics and URL Inspection once authentication is configured. Here is a Node.js REST example using a previously obtained OAuth token:

const property = encodeURIComponent('sc-domain:example.com');
const token = process.env.GOOGLE_ACCESS_TOKEN;
if (!token) throw new Error('Set GOOGLE_ACCESS_TOKEN');

const response = await fetch(
  `https://www.googleapis.com/webmasters/v3/sites/${property}/searchAnalytics/query`,
  {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      startDate: '2026-09-01',
      endDate: '2026-09-07',
      dimensions: ['date'],
      rowLimit: 25000,
    }),
  },
);
if (!response.ok) {
  throw new Error(`Search Console API ${response.status}: ${await response.text()}`);
}
console.log(await response.json());

For URL Inspection using REST, send a POST to the URL Inspection endpoint with an inspectionUrl, siteUrl, and optional language code in the JSON body, authenticated with an access token and the required property access. Confirm current paths, scopes, request fields, and method availability in Google’s API reference before deploying.

8. ScreenshotNeo for visual evidence of a page

When an SEO investigation also needs a visual record of what a visitor-facing page looks like, a screenshot can complement API data and Lighthouse. It cannot replace Search Analytics, URL Inspection, or a Lighthouse audit. ScreenshotNeo is a website screenshot API and MCP server for developers.

Or skip the browser setup

One GET request returns a screenshot. This cURL example saves a WebP image; see the ScreenshotNeo API documentation for request options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before the shot; each cleanup step can be turned off. Bot checks, blank pages, and failed loads are never billed, and response headers identify the page verdict and billing status. Its MCP server lets AI agents use screenshot tools. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, with no card.

9. Performance, reliability, and cost considerations

Keep queries focused. Request only the date range, filters, and dimensions needed for a check. Large numbers of segmented queries increase processing and storage work.
Use sensible cadences. Match run frequency to the decision being made. Search performance comparisons need meaningful date windows; selected URL inspections and Lighthouse runs can use a separate schedule.
Handle API failures explicitly. Record failed collection attempts as failures, with status and timestamp, rather than treating missing data as zero. Retry transient errors with bounded backoff and alert if a job remains unsuccessful.
Expect evidence to have limits. Search Console reports, indexed information, live fetches, and audits describe different scopes. Preserve source and collection time so teams can interpret changes accurately.
Control storage and alert noise. Retain enough history to compare periods relevant to your workflow, and avoid collecting dimensions that nobody reviews.
Plan credentials and access. Grant the monitoring identity only the necessary property access and keep tokens or service credentials out of logs and repositories.
Account for service terms and quotas. Check the current API documentation for limits and project configuration rather than assuming a fixed request allowance.

10. Troubleshooting common failures

Symptom	Likely cause	Fix
HTTP 401 or an authentication error	Missing, expired, or invalid OAuth credentials, or incorrect token handling.	Refresh credentials using the configured Google auth flow, verify the token’s scopes, and avoid logging secrets.
HTTP 403 or permission denied	The authenticated account cannot access the property, or the API is not enabled/configured for the project.	Check Search Console property access and project API configuration. Confirm the URL Inspection URL belongs to the supplied property.
URL Inspection rejects the URL/property pair	The URL is outside the declared property, or the property identifier is wrong.	Use the exact URL to inspect and the matching Search Console property identifier; verify access in Search Console.
No Search Analytics rows	The date range or filters returned no rows, the property has no matching data, or the query’s grouping is too narrow.	Check dates, property, and filters. Try a broader date range or fewer dimensions, then inspect the raw response.
Recent metrics look lower than expected	The selected period may not be complete, or the comparison windows differ.	Compare equivalent completed date ranges and record the requested window with each result.
Live test and indexed result disagree	One describes a current fetch and the other Google’s indexed information; they can differ.	Report both separately with timestamps. Investigate the page and indexing context instead of treating either result as a guarantee.
Lighthouse results vary between runs	Audit conditions or the page environment changed between runs.	Use consistent URLs and CI setup, compare repeated results, and investigate a sustained regression rather than overreacting to one run.
Scheduled job reports success but data is absent	The job may be swallowing request errors or converting missing fields to zero.	Check HTTP status and response parsing; persist collection errors distinctly from valid empty results.

11. A practical rollout checklist

Choose a small set of questions and map each to Search Analytics, Sitemaps, URL Inspection, or Lighthouse.
Verify property access and configure authentication in the job environment.
Run each request manually once and retain the unmodified response for debugging.
Schedule collection and save dated records with source, scope, and timestamps.
Set thresholds tied to decisions, then include the evidence and relevant URL or segment in each alert.
Review alerts before making SEO changes; document the investigation and outcome.

Frequently asked questions

Can an API tell me whether a URL will rank?

No. URL Inspection provides indexed information or a live test, neither of which guarantees that a page appears in Search results.

Can I use URL Inspection to check every URL on my site?

Use it for selected URLs that matter to your workflow. The API reports URL-level information and should not be described as a complete live crawl of the site.

Should Lighthouse replace Search Console monitoring?

No. Lighthouse audits pages, while Search Console provides search-performance, sitemap, and indexed URL information. Use the source that matches the question.

Does a successful live test prove there are no indexing problems?

No. Google’s documentation says the live test does not check every possible indexing issue, including whether a URL is a duplicate or alternate page.