How to Scrape Google AI Mode: Answers, Citations, and Links as JSON
Get structured answers with source links and citation spans, and understand the limits of scraping Google AI Mode.
If you need an answer with source URLs and citation spans in structured form, use Gemini API Grounding with Google Search and extract its response annotations. That gives you a Google API response containing Gemini’s grounded answer and citation metadata; it is not the consumer Google AI Mode answer or an “AI Mode API.” If you need the consumer interface itself, your options are an eligibility-gated, non-commercial Google research API that returns browser-style Search HTML or a vendor’s interface-extraction service with its own variable response contract.
This distinction matters before you write a parser. The three routes return different things, have different access rules, and carry different maintenance costs. This guide starts with the supported structured-answer path, then explains what the research API and vendor extraction can and cannot provide.
1. Choose the output you actually need
| Route | What you receive | Best fit | Main constraint |
|---|---|---|---|
| Gemini API Grounding with Google Search | Model output text with URL citation annotations and offsets; response steps can include search queries and results | An API-generated answer plus sources in structured form | It is Gemini output, not a copy of consumer AI Mode |
| Search Researcher Result API (SRR) | HTML Google would return to a browser for Search URLs | Eligible academic research that can work with returned HTML | Application-gated and restricted to non-commercial use |
| Third-party AI Mode extraction service | Provider-parsed answer blocks and references, sometimes with HTML | A project that specifically needs consumer-interface extraction and accepts provider-specific fields | Response fields and Google markup can change |
Google describes AI Mode as an exploratory experience for nuanced questions, comparisons, and links. It may fan a question out into related searches, and Google says AI Mode and AI Overviews can use different models and techniques. The answers and links can therefore vary. Treat citations as sources shown for one particular response, not as a fixed or exhaustive bibliography. See Google’s AI features documentation.
2. Get answer text and citations through Gemini Grounding
When your requirement is “give me a generated answer and the URLs that support particular spans,” use the documented citation annotations. The fields to preserve are the content block’s text and, for each url_citation, its URL, title, start index, and end index. Keep the original response too: a normalized JSON object is useful to downstream code, but should not discard the source response or its capture time.
Google’s current documentation demonstrates the Interactions API with the official google-genai Python package. The model identifier and API availability can change, so check the current Grounding with Google Search documentation before pinning a production configuration. Set GEMINI_API_KEY in the environment and install the SDK with python -m pip install google-genai.
Runnable Python example
import json
import os
from datetime import datetime, timezone
from google import genai
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
response = client.interactions.create(
model="gemini-3.8-flash",
input="Explain how HTTP cache validators work. Include the key distinctions.",
tools=[{"type": "google_search"}],
)
blocks = []
queries = []
for step in response.steps:
if step.type == "google_search_call":
queries.extend(step.arguments.get("queries", []))
elif step.type == "model_output":
for block in step.content:
if block.type != "text":
continue
citations = []
for annotation in block.annotations or []:
if annotation.type != "url_citation":
continue
start = annotation.start_index
end = annotation.end_index
citations.append({
"url": annotation.url,
"title": annotation.title,
"start_index": start,
"end_index": end,
"cited_text": block.text[start:end],
})
blocks.append({"text": block.text, "citations": citations})
result = {
"captured_at": datetime.now(timezone.utc).isoformat(),
"model": "gemini-3.8-flash",
"answer_blocks": blocks,
"search_queries": queries,
}
print(json.dumps(result, ensure_ascii=False, indent=2))The example keeps citations attached to the text block where Google returned them. The offsets refer to a span in that block. Preserve those offsets even if you also compute cited_text: they let a renderer link the source to the corresponding answer passage. If you transform, trim, or concatenate the answer, the old offsets no longer describe the new text.
For a quick inspection, print the text and citations as received. For a production API, serialize the normalized object to a file, database, or your own endpoint; validate the offsets before slicing, since an unexpected or missing annotation should not crash the whole job. The response may contain several model-output steps or text blocks, so do not assume there is exactly one.
Equivalent cURL request
The SDK is the clearest route for typed response objects. If you want to inspect the raw HTTP response instead, send a REST request using your Gemini API key. Store the key in an environment variable; do not commit it to source control.
curl -sS \
-H "Content-Type: application/json" \
-H "x-goog-api-key: ${GEMINI_API_KEY}" \
-X POST \
"https://generativelanguage.googleapis.com/v1beta/interactions" \
-d '{
"model": "gemini-3.8-flash",
"input": "Explain how HTTP cache validators work.",
"tools": [{"type": "google_search"}]
}' | jq .Read the current API docs for the active endpoint and model before deploying this raw request. The response is an object with steps; locate model_output content blocks and their annotations rather than expecting a top-level citations array. To persist a normalized JSON record, apply the same extraction rules as the Python example.
Equivalent Node.js request
This raw fetch example uses Node.js 18 or later, where fetch is available globally. It outputs the raw JSON response so you can inspect all steps and annotations before mapping them into your application’s schema.
const apiKey = process.env.GEMINI_API_KEY;
if (!apiKey) throw new Error("Set GEMINI_API_KEY first");
const response = await fetch(
"https://generativelanguage.googleapis.com/v1beta/interactions",
{
method: "POST",
headers: {
"Content-Type": "application/json",
"x-goog-api-key": apiKey,
},
body: JSON.stringify({
model: "gemini-3.8-flash",
input: "Explain how HTTP cache validators work.",
tools: [{ type: "google_search" }],
}),
}
);
if (!response.ok) {
throw new Error(`Gemini request failed: ${response.status} ${await response.text()}`);
}
const payload = await response.json();
console.log(JSON.stringify(payload, null, 2));For the application’s JSON format, loop through payload.steps, select type === "model_output", then read each text content block’s text and annotations. Retain only url_citation annotations for source links, and record any search-call query steps separately. Confirm field casing against the API response version you use.
3. Normalize citations without losing their meaning
A practical record should preserve both source data and its relationship to the answer. One possible shape is:
{
"captured_at": "2026-09-29T12:00:00+00:00",
"answer_blocks": [{
"text": "The answer text returned by the model...",
"citations": [{
"url": "https://example.com/source",
"title": "Source title",
"start_index": 0,
"end_index": 42,
"cited_text": "The answer text returned by the model"
}]
}],
"search_queries": ["example query"]
}The values above show a schema example, not a captured response. Avoid flattening citations into a detached list if users need to know which passage each source supports. If you do deduplicate URLs for display, keep the original annotation list too: one source might annotate more than one span.
- Validate that each index is an integer, non-negative, ordered, and within the associated text’s length before slicing.
- Allow no annotations. A grounded request can return useful text without the citation annotations your code expected.
- Store the exact prompt, model identifier, capture time, and raw response according to your data policy so a later audit can distinguish changing answers.
- Escape titles and URLs before inserting them into HTML; treat returned text and links as untrusted input.
- Do not infer that a source supports an entire answer just because it appears in the response. Use the annotated span as the association supplied by the API.
4. When the literal Search interface is required
Google Search Researcher Result API
Google’s SRR API is an official route for approved academic researchers to request Search results and receive HTML nearly like a browser request. Its documentation says some third-party features may be absent, invalid parameters or non-Search URLs can error, and project request limits apply over a rolling 24-hour period. Eligibility depends on the researcher program and application. Google explicitly says research cannot be made available for commercial sale, and SRR use is non-commercial.
That makes SRR unsuitable as a general commercial scraping API. It also does not provide a documented, stable JSON schema for AI Mode answers and citations. If you are eligible and have a research use, follow the application and setup links on the Search Researcher Program page, then parse returned HTML defensively. Do not present SRR as approved commercial access to AI Mode.
Vendor-specific extraction
Third-party services may sell an endpoint that returns parsed AI Mode answer blocks and references, sometimes alongside HTML. Scrape.do documents one such provider-specific endpoint. Its documentation describes optional fields, variable references and shopping cards, and warns that Google’s raw markup class names can change without warning. It also notes that raw HTML can be large and that an answer container may be absent when Google returns no AI Mode content. These are statements about that provider’s documented service, not a guarantee from Google or an independent assessment of reliability.
Because the reviewed documentation does not establish a stable Google endpoint or a universal vendor request contract, do not copy an invented URL or assume that another provider uses the same parameter names or response fields. Before integrating a service, read its current endpoint docs and terms, verify its commercial-use rights and data handling, and write a small adapter around its actual response. Make missing answer and citation fields valid outcomes. Capture timestamps and retain enough raw response data to investigate parser changes.
5. Reliability, performance, and cost
Grounding adds a search step to model generation, so it may take longer than a model-only response; actual latency depends on the request and current service. Do not optimize from an assumed benchmark. Measure end-to-end duration for your own prompts, track errors by status and step type, and set a timeout appropriate to the calling workflow. Retry transient failures with bounded exponential backoff and jitter, but avoid automatically replaying permanent client errors. If a request times out after the service may have completed it, avoid unbounded retries that create duplicate work.
Grounded generation has API pricing and usage terms that can change. Check the current Gemini API pricing and terms before estimating costs; this guide does not state a price. SRR has program quotas and eligibility rules, not an ordinary commercial quota for production scraping. Third-party services set their own rate limits and billing; inspect those contracts directly. Cache only when the application’s freshness, privacy, and applicable terms allow it. If consumers rely on current citations, give the cache a short, explicit TTL or turn it off.
For batch jobs, queue requests with concurrency limits, persist each completed result independently, and make failures visible rather than silently emitting empty JSON. Track missing-answer, no-citation, malformed-annotation, timeout, quota, and provider error cases separately. They have different fixes. Never tell downstream users that a missing citation proves a claim is unsupported; it means the response did not supply that annotation.
6. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| 401 or permission error | Missing, invalid, or restricted API key | Check the environment variable, key restrictions, and API setup; never log the secret. |
| 400 invalid request or model | Stale model identifier, unsupported tool setting, or malformed body | Compare the request to the current Gemini documentation and start with the smallest documented example. |
| Answer exists but citation list is empty | No URL citation annotation was returned for that text block | Keep the answer and emit an empty list; don’t fabricate links or assume the response schema is broken. |
| Index error while extracting cited text | Unexpected offsets, wrong text block, or a transformed string | Validate bounds against the original block and preserve offsets before modifying text. |
| SRR request is rejected | Project is not approved, URL is outside Search scope, parameter is rejected, or quota is exhausted | Check eligibility, endpoint instructions, accepted parameters, and the rolling quota. |
| Third-party parser returns no answer | Google supplied no AI Mode answer for that page, or interface structure changed | Represent absence explicitly, preserve capture metadata, inspect provider diagnostics, and update the adapter to current docs. |
| JSON output breaks a consumer | Consumer assumed one text block or required citations | Handle arrays, empty arrays, and absent optional fields; version your normalized schema. |
7. Or skip the browser setup
ScreenshotNeo is a screenshot API and MCP server for developers. It is useful when the job is to capture a page as an image or PDF, rather than extract AI Mode’s structured answer and citation annotations. A single request returns a screenshot or PDF; it does not turn the screenshot into the grounded answer JSON described above. See ScreenshotNeo and the API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpCookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed. An MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.
Sign up for ScreenshotNeo’s free plan.
8. FAQ
Is Gemini Grounding the Google AI Mode API?
No. It is an official API feature that returns Gemini model output grounded with Google Search. The reviewed documentation does not say that its answer will match what consumer AI Mode would return for the same prompt.
Can SRR be used for a commercial SERP product?
Google’s program materials say SRR is for non-commercial research and research cannot be made available for commercial sale. Do not use it as a commercial scraping route.
Does Google require special schema to appear in AI Mode?
Google Search Central says there are no special technical requirements or special schema markup for AI Overviews or AI Mode. Ordinary Search eligibility still applies, and meeting requirements does not guarantee crawling, indexing, or inclusion. See the official guidance.
Are the citations a complete bibliography?
No. Preserve the links returned for that response and their associated spans. Search behavior and links can vary between responses.
Does Search Console report AI Mode separately?
Google says AI-feature appearances are included in overall Search traffic within the Web search type; it does not provide a separate AI Mode traffic filter in the cited guidance.
