How to Scrape YouTube Comments for Insights and Analysis
Collect YouTube comments with the official Data API, retrieve replies, handle quota, and turn a documented sample into reliable insights.

Use the YouTube Data API to collect comments. For a video, start with commentThreads.list, follow its page tokens, and request comments.list with a top-level comment’s parentId when you need complete replies. Document the videos, dates, pagination boundary, reply handling, exclusions, and language rules so your analysis describes the collected sample accurately.
Do not scrape YouTube pages for data. Google’s YouTube API Services Developer Policies state: “You and your API Clients must not, and must not encourage, enable, or require others to, directly or indirectly, scrape YouTube Applications or Google Applications, or obtain scraped YouTube data or content.” See the current YouTube API Services policies.
1. Choose the collection scope
Define the unit you are studying before making requests:
- One video: pass its ID as
videoIdtocommentThreads.list. - Channel-related threads: use the channel parameters documented for
commentThreads.list, includingchannelIdorallThreadsRelatedToChannelId, when that scope matches your question. - Replies: request
snippet,repliesfor replies included in a thread, then callcomments.listwithparentIdwhen the inline reply set is incomplete.
Save the exact query parameters and collection timestamp. A reproducible record should include video or channel selection, date range, page-token cutoff, whether replies were fetched, duplicate handling, unavailable or disabled comments, language handling, and filtering rules.
2. Create a Google API project and key
- Create or select a project in Google Cloud.
- Enable the YouTube Data API v3 for that project.
- Create an API key and restrict it according to your deployment needs.
- Store the key in an environment variable; do not commit it to source control.
export YOUTUBE_API_KEY='YOUR_API_KEY'
export VIDEO_ID='dQw4w9WgXcQ'
API behavior and quota can change. Check the current YouTube Data API guide before a large run.
3. Collect top-level comments with pagination
commentThreads.list returns comment threads. Request only the resource parts you need. Use snippet for top-level comment data and add replies when you want replies that are present in the returned thread.

cURL
curl -G 'https://www.googleapis.com/youtube/v3/commentThreads' \
--data-urlencode "part=snippet,replies" \
--data-urlencode "videoId=$VIDEO_ID" \
--data-urlencode 'maxResults=100' \
--data-urlencode "key=$YOUTUBE_API_KEY" \
-o page-1.json
Python
import os
import requests
API_URL = "https://www.googleapis.com/youtube/v3/commentThreads"
params = {
"part": "snippet,replies",
"videoId": os.environ["VIDEO_ID"],
"maxResults": 100,
"key": os.environ["YOUTUBE_API_KEY"],
}
response = requests.get(API_URL, params=params, timeout=30)
response.raise_for_status()
data = response.json()
for thread in data.get("items", []):
top = thread["snippet"]["topLevelComment"]["snippet"]
print(top.get("authorDisplayName"), top.get("textDisplay"))
next_page_token = data.get("nextPageToken")
print("nextPageToken:", next_page_token)
Node.js
const api = new URL('https://www.googleapis.com/youtube/v3/commentThreads');
api.search = new URLSearchParams({
part: 'snippet,replies',
videoId: process.env.VIDEO_ID,
maxResults: '100',
key: process.env.YOUTUBE_API_KEY
});
const res = await fetch(api);
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
const data = await res.json();
for (const thread of data.items ?? []) {
const top = thread.snippet.topLevelComment.snippet;
console.log(top.authorDisplayName, top.textDisplay);
}
console.log('nextPageToken:', data.nextPageToken);
Follow every page you need
Each response can include nextPageToken. Send that token as pageToken on the next request. Stop at a documented boundary such as a maximum number of pages, a collection date, or a fixed number of comments.
import os
import requests
url = "https://www.googleapis.com/youtube/v3/commentThreads"
params = {
"part": "snippet,replies",
"videoId": os.environ["VIDEO_ID"],
"maxResults": 100,
"key": os.environ["YOUTUBE_API_KEY"],
}
all_threads = []
while True:
r = requests.get(url, params=params, timeout=30)
r.raise_for_status()
page = r.json()
all_threads.extend(page.get("items", []))
token = page.get("nextPageToken")
if not token:
break
params["pageToken"] = token
print(f"Collected {len(all_threads)} threads")
4. Retrieve complete replies for a thread
A thread response may not contain every reply. For a specific top-level comment, call comments.list with its parentId. The method accepts up to 100 results per page and returns another nextPageToken when more replies are available.
curl -G 'https://www.googleapis.com/youtube/v3/comments' \
--data-urlencode 'part=snippet' \
--data-urlencode 'parentId=TOP_LEVEL_COMMENT_ID' \
--data-urlencode 'maxResults=100' \
--data-urlencode "key=$YOUTUBE_API_KEY" \
-o replies-page-1.json
In a complete collector, inspect the thread’s reply metadata, then paginate comments.list for each top-level comment whose replies matter to your research question. This increases request count, so record whether your dataset contains inline replies only or fully paginated replies.
5. Build a reproducible dataset
Store raw API responses before cleaning. A practical normalized record contains:
- video or channel identifier and collection timestamp;
- comment ID, parent ID, and whether the record is top-level or a reply;
- comment text in the representation returned by the API;
- author display name and timestamps when your permitted use case requires them;
- the request parameters, page token sequence, and API errors;
- your deduplication, language, date, spam, and exclusion rules.
Keep raw and derived fields separate. If you edit HTML entities, normalize whitespace, detect language, or remove duplicates, retain the original value and document the transformation.
6. Analyze comments without overstating the result
Questions comment data can answer
- Which questions recur across a video or selected set of videos?
- Which feature requests, objections, or support problems repeat?
- What themes appear in praise, criticism, and follow-up discussion?
- How do reply chains develop around a topic?
- What language, terminology, or misunderstandings should future content address?
Use samples as samples
Comments are not a census of viewers. State exactly what you collected and avoid presenting the result as the opinion of every viewer. YouTube’s derived-metrics policy allows aggregate viewer sentiment analysis from comment analysis subject to its conditions, but it prohibits inferring sensitive protected attributes. Do not estimate protected characteristics from comment text, names, profile details, or writing style.
Validate automated labels
Sentiment and topic classifiers can misread irony, slang, profanity, multilingual comments, short replies, quoted text, and domain-specific terms. Create a coding guide, manually review a sample of labels, and report uncertainty or disagreement. For high-stakes decisions, use human review rather than treating an automated label as ground truth.
Published studies illustrate why scope matters. Shajari, Agarwal, and Alassad’s 2023 study analyzed 20 channels, 7,782 videos, 294,199 commenters, and 596,982 comments; those figures describe that study’s dataset, not YouTube as a whole. The 2019 “YouTube Chatter” paper likewise reports findings for its selected political and apolitical channel groups, not a platform-wide baseline.
7. Quota, performance, and cost planning
| Concern | What to do |
|---|---|
| Page size | Use the largest page size appropriate for your run; comments.list permits 1–100 results per page. |
| Quota | The comments.list method costs one quota unit per call. The API overview documents a default 10,000-unit daily allocation for most endpoints at the time of publication, subject to change. |
| Reply expansion | Fetching every reply can multiply calls. Expand only the threads required by your research design. |
| Retries | Retry transient transport failures with bounded exponential backoff. Log status, request parameters, and the page token so a run can resume safely. |
| Resuming | Persist completed pages and tokens. Do not assume a failed request means the previous pages should be discarded. |
| Large projects | Estimate calls before starting, check the project quota, and request a quota extension when your documented use case requires more capacity. |
Invalid API requests can still consume at least one quota point. The current quota documentation and comments.list reference are the authority for current limits and costs.
8. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| 401 or key error | The key is missing, invalid, restricted incorrectly, or sent under the wrong parameter. | Verify key, enable YouTube Data API v3, and check Cloud Console restrictions. |
| 403 quota exceeded | The project has used its available daily quota. | Stop the run, inspect quota usage, reduce reply expansion, resume after reset, or request an extension. |
| 400 invalid request | A required identifier or parameter is missing or malformed. | Check videoId, parentId, part, and URL encoding. Log the complete redacted request. |
| No comments returned | Comments may be disabled, unavailable to the API request, or absent for the selected scope. | Record the empty result as an observation; do not substitute page scraping. |
| Replies are missing | Inline replies did not include every reply. |
Call comments.list with the top-level comment’s parentId and follow its page tokens. |
| Duplicate records | A resumed job reprocessed a page or merged inline and expanded replies without a key. | Deduplicate by comment ID and retain the original request provenance. |
| Labels look wrong | Irony, slang, multilingual text, spam, or domain terminology confused the classifier. | Review a labeled sample, refine the coding guide, and report validation results. |
9. Or skip the browser setup
ScreenshotNeo is useful when your research workflow also needs a visual record of the public page around a video or report. It is a screenshot API, not a replacement for the YouTube Data API comment endpoints.

One request returns an image or PDF. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://www.youtube.com/watch?v=dQw4w9WgXcQ -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
- Cookie banners, newsletter popups, and chat widgets are removed before the shot.
- Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed; response headers identify the page verdict and billing result.
- An MCP server lets Claude, Cursor, and other MCP clients call
take_screenshot,get_page_info, andcapture_pdf. - 1,000 screenshots per month are free with no card; paid plans start at $5 for 3,000 shots.
Create a free ScreenshotNeo account to add visual captures to your documented comment-analysis workflow.
10. Collection checklist
- Use the official Data API rather than scraping YouTube pages.
- Define video or channel scope and save the selection rule.
- Request only needed parts and use page tokens until your documented boundary.
- Expand replies with
comments.listwhen complete replies are required. - Persist raw responses, IDs, tokens, timestamps, and errors.
- Check quota before large runs and account for one unit per
comments.listcall. - Validate automated classifications and avoid protected-attribute inference.
- Describe findings as properties of the collected sample.
FAQ
Can I collect every comment on a video?
Only claim the boundary you actually reached. Pagination, unavailable comments, disabled comments, quota, and reply expansion affect completeness.
Does commentThreads.list include every reply?
No. Use comments.list with the top-level comment’s parentId when you need a complete reply collection for that thread.
How should I report sentiment results?
Report the sample, collection rules, classifier or coding method, validation process, and uncertainty. Do not generalize the result to all viewers.
Is browser scraping an acceptable fallback?
No. The YouTube API Services policies prohibit directly or indirectly scraping YouTube or Google applications or obtaining scraped YouTube data. Use the official API for comments.
What is ScreenshotNeo doing in this workflow?
It captures a cleaned visual record of a page or report when you need screenshots or PDFs alongside API-collected comment data; it does not replace YouTube’s comment endpoints.


