How to Build a News Aggregator
Build a reliable RSS and Atom news aggregator with polling, parsing, deduplication, storage, search, and safe publishing practices.

Direct answer: Build a news aggregator by subscribing to RSS 2.0 and Atom 1.0 feed URLs, fetching them on a controlled schedule, parsing entries into one internal schema, deduplicating by publisher ID and normalized URL, storing source and publication metadata, and presenting a searchable reading view. Start with headline, source, timestamp, summary, and outbound link fields. Add conditional HTTP requests, retries, feed health monitoring, and safe HTML sanitization before you scale.
1. Decide what your aggregator should do
RSS and Atom are syndication formats, not article-search services. RSS is XML with channel metadata and items; Atom is XML with entries and attached metadata. The RSS 2.0 specification calls RSS a web content syndication format, while RFC 4287 defines Atom for syndicating web content such as weblogs and news headlines.
Choose the product shape before writing the fetcher:
- Personal reader: subscriptions, unread/read state, search, source filters, saved items, and refresh status.
- Team dashboard: shared feeds, topic labels, assignments, and audit history.
- Public discovery site: source attribution, outbound links, publisher terms, crawlable pages, and a clear policy for excerpts or licensed content.
Decide whether each card contains only a headline and link, a short publisher-provided excerpt, or licensed article text. Feed availability does not grant blanket permission to republish full articles, photographs, or other content. Keep attribution and the original publisher link visible.
2. Create a durable data model
Keep feeds and entries in separate tables. Preserve original values alongside normalized values so a parser change or deduplication bug can be investigated later.
| Table | Important fields | Purpose |
|---|---|---|
| feeds | id, feed_url, publisher_name, category, enabled, fetch_interval, etag, last_modified, last_checked_at, last_status, parse_error | Subscription settings, HTTP validators, and health state |
| entries | feed_id, publisher_item_id, original_url, normalized_url, title, author, published_raw, published_at, summary, content, first_seen_at, last_seen_at, is_read, is_saved | One normalized record per feed item |
Add a uniqueness constraint on (feed_id, publisher_item_id) when a stable ID exists. RSS commonly uses guid; Atom uses entry IDs. Neither should be assumed globally unique across publishers. For items without a stable ID, use (feed_id, normalized_url) as a fallback and log collisions.
3. Fetch feeds with validators and backoff
Store the response’s ETag and Last-Modified headers for every feed. Send them on the next request as If-None-Match and If-Modified-Since. An unchanged feed can return HTTP 304 with no body, reducing bandwidth and parsing work. The feedparser documentation describes validator handling and 304 responses.
Use a per-host concurrency limit, a finite timeout, and bounded retries with exponential backoff. Avoid synchronized polling bursts by adding jitter. Respect cache guidance supplied by the publisher when available, and give operators a way to pause a feed that repeatedly fails.
There is no universal best polling interval. Faster polling reduces visible delay but increases requests; slower polling reduces load but makes updates less timely. Let users or operators set intervals by feed class, then measure freshness and error rates.
4. Parse and normalize RSS and Atom
Use a mature parser instead of writing XML rules for every publisher. A parser should handle format detection, namespaces, character encoding, relative links, content fields, and varied date formats. Treat all feed text and markup as untrusted input.

python -m pip install feedparser requests python-dateutil
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.parse import urljoin, urlsplit, urlunsplit
def normalize_url(url: str, base: str) -> str | None:
if not url:
return None
absolute = urljoin(base, url.strip())
parts = urlsplit(absolute)
if parts.scheme not in ('http', 'https') or not parts.netloc:
return None
# Fragments do not identify a different article.
return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), parts.path or '/', parts.query, ''))
def parse_date(entry):
# feedparser exposes a parsed UTC tuple when it recognizes a date.
for key in ('published_parsed', 'updated_parsed', 'created_parsed'):
value = entry.get(key)
if value:
return datetime(*value[:6], tzinfo=timezone.utc)
return None
def normalize_entry(entry, feed_url):
link = normalize_url(entry.get('link'), feed_url)
item_id = entry.get('id') or entry.get('guid')
published_at = parse_date(entry)
return {
'publisher_item_id': item_id,
'original_url': entry.get('link'),
'normalized_url': link,
'title': (entry.get('title') or '').strip(),
'author': entry.get('author'),
'published_raw': entry.get('published') or entry.get('updated'),
'published_at': published_at.isoformat() if published_at else None,
'summary': entry.get('summary') or entry.get('description'),
'content': entry.get('content', [{}])[0].get('value') if entry.get('content') else None,
}
Keep published_raw even when parsing succeeds. If a date is missing or malformed, do not invent a precise publication time. Use first-seen time for ordering and label that ordering as inferred.
5. Implement conditional fetching
This worker stores validators and handles 304 responses. In production, persist the feed state in your database rather than a process dictionary.
import time
import requests
import feedparser
def fetch_feed(feed):
headers = {'User-Agent': 'MyNewsReader/1.0 (+https://example.com/contact)'}
if feed.get('etag'):
headers['If-None-Match'] = feed['etag']
if feed.get('last_modified'):
headers['If-Modified-Since'] = feed['last_modified']
response = requests.get(feed['url'], headers=headers, timeout=20)
feed['last_checked_at'] = time.time()
feed['last_status'] = response.status_code
if response.status_code == 304:
return []
response.raise_for_status()
feed['etag'] = response.headers.get('ETag')
feed['last_modified'] = response.headers.get('Last-Modified')
parsed = feedparser.parse(response.content)
if parsed.bozo and not parsed.entries:
raise ValueError(f'Feed could not be parsed: {parsed.bozo_exception}')
return [normalize_entry(item, feed['url']) for item in parsed.entries]
Retry only transient failures such as connection resets, timeouts, and selected 5xx responses. Do not retry a permanent 404 indefinitely. Record the final error and schedule the next attempt according to a backoff policy.
6. Deduplicate without hiding legitimate stories
- Within one feed, use the publisher item ID or GUID first.
- If no stable ID exists, use the normalized URL scoped to that feed.
- Across feeds, compare normalized URLs to identify syndicated copies.
- Do not merge solely on title similarity unless you add an explainable grouping layer.
Different publishers may syndicate the same story under different URLs, and unrelated stories can share a headline. Keep every source link available inside a grouped story. Store the original identifier and URL so users can inspect where an item came from.
7. Build the reading interface
A useful first release includes:
- source enable/disable controls;
- topic or category filters;
- full-text search over title, summary, and author;
- newest and oldest ordering;
- read, unread, and saved states;
- a visible publisher name, publication time, and outbound link.
Sanitize feed-provided HTML before rendering. Strip scripts, event-handler attributes, unsafe URLs, and dangerous embedded objects. If you display excerpts, cap their length and preserve the publisher attribution. Open original articles in a new page only when that matches your product’s privacy and navigation policy.
8. Make refresh jobs reliable
A small synchronous worker can handle a personal reader. A queue-backed design is easier to scale when you have many feeds or need isolated retries.
| Architecture | Strength | Tradeoff |
|---|---|---|
| One scheduled worker | Simple deployment and debugging | A slow or broken source can delay others |
| Queue plus workers | Per-feed retries, concurrency limits, and horizontal scaling | More operational components and monitoring |
| On-demand fetch | Freshness when a reader opens a source | Higher latency and request spikes |
Add metrics for fetch duration, response status, 304 rate, parse failures, entries added, duplicate rate, and age of the newest item per feed. Alert on repeated failures or a feed that suddenly returns zero entries.
9. Respect crawler rules and publishing rights
RFC 9309 explains that robots.txt rules are requested crawler behavior and are not access authorization. Follow robots.txt as a matter of crawler discipline, but do not treat it as a license to republish content. Check each publisher’s terms and applicable rights before displaying full text, images, or commercial excerpts.
If you operate a public site, separate aggregator concerns from publisher SEO. Google’s publisher guidance recommends stable section pages and crawlable HTML article links for news sites. Its sitemap guidance says RSS and Atom feeds can describe recent URLs, but submitting a sitemap does not guarantee crawling or indexing. These recommendations apply to a publisher’s own site; they do not guarantee inclusion of a third-party aggregator.
10. Test edge cases before launch
- Feed returns HTTP 304 with an empty body.
- Feed changes from RSS to Atom or introduces a namespace.
- Entry has no link, no ID, or a relative link.
- Date is missing, invalid, or in a timezone you do not expect.
- Same article appears with tracking parameters or URL fragments.
- Feed contains malformed HTML, scripts, or unsafe URLs.
- Publisher returns a login page, CAPTCHA, or an HTML error document with status 200.
- Large feeds exceed your response or parsing limits.
- Publisher removes old entries and your retention policy must preserve them.
11. Troubleshooting common failures
| Symptom | Likely cause | Fix |
|---|---|---|
| No new items after refresh | Unchanged feed or incorrect validator handling | Check for 304, retain validators per feed, and inspect the stored last-checked time. |
| Duplicate cards on every poll | Insert has no stable uniqueness key | Use feed-scoped ID first, normalized URL second, and enforce a database constraint. |
| Stories sorted in the wrong order | Malformed or missing dates | Retain raw dates, parse updated as a fallback, then use first-seen time with an inferred-date label. |
| Some feeds parse as empty | Encoding, namespace, or malformed XML issue | Save the response for diagnosis, update the parser, and record the parse exception. |
| Unsafe markup appears in cards | Rendering feed HTML without sanitization | Sanitize server-side and apply an allowlist for tags and URL schemes. |
| One host consumes all workers | No per-host concurrency control | Use host-level limits, jitter, and a queue with fair scheduling. |
| Publisher blocks requests | Missing identity, excessive rate, or restricted endpoint | Use a descriptive User-Agent, slow polling, follow publisher instructions, and remove feeds you cannot fetch lawfully. |
12. Performance, reliability, and cost notes
Conditional requests usually provide the largest bandwidth improvement because unchanged feeds return headers without a body. Cache parsed entries and avoid reparsing identical content. Bound response size and parsing time to protect workers from unexpectedly large documents. Use database indexes on feed_id, normalized_url, published_at, and read state.

Reliability comes from isolation and observability: one bad feed should not stop the schedule, and every failure should have a stored status and next-attempt time. Cost is driven by request count, worker time, storage, and any queue or database service. Lowering polling frequency, honoring 304 responses, and disabling abandoned feeds reduce all four.
Or skip the browser setup
A news aggregator often needs preview images or PDF snapshots of linked pages. You can run and maintain a headless browser yourself, or use ScreenshotNeo as a single HTTP call. It accepts a URL and returns PNG, JPEG, WebP, or PDF. Before capture it accepts cookie and consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be turned off.
Only clean shots are billed. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the result with X-Page-Verdict and X-Billed headers. ScreenshotNeo also provides an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
See the ScreenshotNeo API documentation for all options, including full-page capture, CSS element selection, dark mode, device presets, retina scale, PDF paper sizes and page ranges, custom CSS and JavaScript, waits, request blocking, headers, cookies, user agents, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, async webhooks, bulk capture, and usage data.
curl -G 'https://api.screenshotneo.com/v1/shot' -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get('https://api.screenshotneo.com/v1/shot', params={'access_key': 'YOUR_API_KEY', 'url': 'https://stripe.com'}, timeout=90)
r.raise_for_status()
open('shot.webp', 'wb').write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot failed: ${res.status}`);
const data = Buffer.from(await res.arrayBuffer());
await fs.promises.writeFile('shot.webp', data);
The free plan includes 1,000 screenshots each month with no card. Paid plans start at $5 for 3,000 shots, and every feature is available on every plan. Create a free ScreenshotNeo account.
FAQ
How do I aggregate RSS feeds?
Store feed URLs, fetch them on a schedule with ETag and Last-Modified validators, parse entries, normalize their fields, deduplicate, and save the results.
Should I store full article text?
Only when the publisher’s terms or a license allows it. A headline, short supplied summary, attribution, and outbound link are safer defaults.
How often should a news aggregator check feeds?
Choose intervals based on freshness needs, request load, publisher guidance, and observed update frequency. There is no universal interval supported by the standards.
Are RSS and Atom IDs globally unique?
No. Scope identifiers to their feed or publisher, and retain the source identity with every entry.
Can robots.txt authorize republication?
No. RFC 9309 describes crawler behavior rules, not access authorization or republication rights.


