Getting Started with Web Scraping in C#

To scrape a website in C#, start with HttpClient to request a page, check the HTTP response, and parse its HTML with a library such as AngleSharp. Use browser automation such as Playwright for .NET only when the content depends on JavaScript or browser behavior. Check the site’s robots.txt and relevant permissions, make requests at a restrained pace, and stop when the page denies access or fails repeatedly.

This guide builds a small, asynchronous scraper that extracts links from a permitted page. It also covers selectors, pagination, browser-rendered content, error handling, and practical operating costs.

1. Choose the right tool for each step

Scraping has separate jobs. An HTTP client retrieves a response, an HTML parser turns markup into a queryable document, and browser automation runs a browser. These tools solve different problems.

Need	Start with	Why
Retrieve a page or endpoint	`HttpClient`	Asynchronous requests and explicit status and response handling.
Read content in returned HTML	AngleSharp or Html Agility Pack	Parse markup into a document that can be queried.
Read browser-dependent content	Playwright for .NET	Automates Chromium, Firefox, and WebKit when page behavior requires a browser.

AngleSharp provides a standards-oriented DOM and familiar querySelector and querySelectorAll methods. Parsing HTML does not, by itself, execute arbitrary page JavaScript. Html Agility Pack is another option identified by Microsoft. Choose based on your project and the markup you need to handle.

2. Check the site and page first

Choose a page that you are permitted to access for your intended use. Review the site’s terms and any other applicable permissions.
Check the site’s robots.txt instructions for the paths you plan to crawl and follow them. RFC 9309 defines the robots protocol, but explicitly says its rules are not access authorization. A permissive robots file does not grant permission by itself.
Open the page normally and inspect whether the target content appears in the returned HTML. If it does, an HTTP request and parser are usually the simpler starting point.
Identify the specific elements and fields you need. Avoid collecting unrelated page content.

There is no universal request rate that is appropriate for every site. Start slowly, avoid bursts, add delays between requests, and stop if the site returns denial responses or repeated failures. Do not use a scraper to work around authentication or access controls.

3. Create a small C# scraper

The example below uses .NET’s built-in HttpClient and the AngleSharp NuGet package. It fetches one page and extracts each link’s visible text and absolute URL. In a terminal, create a console app and add AngleSharp:

A basic scraper fetches HTML, parses a document tree, then extracts only the fields it needs.

dotnet new console -n HtmlScraper
cd HtmlScraper
dotnet add package AngleSharp

Replace Program.cs with this complete example. Change the target to a page you are allowed to access.

using AngleSharp;
using System.Net;

var target = new Uri("https://example.com/");
using var cancellation = new CancellationTokenSource(TimeSpan.FromSeconds(30));

// Reuse this client for requests during the lifetime of this small program.
using var handler = new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(5)
};
using var client = new HttpClient(handler)
{
    Timeout = TimeSpan.FromSeconds(20)
};
client.DefaultRequestHeaders.UserAgent.ParseAdd("ExampleResearchBot/1.0 (+https://example.com/contact)");

try
{
    using var response = await client.GetAsync(
        target,
        HttpCompletionOption.ResponseHeadersRead,
        cancellation.Token);

    Console.WriteLine($"HTTP {(int)response.StatusCode} {response.ReasonPhrase}");
    response.EnsureSuccessStatusCode();

    var mediaType = response.Content.Headers.ContentType?.MediaType;
    if (mediaType is not null && mediaType != "text/html" && mediaType != "application/xhtml+xml")
    {
        throw new InvalidOperationException($"Expected HTML but received {mediaType}.");
    }

    var html = await response.Content.ReadAsStringAsync(cancellation.Token);
    var context = BrowsingContext.New(Configuration.Default);
    var document = await context.OpenAsync(request => request
        .Address(target)
        .Content(html));

    foreach (var anchor in document.QuerySelectorAll("a[href]"))
    {
        var href = anchor.GetAttribute("href");
        if (string.IsNullOrWhiteSpace(href)) continue;

        var absolute = Uri.TryCreate(target, href, out var resolved)
            ? resolved.AbsoluteUri
            : href;
        var label = string.Join(" ", anchor.TextContent.Split(
            (char[]?)null, StringSplitOptions.RemoveEmptyEntries));
        Console.WriteLine($"{label}\t{absolute}");
    }
}
catch (OperationCanceledException)
{
    Console.Error.WriteLine("The request was canceled or exceeded its time limit.");
}
catch (HttpRequestException ex)
{
    Console.Error.WriteLine($"HTTP request failed: {ex.Message}");
}
catch (InvalidOperationException ex)
{
    Console.Error.WriteLine($"Could not process the response: {ex.Message}");
}

Run it with dotnet run. The example prints status before checking success so you can see the server response, then rejects non-success status codes and unexpected content types rather than quietly parsing an error page as if it were the target document. The contact-style User-Agent is an example; use an accurate identifier and contact URL appropriate to your crawler.

4. Understand the important parts

HTTP request and response

HttpClient.GetAsync sends the request asynchronously. ResponseHeadersRead lets the program inspect headers before reading the body. EnsureSuccessStatusCode raises an exception for unsuccessful HTTP status codes. For workflows that need custom handling, inspect response.StatusCode and decide how to handle each result explicitly.

Microsoft describes HttpClient as sending requests and receiving responses for a URI. Its guidance recommends reusing clients rather than constructing and disposing one for each request. A long-lived client can use a suitable PooledConnectionLifetime; applications using dependency injection can instead use IHttpClientFactory. The short console example owns one client for its process. In a service, register and inject a client or factory-managed client rather than creating one inside every scraping operation.

HTML parsing and selectors

The parser builds a DOM from the returned HTML. CSS selectors keep extraction focused: a[href] selects anchors with an href, while .product-card h2 would select headings inside elements with that class. Read text from TextContent and attributes with GetAttribute. Normalize whitespace and handle missing attributes because real pages can contain empty or malformed elements.

Selectors are tied to the page’s structure. Inspect the actual response and adjust selectors when the site changes. Do not assume that a visually prominent element has a stable class name or that every matching node has all expected fields.

Relative links

Links commonly use relative values such as /about or ../next. Resolve them against the page URI, as the example does with Uri.TryCreate. Before crawling a discovered link, check its host and path against your intended scope. Otherwise, a single page can send a crawler to unrelated domains or sections.

5. Extract structured records and crawl carefully

For a data extraction task, turn each matching element into a record rather than printing every node. For example, with cards that contain a title and link:

var cards = document.QuerySelectorAll(".listing-card");
foreach (var card in cards)
{
    var title = card.QuerySelector("h2")?.TextContent.Trim();
    var href = card.QuerySelector("a[href]")?.GetAttribute("href");

    if (string.IsNullOrWhiteSpace(title) || string.IsNullOrWhiteSpace(href))
        continue;

    if (Uri.TryCreate(target, href, out var link))
        Console.WriteLine($"{title}\t{link.AbsoluteUri}");
}

Change .listing-card and the child selectors to match the permitted target page. For multiple pages, use a bounded queue and a visited-URL set. Add a delay between requests, limit concurrency, keep an explicit allowed host/path scope, and define a maximum page count or other stop condition. Persist progress if the job may run for a long time. These are practical safeguards, not a universal rate specification.

For pagination, follow only the expected next-page link and stop when it is absent, repeated, outside scope, or beyond your configured page limit. Deduplicate records using a stable field when possible. Keep response status and URL with the extracted record so you can trace incomplete results.

6. When the page needs a browser

If the HTTP response contains a shell but not the content you need, the page may populate it after JavaScript runs. First confirm this by inspecting the fetched HTML and the page’s behavior. If browser execution is genuinely needed, Playwright for .NET automates Chromium, Firefox, and WebKit through one API. Browser automation is a heavier approach because it requires running and maintaining a browser runtime.

Use browser automation when the needed content appears only after the page executes in a browser.

A minimal Playwright workflow is: install the Playwright package in a .NET project, install the browser binaries using the command documented for the package version, launch a browser, navigate to the permitted page, wait for the specific content you need, then read its DOM. Consult the official Playwright for .NET project documentation for current installation commands and browser requirements; browser versions and installation details can change.

Prefer waiting for a relevant selector over an arbitrary long sleep. Browser pages may never become fully idle because of analytics or long-lived requests. Set navigation and overall timeouts, close pages and browsers in finally or with await using patterns supported by the API, and avoid opening many browser contexts at once. Do not treat browser automation as a way around a login wall, CAPTCHA, or other access control.

7. Reliability, performance, and cost

Reuse connections: One reused HttpClient avoids creating a fresh client for each request. For long-running applications, select connection lifetime and client factory patterns to match the service.
Bound work: Set request timeouts, cancellation, page limits, and a restrained concurrency limit. Retry only transient failures, with a bounded delay and retry count; do not hammer a site that is denying requests.
Read only what you need: Select the smallest useful fields, avoid downloading unnecessary resources, and store results incrementally for long jobs.
Separate failure types: Record HTTP status, timeout, parsing failure, and missing fields distinctly. A successful HTTP response does not guarantee the expected page content.
Account for browser overhead: A real browser uses more runtime resources than a plain HTTP request and parser. Use it only for browser-dependent behavior.
Estimate your own costs: The C# libraries named here do not establish a universal per-page price. Your hosting, bandwidth, storage, and browser runtime costs depend on deployment and workload. Measure those in your environment rather than relying on generic benchmarks.

8. Troubleshooting

Symptom	Likely cause	What to do
401 or 403 response	The resource requires authorization or the request is denied.	Check that you have permission and use only documented access methods. Stop if access is denied; do not attempt to bypass controls.
404 response	The URL or path is wrong, or the content moved.	Verify the URL manually and inspect redirects and the final response URI.
Timeout or cancellation	The server is slow, the network is unavailable, or the configured deadline is too short.	Check connectivity, use a bounded timeout appropriate to the task, and reduce request concurrency. Do not retry indefinitely.
No selector matches	The selector does not match the response DOM, markup differs, or the target is injected by JavaScript.	Save or inspect the returned HTML, verify the selector against it, then consider Playwright only if browser execution is required.
Wrong characters or garbled text	Unexpected encoding or the response is not the page you assumed.	Inspect response headers and content, and confirm the media type and document encoding before extraction.
Parser sees an error page	The server returned an HTML error or challenge page instead of the intended content.	Check status and page title/body before parsing records; stop on access challenges.
Too many connections or unstable service	A new client is created per request or concurrency is excessive.	Reuse `HttpClient`, use a suitable connection lifetime or `IHttpClientFactory`, and reduce concurrency.
Playwright cannot launch	Required browser binaries are not installed or the runtime environment lacks dependencies.	Follow the current Playwright .NET installation documentation for the package version and environment.

9. Or skip the browser setup

If your task is to capture a page as an image or PDF rather than build a custom crawler, ScreenshotNeo provides a website screenshot API and MCP server. Its one-call GET API accepts a URL and returns PNG, JPEG, WebP, or PDF. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await Bun.write('shot.webp', res);

Cookie banners are accepted and removed before capture, along with known newsletter popups and chat widgets; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and the response identifies the page verdict and billing status in headers. Its MCP server exposes screenshot, page-info, and PDF capture tools for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, no card required.

10. FAQ

How do I parse HTML in C#?

Fetch the HTML, parse it into a DOM with AngleSharp or Html Agility Pack, then query elements and read their text or attributes.

Can HttpClient run JavaScript?

No. It retrieves HTTP responses; use a browser automation tool when page content depends on browser execution.

Is robots.txt permission to scrape?

No. RFC 9309 says robots rules are not access authorization. Check permissions and applicable site terms separately.

Should I use AngleSharp or Playwright?

Use AngleSharp to query returned markup. Use Playwright when you need a browser to execute page behavior. They address different stages of a workflow.

Can I run this as a recurring scraper?

Yes, if the access is permitted. Add bounded scheduling, pacing, deduplication, persistent progress, clear logging, and a stop condition, then monitor failures and page structure changes.