Web Scraping and Browser Automation with Crawlee
Learn when to use CheerioCrawler or PlaywrightCrawler, install Crawlee correctly, handle sessions and proxies, and build reliable scraping workflows.
Blog
Guides for capturing the web
How to take clean screenshots, turn HTML into images and PDFs, run headless browsers, and give AI agents eyes. Written by the team that builds ScreenshotNeo.
Using Webhooks in Browser Automation Functions
Connect webhook events to browser automation reliably. Learn the trigger patterns, runnable examples, security, retries, and ways to handle long-running browser tasks.
Fintech Browser Automation on Your Own Infrastructure
Run browser automation for financial workflows on infrastructure you control. Set up Playwright, isolate credentials and sessions, and plan for operational and policy limits.
How to Associate a User with a Website Screenshot
Link every screenshot to an account or pseudonymous ID in your app while keeping attribution, analytics, and privacy boundaries clear.
Residential Proxies for Browser Automation: Uses and Considerations
Learn when residential proxies help browser automation, how to configure them in Playwright, and how to handle routing, sessions, security, cost, and compliance.
How to Scrape Websites Without Getting Blocked
Learn a reliable, permission-first scraping workflow: robots.txt, rate limits, retries, refusal handling, and safer screenshot automation.
How to Scrape Brain Product Pages
Use BRAIN’s partner API for authorized product data. When you need public page fields, extract Product JSON-LD first and render JavaScript when required.
How to Set Domain Policies for Browser Automation
Configure Chrome domain allowlists for Playwright and other automation, verify policy scope, troubleshoot failures, and use a managed screenshot API.
Browser Automation for Fintech
Learn how to automate fintech browser workflows safely with Playwright, isolated accounts, protected session state, and auditable actions.
How to Scrape Instagram Without Getting Blocked
There is no reliable block-proof Instagram scraper. Learn the authorized API route, enforcement limits, compliant workflows, and safer screenshot options.
Deploy MCP Servers with Browser Automation
Deploy Playwright MCP locally or over HTTP, choose the right browser connection, protect session state, and troubleshoot reliable automation servers.
How to Fix a “Your IP Has Been Blocked” Error
Learn what Cloudflare 1006, 1020 and 1015 mean, how to unblock access safely, and what to send the site owner.
How to Select Values Between Two Nodes in BeautifulSoup and Python
Use BeautifulSoup’s sibling methods for values beside a label, or document-order traversal when the target is elsewhere. Includes runnable examples and fixes for common parsing surprises.
Screenshot Webpages as JPEG in TypeScript
Use Playwright in TypeScript to capture webpages as JPEG, control quality and scale, save files or buffers, and handle full-page and element screenshots.
How to Use Tor as a Proxy for Web Scraping in 2026
Configure Tor SOCKS5 for Python and browser scraping, prevent DNS leaks, and understand Tor’s limits, safe request rates, and alternatives.
How to Configure Browser Automation Sessions
Configure Playwright or Selenium sessions with the right browser, profile state, proxy, credentials and timeouts. Includes runnable examples and fixes for common failures.
How to Parse Web Data With Python and Beautiful Soup
Learn how to fetch HTML, parse it with Beautiful Soup, extract text, links and structured fields, and handle JavaScript, malformed markup and crawl limits.
How to Find Sibling HTML Nodes Using BeautifulSoup and Python
Use Beautiful Soup sibling navigation to find the next or previous HTML node, skip whitespace, and select matching tags reliably.