ScreenshotNeo

BlogHow-to

How to Pass Custom Parameters to Scrapy Spiders

Pass custom Scrapy spider parameters from the CLI or Python, parse values safely, choose runners correctly, and troubleshoot common errors.

By the ScreenshotNeo team30 September 20268 min read

How to Pass Custom Parameters to Scrapy Spiders

Use -a name=value with scrapy crawl:

scrapy crawl myspider -a category=electronics -a region=west

Scrapy exposes supplied arguments as attributes on the spider. From Python, pass keyword arguments to CrawlerProcess.crawl() or CrawlerRunner.crawl(). Spider arguments arrive as strings, so parse and validate lists, numbers, booleans, JSON, and other structured values yourself.

This behavior is documented in the Scrapy spider arguments guide. The current Core API documentation covers runner APIs.

1. Pass parameters from the command line

Add one -a name=value option for every argument. The argument name becomes an attribute on the spider instance.

Spider arguments flow from the command line into request construction and parsed output.
Spider arguments flow from the command line into request construction and parsed output.
scrapy crawl product_spider \
  -a category=electronics \
  -a region=west \
  -a max_pages=5

For a simple argument, you can read the attribute directly:

import scrapy


class ProductSpider(scrapy.Spider):
    name = "product_spider"

    def start_requests(self):
        category = getattr(self, "category", "all")
        region = getattr(self, "region", "global")
        url = f"https://example.com/products?category={category}&region={region}"
        yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        yield {"url": response.url}

getattr() supplies a default when the argument is optional. If an argument is mandatory, fail early with a clear error instead:

class ProductSpider(scrapy.Spider):
    name = "product_spider"

    def __init__(self, category=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if not category:
            raise ValueError("Pass a category with -a category=...")
        self.category = category

2. Use arguments in the spider initializer

The default spider initializer accepts keyword arguments and copies them onto the spider. A custom __init__ method is therefore optional for simple access. When you do define one, accept *args and **kwargs, then call super().__init__(*args, **kwargs).

The initializer is useful when you want to normalize values once and construct start_urls from an argument:

import scrapy
from urllib.parse import quote


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def __init__(self, tag=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.tag = tag
        if tag:
            self.start_urls = [f"https://quotes.toscrape.com/tag/{quote(tag)}/"]
        else:
            self.start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote_item in response.css("div.quote"):
            yield {
                "text": quote_item.css("span.text::text").get(),
                "tag": self.tag,
            }

Current-style spiders can also read arguments from an asynchronous start() method:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    async def start(self):
        tag = getattr(self, "tag", None)
        url = "https://quotes.toscrape.com/"
        if tag is not None:
            url += f"tag/{tag}"
        yield scrapy.Request(url, self.parse)

    def parse(self, response):
        yield {"url": response.url}

3. Treat every spider argument as a string

Scrapy does not automatically convert command-line values to Python types. This command passes the characters 1, 0, 0 as a string:

scrapy crawl product_spider -a max_pages=100

Convert and validate it before using it:

class ProductSpider(scrapy.Spider):
    name = "product_spider"

    def __init__(self, max_pages="10", *args, **kwargs):
        super().__init__(*args, **kwargs)
        try:
            self.max_pages = int(max_pages)
        except (TypeError, ValueError) as exc:
            raise ValueError("max_pages must be an integer") from exc
        if self.max_pages < 1:
            raise ValueError("max_pages must be at least 1")

Booleans

Do not use bool(value): bool("false") is True. Parse explicit spellings:

def parse_bool(value):
    normalized = value.strip().lower()
    if normalized in {"1", "true", "yes", "on"}:
        return True
    if normalized in {"0", "false", "no", "off"}:
        return False
    raise ValueError("Expected true/false, yes/no, or 1/0")

Lists and dictionaries

A value such as start_urls=https://a.example,https://b.example is still one string. Split only when your input format cannot contain the separator:

class MultiSiteSpider(scrapy.Spider):
    name = "multi_site"

    def __init__(self, start_urls="", *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = [url.strip() for url in start_urls.split(",") if url.strip()]
        if not self.start_urls:
            raise ValueError("Pass at least one URL with -a start_urls=url1,url2")

For nested data, use a documented JSON format and json.loads():

import json

class ConfigSpider(scrapy.Spider):
    name = "config"

    def __init__(self, filters="{}", *args, **kwargs):
        super().__init__(*args, **kwargs)
        try:
            self.filters = json.loads(filters)
        except json.JSONDecodeError as exc:
            raise ValueError("filters must be valid JSON") from exc

ast.literal_eval() can parse trusted Python-literal syntax, but JSON is usually clearer for shell scripts and CI jobs. Never evaluate an arbitrary argument with eval().

4. Start a crawl from a Python script

Use CrawlerProcess when your script owns the Twisted reactor and no other reactor is running:

The CLI and Python APIs provide the same spider initialization arguments.
The CLI and Python APIs provide the same spider initialization arguments.
from scrapy.crawler import CrawlerProcess
from myproject.spiders.product import ProductSpider

process = CrawlerProcess()
process.crawl(
    ProductSpider,
    category="electronics",
    region="west",
    max_pages="5",
)
process.start()

These keyword arguments are passed to the spider initializer just like -a values. Keep conversions in the spider so CLI and Python launches behave consistently.

5. Use CrawlerRunner inside an existing application

Choose CrawlerRunner when another part of your program already owns the reactor. Its crawl() method accepts the spider class or name plus initialization arguments:

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from myproject.spiders.product import ProductSpider


@defer.inlineCallbacks
def run():
    runner = CrawlerRunner(get_project_settings())
    yield runner.crawl(
        ProductSpider,
        category="electronics",
        region="west",
    )
    reactor.stop()

run()
reactor.run()

Do not call reactor.run() a second time from a host application that already started it. For coroutine-based applications, current Scrapy documentation also describes AsyncCrawlerProcess and AsyncCrawlerRunner; follow the reactor and event-loop requirements for your deployment.

6. Choose spider arguments or settings

Use arguments for values that vary between runs: a category, tenant, date range, seed URL, or one-off crawl limit. Use settings for stable project behavior such as downloader middleware, pipelines, concurrency, and retry policy. Scrapy’s FAQ describes this as a practical choice rather than a strict rule.

Input Best location Example
Changes every run Spider argument -a category=books
Shared across spiders Project setting DOWNLOAD_DELAY
Secret credential Environment variable or secret store Read in settings or the spider
Large structured configuration Versioned file or external config Pass a file path as an argument

7. Make parameterized spiders reliable

  • Validate required arguments before yielding requests.
  • Normalize whitespace and casing once in __init__.
  • URL-encode user values with urllib.parse.quote or query parameters instead of string concatenation.
  • Log the effective, non-secret configuration at startup.
  • Never log API keys, passwords, cookies, or authorization headers.
  • Use deterministic defaults so a missing optional argument has a predictable result.
  • Keep argument names stable when CI jobs or Scrapyd clients depend on them.

8. Troubleshooting

“Unknown option” or the argument is ignored

Cause: The option was not passed with -a, or it was placed before the spider name.

Fix: Use scrapy crawl spider_name -a key=value. Confirm the spider name with scrapy list.

The attribute does not exist

Cause: Code accessed self.category although the argument was optional and was not supplied.

Fix: Use getattr(self, "category", default) or reject missing input explicitly.

A number behaves like text

Cause: Spider arguments are strings.

Fix: Convert with int(), float(), or a dedicated parser and validate the range.

Only one character or malformed URLs are processed

Cause: A comma-separated or JSON value was used as a collection without parsing.

Fix: Split the string or call json.loads() before iterating.

TypeError: __init__() got an unexpected keyword argument

Cause: A custom initializer does not accept the supplied keyword.

Fix: Add the named parameter or accept **kwargs and call the base initializer.

The crawl hangs when started from Python

Cause: The reactor is started twice, or CrawlerProcess was used inside an application that already owns the event loop.

Fix: Use CrawlerRunner in an existing reactor, and let only the top-level application start and stop it.

Shell quoting changes the value

Cause: Spaces, ampersands, dollar signs, or JSON punctuation were interpreted by the shell.

Fix: Quote the complete assignment, for example -a 'filters={"brand":"Acme"}', and prefer a file for complex configuration.

9. Performance, reliability, and cost considerations

Passing an argument adds negligible work; crawl performance is determined by requests, parsing, concurrency, throttling, retries, and the target site. Avoid creating a separate process for every parameter when one process can run several crawlers or requests.

  • Use a bounded page or item limit to prevent accidental unbounded crawls.
  • Keep retries and timeouts in settings so every run follows the same reliability policy.
  • Use caching during development to avoid repeatedly downloading unchanged pages.
  • Record the parameter set with crawl results so output can be reproduced.
  • Estimate target-site load before increasing concurrency or launching many parameter combinations.

Scrapy itself does not charge for spider arguments. Your costs come from infrastructure, bandwidth, storage, proxies, and any external services used by the crawl.

10. Complete example

import json
import scrapy
from urllib.parse import quote


class CatalogSpider(scrapy.Spider):
    name = "catalog"

    def __init__(self, category=None, regions="[\"west\"]", limit="20", *args, **kwargs):
        super().__init__(*args, **kwargs)
        if not category:
            raise ValueError("category is required")
        self.category = category
        self.regions = json.loads(regions)
        self.limit = int(limit)
        if not isinstance(self.regions, list) or not self.regions:
            raise ValueError("regions must be a non-empty JSON list")
        if self.limit < 1:
            raise ValueError("limit must be positive")

    def start_requests(self):
        for region in self.regions:
            url = (
                "https://example.com/catalog/"
                f"{quote(self.category)}/{quote(region)}"
            )
            yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        for product in response.css("article.product")[: self.limit]:
            yield {
                "category": self.category,
                "url": response.url,
                "name": product.css("h2::text").get(),
            }

Run it with:

scrapy crawl catalog \
  -a category=electronics \
  -a 'regions=["west","east"]' \
  -a limit=20

Or skip the browser setup

If your next step is turning crawl results or any URL into a clean image or PDF, ScreenshotNeo provides a single screenshot request. See the ScreenshotNeo API documentation for all options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers identify the page verdict and billing status. An MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

FAQ

Can I pass the same argument more than once?

Use one argument name with a format you parse, such as a comma-separated list or JSON array. Repeated command-line options are not a substitute for defining a clear input format.

Are spider arguments available in callbacks?

Yes. They are attributes on the spider instance, so callbacks can read normalized values such as self.category.

Should credentials be spider arguments?

Usually no. Keep secrets in environment variables or a secret manager and avoid exposing them in shell history and logs.

Can Scrapyd receive spider arguments?

Yes. Scrapyd accepts spider initialization arguments when scheduling a job; use the same names and string parsing rules as the CLI.

What is the safest format for complex values?

Use JSON, validate it with json.loads(), and move very large configurations into a file referenced by an argument.