Make Concurrent Requests in Ruby

For independent, I/O-bound HTTP requests, use the async gem with Ruby’s Fiber Scheduler: start one child task per request, then collect each task’s result. The scheduler can suspend fibers while supported network operations wait, allowing other requests to progress. This is cooperative concurrency; it does not make every Ruby library non-blocking, speed up CPU-bound work, or establish a universally safe request limit. Async’s guide and Ruby’s Ruby 3.0 release example show this pattern with Net::HTTP.

1. Make concurrent requests with Async and Net::HTTP

Install the gem, save this as concurrent_get.rb, and run it with the Ruby version used by your application:

gem install async
ruby concurrent_get.rb

require "async"
require "net/http"
require "uri"

urls = [
  "https://example.com/one",
  "https://example.com/two",
  "https://example.com/three"
]

Async do
  tasks = urls.map do |url|
    Async do
      response = Net::HTTP.get_response(URI(url))
      unless response.is_a?(Net::HTTPSuccess)
        raise "GET #{url} returned #{response.code} #{response.message}"
      end
      { url: url, body: response.body }
    end
  end

  results = tasks.map(&:wait)
  results.each do |result|
    puts "#{result[:url]}: #{result[:body].bytesize} bytes"
  end
end

The outer Async block starts the scheduler and acts as the parent task. Each inner block creates a child task. Mapping creates all the children before the parent waits for their results, so supported I/O can overlap. wait returns a task’s value and surfaces task failures. The results remain in input order because the code collects tasks in the same order it created them; that does not imply the requests finished in that order.

For a small demonstration, Net::HTTP.get(URI(url)) is enough. get_response gives you the status and headers so you can make an explicit decision about non-success responses. A real program should decide which status codes count as acceptable for its endpoint instead of treating every response as success.

2. What the Fiber Scheduler does—and does not do

Ruby defines a Fiber Scheduler interface that lets a scheduler intercept supported blocking operations and suspend a fiber while it waits. Async supplies an event loop and task APIs around that mechanism. When a request yields during network I/O, another runnable task can make progress. See the Fiber Scheduler reference for the interface.

The important qualification is “supported.” A dependency that performs blocking work without using scheduler hooks can block the thread running the event loop, holding up other fibers. Likewise, placing CPU-heavy calculations inside Async blocks does not make those calculations run in parallel. Check the documentation and behavior of the HTTP client and any wrappers, middleware, DNS, TLS, and instrumentation used by your application. Test against the Ruby and gem versions you actually deploy: the Ruby 3.0 announcement is an introduction-era example, while current master documentation can describe development-version behavior.

3. Bound fan-out for large URL lists

The compact example creates one task per URL. That is reasonable for a few independent calls, but a very large input can create excessive in-flight work, memory use, open connections, or load on the remote service. There is no universal safe number: choose a bound based on the remote API’s rate limits, your latency and timeout budget, and local capacity.

One straightforward option is to process URLs in batches. This limits the number of tasks created at once without depending on an unverified semaphore API:

require "async"
require "net/http"
require "uri"

urls = (1..100).map { |n| "https://example.com/items/#{n}" }
batch_size = 8 # Example only; choose a limit for your service and workload.

Async do
  urls.each_slice(batch_size) do |batch|
    tasks = batch.map do |url|
      Async do
        Net::HTTP.get_response(URI(url))
      end
    end

    responses = tasks.map(&:wait)
    responses.each do |response|
      puts response.code
    end
  end
end

This runs at most one batch at a time. A parent task can also coordinate children with a semaphore or barrier; see the Async getting-started guide. If the service publishes a concurrency or rate limit, design for it. Concurrency limits and requests-per-second limits are different controls: fast responses can still produce a high request rate.

Only fan out requests that are independent. If request B needs an ID returned by request A, await A, extract the ID, then issue B. Starting both at once would change the program’s meaning, not just its timing.

4. Errors, timeouts, retries, and partial results

Concurrency makes failure policy more visible. A child can raise while its siblings are running, and waiting on a task can raise that failure in the parent. Decide whether one failed URL should stop the batch, be recorded while other URLs finish, or be retried. Do not silently turn all exceptions into empty response bodies: that makes a failed request look like a valid result.

• Status codes: inspect the response and classify success, retryable responses, and permanent failures according to the API contract.
• Timeouts: configure connection and read timeouts through the HTTP client API supported by your Ruby version. A timeout prevents a stalled remote endpoint from consuming a task indefinitely; verify the exact options against the installed Net::HTTP documentation.
• Retries: retry only when appropriate for the method and operation. Use a bounded retry count and backoff, and account for service rate limits. Retrying every error immediately can amplify overload.
• Partial failure: if results are independently useful, capture each task’s success or error as a per-URL outcome and continue collecting siblings. If the operation must be all-or-nothing, fail the parent explicitly.
• Cancellation: define what should happen to outstanding requests when the caller no longer needs the results. Confirm cancellation behavior with the Async and HTTP-client versions in use.

The examples show the concurrency mechanics, not a universal retry, cancellation, or timeout policy. Those choices depend on whether requests are safe to repeat, the API’s semantics, and the client version.

5. Connection reuse with Net::HTTP

Net::HTTP.get is a convenient one-request call. For repeated requests to the same host, Net::HTTP documents sessions that can carry multiple requests, and recommends block-form Net::HTTP.start when you want the session closed automatically at block exit. Read the Net::HTTP reference for session lifecycle and version-specific details.

Connection reuse within a session is distinct from sharing one mutable session object across concurrent tasks. The cited session documentation does not establish that simultaneous use of one session is safe. Keep ownership clear: do not assume fibers can concurrently use the same session unless the client’s documentation for your version explicitly supports it. If using a connection pool or a different HTTP client, check its scheduler compatibility and concurrency guarantees.

6. When threads are a better fit

Threads or a thread pool can be a better fit for existing blocking libraries or operations that should run outside the fiber event loop. Async’s guide describes using a background thread for code that is otherwise unsafe to call in its task context. Concurrent Ruby provides thread pools and concurrency primitives.

Threads bring their own resource and coordination costs. If multiple workers mutate shared data, protect it with a suitable synchronization strategy or give each worker independent state. Races can corrupt assumptions, and lock misuse can deadlock. Pool sizing is an application decision; do not assume more threads always improve throughput. Ractors offer isolated parallel execution, but add object-sharing constraints and are usually more complexity than typical HTTP fan-out requires.

7. Troubleshooting

Symptom	Likely cause	What to check
Requests still appear sequential	A client or dependency blocks without yielding to the active scheduler, or the requests are dependent on each other.	Check scheduler compatibility for the HTTP client and wrappers. Confirm tasks are created before waiting, and inspect whether the work is actually waiting on I/O.
One slow call stalls every task	Blocking work is running on the event-loop thread.	Identify the blocking operation. Use a scheduler-compatible library or move otherwise unsafe work to a background thread, following the library’s guidance.
A task raises during wait	The request failed, parsing raised, or application code raised inside the child.	Log the URL and exception context, inspect HTTP status separately from transport errors, and choose an explicit per-task or fail-fast policy.
Memory or open connections grow with input size	All URLs were scheduled at once.	Use bounded batches or parent-task coordination. Select a bound based on service limits and local capacity.
Repeated calls are slower than expected	Calls may each create a separate connection, or waiting and CPU work may dominate.	For same-host sequences, review Net::HTTP session reuse. Measure your own workload and keep connection ownership safe.
Timeout option is rejected or has no effect	The option may belong to a different client/version or only cover one phase of a request.	Check the installed Ruby version’s Net::HTTP reference and distinguish connect, read, and overall deadlines.
Remote service returns rate-limit errors	Concurrency or request rate exceeds the service policy.	Honor the service’s documented limits and retry guidance; reduce fan-out and use bounded backoff where retries are appropriate.

8. Performance, reliability, and cost

Concurrency can reduce elapsed time when independent requests spend much of their time waiting on I/O, because their waiting periods can overlap. It does not guarantee a speedup: scheduler-incompatible calls, CPU work, connection setup, service throttling, and local resource contention can erase the benefit. The reviewed Ruby and Async references do not provide a generally applicable benchmark comparing fibers, threads, and clients, so measure with your own endpoints and deployment conditions.

Reliability comes from bounded work, explicit timeouts, status and exception handling, and a deliberate retry policy—not from concurrency itself. Keep an eye on partial failures and remote rate limits. Net::HTTP session reuse may avoid repeatedly managing a connection for same-host requests, but manage session lifetime and concurrent ownership carefully.

For direct Ruby calls, the cost considerations are your application’s compute, memory, network resources, and any limits or charges imposed by the remote API. A screenshot API is useful when the output you need is an image or PDF of a rendered page rather than an HTTP response body.

9. Or skip the browser setup

If your concurrent requests are meant to capture rendered pages, ScreenshotNeo provides a screenshot API and MCP server. A GET request returns a PNG, JPEG, WebP, or PDF. See the API documentation for its request options.

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo accepts cookie and consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers indicate the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up free for 1,000 screenshots a month, no card required.

10. FAQ

Does Ruby concurrency require multiple OS threads?

Not for this pattern. Async schedules fibers cooperatively on a scheduler. Threads are a separate option for blocking code or other workload needs.

Can I use Async for requests that depend on one another?

You can, but dependent requests must wait for the result they need. Concurrency helps independent work; it cannot remove a real data dependency.

Should I use Ractors for HTTP fan-out?

Usually this is more complexity than needed for I/O-bound calls. Start by checking scheduler compatibility and use Async or a thread-based design that fits the client and workload.

How many requests should I run at once?

There is no universal number. Follow the remote service’s limits and choose a bound that fits local resources, response times, and failure behavior.