How to Use Python to Connect and Interact With APIs

To connect to an HTTP API from Python, choose the endpoint and method from that API’s documentation, send the request with a finite timeout, authenticate exactly as documented, check the HTTP status, and only then parse the response. The requests library is the shortest practical path; Python’s built-in urllib.request avoids a dependency.

An API call is a request and response: your client supplies a method, URL, parameters or body, headers, and credentials; the server returns a status code, headers, and usually a body. HTTP method meaning matters, so never substitute GET for a documented POST or assume every API uses the same authentication or pagination scheme.

1. Install a client and make your first request

Requests

Install Requests in the environment that runs your program:

python -m pip install requests

This complete example sends query parameters, sets an Accept header, applies a timeout, checks status, and handles network, HTTP, and JSON failures. Replace the URL and parameters with the target API’s documentation.

import requests

url = 'https://api.example.com/v1/items'
try:
    response = requests.get(url, params={'limit': 10}, headers={'Accept': 'application/json'}, timeout=10)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.Timeout:
    print('The API request timed out')
except requests.exceptions.HTTPError as exc:
    print(f'Unsuccessful HTTP status: {exc}')
except requests.exceptions.JSONDecodeError:
    print('The response body was not valid JSON')
except requests.exceptions.RequestException as exc:
    print(f'The request failed: {exc}')
else:
    print(data)

Requests supports method helpers, query parameters through params, JSON bodies through json, headers, authentication, sessions, and timeouts. See the Requests documentation.

Standard-library alternative

Use urllib.request when adding a third-party package is not possible:

import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError

query = urlencode({'limit': 10})
request = Request(f'https://api.example.com/v1/items?{query}', headers={'Accept': 'application/json'}, method='GET')
try:
    with urlopen(request, timeout=10) as response:
        if not 200 <= response.status < 300:
            raise RuntimeError(f'Unexpected HTTP status: {response.status}')
        data = json.loads(response.read().decode(response.headers.get_content_charset() or 'utf-8'))
except HTTPError as exc:
    print(f'HTTP error {exc.code}: {exc.reason}')
except URLError as exc:
    print(f'Connection error: {exc.reason}')
else:
    print(data)

Python documents urllib.request as a URL-opening interface with support for authentication, redirects, cookies, and proxies. Requests is generally more concise when you need sessions or higher-level options.

2. Build the request correctly

Methods and URLs

Method	Typical intent	Retry consideration
`GET`	Read a representation	Safe and idempotent
`HEAD`	Read headers without a body	Safe and idempotent
`POST`	Process submitted content	Do not repeat automatically unless safe
`PUT`	Replace a representation	Idempotent by HTTP semantics
`DELETE`	Request removal	Idempotent by HTTP semantics

These protocol meanings come from RFC 9110; an API can impose additional rules.

Parameters, forms, and JSON

r = requests.get(url, params={'q': 'python api', 'page': 2}, timeout=10)
r = requests.post(url, data={'name': 'Ada'}, timeout=10)
r = requests.post(url, json={'name': 'Ada', 'active': True}, headers={'Accept': 'application/json'}, timeout=10)

Use params for query values, data for form encoding or raw bytes, and json for a JSON body. Do not concatenate unescaped query strings.

Headers and authentication

import os
headers = {'Accept': 'application/json', 'Authorization': f"Bearer {os.environ['API_TOKEN']}"}
r = requests.get(url, headers=headers, timeout=10)
r.raise_for_status()

Some services expect an API key in a different header or query parameter. Requests also documents Basic and Digest authentication:

r = requests.get(url, auth=(os.environ['API_USER'], os.environ['API_PASSWORD']), timeout=10)
r.raise_for_status()

For OAuth, follow the provider’s flow and its documented client integration. Keep secrets in environment variables or a deployment secret store, never in source control, URLs, logs, or exception messages.

Cookies, sessions, proxies, and TLS

session = requests.Session()
session.headers.update({'Accept': 'application/json'})
session.cookies.update({'session_id': os.environ['SESSION_ID']})
response = session.get('https://api.example.com/v1/me', timeout=(3.05, 20))
response.raise_for_status()

A Session persists cookies and reuses pooled connections. A timeout tuple separates connection and read limits. Keep TLS verification enabled; configure a proxy only when your network requires it.

3. Read and validate the response

Check status before parsing. A server can return valid JSON describing an error, and a 204 response can have no JSON body.

response = requests.get(url, timeout=10)
print(response.status_code)
print(response.headers.get('content-type'))
response.raise_for_status()
result = None if response.status_code == 204 or not response.content else response.json()

Status classes are 1xx informational, 2xx successful, 3xx redirection, 4xx client error, and 5xx server error. Inspect response bodies and provider request-ID headers when available. JSON decoding is a separate operation and can fail after a successful status.

4. Use a reusable API client

import os
import requests

class ApiError(RuntimeError):
    pass

class ApiClient:
    def __init__(self, base_url, token):
        self.base_url = base_url.rstrip('/')
        self.http = requests.Session()
        self.http.headers.update({'Accept': 'application/json', 'Authorization': f'Bearer {token}'})

    def get_json(self, path, params=None, timeout=(3.05, 20)):
        try:
            response = self.http.get(f'{self.base_url}/{path.lstrip("/")}', params=params, timeout=timeout)
            response.raise_for_status()
            return None if not response.content else response.json()
        except requests.exceptions.JSONDecodeError as exc:
            raise ApiError('The API returned invalid JSON') from exc
        except requests.exceptions.RequestException as exc:
            raise ApiError(f'API request failed: {exc}') from exc

client = ApiClient('https://api.example.com/v1', os.environ['API_TOKEN'])
print(client.get_json('items', params={'limit': 10}))

Centralizing the base URL, headers, timeout, error handling, and session makes behavior consistent. Redact credentials and sensitive data before logging.

5. Pagination, uploads, and downloads

Pagination

Pagination is not uniform. An API may use pages, offsets, cursors, continuation tokens, or links. Implement its documented signal:

items = []
cursor = None
while True:
    params = {'limit': 100}
    if cursor:
        params['cursor'] = cursor
    page = client.get_json('items', params=params)
    items.extend(page['items'])
    cursor = page.get('next_cursor')
    if not cursor:
        break

Uploads and downloads

with open('report.csv', 'rb') as handle:
    response = requests.post('https://api.example.com/v1/imports', files={'file': ('report.csv', handle, 'text/csv')}, timeout=60)
response.raise_for_status()

with requests.get('https://api.example.com/v1/archive.zip', stream=True, timeout=60) as response:
    response.raise_for_status()
    with open('archive.zip', 'wb') as output:
        for chunk in response.iter_content(chunk_size=1024 * 1024):
            if chunk:
                output.write(chunk)

6. Retries, reliability, and performance

Use finite timeouts on every call. RFC 9110 defines GET, HEAD, OPTIONS, and TRACE as safe and also describes PUT and DELETE as idempotent. A POST that creates a record or triggers payment may not be safe to repeat. A network disconnect does not prove that a mutation was not applied. Prefer an API-provided idempotency key or operation-status lookup.

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(total=3, connect=3, read=3, status=3, backoff_factor=0.5, status_forcelist=(429, 502, 503, 504), allowed_methods=frozenset({'GET', 'HEAD', 'OPTIONS'}), respect_retry_after_header=True)
session = requests.Session()
session.mount('https://', HTTPAdapter(max_retries=retry))

This retries only selected safe methods and honors Retry-After when supported. Respect rate limits, use bounded concurrency, and cache permitted GET responses. Sessions reuse connections, but no comparative benchmark establishes a universal performance winner.

Cost and quota discipline

Read quota and billing rules before adding retries.
Request only needed fields and page sizes.
Cache stable GET responses and avoid unnecessary polling.
Record status, latency, and request IDs without recording tokens or sensitive bodies.

7. Troubleshooting checklist

Symptom	Cause	Fix
401 or 403	Missing, expired, wrongly formatted credentials, or insufficient scope	Recheck the documented scheme, token scope, and required header.
400 or 422	Wrong method, parameter, content type, or JSON shape	Compare the serialized request with the provider example.
404	Wrong path, version, or resource ID	Verify the exact URL and API version.
429	Rate limit or quota exceeded	Honor `Retry-After`, reduce concurrency, and check quota.
5xx	Provider-side failure	Retry only safe operations with bounded backoff.
Timeout	Slow server, DNS, or network	Set connect/read timeouts and retry safe calls sparingly.
JSON decode error	Empty body, HTML error, or non-JSON content	Check status and `Content-Type` before parsing.
Works in curl, fails in Python	Different headers, encoding, proxy, or body	Compare the complete request and use `params`/`json`.

8. Command-line comparison

curl --fail-with-body --connect-timeout 5 --max-time 30 \
  -H 'Accept: application/json' \
  -H "Authorization: Bearer $API_TOKEN" \
  --get 'https://api.example.com/v1/items' \
  --data-urlencode 'limit=10'

import os, requests
r = requests.get('https://api.example.com/v1/items', params={'limit': 10}, headers={'Accept': 'application/json', 'Authorization': f"Bearer {os.environ['API_TOKEN']}"}, timeout=30)
r.raise_for_status()
print(r.json())

const q = new URLSearchParams({ limit: '10' });
const res = await fetch(`https://api.example.com/v1/items?${q}`, {headers: {Accept: 'application/json', Authorization: `Bearer ${process.env.API_TOKEN}`}, signal: AbortSignal.timeout(30000)});
if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
console.log(await res.json());

9. Or skip the browser setup: ScreenshotNeo

If the API interaction you need is a website screenshot, ScreenshotNeo provides one GET request that returns PNG, JPEG, WebP, or PDF. It accepts consent banners as a visitor and removes 60+ known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the result with X-Page-Verdict and X-Billed headers.

See the ScreenshotNeo API documentation:

import requests
r = requests.get('https://api.screenshotneo.com/v1/shot', params={'access_key': 'YOUR_API_KEY', 'url': 'https://stripe.com'}, timeout=90)
open('shot.webp', 'wb').write(r.content)

curl -G 'https://api.screenshotneo.com/v1/shot' -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Options include full-page lazy-image loading, CSS-selector element capture, dark mode, 12 device presets or any viewport, retina scale, PDF paper size/margins/landscape/page ranges, HTML/CSS to image, custom CSS and JavaScript, pre-capture clicks, hidden selectors, selector/delay/network-idle waits, ad/tracker/request/resource blocking, custom headers/cookies/user agent/Authorization, timezone and geolocation, transparent backgrounds, resizing, a chosen cache TTL, signed image links, async jobs with signed webhooks, bulk capture of 100 URLs per call, a usage API, and an OpenAPI spec. Parameter names used by other screenshot APIs also work. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.

Plans include 1,000 shots per month free with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

10. FAQ

Should I use Requests or urllib?

Use urllib.request when dependencies are restricted. Choose Requests when sessions, concise JSON calls, authentication helpers, and familiar exceptions help.

Can valid JSON mean the call failed?

Yes. Check status before parsing because error responses can also be valid JSON.

What timeout should I choose?

Set finite connect and read limits based on the endpoint’s expected latency and your application’s deadline.

Can I retry every failed request?

No. Retry only safe operations, or use the provider’s idempotency mechanism and status lookup for mutations.

How do I handle pagination?

Follow that API’s documented page, offset, cursor, continuation-token, or link convention.