Access Secured Pages in Python with httplib2

To access an HTTP-authenticated page with httplib2, create an httplib2.Http client, register credentials with add_credentials(), and call request() with the page URL and method. For a Basic-authenticated HTTPS endpoint, the minimal pattern is:

import httplib2

http = httplib2.Http()
http.add_credentials("your_username", "your_password")
response, content = http.request(
    "https://example.org/protected",
    "GET",
)

print(response.status)
print(content.decode("utf-8", errors="replace"))

This GET example adapts the pattern in the httplib2 documentation, which demonstrates an HTTPS Basic-authenticated PUT request. It applies when the server uses an HTTP authentication scheme supported by httplib2. It does not automate a website’s HTML login form, browser session, OAuth authorization flow, or other interactive sign-in process.

1. Confirm what kind of protection the endpoint uses

“Secured page” can mean several different things. Before writing client code, identify the mechanism expected by the server or API documentation:

Mechanism	What it means	Does add_credentials() apply?
HTTP Basic, Digest, or WSSE	The server challenges the HTTP request using an authentication scheme.	Yes. These are the authentication types identified by httplib2’s project documentation.
HTML form login	A page accepts form fields and may establish a cookie session or require CSRF tokens.	No general form-login flow is established by add_credentials().
OAuth or bearer token	The service expects a token obtained or issued through its authorization process.	Do not assume username/password challenge handling is the right mechanism. Follow the service’s token instructions.
Client TLS certificate	The TLS connection requires a certificate from the client.	No. This is separate from HTTP credentials; httplib2 documents an add_certificate(key, cert, domain) helper.

Use only credentials and permissions issued for the resource. An authentication helper cannot grant access the account does not have, and this guidance is for endpoints you are authorized to use.

2. Install httplib2 and prepare the request

Install the package into the Python environment that will run the script:

python -m pip install httplib2

PyPI’s listing at the research date identifies httplib2 0.32.0, released June 26, 2026, and requires Python 3.8 or newer. Package metadata can change; consult the current PyPI project page for the version and interpreter requirements applicable to your environment.

Save the minimal example as fetch_protected.py, replace the URL and credentials, then run python fetch_protected.py. The request returns a response object and a byte string. The status is available as response.status; the body is content. Decode text only when the resource is actually textual and use an appropriate encoding for that response.

3. Understand the authentication challenge

With HTTP Basic authentication, the server can first respond with status 401 and a WWW-Authenticate header that identifies Basic authentication and a realm. The realm describes the protected area. The client then retries with credentials for that challenge. This is the challenge-response flow described in the Python Basic Authentication HOWTO.

Register credentials before making the request so the client can use them when the server challenges. The helper’s documented shape is add_credentials(name, password[, domain]). The optional domain argument scopes where credentials are used; consult the current httplib2 docs for its precise behavior and use it when you need to constrain credential scope.

The minimum example assumes the server’s authentication scheme and credential format match the credentials supplied. If the server requires Digest or WSSE, verify the endpoint’s instructions and use the supported scheme it expects. Do not interpret a 401 as proof that a browser login is needed: inspect the response headers and server documentation first.

4. Check the response instead of assuming success

A completed HTTP request is not necessarily an authorized page load. Inspect the status and relevant headers before treating the body as the protected content:

import httplib2

http = httplib2.Http()
http.add_credentials("your_username", "your_password")
response, content = http.request("https://example.org/protected", "GET")

print("HTTP status:", response.status)
print("Content-Type:", response.get("content-type"))
print("WWW-Authenticate:", response.get("www-authenticate"))

if response.status == 200:
    print(content.decode("utf-8", errors="replace"))
elif response.status == 401:
    print("Authentication was not accepted or the challenge was not satisfied.")
elif response.status == 403:
    print("The server refused access to this resource.")
else:
    print("The server returned another status; inspect headers and body.")

Exact status behavior depends on the server. A 401 commonly signals that authentication is missing or rejected; a 403 commonly means the server understood the request but does not permit it. Applications can use statuses differently, so consult the endpoint’s documentation. Avoid printing response bodies or headers into production logs without checking for sensitive data.

5. Keep credentials and transport safe

• Use an https:// URL for requests carrying credentials. The official httplib2 example combines Basic authentication with HTTPS. Do not send passwords over plain HTTP.
• Load secrets from a secret manager or protected environment configuration rather than committing literal passwords to source control. Do not echo secrets into logs, shell history, exception reports, or shared notebooks.
• Keep credential scope as narrow as the library and endpoint allow. Use the optional domain argument where suitable and verify behavior against the current documentation.
• Do not disable TLS certificate verification to work around connection errors. The reviewed project material does not establish detailed current certificate-validation defaults or CA setup; check the current project docs and your deployment’s TLS requirements.
• Use the authentication method the service documents. Do not use credentials to evade bot checks, CAPTCHAs, access controls, or account restrictions.

6. Practical variations and boundaries

Use a different HTTP method

The same client and credential setup can be used with the method required by the endpoint. The project docs show a Basic-authenticated HTTPS PUT; for example, adapt the request call as follows when the service explicitly expects PUT:

response, content = http.request(
    "https://example.org/resource",
    "PUT",
    body=b"payload",
    headers={"Content-Type": "application/octet-stream"},
)

This is illustrative: select the method, body format, and headers from the endpoint contract. The library documentation lists support for arbitrary HTTP methods, but an endpoint can still reject a method it does not allow.

Use a client certificate only when TLS requires it

A client certificate participates in TLS negotiation, while Basic, Digest, and WSSE are HTTP-level authentication mechanisms. The httplib2 docs list add_certificate(key, cert, domain) separately from add_credentials(). If a service requires mutual TLS, follow its certificate and key instructions; registering a username and password does not replace the client certificate.

Do not treat a browser page as a raw authenticated API response

A site may return HTML, redirect to an identity provider, require JavaScript, or maintain state in cookies. The documented credential helper covers HTTP authentication challenges. It does not establish that httplib2 will execute browser JavaScript, submit a login form, or complete a multi-step identity flow. Use the service’s supported API or authentication client for those cases.

7. Troubleshooting

Symptom	Likely cause	What to check or change
401 Unauthorized	Wrong username or password, unsupported/mismatched scheme, or credentials not sent for the challenge.	Inspect WWW-Authenticate, confirm the documented scheme and realm, check for typos, and register credentials before request(). Confirm that the account is allowed to use this endpoint.
403 Forbidden	The server refuses this account or request even if authentication succeeded.	Check account permissions, resource-level policy, required scopes, and endpoint rules with the service owner.
Redirect to a login page	The URL may use form-based sign-in or an identity-provider flow rather than HTTP Basic/Digest/WSSE.	Check the final response URL and service documentation. Use the documented session or authorization flow; do not keep resending a password to an unrelated login page.
SSL or certificate error	Transport trust configuration, hostname mismatch, expired certificate, or a required client certificate may be involved.	Check the URL hostname, system trust store, certificate validity, and whether the endpoint requires mutual TLS. Do not turn off certificate checks as a shortcut.
Works in a browser, fails in Python	The browser may already have a cookie session, use an interactive sign-in, or send other required headers.	Determine the actual authentication mechanism from the endpoint owner. A successful browser session does not imply HTTP challenge authentication.
Empty or unexpected content	The request may have reached an error page, an unexpected resource, or a response encoded differently than assumed.	Check status, content type, redirect behavior, and a safely inspected portion of the body before decoding or parsing.
Method not allowed	The endpoint does not accept the chosen HTTP method.	Use the method documented by the server. httplib2 supports arbitrary methods at the client level, but the server defines what is valid.

8. Performance, reliability, and cost

httplib2’s project documentation lists connection keep-alive, caching, safe GET redirects, and gzip/deflate compression among its HTTP client features. Those features can help with repeated requests or compressed responses, but they do not guarantee a particular latency, cache policy, retry behavior, or availability. The reviewed sources provide no benchmark or uptime figure, so plan and measure against the actual endpoint and workload.

For a reliable integration, handle non-success statuses explicitly, keep timeouts and retry policy aligned with the needs of your application and the current library API, and avoid blindly retrying requests that may have side effects. The sources reviewed for this article do not establish exact timeout defaults or retry semantics; verify them in the current documentation before relying on them. Consider whether cached responses are acceptable for protected or frequently changing data, and follow the server’s cache directives and your organization’s data-handling rules.

httplib2 is an installable Python software library; PyPI does not describe it as a hosted per-request service in the cited package metadata. Your direct costs therefore depend on your runtime, infrastructure, network usage, and the service you call. The sources do not provide a universal cost estimate.

9. If the job is a screenshot, use a capture API

HTTP authentication retrieves an HTTP response; it does not render a page in a browser or produce a screenshot. If your actual task is to capture a public page, you can use a browser automation stack and manage browser installation, rendering waits, and image output yourself. For a screenshot without that setup, ScreenshotNeo is a website screenshot API and MCP server from Yorker Media. Its API accepts a URL in one GET request and returns PNG, JPEG, WebP, or PDF. This is for pages the service can access; do not assume it signs into arbitrary protected pages on your behalf.

Or skip the browser setup

Make one GET request with your ScreenshotNeo API key and target URL. See the ScreenshotNeo API documentation for request options.

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

• Cookie and consent banners, newsletter popups, and chat widgets are removed before the shot; each cleanup step can be turned off.
• Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed. Responses identify page verdict and billing status in headers.
• An MCP server gives AI agents tools to take screenshots, get page information, and capture PDFs.
• The Free plan includes 1,000 screenshots a month with no card. Paid plans start at $5 for 3,000 screenshots; every feature is on every plan.

Sign up free for 1,000 screenshots a month, with no card required.

10. FAQ

Can httplib2 make an authenticated HTTPS GET request?

Yes. The documented client, credential registration, and request pattern can be adapted to GET, as shown above. The endpoint must use an HTTP authentication mechanism the library supports.

Does adding credentials make a website login work?

Only when the server uses a supported HTTP authentication challenge. It does not establish a general solution for forms, cookies, OAuth, or JavaScript-driven sign-in.

Does add_credentials() configure certificate authentication?

No. The project documents a separate add_certificate() helper for an SSL client certificate. Follow the service’s mutual TLS instructions when that is required.

Where can I find the current httplib2 version?

Check the PyPI project listing and the project documentation. Version and interpreter requirements may change after the dated package context noted above.