HOWTO · Python
How to Solve Urllib HTTP Error 403 Forbidden Message in Python
This tutorial explains how to diagnose and handle HTTP 403 responses raised by Python's urllib module.
On this page
An HTTP 403 response means a server understood a request but refused to fulfill it. In Python, urllib.request.urlopen() reports that response as urllib.error.HTTPError. This is an HTTP response from the server, not evidence that urllib itself is broken. The status alone does not reveal why access was refused: possible reasons include an account or permission requirement, a site access policy, a blocked network address, or a request that the server does not accept. The site operator is the reliable source for the specific reason.
Understand what urllib.error.HTTPError: HTTP Error 403: Forbidden means
urllib.error.HTTPError is both an exception and a response-like object. A 403 differs from a temporary server failure: repeating an unchanged request is unlikely to help when the cause is authorization or policy. Do not assume a browser can access the same resource under the same conditions, or that changing Python clients will grant permission.
Check the server response and access policy
Inspect the exception’s status, headers, and body when the server provides them. The body may contain a useful explanation, but some servers return a generic page or no readable detail. Avoid printing sensitive response data into shared logs.
robots.txt is a crawler guidance file. It can tell a compliant crawler which paths a site asks it to avoid, but it is not an access-control system, does not grant permission, and does not prove why a 403 occurred. Check the site’s terms and obtain permission where required. This local example serves a small robots.txt file and checks a target URL with urllib.robotparser:
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.robotparser import RobotFileParser
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"User-agent: *\nDisallow: /private/\n"
self.send_response(200)
self.send_header("Content-Type", "text/plain; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
pass
server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
origin = f"http://127.0.0.1:{server.server_port}"
parser = RobotFileParser()
parser.set_url(f"{origin}/robots.txt")
parser.read()
allowed = parser.can_fetch("ExampleBot", f"{origin}/public/report.html")
print(allowed)
finally:
server.shutdown()
server.server_close()
thread.join()
An allowed result only reflects that file’s crawler guidance; it does not grant access or override the site’s terms, authorization requirements, or other policies.
Send an authorized request with the required headers
If the site permits the request, send the headers its documentation requires. A descriptive User-Agent identifies your client; Accept can state which response formats it understands. Use only credentials or cookies issued for your authorized access, and protect them from source control and logs. Headers and cookies do not guarantee access, and should not be changed to disguise automation or evade a restriction. If permission is unclear, stop and contact the site owner instead of trying different identities or proxies.
The following self-contained example starts a local server that grants a request, then fetches that response with urllib. It demonstrates request construction without relying on a third-party service. Run this example; the local server prints:
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.request import Request, urlopen
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"Access granted locally"
self.send_response(200)
self.send_header("Content-Type", "text/plain; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
pass
server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
request = Request(
f"http://127.0.0.1:{server.server_port}/",
headers={"User-Agent": "AuthorizedExample/1.0"},
)
with urlopen(request, timeout=5) as response:
print(response.status)
print(response.read().decode("utf-8"))
finally:
server.shutdown()
server.server_close()
thread.join()
200
Access granted locally
Handle HTTP errors and inspect the response
Catch urllib.error.HTTPError before the broader urllib.error.URLError when you need status-specific handling. Read only the response details you need, then close the error response. Handle other network failures separately: a timeout or name-resolution failure is not a 403 response. Do not automatically retry a 403. Retry only failures that are plausibly transient, with a limit and delay; for a refusal, check permissions and policy or ask the server operator for guidance. A retry loop cannot make an unauthorized request acceptable.
A controlled local refusal is a useful boundary case: the request reaches a server, but the server returns HTTP 403. This example reads the available body from HTTPError; it does not imply that changing a request can resolve a real site’s access policy.
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import urlopen
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"Access denied locally"
self.send_response(403)
self.send_header("Content-Type", "text/plain; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
pass
server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
try:
with urlopen(f"http://127.0.0.1:{server.server_port}/", timeout=5) as response:
print(response.status)
except HTTPError as error:
try:
print(error.code)
print(error.headers.get("Content-Type"))
print(error.read().decode("utf-8"))
finally:
error.close()
finally:
server.shutdown()
server.server_close()
thread.join()
403
text/plain; charset=utf-8
Access denied locally
When to use requests instead of urllib
requests.Session can be convenient when an application needs a persistent client session, shared headers, or a familiar API, provided the request is authorized. It does not bypass the server’s decision, and a 403 from requests has the same basic meaning as one from urllib. Choose the library that fits your application; for either client, inspect the response, follow the site’s rules, and stop when access is denied. Neither a different library nor a browser-like header is a general fix for an HTTP 403.