How to Get HTML With HTTP Requests in Python

HTTP or Hypertext Transfer Protocol is the foundation of the internet or the World Wide Web (WWW). Essentially, it is an application-layer protocol meant for transferring data from web servers to web browsers such as Chrome, Firefox, and Brave. HTTP requests are sent to unique web addresses or URLs to fetch required resources and hypermedia documents such as HTML (Hypertext Markup Language). A URL or Uniform Resource Locator is a unique web address used to locate content over the internet. The content targetted can be HTML files and text files, images, audios, videos, zip files, and executables.

In this guide, we will learn the ways to fetch HTML at a URL with the help of HTTP requests using Python.

Send HTTP Requests With requests Module in Python

The requests is a Python package that allows us to send HTTP requests over the internet. We can use this module to send HTTP requests to a URL and fetch HTML and other essential details. The requests module has a get() method that sends a GET request to the specified URL and returns a requests.Response type object. We will use this method and the returned response to get HTML content. Refer to the following code for the Python program.

import requests

r = requests.get("https://www.google.com")
print("Status Code:", r.status_code)
print("URL:", r.url)
print("HTML:\n", r.text)

Output:

Status Code: 200
URL: https://www.google.com/
HTML:
...

In the above code, we first made a GET request to https://www.google.com, and the response returned was stored in the variable r. The text attribute in the response returns the HTML content. The output does not show the HTML content, but ... because it was too big for better readability of the article.