HTTP or Hypertext Transfer Protocol is the foundation of the internet or the World Wide Web (WWW). Essentially, it is an application-layer protocol meant for transferring data from web servers to web browsers such as Chrome, Firefox, and Brave. HTTP requests are sent to unique web addresses or URLs to fetch required resources and hypermedia documents such as HTML (Hypertext Markup Language). A URL or Uniform Resource Locator is a unique web address used to locate content over the internet. The content targetted can be HTML files and text files, images, audios, videos, zip files, and executables.
In this guide, we will learn the ways to fetch HTML at a URL with the help of HTTP requests using Python.
Send HTTP Requests With
requests Module in Python
requests is a Python package that allows us to send HTTP requests over the internet. We can use this module to send HTTP requests to a URL and fetch HTML and other essential details. The
requests module has a
get() method that sends a GET request to the specified URL and returns a
requests.Response type object. We will use this method and the returned response to get HTML content. Refer to the following code for the Python program.
import requests r = requests.get("https://www.google.com") print("Status Code:", r.status_code) print("URL:", r.url) print("HTML:\n", r.text)
Status Code: 200 URL: https://www.google.com/ HTML: ...
In the above code, we first made a GET request to
https://www.google.com, and the response returned was stored in the variable
text attribute in the response returns the HTML content. The output does not show the HTML content, but
... because it was too big for better readability of the article.