How to Use Cookies in Python Requests

1. Use requests.get() to Get Cookies in Python
2. Use Session() Class to Stay Connected With Pages After Login

This tutorial demonstrates the use of requests.get() to get cookies with the help of the cookies parameter in Python and how to access a particular web page that requires a login.

Further, we will also learn the use of requests.Session() to stay connected with that page.

Use requests.get() to Get Cookies in Python

The requests library lets create and read HTTP cookies from the server. According to the MDN documentation, a cookie is “a small piece of data that a server sends to the user’s web browser.”

We can store the cookie in the browser and send it back to the server. Cookies typically serve one of the following purposes:

• Session management for login
• Shopping carts
• User personalization, or tracking user behavior

This section demonstrates how to work with cookies, and in the next section, we will use the first purpose (session management for login).

To send and receive cookies with the requests library, we will start by importing requests. Following the import, we will set a variable named URL to pass the link as a string.

We will use this URL because it provides a simple response body to see the cookie information we sent to the server. For example, let’s say we want to customize our user experience based on a user’s location.

This way, we can provide shipping estimates for our products without the user having to enter an address. To do this, we will request to know the user’s location in the cookie, and if they agree, we will store the information in a cookie.

To mimic the content received in this event, we will declare a dictionary called CK with the key of location and a value is New York.

Next, we will declare our response object response and set it using the requests.get() method. We will enter our URL variable as the first parameter, followed by the cookies keyword that will be set equal to our CK dictionary.

Now that the request is set up, we can print the text from the response object using response.text. When a script is run, we can see the cookie we just sent is within the response body.

import requests

URL = "https://httpbin.org/cookies"
CK = {"location": "New York"}
response = requests.get(URL, cookies=CK)
print(response.text)

Output:

{
  "cookies": {
    "location": "New York"
  }
}

To receive a cookie from a server, we will use a cookie already set up on the google website.

Use Session() Class to Stay Connected With Pages After Login

Using the inspect element tool in our browser, we can see where the login request is sent and mimic that in our program. And the session part allows us to stay alive and access all the pages behind the login.

Before writing our code, we must do a few things; however, we need to find the login URL and see what parameters are sent with that post request.

We need the login credentials, so we will share the login information with you in this example because we are using a dummy site.

We have a simple login form with a username and password required; if we log into this using the given information, we will be logged in and get a secure area.

We will see the request and response by using the inspect element and going to the network tab. We can see here if we click the Login button, several requests pops-up.

Before login into the page, you should check the Preserve log so that we can see everything in the network tab. After opening the authenticate request, we will get some information, and one of them is a post request.

A post request is a request sent to the server from the web browser, and a get request is the information coming back to find out the URL that is being posted.

We will copy the post request URL because we will send our post request from our python script. We can find our form data inside the payload.

Sometimes you find more information inside the payload, so you must ensure that all parameters go along with the post request.

But, in our case, we see there is only a username and password. So, after login, we got directed back to the secure page, and this is our get request; by using a secure response, we will copy the request URL to use in our python script.

Now we will jump into the code editor and start writing our code, the first thing we need to do, as always, is import requests, and we need to set our login URL for the post request and secure URL for get request.

Now we need to call the post() method to send the username & password and get authenticated with the server. To do that, we need to send some payload using a dictionary called ACCESS_DATA.

We wrapped up the code inside the Login_Dummy_web() and created another python file called data.py.

After importing the data file, we accessed the username and password from this file while calling the function and returning the RESULT.text of the secure page.

Let’s try to get back to the secure page using the get() method; it will send us back to a login page because we are not authenticated.

import requests
import data


def Login_Dummy_web(username, password):
    LOGIN_URL = "https://the-internet.herokuapp.com/authenticate"
    SECURE_URL = "https://the-internet.herokuapp.com/secure"

    ACCESS_DATA = {"username": username, "password": password}

    RESULT = requests.post(LOGIN_URL, data=ACCESS_DATA)
    RESULT2 = requests.get(SECURE_URL)
    return RESULT2.text


print(Login_Dummy_web(data.username, data.password))

Output:

We already logged in using the post() method, but still, it denies use to send the secure page because we do not have our session. The session allows us to persist cookies for a particular website; to get a secure page, we need to use requests.Session().

Using the with keyword, we use context manager; context manager is handy because it will allow us to stay connected and stay logged in as long as we remain.

Now we have to call get() and post() methods from the session variable called ss because now we are accessing it from the Session() class.

import requests
import data


def Login_Dummy_web(username, password):
    LOGIN_URL = "https://the-internet.herokuapp.com/authenticate"
    SECURE_URL = "https://the-internet.herokuapp.com/secure"

    ACCESS_DATA = {"username": username, "password": password}
    with requests.Session() as ss:
        ss.post(LOGIN_URL, data=ACCESS_DATA)
        RESULT = ss.get(SECURE_URL)
    return RESULT.text


print(Login_Dummy_web(data.username, data.password))

Now it says (Welcome to the Secure Area), which proves we have logged in there; using the requests.Session() variable called ss, we can get another URL content.

The code of the data.py file is given below:

username = "tomsmith"
password = "SuperSecretPassword!"

For more details, you can visit this website.