How to Set Maximum Retries for Requests in Python

  2. HowTo
  3. Python How-To's
  4. How to Set Maximum Retries for Requests …
Haider Ali Feb 02, 2024
Python Python Requests
How to Set Maximum Retries for Requests in Python

This tutorial describes why we get an error saying maximum retries exceeded and how we can set max_retries for requests in Python. It also provides us with tips if a load on the server causes this error.

Set max_retries for Requests Error, Its Causes & Solutions in Python

Before we go ahead and see how we can set max_retries for requests in Python, let’s address some of the issues first.

  • First, this error can occur if your URL is not correct. So, you need to check whether the URL you are requesting is valid or not.
  • This error can also occur due to the internet connection, so ensure you are clear of all such issues.
  • We also get this error when there’s a server overload. When the servers are busy, you can encounter this type of error.

That’s where the phenomenon of increasing the number of retries for requesting the URL comes in to assist us. So let’s learn how you can do that. But, first, take a look at the following code.

Example Code:

import requests

# increass retries number
retries = 4  # enter the number of retires you want to increase
requests.adapters.DEFAULT_RETRIES = retries

This is how you can increase the number of retries for a URL request in Python, but if you are getting this error due to a load on the server, you can use the following tips.

Note: You need to install the requests module in your Python using the following command:

python -m pip install requests

Tips to Get Rid of Set max_retries for Requests in Python

You can use any of the following tips if the load on the server causes this error.

  • Disable keep_alive

    You can disable the keep_alive functionality of the session. The following piece of code explains how you can do that.

    request_session = requests.session()
# disable keep_alive
request_session.keep_alive = False
request_session.get(your_url)
  • Use timeout

    When the server is overloaded, busy, or far away from the location, this can become the reason for you getting the same error. To avoid this, you can increase the time for response.

    You can do it with both the POST and GET requests from the server. For example, take a look at the following code.

    import requests

# the time in seconds in this example is increasing time by 5
req = requests.get(your_url, timeout=5)
req = requests.post(your_url, timeout=5)

    You can also use the tuple in the timeout parameters. The first parameter will increase the build time, and the second parameter will increase the response time. The code example is given below.

    req = requests.get(your_url, timeout=(3, 6))
  • Set a backoff_factor (Delay/Sleep)

    backoff_factor is a urllib3 argument used by requests to initially set up a network connection.

    You can also set up the backoff_factor to increase each attempt’s number of tries and sleep time.

    It will avoid getting this error. With this, you are stopping the piece of code and making it wait until it receives the server’s response.

    Here’s an illustration of how backoff_factor can delay service requests whenever one fails.

    # you can set a backoff factor means delay/sleep time in each retry
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

# initailize the request session
request_session = requests.Session()
# initailizing retry object
# you can increase the number of total retires and sleep time of each retry
retries = Retry(total=3, backoff_factor=1)
adapter = HTTPAdapter(max_retries=retry)
request_session.mount("http://", adapter)
request_session.get(your_url)

    According to the urllib3 documentation, backoff_factor is a base value that the library uses to determine the sleep interval between retries.

    For example, after each unsuccessful connection attempt, urllib3 will sleep for {backoff_factor} * (2 (number of total retries - 1)) seconds.

    For instance, sleep() will sleep for 0.0s, 0.2s, 0.4s,… between retries if the backoff_factor is set to 0.01. Backoff is inactive by default (set to 0).

    If the status code returned is 500, 502, 503, or 504, it will additionally demand a retry. To have even finer control over retries, you can modify the retry.

Author: Haider Ali
Haider Ali avatar Haider Ali avatar

Haider specializes in technical writing. He has a solid background in computer science that allows him to create engaging, original, and compelling technical tutorials. In his free time, he enjoys adding new skills to his repertoire and watching Netflix.

LinkedIn

Related Article - Python Requests