Parallel for Loop in Python

1. Use the multiprocessing Module to Parallelize the for Loop in Python
2. Use the joblib Module to Parallelize the for Loop in Python
3. Use the asyncio Module to Parallelize the for Loop in Python

Parallelizing the loop means spreading all the processes in parallel using multiple cores. When we have numerous jobs, each computation does not wait for the previous one in parallel processing to complete. Instead, it uses a different processor for completion.

In this article, we will parallelize a for loop in Python.

Use the multiprocessing Module to Parallelize the for Loop in Python

To parallelize the loop, we can use the multiprocessing package in Python as it supports creating a child process by the request of another ongoing process.

The multiprocessing module could be used instead of the for loop to execute operations on every element of the iterable. It’s multiprocessing.pool() object could be used, as using multiple threads in Python would not give better results because of the Global Interpreter Lock.

For example,

import multiprocessing


def sumall(value):
    return sum(range(1, value + 1))


pool_obj = multiprocessing.Pool()

answer = pool_obj.map(sumall, range(0, 5))
print(answer)

Output:

0, 1, 3, 6, 10

Use the joblib Module to Parallelize the for Loop in Python

The joblib module uses multiprocessing to run the multiple CPU cores to perform the parallelizing of for loop. It provides a lightweight pipeline that memorizes the pattern for easy and straightforward parallel computation.

To perform parallel processing, we have to set the number of jobs, and the number of jobs is limited to the number of cores in the CPU or how many are available or idle at the moment.

The delayed() function allows us to tell Python to call a particular mentioned method after some time.

The Parallel() function creates a parallel instance with specified cores (2 in this case).

We need to create a list for the execution of the code. Then the list is passed to parallel, which develops two threads and distributes the task list to them.

See the code below.

from joblib import Parallel, delayed
import math


def sqrt_func(i, j):
    time.sleep(1)
    return math.sqrt(i ** j)


Parallel(n_jobs=2)(delayed(sqrt_func)(i, j) for i in range(5) for j in range(2))

Output:

[1.0,
 0.0,
 1.0,
 1.0,
 1.0,
 1.4142135623730951,
 1.0,
 1.7320508075688772,
 1.0,
 2.0]

Use the asyncio Module to Parallelize the for Loop in Python

The asyncio module is single-threaded and runs the event loop by suspending the coroutine temporarily using yield from or await methods.

The code below will execute in parallel when it is being called without affecting the main function to wait. The loop also runs in parallel with the main function.

import asyncio
import time


def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped


@background
def your_function(argument):
    time.sleep(2)
    print("function finished for " + str(argument))


for i in range(10):
    your_function(i)


print("loop finished")

Output:

ended execution for 4
ended execution for 8
ended execution for 0
ended execution for 3
ended execution for 6
ended execution for 2
ended execution for 5
ended execution for 7
ended execution for 9
ended execution for 1