Pool Map With Multiple Arguments in Python

Pool Map With Multiple Arguments in Python

  1. Parallel Function Execution Using the pool.map() Method
  2. Parallel Function Execution With Multiple Arguments Using the pool.starmap() Method

This article will explain different methods to perform parallel function execution using the multiprocessing module in Python.

The multiprocessing module provides the functionalities to perform parallel function execution with multiple inputs and distribute input data across different processes.

We can parallelize the function’s execution with different input values by using the following methods in Python.

Parallel Function Execution Using the pool.map() Method

The pool.map(function, iterable) method returns an iterator that applies the function provided as input to each item of the input iterable. Therefore, if we want to perform parallel execution of the function with different inputs, we can use the pool.map() method.

The below example code demonstrates how to use the pool.map() method to parallelize the function execution in Python.

from multiprocessing import Pool

def myfunc(x):
    return 5 + x

if __name__ == '__main__':
    with Pool(3) as p:
        print(p.map(myfunc, [1, 2, 3]))

Output:

[6, 7, 8]

If the input function has multiple arguments, we can execute the function in parallel using the pool.map() method and partial() function with it.

The below example demonstrates how to parallelize the function execution with multiple arguments using the pool.map() in Python.

from multiprocessing import Pool
from functools import partial

def multiply(x, y):
    print(x*y)

if __name__ == '__main__':
    with Pool(3) as p:
      p.map(partial(multiply, 5), [1, 2, 3])

Output:

5
10
15

As can be noticed in the above example, the shortcoming of this method is that we can not change the value of the first argument.

Parallel Function Execution With Multiple Arguments Using the pool.starmap() Method

If we want to execute a function parallel with multiple arguments, we can do so using the pool.starmap(function, iterable) method.

Like the pool.map(function, iterable) method, the pool.starmap(function, iterable) method returns an iterator that applies the function provided as input to each item of the iterable. Still, it expects each input item iterable to be arranged as input function argument iterables.

By using the pool.starmap() method we can provide different values to all arguments of the function, unlike the pool.map() method.

We can perform parallel function execution with multiple arguments in Python using the pool.starmap() method in the following way.

from multiprocessing import Pool

def print_name(name, lname):
    print('full name =', name, lname)

if __name__ == '__main__':
    with Pool(3) as p:
      p.starmap(print_name, [('Thomas', 'Scott'), ('Ali', 'Khan')])

Output:

full name = Thomas Scott
full name = Ali Khan

Related Article - Python Multiprocessing

  • Python Multiprocessing Shared Object
  • Python Multiprocessing Logging
  • Multiprocessing vs Threading in Python