- Use the Processing Pool and Its Methods to Perform Multiprocessing in Python
apply_async()Function to Perform Multiprocessing in Python
map_sync()Functions to Perform Multiprocessing in Python
This article explains how to execute multiple processes in parallel in Python.
Use the Processing Pool and Its Methods to Perform Multiprocessing in Python
To perform multiprocessing in Python, do the following steps.
- First, import the
- Use the
Poolclass to define the Python Processing Pool. This Pool represents a pool of worker processes. Take the code snippet below as an example.
Pool class will allow the
n number of processes to run simultaneously. This class contains methods that enable the delegation of tasks to the Pool’s worker processes.
The followings are some of the methods of the
Pool.map()- Use this blocking method of the Pool to block until the result is ready. This method chops the associated one iterable into several chunks. Then, it submits the chunks to the process pool as separate tasks.
Use the following non-blocking methods of the
Poolclass to submit the processes simultaneously at once. Use the
get()method to obtain the function’s results.
To determine the number of processes that your machine can handle for multiprocessing, you should know the number of processors you have in your device; you can check this by using the following code.
import multiprocessing as mp print("Number of processors: ", mp.cpu_count())
Number of processors: 4
apply_async() Function to Perform Multiprocessing in Python
Here is an example demonstrating multiprocessing using the processing pool in Python while determining the square of a range on numbers from 0-9.
Note that here, we have even printed the Process worker’s name, which is optional.
from multiprocessing import Pool from multiprocessing import Process, current_process import time from time import sleep import random def fnc_square(x): print(current_process().name) return x * x if __name__ == "__main__": pool = Pool(processes=4) result = pool.apply_async(fnc_square, ) print(result.get(timeout=1)) print(pool.map(fnc_square, range(10)))
SpawnPoolWorker-1 36 SpawnPoolWorker-2 SpawnPoolWorker-1 SpawnPoolWorker-2 SpawnPoolWorker-1 SpawnPoolWorker-2 SpawnPoolWorker-1 SpawnPoolWorker-2 SpawnPoolWorker-1 SpawnPoolWorker-2 SpawnPoolWorker-1 [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
map_sync() Functions to Perform Multiprocessing in Python
map_async function is non-blocking, whereas the
map function is blocking. Below is an example that demonstrates the difference between these two commands.
from multiprocessing import Pool import time def f(x): print(x * x) if __name__ == "__main__": pool = Pool(processes=1) pool.map(f, range(10)) r = pool.map_async(f, range(10)) print("First print") print("Second print") r.wait() print("Third print")
Here, we can observe that the
pool.map() function will wait for the 10 operation calls to complete. Hence, we see the result printed in order.
pool.map_async() will execute the 10 function calls asynchronously. When the
r.wait() function is called, it blocks the processor for execution. Hence, in the output, we see the print messages as
First print and
Second print in between, but the
Third print will always be at the end.
0 1 4 9 16 25 36 49 64 81 First print Second print 0 1 4 9 16 25 36 49 64 81 Third print