Python Multiprocessing Logging

1. Multiprocessing
2. Multiprocessing in Python
3. Multiprocessing Logging in Python

This article will discuss the concept of multiprocessing. After this, we will discuss multiprocessing in Python and log handling for multiprocessing using Python code.

Multiprocessing

Multiprocessing is a computing paradigm in which more than one processor simultaneously processes different parts of the same program.

The application running in multiprocessing operating systems are divided into smaller subroutines that run independently by allocating into different processors to improve performance.

Multiprocessing has two different types:

1. Symmetric Multiprocessing: In this multiprocessing technique, a single operating system manages more than one computer processor with shared main memory. Generally, a computer system having multiprocessing power support symmetric multiprocessing.
2. Asymmetric Multiprocessing: In this multiprocessing, a computer processor is assigned to handle operating system-related tasks, and another computer processor is assigned application-related tasks. It is considered inefficient compared to symmetric multiprocessing because one processor may idle, and another may be completely busy in the same time interval in asymmetric multiprocessing.

Multiprocessing in Python

In Python, the multiprocessing library is used for multiprocessing tasks.

Consider the following example:

import multiprocessing


def func1(arg):
    print("func1: with parameter", arg)


def func2(arg):
    print("func2: with parameter", arg)


if __name__ == "__main__":
    process1 = multiprocessing.Process(target=func1, args=(1,))
    process2 = multiprocessing.Process(target=func2, args=(2,))

    process1.start()
    process2.start()

    process1.join()
    process2.join()
    print("Processes Ended")

In the above code, we use import multiprocessing to include the multiprocessing module. A Process class object is used to create a process.

Process class object receives target (A function to run in the process) as a parameter and args as the argument of the target function.

A start method of the Process class is used to start the process. In the above example, we started two processes.

We use the join method to keep executing the current program until process1 and process2 are not terminated.

Once process1 and process2 complete their tasks, the current program runs the print("Processes Ended") statement.

The output of the above program is as follows:

func1: with parameter 1
func2: with parameter 2
Processes Ended

Process ID in Python During Multiprocessing

We can also print the process ID (PID) using two different methods.

1. os.getpid()
2. Process class object member variable pid

Consider the following code:

import multiprocessing
import os


def func1(arg):
    print("func1: with parameter ", arg)
    print("ID of func1 process:", os.getpid())


def func2(arg):
    print("func2: with parameter ", arg)
    print("ID of func2 process:", os.getpid())


if __name__ == "__main__":
    process1 = multiprocessing.Process(target=func1, args=(1,))
    process2 = multiprocessing.Process(target=func2, args=(2,))

    process1.start()
    process2.start()

    print("Process 1 / function 1 PID: ", process1.pid)
    print("Process 2 / function 2 PID: ", process2.pid)
    process1.join()
    process2.join()
    print("Processes Ended")

In the above example, os.getpid() and process1.pid shows the process IDs. The output of the following code is as follows:

Process 1 / function 1 PID:  11368
Process 2 / function 2 PID:  14876
func1: with parameter  1
ID of func1 process: 11368
func2: with parameter  2
ID of func2 process: 14876
Processes Ended

The os.getpid() and process1.pid shows the same process ID.

Multiprocessing With Pool Class

A Pool class of Python multiprocessing module is used to parallelly execute the same function with different input values. For example, consider the following code:

import multiprocessing


def func1(arg):
    print("func1: with parameter ", arg)


if __name__ == "__main__":
    process_pool = multiprocessing.Pool(3)
    process_pool.map(func1, [1, 2, 3])
    process_pool.close()
    process_pool.join()

In the above code, multiprocessing.Pool creates three processes to call func1 with different arguments. The output of the following code is as follows:

func1: with parameter  1
func1: with parameter  2
func1: with parameter  3

Multiprocessing Logging in Python

We can log information from multiple processes using Python’s multiprocessing library. There are different methods for logging multiprocessing.

We can use logging.handlers, QueueHandler, and QueueListener classes in Python for logging.

Consider the following code:

import multiprocessing
import logging
import os
from logging.handlers import QueueHandler, QueueListener


def func(arg):
    logging.info(
        "Process/function with argument {} and PID {}".format(arg, os.getpid())
    )


def Process_init(q):
    queue_handler = QueueHandler(q)
    logger = logging.getLogger()
    logger.setLevel(logging.DEBUG)
    logger.addHandler(queue_handler)


if __name__ == "__main__":
    print("Main Started")
    mp_queue = multiprocessing.Queue()
    lg_handler = logging.StreamHandler()

    lg_handler.setFormatter(
        logging.Formatter("%(levelname)s: %(asctime)s - %(process)s - %(message)s")
    )

    queue_listener = QueueListener(mp_queue, lg_handler)
    queue_listener.start()

    process_pool = multiprocessing.Pool(2, Process_init, [mp_queue])
    process_pool.map(func, [1, 2])
    process_pool.close()
    process_pool.join()

    queue_listener.stop()
    print("Main Ended")

The above code defines a function func accepting one argument.

The func method logs a string having the argument value and the process ID. We get process ID using os.getpid().

In the main method, we create multiprocessing.Queue() and logging.StreamHandler() object for QueueListener. The handler.setFormatter is used to set the log format.

We create two processes using the multiprocessing.Pool object and call the func method in both processes with 1 and 2 as the argument value.

The Process_init method is used to initialize the queue_handler. The output of the above code is as follows:

The above code shows the log information for two processes having process ID 7024 and 15680.

We can also log the information into a log file by replacing lg_handler = logging.StreamHandler() statement with lg_handler = logging.FileHandler('info.log').

The logging.FileHandler creates a file info.log and stores the logs in the info.log file.