How to Read Stdout While Running in Python Subprocess

1. Python Subprocess Read stdout While Running
2. Approach 1: Use check_call to Read stdout of a subprocess While Running in Python
3. Approach 2: Poll the Process to Read stdout of a subprocess While Running in Python

The main aim of this article is to demonstrate how to read the stdout of a subprocess being executed in Python.

Python Subprocess Read stdout While Running

As with many other built-in modules, Subprocess is also a built-in module that comes pre-installed with a “normal” Python installation.

It is used mostly when you want to run tasks, processes, and programs in a new process, perform a specific set of tasks and return the result.

One of its many reasons for wide usage is that it allows the execution of external programs and executables right from the program as a separate process.

While executing a program using the Subprocess library, it may be required that the output of that external program be shown in real-time. This can be a requirement for plenty of reasons, such as when a program may be real-time and depends on calculations after a short amount of time.

Consider the following program:

import subprocess


def execute(command):
    process = subprocess.Popen(
        command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
    )
    output = process.communicate()[0]
    exitCode = process.returncode

    if exitCode == 0:
        return output
    else:
        raise Exception(command, exitCode, output)


if __name__ == "__main__":
    print(execute("cd C:\\ && C: && tree").decode("unicode_escape"))

Output:

Folder PATH listing
Volume serial number is 0213-B7F2
C:.
+---DRIVERS
¦   +---bluetooth
¦       +---U1BLT07AVVASDSAP
¦           +---Custom
¦           ¦   +---EULA
¦           +---Win64
¦               +---LD
¦               +---svcpack
+---flutter
¦   +---.git
¦   ¦   +---hooks
¦   ¦   +---info
¦   ¦   +---logs
¦   ¦   ¦   +---refs
¦   ¦   ¦       +---heads
¦   ¦   ¦       +---remotes
¦   ¦   ¦           +---origin
¦   ¦   +---objects
¦   ¦   ¦   +---info
¦   ¦   ¦   +---pack
¦   ¦   +---refs
¦   ¦       +---heads
¦   ¦       +---remotes
¦   ¦       ¦   +---origin
¦   ¦       +---tags
¦   +---.github
¦   ¦   +---ISSUE_TEMPLATE
¦   ¦   +---workflows
¦   +---.idea
¦   ¦   +---runConfigurations
...

We can see the output in the program but not in real-time. The output will only be shown after the whole command (tree in our case) has finished executing.

The output will not show something until the program (or command) is executed in a separate process using the Subprocess command.

In our case, since it is required that we should get the output in real-time, we need to create another solution, the one which shows the output of the program as it is written to the stdout.

The solution can be approached in multiple ways, some of which are mentioned below.

Approach 1: Use check_call to Read stdout of a subprocess While Running in Python

Consider the following code:

import subprocess
import sys


def execute(command):
    subprocess.check_call(
        command, shell=True, stdout=sys.stdout, stderr=subprocess.STDOUT
    )


if __name__ == "__main__":
    print(execute("cd C:\\ && C: && tree").decode("unicode_escape"))

Output:

Folder PATH listing
Volume serial number is 0213-B7F2
C:.
├───DRIVERS
│   └───bluetooth
│       └───U1BLT0720US14CMP
│           ├───Custom
│           │   └───EULA
│           └───Win64
│               ├───LD
│               └───svcpack
├───flutter
│   ├───.git
│   │   ├───hooks
│   │   ├───info
│   │   ├───logs
│   │   │   └───refs
│   │   │       ├───heads
│   │   │       └───remotes
│   │   │           └───origin
│   │   ├───objects
│   │   │   ├───info
│   │   │   └───pack
│   │   └───refs
│   │       ├───heads
│   │       ├───remotes
│   │       │   └───origin
│   │       └───tags
│   ├───.github
│   │   ├───ISSUE_TEMPLATE
│   │   └───workflows
│   ├───.idea
│   │   └───runConfigurations
│   ├───.pub-cache
│   │   ├───hosted
│   │   │   └───pub.dartlang.org
│   │   │       ├───.cache
...

If the main requirement is to print the program’s output in real-time, check_call can be used to achieve this. This simple, clean and elegant solution is concise and “just perfect” for very simple programs, like when only the output needs to be printed.

The check_call also supports passing parameters, so if your program needs any arguments to work, they can be passed easily to the function without any hassle. This method waits for the program to complete.

Based on the completion of the program, the method returns; otherwise, it raises a CalledProcessError exception. The CalledProcessError will have the failure return code, which can be accessed using the returncode attribute.

In the code mentioned above, the command is passed to the check_call method, which contains the command (or the program) to be executed.

The shell parameter was set to true to execute the process using a shell, and the process’ stdout is set to the stdout of our program, so it writes directly to our stdout, and we can see the changes as they happen in our stdout.

Lastly, the stderr is set to the stdout of the spawned process.

Approach 2: Poll the Process to Read stdout of a subprocess While Running in Python

Consider the following code:

import subprocess
import sys


def execute(command):
    process = subprocess.Popen(
        command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
    )

    # Poll process for new output until finished
    while True:
        nextline = process.stdout.readline().decode("unicode_escape")
        if nextline == "" and process.poll() is not None:
            break
            sys.stdout.write(nextline)
            sys.stdout.flush()

        output = process.communicate()[0]
        exitCode = process.returncode

        if exitCode == 0:
            return output
        else:
            raise Exception(command, exitCode, output)


if __name__ == "__main__":
    print(execute("cd C:\\ && C: && tree").decode("unicode_escape"))

Output:

...
¦   ¦   ¦       ¦   ¦       +---UuidUtil
¦   ¦   ¦       ¦   +---example
¦   ¦   ¦       ¦   +---lib
¦   ¦   ¦       ¦   +---test
¦   ¦   ¦       +---vector_math-2.1.2
¦   ¦   ¦       ¦   +---benchmark
¦   ¦   ¦       ¦   +---bin
¦   ¦   ¦       ¦   +---lib
¦   ¦   ¦       ¦   ¦   +---src
¦   ¦   ¦       ¦   ¦       +---vector_math
¦   ¦   ¦       ¦   ¦       ¦   +---third_party
¦   ¦   ¦       ¦   ¦       +---vector_math_64
¦   ¦   ¦       ¦   ¦       ¦   +---third_party
¦   ¦   ¦       ¦   ¦       +---vector_math_geometry
¦   ¦   ¦       ¦   ¦       ¦   +---filters
¦   ¦   ¦       ¦   ¦       ¦   +---generators
¦   ¦   ¦       ¦   ¦       +---vector_math_lists
¦   ¦   ¦       ¦   ¦       +---vector_math_operations
¦   ¦   ¦       ¦   +---test
¦   ¦   ¦       ¦   +---tool
¦   ¦   ¦       +---video_player-2.2.11
¦   ¦   ¦       ¦   +---android
¦   ¦   ¦       ¦   ¦   +---gradle
...

To ensure that the program’s output spawned using the Subprocess is printed as soon as it is written to the stdout, we need to poll the process for output and continue reading the last line of the stdout of the program’s stdout.

In an infinite loop, we continue to read the output of the spawned process using readline(). Since the output is encoded, we need to decode it (utf-escape in our case) to be represented suitably.

This can be done using the decode method and passing the relevant encoding scheme.

Once the program has finished executing, we must also escape the “infinite” loop. We can do so by checking for two things:

1. The current stdout line is empty.
2. The process has terminated or died.

This can be done quite simply, with a simple string comparison for the first condition and the poll() method to check if the process has terminated or not. If the program has terminated, it returns the returncode; otherwise, it returns None.

A simple comparison with None can also give us information on whether the program has finished executing or not.