Python - Tail a Log File and Compare Blocking & Non-Blocking Tail Functions

1. Overview of Python tail() Function
2. Tail a Log File in Python
3. Blocking vs Non-Blocking Tail() Functions in Python
4. Log File for the Above Examples

Today’s tutorial provides an overview of the tail() function in Python that leads to its working and demonstration of how to tail a log file.

It also compares Python’s blocking and non-blocking tail functions and highlights the differences.

Overview of Python tail() Function

In Python, the data frame’s final five rows are shown by default when using the tail() function. There is only one input parameter, which is the number of rows.

This option allows us to display a specific amount of rows. Additionally, the tail() function also accepts a negative number, as shown in the below example.

In such a situation, all rows are returned but not the first rows. The primary distinction between the head() and the tail() is that, when empty parameters are passed, the head() and tail() both return five rows.

Remember that the head() and tail() functions produce ordered data, whereas a sample() produces unordered data.

Syntax of tail() Function:

dataframe.tail(n=5)

Working of the Tail() Function in Python

If we pass the negative value of n to the tail() function, it will exclude the first n (see the following example).

When we run print(df.tail(2)), it displays the last two rows, and when we execute print(df.tail(-6)), it shows all rows excluding the first 6.

Example Code:

import pandas as pd

df = pd.DataFrame(
    {
        "Colours": [
            "Purple",
            "White",
            "Black",
            "Brown",
            "Pink",
            "Orange",
            "Blue",
            "Red",
            "Yellow",
        ]
    }
)
print("Complete Data Frame:")
print(df)

print("When we pass the value of `n`:")
print(df.tail(2))

print("\nWhen we pass the -ve value of `n`:")
print(df.tail(-6))

Output:

Complete Data Frame:
   Colours
0  Purple
1   White
2   Black
3   Brown
4    Pink
5  Orange
6    Blue
7     Red
8  Yellow

When we pass the value of `n`:
   Colours
7     Red
8  Yellow

When we pass the -ve value of `n`:
   Colours
6    Blue
7     Red
8  Yellow

Tail a Log File in Python

We have created a log file named std.log in which we have stored some data, which we will see in the output below. To tail a log file, we can run the tail from the sh module.

To perform an infinite loop and display the output line, we call tail() with the file name and _iter set to True.

Example Code:

from sh import tail

for line in tail("-f", "std.log", _iter=True):
    print(line)

Output:

2022-08-25 21:44:10,045 This is just a reminder
2022-08-25 21:44:10,046 Meeting is at 2 pm
2022-08-25 21:44:10,046 After the lunch break

Blocking vs Non-Blocking Tail() Functions in Python

Let’s start with non-blocking tail() function.

Non-Blocking Tail() Function

When a function is blocked, it can potentially postpone the completion of subsequent activities. Additionally, we may suffer the system’s overall performance.

In other words, your program will block and prevent anything else from running. To launch the subprocess and connect to its output stream, we are utilizing the subprocess module (stdout).

Example Code:

import subprocess
import select
import time

f = subprocess.Popen(
    ["tail", "-F", "log.txt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
p = select.poll()
p.register(f.stdout)

while True:
    if p.poll(1):
        print(f.stdout.readline())
    time.sleep(1)

Output:

b'2022-08-25 21:44:10,045 This is just a reminder\n'

Blocking Tail() Function

The following code will also display new lines as they are added, but you may utilize the subprocess module without the additional select module calls. It blocks until the tail program is terminated with the f.kill() command.

Example Code:

import subprocess

f = subprocess.Popen(
    ["tail", "-F", "log.txt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
while True:
    line = f.stdout.readline()
    print(line)

Output:

b'2022-08-25 21:44:10,045 This is just a reminder\\n'
b'2022-08-25 21:44:10,046 Meeting is at 2 pm\\n'

Log File for the Above Examples

The following code is for creating log.txt, which is then used in the blocking and non-blocking code.

Example Code:

import logging

logging.basicConfig(
    filename="log.txt", level=logging.DEBUG, format="%(asctime)s %(message)s"
)
logging.debug("This is just a reminder")
logging.info("Meeting is at 2 pm")
logging.info("After the lunch break")

Output:

2022-08-25 21:44:10,045 This is just a reminder
2022-08-25 21:44:10,046 Meeting is at 2 pm
2022-08-25 21:44:10,046 After the lunch break