Chunksize in Pandas

The pandas library organizes data into DataFrames of rows and columns, and readers such as read_csv normally load a whole file into one DataFrame. When a file is too large to load comfortably — for example a dataset like the MovieLens ratings export with millions of rows — the chunksize parameter lets you read and process it in batches of rows instead.

How chunksize works in pandas

Passing chunksize=n to a supported pandas reader changes its return value. Instead of a DataFrame, read_csv() returns a TextFileReader, an iterator-like object that yields one DataFrame per batch. Each batch contains at most n rows, and the last batch can contain fewer when the row count is not an exact multiple of the chunk size.

Two details matter when adapting this pattern. First, chunksize counts rows, not bytes: every chunk keeps all of the file’s columns, and only the row count varies. Second, the parameter does not make parsing faster by itself. It changes when memory is allocated, so whether it helps depends on what your program keeps from each chunk.

Read a CSV file in chunks

To read a file in batches, pass chunksize to read_csv and loop over the result. The example below writes a small ratings-style CSV to a temporary file and then reads it back three rows at a time, printing the shape of each yielded DataFrame so you can see the batching.

Run this example:

import os
import tempfile

import pandas as pd

rows = ["userId,movieId,rating"]
for i in range(7):
    rows.append(f"{i + 1},{100 + i},{1.0 + (i % 5) * 0.5}")

path = os.path.join(tempfile.mkdtemp(), "ratings.csv")
with open(path, "w") as f:
    f.write("\n".join(rows) + "\n")

reader = pd.read_csv(path, chunksize=3)
for number, chunk in enumerate(reader, start=1):
    print(f"chunk {number}: shape={chunk.shape}")

chunk 1: shape=(3, 3)
chunk 2: shape=(3, 3)
chunk 3: shape=(1, 3)

The file holds seven data rows and chunksize=3, so the TextFileReader yields two full chunks followed by a one-row remainder. The column count stays at three for every chunk, confirming that batching applies to rows only.

Process each chunk

The useful pattern is to accumulate a small result per chunk — a count, a sum, or a filtered subset — rather than collecting the chunks themselves. This example aggregates the same file, tracking a running row count and rating total so the whole file never has to sit in memory at once.

Run this example:

import os
import tempfile

import pandas as pd

rows = ["userId,movieId,rating"]
for i in range(7):
    rows.append(f"{i + 1},{100 + i},{1.0 + (i % 5) * 0.5}")

path = os.path.join(tempfile.mkdtemp(), "ratings.csv")
with open(path, "w") as f:
    f.write("\n".join(rows) + "\n")

total_rows = 0
rating_sum = 0.0
for chunk in pd.read_csv(path, chunksize=3):
    total_rows += len(chunk)
    rating_sum += chunk["rating"].sum()

print(f"rows read: {total_rows}")
print(f"average rating: {rating_sum / total_rows:.2f}")

rows read: 7
average rating: 1.79

Any DataFrame operation works inside the loop — filtering, column selection, or groupby aggregations — so you can transform each chunk before folding it into the accumulator.

Understand chunk size and memory tradeoffs

Reading in chunks can reduce peak memory, but it is not automatic. Peak usage is roughly the size of the largest chunk plus whatever your program retains. If you append every chunk to a list and concatenate them afterward, you hold the entire dataset again and can use more memory than a single plain read.

Choose the value by balancing two pressures. A larger chunksize means fewer loop iterations and less per-iteration overhead, but each chunk occupies more memory. A smaller value lowers the peak at the cost of more passes through the loop. Also write loop code that does not assume a fixed row count, since the final chunk is usually shorter.

Use chunked reading with other data sources

chunksize is not exclusive to read_csv. Other pandas readers that accept it include pandas.read_json (with lines=True), pandas.read_sql and pandas.read_sql_table, pandas.read_stata, and pandas.read_sas; some readers, such as pandas.read_hdf, offer an equivalent iterator mode instead. Support and the accepted options vary by reader and pandas version, so check the official documentation for the function you plan to use.

The pattern stays the same wherever chunked reading is available: request batches, loop over them, and fold each batch into a small result.