How to Find Duplicate Rows in a DataFrame Using Pandas

Use DataFrame.duplicated() to find duplicate rows in a Pandas DataFrame. The method returns a Boolean Series aligned with the DataFrame’s index; use that Series as a mask to select the rows marked True. By default, Pandas keeps the first occurrence unmarked and marks later matching rows. You can compare every column or choose comparison columns, and you can change which occurrences are marked with keep.

Duplicate-row checks are useful when inspecting or cleaning tabular data. Finding duplicates does not delete or change rows: it only identifies them. Review the selected rows before deciding whether to remove any records.

Use DataFrame.duplicated() to Identify Duplicate Rows

The method accepts subset and keep:

• subset is a column label or list of labels to compare. With its default value, None, Pandas compares all columns.
• keep="first" is the default. The first row in each matching group is marked False; subsequent rows are True.
• keep="last" leaves the last row in each group unmarked and marks earlier occurrences True.
• keep=False marks every row in a group that has more than one matching row as True.

The result is a Series of flags, not a DataFrame of duplicate rows. Apply it with df[df.duplicated()] to get the rows marked as duplicates. The DataFrame’s index labels are retained in the filtered result, which helps locate each row in the original data.

Create a DataFrame With Duplicate Rows

This example includes three identical John/Austin employee records at indexes 1, 3, and 4; index 1 is the first occurrence, while indexes 3 and 4 are marked as duplicates by default. Another row shares the same name and age but has a different city, and other records provide additional comparison cases. The example prints the default flags, rows selected using all columns, and rows selected using one or two comparison columns. Run this example:

import pandas as pd

employees = [
    ("Joe", 28, "Chicago"),
    ("John", 32, "Austin"),
    ("Melvin", 25, "Dallas"),
    ("John", 32, "Austin"),
    ("John", 32, "Austin"),
    ("John", 32, "Houston"),
    ("Melvin", 40, "Dehradun"),
    ("Hazel", 32, "Austin"),
]
df = pd.DataFrame(employees, columns=["Name", "Age", "City"])

print("Flags for all columns:")
print(df.duplicated().tolist())
print("\nDuplicate rows across all columns:")
print(df[df.duplicated()])
print("\nRows repeated by City:")
print(df[df.duplicated(subset="City")])
print("\nRows repeated by Name and Age:")
print(df[df.duplicated(subset=["Name", "Age"])])
print("\nAll rows in exact duplicate groups:")
print(df[df.duplicated(keep=False)])

Flags for all columns:
[False, False, False, True, True, False, False, False]

Duplicate rows across all columns:
   Name  Age    City
3  John   32  Austin
4  John   32  Austin

Rows repeated by City:
    Name  Age    City
3   John   32  Austin
4   John   32  Austin
7  Hazel   32  Austin

Rows repeated by Name and Age:
   Name  Age     City
3  John   32   Austin
4  John   32   Austin
5  John   32  Houston

All rows in exact duplicate groups:
   Name  Age    City
1  John   32  Austin
3  John   32  Austin
4  John   32  Austin

Select Duplicate Rows by All or Selected Columns

Calling df.duplicated() with no subset compares the values in every column. In the example, the three rows for John in Austin have identical values, so the default flags leave index 1 unmarked and mark indexes 3 and 4. The John row in Houston is not an all-column duplicate because its city differs.

Use df.duplicated(subset="City") when the comparison should use only City. Since Austin appears in multiple rows, the default marks each Austin row after the first, even when its name and age differ. This is a match by city, not an assertion that those full records are identical. Pass a list such as subset=["Name", "Age"] to compare a combination of columns; in this data, John in Austin and John in Houston belong to the same comparison group, so later occurrences in that group are marked.

To retain the last occurrence instead, filter with df[df.duplicated(keep="last")]. To select every row belonging to a repeated group, use df[df.duplicated(keep=False)]. Choose the setting based on which records you need to inspect; the method itself leaves the DataFrame unchanged.

Conclusion

Use DataFrame.duplicated() to produce duplicate flags, then apply the Boolean Series to select rows. Leave subset unset to compare all columns or provide the columns that define a match. Set keep to "first", "last", or False to control which members of each repeated group are marked.