set()Function to Remove Duplicates in a Python List
iteration_utilsModule to Find Duplicate in a Python List
This tutorial shows you how to look for duplicates within a list in Python.
set() Function to Remove Duplicates in a Python List
set() is a function to convert a list into a set. Based on the description of a set and a list, converting a list to a set would mean removing all the duplicates in the list.
However, what we want is to search for the entries within the list that are duplicates. To do that, we still use the
set() function, but we only include those with more than one entry in the list, which means they are entries that have duplicates.
Declare a function that looks for duplicates within a list and store them as a set.
def listToSet(listNums): set([num for num in listNums if listNums.count(x) > 1])
Another approach, if you want to keep the result as a list instead of a set, is to use
add() to find duplicates within the list and re-convert it to a list during return.
def list_duplicates(listNums): once = set() seenOnce = once.add twice = set( num for num in listNums if num in once or seenOnce(x) ) return list( twice )
This function adds the entry to
once if the number appears for the first time, and doesn’t exist within the set
Both functions will return the same output, although one is a set, and another is a list. The result will output all the duplicate entries in
iteration_utils Module to Find Duplicate in a Python List
iteration_utils has two functions that can search for duplicates within a pre-defined function:
We’re going to use the same list definition
iteration_utils and use it to return a list of all the duplicates within
from iteration_utils import duplicates listNums = [1,1,2,3,3,4,5,5,5,5,6,8,8] def listDups(listNums): return list(duplicates(listNums))
The output of the above function will return a list of all the duplicate entries within
Although, the list also includes the iterations of the duplicate entries. This means it will return every instance of the duplicated entry.
- This is where
unique_everseencomes in. This function processes the list to remove all the duplicate instances that
from iteration_utils import duplicates from iteration_utils import unique_everseen listNums = [1,1,2,3,3,4,5,5,5,5,6,8,8] def listDupsUnique(listNums): return list(unique_everseen(duplicates(listNums)))
This function will then return:
In summary, there are 2 easy solutions to look for duplicates within a list in Python. The first is using
set() and other utility functions of sets in Python to look for duplicates and store them in another variable. Another one is by
iteration_utils module by using
unique_everseen, which more or less does the same thing and produces the same output.