How to Remove Duplicate Values From a PowerShell Array

1. Use Select-Object to Remove Duplicate Values From a PowerShell Array
2. Use Sort-Object to Remove Duplicate Values From a PowerShell Array
3. Use Get-Unique to Remove Duplicates From PowerShell Array
4. Use Hashtable to Remove Duplicates From PowerShell Array
5. Conclusion

Arrays are fundamental data structures in PowerShell that allow you to store collections of items, whether those items are of the same type or different types. In various scripting scenarios, you may encounter the need to eliminate duplicate items from an array.

This tutorial will introduce different methods to remove duplicates from a PowerShell array.

Use Select-Object to Remove Duplicate Values From a PowerShell Array

The Select-Object cmdlet is used to select specified properties of an object or set of objects. You can also select unique objects in a specified position in an array.

The -Unique parameter specifies that only a single member will be selected if identical properties and values are in objects. It is case-sensitive, so strings that only differ in character casing are also considered unique values.

We have created an array $data having duplicate items with different cases.

$data = @("powershell", "arrays", "powershell", "Arrays", "PowerShell")

The following example selects unique values from the array and saves them to the $data variable. As a result, the duplicate elements will be removed from an array.

Command:

$data = $data | Select-Object -Unique
$data

Output:

powershell
arrays
Arrays
PowerShell

It removes the duplicate string "powershell" but does not remove "PowerShell" because there is a capitalized letter in the string.

Use Sort-Object to Remove Duplicate Values From a PowerShell Array

The Sort-Object cmdlet sorts objects in ascending or descending order based on their property values. With Sort-Object also, you can use the -Unique parameter to remove the duplicates from the objects.

But it is case-insensitive, so strings that only differ in character casing are also considered the same. The following example removes the duplicate items from an array and saves them to the variable $a.

Command:

$a = @("powershell", "arrays", "powershell", "Arrays", "PowerShell")
$a = $a | Sort-Object -Unique
$a

Output:

Arrays
PowerShell

It removes "Arrays" and "PowerShell", although they only differ by character cases. If you want to make the Sort-Object case-sensitive, you can use the -CaseSensitive parameter.

Command:

$b = @("powershell", "arrays", "powershell", "Arrays", "PowerShell")
$b = $b | Sort-Object -Unique -CaseSensitive
$b

Output:

arrays
Arrays
powershell
PowerShell

Now, the output only removes the "powershell" since it is the only variable that has a duplicate (in case-sensitive).

Use Get-Unique to Remove Duplicates From PowerShell Array

The Get-Unique cmdlet gets unique items from a sorted list. It compares each item in a sorted list to the next item, removes duplicates, and returns only one member of each item.

This cmdlet only works if the list is sorted. You can use the Sort-Object cmdlet to sort objects by property values.

The following example uses the Get-Unique cmdlet to remove duplicate items from the array $sort after sorting the array elements with Sort-Object.

Command:

$c = @("powershell", "arrays", "powershell", "Arrays", "PowerShell")
$c = $c | Sort-Object | Get-Unique
$c

Output:

Arrays
arrays
PowerShell
powershell

In this output, the "powershell" is removed.

The Get-Unique is case-sensitive, so strings that differ only in character casing are also considered unique values. In this way, you can easily remove duplicate elements from an array in PowerShell.

Use Hashtable to Remove Duplicates From PowerShell Array

PowerShell allows you to utilize a Hashtable to remove duplicates efficiently. A Hashtable stores key-value pairs where keys are unique, and by creating a Hashtable and using the array items as keys, you can automatically remove duplicates.

The following example initializes an empty Hashtable ($uniqueHash) and then iterates through the array, setting each item as a key in the Hashtable. Since keys must be unique, duplicates are automatically eliminated.

The result is stored in the $uniqueArray variable.

$array = @("powershell", "arrays", "powershell", "Arrays", "PowerShell")
$uniqueHash = @{}
$array | ForEach-Object { $uniqueHash[$_] = $true }
$uniqueArray = $uniqueHash.Keys
$uniqueArray

Output:

powershell
arrays

In this output, duplicates are removed using a Hashtable and prints the remaining values. Please note that using a Hashtable to remove duplicates does not consider the case sensitivity of the values.

Conclusion

In PowerShell, managing duplicate values in arrays is a common task, and this article has introduced several methods to achieve it. By using the Select-Object, Sort-Object, and Get-Unique cmdlets, as well as utilizing Hashtables, you can tailor your approach to suit your specific requirements, whether you need case-sensitive or case-insensitive removal of duplicates.