PythonBeginner.help

How to Remove Duplicates from a List in Python

Removing duplicates from a list means keeping only one copy of each value.

In Python, there are a few simple ways to do this:

  • Use dict.fromkeys() to remove duplicates and keep the original order
  • Use set() if you only care about unique values
  • Use a loop if you want to understand the logic or need more control

For most beginners, dict.fromkeys() is a good default choice.

Quick fix #

items = [1, 2, 2, 3, 1, 4]
unique_items = list(dict.fromkeys(items))
print(unique_items)

Output:

[1, 2, 3, 4]

This removes duplicates and keeps the original order.

What removing duplicates means #

A duplicate is a value that appears more than once in a list.

For example:

items = [1, 2, 2, 3, 1, 4]

In this list:

  • 2 appears more than once
  • 1 appears more than once

If you remove duplicates, you keep one copy of each value:

[1, 2, 3, 4]

The important thing to remember is this:

  • Some methods keep the original order
  • Some methods do not keep the original order

If order matters, choose your method carefully.

Use dict.fromkeys() to keep order #

This is one of the easiest ways to remove duplicates while keeping the original order.

items = [1, 2, 2, 3, 1, 4]
unique_items = list(dict.fromkeys(items))

print(unique_items)

Output:

[1, 2, 3, 4]

How it works #

  • dict.fromkeys(items) creates a dictionary using the list values as keys
  • Dictionary keys must be unique
  • Duplicate values are removed automatically
  • list(...) turns the keys back into a list

This works well for:

  • strings
  • numbers
  • tuples

Example with strings:

names = ["Ana", "Bob", "Ana", "Cara", "Bob"]
unique_names = list(dict.fromkeys(names))

print(unique_names)

Output:

['Ana', 'Bob', 'Cara']

If you want to understand sets better, see Python sets explained and the set() function reference.

Use set() when order does not matter #

A set stores only unique values.

That means you can remove duplicates like this:

items = [1, 2, 2, 3, 1, 4]
unique_items = list(set(items))

print(unique_items)

Possible output:

[1, 2, 3, 4]

This method is:

  • short
  • common
  • often fast

But there is one important warning:

  • The final order may not match the original list

For example:

letters = ["b", "a", "c", "a", "b"]
unique_letters = list(set(letters))

print(unique_letters)

The output may be in a different order than the original list.

Use this only when order does not matter.

Use a loop to understand the logic #

A loop is easy to read and helps you understand what duplicate removal is doing.

items = [1, 2, 2, 3, 1, 4]
unique_items = []

for item in items:
    if item not in unique_items:
        unique_items.append(item)

print(unique_items)

Output:

[1, 2, 3, 4]

What the code does #

  • Start with an empty list called unique_items
  • Loop through each item in the original list
  • Check whether the item is already in unique_items
  • If not, add it with append()

This keeps the original order and is beginner-friendly.

If you want a quick refresher on adding values to lists, see Python list append() method.

When to use this method #

Use a loop when:

  • you want to understand the logic clearly
  • your list contains values that do not work well with set() or dict.fromkeys()
  • you need custom rules for what counts as a duplicate

This approach is usually slower for large lists because item not in unique_items checks the list again each time. For small beginner examples, that is often fine.

Handle lists with unhashable items #

set() and dict.fromkeys() work best with hashable values such as:

  • strings
  • numbers
  • tuples

Some values are unhashable, including:

  • lists
  • dictionaries

That means this will fail:

items = [[1, 2], [1, 2], [3, 4]]
unique_items = list(set(items))

You will get an error because lists cannot be used as set values.

In cases like this, use a loop:

items = [[1, 2], [1, 2], [3, 4]]
unique_items = []

for item in items:
    if item not in unique_items:
        unique_items.append(item)

print(unique_items)

Output:

[[1, 2], [3, 4]]

Example with dictionaries #

A list of dictionaries usually needs a custom rule.

items = [
    {"id": 1, "name": "Ana"},
    {"id": 2, "name": "Bob"},
    {"id": 1, "name": "Ana"}
]

unique_items = []

for item in items:
    if item not in unique_items:
        unique_items.append(item)

print(unique_items)

Output:

[{'id': 1, 'name': 'Ana'}, {'id': 2, 'name': 'Bob'}]

Sometimes you do not want to compare whole dictionaries. You may only want to treat items as duplicates if the "id" matches. In that case, you need a custom check.

Choose the right method #

Here is a simple guide:

  • Use dict.fromkeys() if you want to keep order
  • Use set() if order does not matter
  • Use a loop if you want beginner-friendly logic or need custom checks

In short:

  • Best simple default: list(dict.fromkeys(items))
  • Shortest option: list(set(items))
  • Most flexible option: a loop

If you are also checking values before adding them, how to check if a value exists in a list in Python may help.

Common mistakes #

Here are some common problems when removing duplicates from a list.

Using set() and expecting the original order to stay the same #

This is a very common mistake.

items = [3, 1, 2, 1, 3]
unique_items = list(set(items))
print(unique_items)

The values will be unique, but the order may change.

If you need order, use:

unique_items = list(dict.fromkeys(items))

Trying set() on a list that contains other lists or dictionaries #

This will usually fail because lists and dictionaries are unhashable.

Bad example:

items = [[1, 2], [1, 2]]
unique_items = list(set(items))

Use a loop instead.

Removing duplicates without assigning the result #

This does not change the original list by itself:

items = [1, 2, 2, 3]
list(dict.fromkeys(items))
print(items)

Output:

[1, 2, 2, 3]

You need to store the result:

items = [1, 2, 2, 3]
unique_items = list(dict.fromkeys(items))
print(unique_items)

Thinking duplicate removal changes the original list automatically #

Most duplicate-removal approaches create a new list.

If you want to keep the result, assign it to a variable.

You can also replace the original variable:

items = [1, 2, 2, 3]
items = list(dict.fromkeys(items))
print(items)

If you want to preserve the original list before changing it, see how to copy a list in Python.

Helpful debugging checks #

If your result is not what you expected, these checks can help:

print(items)
print(type(items))
print(unique_items)
print(len(items), len(unique_items))
for item in items:
    print(repr(item))

These help you:

  • confirm the input list
  • check the data type
  • compare the original and new lists
  • see whether values only look the same but are actually different

FAQ #

How do I remove duplicates and keep the original order? #

Use list(dict.fromkeys(your_list)). It is short and keeps order in modern Python.

What is the shortest way to remove duplicates? #

Use list(set(your_list)), but only if you do not care about order.

Why did my list change order after removing duplicates? #

You probably used set(). Sets do not keep list order.

Can I remove duplicates from a list of dictionaries? #

Not directly with set() in most cases. Use a loop and define your own rule for duplicates.

See also #

Press Esc to close