Remove duplicates from a list

Question

Hello i want to remove duplicates from a list i do this but not working

List<Customer> listCustomer = new ArrayList<Customer>();    
for (Customer customer: tmpListCustomer)
{
  if (!listCustomer.contains(customer)) 
  {
    listCustomer.add(customer);
  }
 }

Answer 1

I suspect you might not have Customer.equals() implemented properly (or at all).

List.contains() uses equals() to verify whether any of its elements is identical to the object passed as parameter. However, the default implementation of equals tests for physical identity, not value identity. So if you have not overwritten it in Customer, it will return false for two distinct Customer objects having identical state.

Here are the nitty-gritty details of how to implement equals (and hashCode, which is its pair - you must practically always implement both if you need to implement either of them). Since you haven't shown us the Customer class, it is difficult to give more concrete advice.

As others have noted, you are better off using a Set rather than doing the job by hand, but even for that, you still need to implement those methods.

Answer 2

If that code doesn't work, you probably have not implemented equals(Object) on the Customer class appropriately.

Presumably there is some key (let us call it customerId) that uniquely identifies a customer; e.g.

class Customer {
    private String customerId;
    ...

An appropriate definition of equals(Object) would look like this:

    public boolean equals(Object obj) {
        if (obj == this) {
            return true;
        }
        if (!(obj instanceof Customer)) {
            return false;
        }
        Customer other = (Customer) obj;
        return this.customerId.equals(other.customerId);
    }

For completeness, you should also implement hashCode so that two Customer objects that are equal will return the same hash value. A matching hashCode for the above definition of equals would be:

    public int hashCode() {
        return customerId.hashCode();
    }

It is also worth noting that this is not an efficient way to remove duplicates if the list is large. (For a list with N customers, you will need to perform N*(N-1)/2 comparisons in the worst case ... when there are no duplicates.) For a more efficient solution you should use something like a HashSet to do the duplicate checking.

Answer 3

Just add all your elements to a Set: it does not allow it's elements to be repeated. If you need a list afterwards, use new ArrayList(theSet) constructor afterwards (where theSet is your resulting set).

Answer 4

Without seeing your object, I can suggest two things:

• don't implement hashCode() and equals()
• implement hashCode() and equals() using the same field(s), that will be your criteria for equality. Let your IDE (Eclipse, NetBeans) generate these methods.

Then, you can do the following, which will eliminate duplicates, because Sets don't allow duplicates

listCustomer = new ArrayList<Customer>(new HashSet<Customer>(listCustomer));

Answer 5

The correct answer for Java is use a Set. If you already have a List<Customer> and want to de duplicate it

Set<Customer> s = new HashSet<Customer>(listCustomer);

Otherise just use a Set implemenation HashSet, TreeSet directly and skip the List construction phase.

You will need to override hashCode() and equals() as well to make sure that the behavior you want actually what you get. equals() can be as simple as comparing unique ids of the objects to as complex as comparing every field. hashCode() can be as simple as returning the hashCode() of the unique id' String representation or the hashCode().

Answer 6

As others have mentioned, you are probably not implementing equals() correctly.

However, you should also note that this code is considered quite inefficient, since the runtime could be the number of elements squared.

You might want to consider using a Set structure instead of a List instead, or building a Set first and then turning it into a list.

Answer 7

Two suggestions:

• Use a HashSet instead of an ArrayList. This will speed up the contains() checks considerably if you have a long list

• Make sure Customer.equals() and Customer.hashCode() are implemented properly, i.e. they should be based on the combined values of the underlying fields in the customer object.

Answer 8

The "contains" method searched for whether the list contains an entry that returns true from Customer.equals(Object o). If you have not overridden equals(Object) in Customer or one of its parents then it will only search for an existing occurrence of the same object. It may be this was what you wanted, in which case your code should work. But if you were looking for not having two objects both representing the same customer, then you need to override equals(Object) to return true when that is the case.

It is also true that using one of the implementations of Set instead of List would give you duplicate removal automatically, and faster (for anything other than very small Lists). You will still need to provide code for equals.

You should also override hashCode() when you override equals().

Answer 9

Does Customer implement the equals() contract?

If it doesn't implement equals() and hashCode(), then listCustomer.contains(customer) will check to see if the exact same instance already exists in the list (By instance I mean the exact same object--memory address, etc). If what you are looking for is to test whether or not the same Customer( perhaps it's the same customer if they have the same customer name, or customer number) is in the list already, then you would need to override equals() to ensure that it checks whether or not the relevant fields(e.g. customer names) match.

Note: Don't forget to override hashCode() if you are going to override equals()! Otherwise, you might get trouble with your HashMaps and other data structures. For a good coverage of why this is and what pitfalls to avoid, consider having a look at Josh Bloch's Effective Java chapters on equals() and hashCode() (The link only contains iformation about why you must implement hashCode() when you implement equals(), but there is good coverage about how to override equals() too).

By the way, is there an ordering restriction on your set? If there isn't, a slightly easier way to solve this problem is use a Set<Customer> like so:

Set<Customer> noDups = new HashSet<Customer>();
noDups.addAll(tmpListCustomer);
return new ArrayList<Customer>(noDups);

Which will nicely remove duplicates for you, since Sets don't allow duplicates. However, this will lose any ordering that was applied to tmpListCustomer, since HashSet has no explicit ordering (You can get around that by using a TreeSet, but that's not exactly related to your question). This can simplify your code a little bit.