python - Remove duplicates from a list of dictionaries when only one of the key values is different

Question

I have seen some similar answers, but I can't find something specific for this case:

I have a list of dictionaries like this:

[
 {"element":Bla, "version":2, "date":"12/04/12"},
 {"element":Bla, "version":2, "date":"12/05/12"},
 {"element":Bla, "version":3, "date":"12/04/12"}
]

The actual dictionary has many other keys, but what I'm trying to do is to delete all the entries that have the exact same key pair values, except for the date. That is, to delete all duplicates (which are not really exact duplicates as only the date varies). In this case, what I would expect to get is:

[
 {"element":Bla, "version":2, "date":"12/04/12"},
 {"element":Bla, "version":3, "date":"12/04/12"}
]

Answer 1

You say you have a lot of other keys in the dictionary not mentioned in the question.

Here is O(n) algorithm to do what you need:

>>> seen = set()
>>> result = []
>>> for d in dicts:
...     h = d.copy()
...     h.pop('date')
...     h = tuple(h.items())
...     if h not in seen:
...         result.append(d)
...         seen.add(h)

>>> pprint(result)
[{'date': '12/04/12', 'element': 'Bla', 'version': 2},
 {'date': '12/04/12', 'element': 'Bla', 'version': 3}]

h is a copy of the dict. date key is removed from it with pop.

Then tuple is created as a hashable type which can be added to set.

If h has never been seen before, we append it to result and add to seen. Additions to seen is O(1) as well as lookups (h not in seen).

At the end, result contains only unique elements in terms of defined h values.