how to return the count of unique documents by using elasticsearch aggregation

Question

I encountered a problem that elasticsearch could not return the count of unique documents by just using terms aggregation on a nested field.

Here is an example of our model:

{
    ...,
    "location" : [
        {"city" : "new york", "state" : "ny"},
        {"city" : "woodbury", "state" : "ny"},
        ...
    ],
    ...
}

I want to do aggregation on the state field, but this document will be counted twice in the 'ny' bucket since 'ny' appears twice in the document.

So I'm wondering if where is a way to grab the count of distinct documents.

mapping:

people = {
  :properties => {
    :location => {
      :type => 'nested',
      :properties => {
        :city => {
          :type => 'string',
          :index => 'not_analyzed',
        },
        :state => {
          :type => 'string',
          :index => 'not_analyzed',
        },
      }
    },
    :last_name => {
      :type => 'string',
      :index => 'not_analyzed'
    }
  }
}

the query is pretty simple:

curl -XGET 'http://localhost:9200/people/_search?pretty&search_type=count' -d '{
  "query" : {
    "bool" : {
      "must" : [
        {"term" : {"last_name" : "smith"}}
      ]
    }
  },
  "aggs" : {
    "location" : {
      "nested" : {
        "path" : "location"
      },
      "aggs" : {
        "state" : {
          "terms" : {"field" : "location.state", "size" : 10}
        }
      }
    }
  }
}'

The response:

{
  "took" : 104,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1248513,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "location" : {
      "doc_count" : 2107012,
      "state" : {
        "buckets" : [ {
          "key" : 6,
          "key_as_string" : "6",
          "doc_count" : 214754
        }, {
          "key" : 12,
          "key_as_string" : "12",
          "doc_count" : 168887
        }, {
          "key" : 48,
          "key_as_string" : "48",
          "doc_count" : 101333
        } ]
      }
    }
  }
}

The doc_count is much larger than the total in hit. So there must be duplicates.

Thanks!

Answer 1

I think you need a reverse_nested aggregation, because you want aggregation based on a nested value, but actually counting the ROOT documents, not the nested ones

{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "last_name": "smith"
          }
        }
      ]
    }
  },
  "aggs": {
    "location": {
      "nested": {
        "path": "location"
      },
      "aggs": {
        "state": {
          "terms": {
            "field": "location.state",
            "size": 10
          },
          "aggs": {
            "top_reverse_nested": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }
}

And, as a result, you would see something like this:

"aggregations": {
      "location": {
         "doc_count": 6,
         "state": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
               {
                  "key": "ny",
                  "doc_count": 4,
                  "top_reverse_nested": {
                     "doc_count": 2
                  }
               },
               {
                  "key": "ca",
                  "doc_count": 2,
                  "top_reverse_nested": {
                     "doc_count": 2
                  }
               }
            ]
         }
      }
   }

And what you are looking for is under top_reverse_nested part. One point here: if I'm not mistaking "doc_count": 6 is the NESTED document count, so don't be confused about these numbers thinking you are counting root documents, the count is on the nested ones. So, for a document with three nested ones that match, the count would be 3, not 1.