Why front Amazon AWS ElasticSearch with AWS Kinesis Firehose

Question

I see several applications where data is being sent to AWS Kinesis Firehose and then automatically transferred to AWS ElasticSearch. You can directly write to AWS ElasticSearch. If I don't need any kind of data transformation and I can directly write data to ElasticSearch does fronting ElasticSearch with AWS Kinesis Firehose still provide any advantage. Like does it protect ElasticSearch from spikes in traffic etc.?

question from:https://stackoverflow.com/questions/66057918/why-front-amazon-aws-elasticsearch-with-aws-kinesis-firehose

Answer 1

Apart from transformations, the following reasons can be considered for having Firehose in front of AWS ES:

1. Better control over streaming data

Since Elasticsearch has limit on the write queue size, if there is a burst in data for few seconds, ES might throw rejects if it wont be able to write the data in that limited data. In this, you will end up loosing the rejected data as well.

However, when Firehose is kept in front, it will handle the retries for you and there will be less chances of data loss.

2. Firehose is one-way to ES

Your ES cluster might contain confidential data and if you are allowing user to make POST requests (required for some writes), you might expose the cluster to more than required users. Firehose can help you in limiting that by only giving write applications/user access to the FH stream instead of the ES cluster.