How to use elastic search EC2 discovery plugin?

By | July 11, 2019

Problem
To increase the performance in Elasticsearch, many times we add new nodes to Elasticsearch cluster and manually change the configuration file with new IP address, etc. so that it gets added into the cluster. This takes up our time too. To make it efficient, we deceided to use AWS EC2 discovery plugin which discovers the Elasticsearch EC2 instances automatically through the host provider ‘ec2’. This plugin made our task so simple. The below changes are first tested in staging and then carefully executed in production without any issues. It takes less than an hour to install AWS EC2 plugin with the upgrade.

The discovery-ec2 plugin allows Elasticsearch to find the master-eligible nodes in a cluster running on AWS EC2 by querying the AWS API for the addresses of the EC2 instances running these nodes.

Install the ElasticSearch EC2 Discovery Plugin
$ cd /home/ec2-user/elasticsearch-7.2.0/bin
$ sudo ./elasticsearch-plugin install discovery-ec2

Enabling EC2 discovery
To enable EC2 discovery, configure Elasticsearch to use the ec2 seed hosts provider in config/elasticsearch.yml:
discovery.seed_providers: ec2

Configuring EC2 discovery
EC2 discovery supports a number of settings. Some settings are sensitive and must be stored in the Elasticsearch keystore. For example, to authenticate using a particular access key and secret key, add these keys to the keystore by running the following commands:

$ bin/elasticsearch-keystore add discovery.ec2.access_key
$ bin/elasticsearch-keystore add discovery.ec2.secret_key

Configuring EC2 discovery
EC2 discovery supports a number of settings. Some settings are sensitive and must be stored in the Elasticsearch keystore. For example, to authenticate using a particular access key and secret key, add these keys to the keystore by running the following commands:

$ bin/elasticsearch-keystore add discovery.ec2.access_key
$ bin/elasticsearch-keystore add discovery.ec2.secret_key

In detailed discription is found in this reference
– https://blog.francium.tech/install-aws-ec2-discovery-plugin-in-elasticsearch-5a973348cad9
– https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html#discovery-ec2-usage

Leave a Reply