A Maven 3.1+ plugin to run instances of Elasticsearch version 5+ during the integration test phase of a build.
Instances are started in forked processes using the runforked goal.
They are terminated using the stop goal and, for extra peace of mind, using a JVM shutdown hook.
Each instance is installed in ${project.build.directory}/elasticsearch${instanceIndex}.
This plugin is known to work with Elasticsearch versions 5.0.0 to 8.0.0 .
For versions 8+, the xpack security is disabled -
there is no need for HTTPS or basic authentication to send requests to the Elasticsearch server.
For Elasticsearch version 1.x.x and 2.x.x support, see version 1.x and 2.x of this plugin.
The Elasticsearch behaviour and properties can be configured through the following plugin configuration parameters:
instanceCount [defaultValue=1]
how many Elasticsearch instances to start (all within the same cluster)
skip [defaultValue=false]
whether to skip the plugin execution or not
clusterName [defaultValue="test"]
the name of the cluster to create
flavour [defaultValue="oss"]
only applicable to Elasticsearch versions 6.3.0 to 7.10.x (inclusive); the flavour of Elasticsearch to install (oss, default); the default is not supported, due to x-pack issues; this property is ignored by all versions outside that range
You can also end your url with "/%s", it will indicate to the plugin to replace the "%s" value by his own computed filename (ideal for proxy like nexus,Artifactory, ...)
on Linux: http://my_private_repository/elasticsearch-downloads/elasticsearch-oss-7.4.2.tar.gz
on Windows: http://my_private_repository/elasticsearch-downloads/elasticsearch-oss-7.4.2.zip
downloadUrlUsername [defaultValue=""]
username supplied as part of basic authentication when downloading
downloadUrlPassword [defaultValue=""]
password supplied as part of basic authentication when downloading
httpPort [defaultValue=9200]
the port to configure Elasticsearch to listen to HTTP traffic to; when configuring multiple instances, they will be assigned subsequent HTTP ports starting with this value (mind the conflicts with the transport ports)
transportPort [defaultValue=9300]
the port for the Elasticsearch node to node communication; when configuring multiple instances, they will be assigned subsequent transport ports starting with this value (mind the conflicts with the HTTP ports)
pathConf [defaultValue=""] (note: common to all instances !!!)
the absolute path (or relative to the maven project) of the custom directory containing configuration files, to be copied to Elasticsearch instances
environmentVariables [defaultValue=""]
the environment variables to set before starting each Elasticsearch instance (see the Environment variables section for details)
pathData [defaultValue=""] - work in progress (note: per instance !!!); while support for this is being implemented, use pathConf to configure this option
the custom data directory to configure in Elasticsearch
pathLogs [defaultValue=""] - work in progress (note: per instance !!!); while support for this is being implemented, use pathConf to configure this option
the custom logs directory to configure in Elasticsearch
plugins [defaultValue=""]
the list of plugins to install in each Elasticsearch instance before starting it (see the Plugins section for details)
instanceSettings [defaultValue=""]
the list of settings to apply to corresponding Elasticsearch instances (see the InstanceSettings section for details)
pathInitScript [defaultValue=""]
the path of the initialization scripts (see the Initialization scripts section for details)
keepExistingData [defaultValue=true]
whether to keep the data and log directories, if they already exist; since the behavior,
before this flag was implemented, was to keep the existing data, the default is true
instanceStartupTimeout [defaultValue=120]
how long to wait (in seconds) for each Elasticsearch instance to start up; since Elasticsearch 8 takes significantly longer than the previous versions to start up, the default was changed from 30 to 120 seconds
clusterStartupTimeout [defaultValue=30]
how long to wait (in seconds) for all instances to form a cluster; this is in addition to the instance startup timeout
clientSocketTimeout [defaultValue=5000]
the default socket timeout (in milliseconds) for requests sent to the Elasticsearch server
setAwait [defaultValue=false]
whether to block the execution once all Elasticsearch instances have started, so that the maven build will not proceed to the next step; use CTRL+C to abort the process
autoCreateIndex [defaultValue=true]
configuration of automatic index creation represented by action.auto_create_index setting
logLevel [defaultValue=INFO]
the log level to be used by the console logger; the valid values are defined in AbstractElasticsearchBaseMojo.getMavenLogLevel() and they are: DEBUG, INFO, WARN, ERROR, FATAL, DISABLED.
To use the plugin, include the following in your pom.xml file and modify the configuration as needed:
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<!-- THE PLUGIN VERSION; FOR THE LIST OF AVAILABLE VERSIONS, SEE https://github.com/alexcojocaru/elasticsearch-maven-plugin/releases-->
<version>6.13</version>
<configuration>
<!-- THE ELASTICSEARCH VERSION; REPLACE WITH THE VERSION YOU NEED -->
<version>7.2.0</version>
<clusterName>test</clusterName>
<transportPort>9300</transportPort>
<httpPort>9200</httpPort>
</configuration>
<executions>
<!-- The elasticsearch maven plugin goals are by default bound to the pre-integration-test and post-integration-test phases-->
<execution>
<id>start-elasticsearch</id>
<phase>pre-integration-test</phase>
<goals>
<goal>runforked</goal>
</goals>
</execution>
<execution>
<id>stop-elasticsearch</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
Environment variables
The environment variables to set before starting each Elasticsearch instance.
Environment variables
may be referenced from the Elasticsearch configuration,
but the main use case for defining environment variables is to set JAVA_HOME,
so that Elasticsearch is run with a different JDK than the one used to run Maven.
The way to define environment variables is as follows:
Instance settings are applied to each corresponding elasticsearch instance (via -E on the commandline)
during startup. If the list is smaller then instanceCount no extra settings
are applied to the remaining instances. If it's larger, the extra items are ignored.
A comma-separated list of initialization script files can be provided using the pathInitScript parameter of the plugin, in which case they will be executed against the local Elasticsearch cluster.
The file extension defines the file format: json for JSON format, anything else for custom format, both formats can be used at the same time.
The script paths will be trimmed and executed in the order they are provided. For example:
And the plugin will always execute script1.json before script2.script
JSON format
The provided JSON files should contain a list of requests to be sent, one by one, to the Elasticsearch cluster.
Each request definition has three properties:
the request method: one of PUT, POST, DELETE
the name (in uppercase) of the request method to be used for the current request
the path part of the URL (should not start with slash)
will be appended to the protocol, hostname and port parts when the full URL is constructed
the payload
it should not be defined for DELETE requests; some Elasticsearch requests do not require a payload (eg. POST index/_refresh in Elasticsearch version 7 or less), in which case define the payload as {}; some Elasticsearch requests do not allow a payload (eg. POST index/_refresh in Elasticsearch version 8), in which case do not define the payload property
Example (see the src/main/test/resources/init.json file for a more complete example):
Error: Could not find or load main class (on OSX 10.13.6)
There seems to be an issue when starting certain versions of Elasticsearch (eg. 5.6.8) on OSX 10.13.6,
directly or via the plugin. The issue is caused by the incorrect quoting of the -cp argument
on the Java command built by the bin/elasticsearch script inside the Elasticsearch package.
A workaround is described here.
In summary, set the ES_JVM_OPTIONS environment variable to -cp "./target/elasticsearch0/lib/*"
in the IDE's run configuration or on the shell environment where maven/Elasticsearch is executed.
Error: Could not reserve enough space for x object heap (on Win 10)
This has the same root cause as the OSX specific error described above, and can be fixed using the same workaround.
Node is killed when running in TravisCI (not applicable; travis-ci.com is not free)
When running your build job in TravisCI, it can happen that your node is being killed without any notice.
To fix that you may have to modify the .travis.yml file as follows:
Error: java.lang.RuntimeException: can not run elasticsearch as root (in Docker)
When running the build in Docker, depending on the Docker image, the current user in the container
maybe be the root user and, because of this, Elasticsearch will fail to start.
The fix is to use a Docker image which does not use the root user. See
this discussion
for details.
Avoid downloading a plugin from the Internet repeatedly
When you want to run integration tests with a given plugin (eg. reindex-client),
elasticsearch-maven-plugin will run behind the scene a command like
bin/elasticsearch-plugin install reindex-client which will download the plugin
from the Internet at every execution.
You can use some Maven magic to avoid the download by first using
maven-dependency-plugin to download the plugin as an artifact which will be
stored in your local .m2 directory, then copy from there to your project
target directory, eg.
Why does the plugin uses a lock file and why does it need to open a server socket?
The plugin, at runtime, looks for the Elasticsearch artifact in the local maven repository and,
if not found, it downloads it into the system temp directory and installs it from there into the
local maven repo. When several Elasticsearch maven plugins are executed at the same time
on a single machine (eg. during parallel builds), and the ES artifact hasn't already been installed
in the local repo, it is not desirable that each plugin downloads and installs the same artifact;
that is because the ES artifact is a large file (~ 250 Mb), and downloading the same file
multiple times would be a waste of bandwidth.
Because of this, the plugin is capable to synchronize with other plugins running at the same time
on the same machine, so that a single one downloads and installs, while the rest wait
for the master plugin to complete in order to move forward with using the artifact.
The synchronization mechanism is described by
this diagram.
The integration tests exist as maven plugins in the src/it directory and are executed via the maven invoker plugin
(see the pom.xml for details on its configuration).
Each test has a maven like structure, with a pom.xml
and a src/test/java directory containing the test sources.
The properties common between all tests are defined in the invoker plugin config under the "testProperties" tag.
During the integration test phase, the invoker plugin copies the tests to target/it and executes
the "clean" and "verify" goals on each them. They are executed in a separate process
with a brand new maven config, defined using the following two invoker plugin properties:
localRepositoryPath (set to target/local-repo) and settingsFile (set to src/it/settings.xml).
The invoker configuration also defines a pre-build hook script
and a post-build hook script
to run before and after executing the test. These are groovy scripts which each test directory must contain.
The structure of an integration test
Because the integration tests are executed as maven projects, they have a maven-like file structure.
pom.xml
The pom.xml is generic and does not contain anything specific to any test - it defines the test project
dependencies and which goals to execute on the elasticsearch maven plugin.
setup.groovy
The pre-build hook script (setup.groovy) does the plugin and context configuration,
by using the ItSetup util to create a map of plugin properties and to save them to the
test.properties file in the test directory (to be picked up by the Java tests via the methods in ItBase).
The properties are also set on the context, for some are needed by the post-build hook script.
Defining the number of ES instances is required in the groovy script.
The ES cluster name and the HTTP and transport protocol ports
are randomly generated by ItSetup.generateProperties to avoid clashes between integration tests.
Any other properties to be passed over to the plugin can be added to the props map
(see src/it/runforked-auto-create-index-false/setup.groovy for an example).
verify.groovy
The standard verification done here is that the ES base directory(ies) were created
and that the ES instance(s) are not running (via the ItVerification util).
This file uses some of the plugin properties set on the context by the pre-build hook script.
Java test file(s)
The actual tests are defined in java files in the src/test/java directory under each integration test directory
(eg. src/it/runforked-auto-create-index-false/src/test/java/com/github/alexcojocaru/mojo/elasticsearch/v2/AutoCreateIndexTest.java).
They will be compiled and executed during the maven invoker plugin execution of the integration test maven project.
All java tests should extend com.github.alexcojocaru.mojo.elasticsearch.v2.ItBase
to get the clusterName and httpPort read from the context (ie. the "test.properties" file created by
the pre-build hook script) and the ES client set up.
NOTE: It is not possible to execute such a test case in an IDE, due to the lack of context
(the test properties must be set in the props file by executing the groovy script,
the elasticsearch maven plugin must be running, etc).
How to write new tests
Copy one of the existing integration tests and modify as needed. It will be picked up
by the invoker plugin due to the wildcard definition in the plugin config in pom.xml.
How to run single integration test
Set the integrationTest env variable to the integration test name when running maven, eg:
There are two ways to obtain more information during the execution of an integration test:
Debugging the maven execution
To have the invoker plugin output detailed information about the integration test execution,
change the debug attribute of the invoker plugin configuration (in the pom.xml) to true.
Debugging the elasticsearch maven plugin execution during the integration test
Set the es.logLevel property of the plugin to DEBUG, by
请发表评论