Summary
Question
In preparation for upgrading Elasticsearch in a future ALM Octane version, the upgrade to ALM Octane CP9 performs maintenance operations on the Elasticsearch indexes. The Elasticsearch index of each ALM Octane space will be split into multiple indexes (each holding data for an ALM Octane feature).
This article lists the steps you must perform before attempting to upgrade to CP9.
The article also explains new, additional steps to take during the upgrade.
Answer
Prerequisites
Perform the following checks on the Elasticsearch server before you start the upgrade process. The checks can be done while ALM Octane and Elasticsearch are up and running.
- Make sure there is 60% disk availability on the Elasticsearch server. You can run the following command from Kibana to check:
GET /_cat/allocation?v
Or the following CURL command:
CURL -XGET "http://{elastic host}:9200/_cat/allocation?v
If you do not have sufficient space on the Elasticsearch server, allocate space before proceeding with the upgrade.
- The Elasticsearch server by default allows scripting. If you have blocked this option, either enable scripting on the Elasticsearch server by adding painless to the script.allowed_types property in the elasticsearch.yml file, or remove this property altogether.
For example, add these lines to the file:
# Allow painless scripting in elastic
script.allowed_types: painless
Changes in upgrade process
Normally, after upgrading all spaces and before releasing ALM Octane for use (ending the downtime), the directive is to stop and start the ALM Octane service. However for CP9, the restart should be done only after the split has completed for all spaces. The section below explains how to monitor the split and to determine that it completed successfully.
Monitoring index split process
Once a space’s upgrade is done, a post upgrader is launched to split the space’s index.
Monitor the spaces’ split post upgrader in the new Post Upgraders tab. Use an Equal to filter: Name: Split Elastic Index.
The description column indicates the split’s current stage and its progress within the stage.
The status will display SUCCESS when completed successfully.
Troubleshooting index split process
In case of an error in the split process, the status will display FAILED. –If this happens, check the error message, fix the indicated problem, and re-run the upgrader. This is done by selecting the failed post upgrader and clicking the Re-run button.
Note that some Elasticsearch failures might be random due to momentary load on the Elasticsearch cluster. Try re-running the post upgrader in such cases. It might succeed this time.
We don’t recommend that you release the spaces for use by users before the post upgrader completes successfully. However, if you do, Elasticsearch-related features will be disabled for that space. A message to this effect will be displayed at the top of the users’ screens.
Successful split is a prerequisite for the next version
When upgrading to the version following CP9, you will not be able to upgrade spaces until ALL spaces have been successfully upgraded to 12.60.35 and successfully split.
Also note that the original space indexes should be left as is. They will be removed in a future version.