Cluster Upgrade

Warning

Rolling upgrades are not supported when upgrading to version 8.x. An upgraded node will not join an existing 7.x cluster. For upgrades to 8.x, all nodes must be stopped and upgraded together. See Upgrading a multi-node cluster from 7.x for the full procedure.

This section describes how to upgrade a multi-node cluster while maintaining availability during upgrades within the same major version (e.g., 7.8 to 7.9). For single-node installations, use the standard procedure described in Version-Specific Upgrades.

In a multi-node deployment, each node is upgraded individually using the same ./install.sh -u script. The API commands in this procedure (shard allocation control, flush) prevent unnecessary shard rebalancing while nodes are being restarted.

Prerequisites

  • Cluster health is green

  • A current backup exists (see Backup and Recovery)

  • All nodes are running the same Energy Logserver version

Verify cluster health before proceeding:

curl -u $USER:$PASSWORD "http://localhost:9200/_cluster/health?pretty"

Procedure

1. Disable shard reallocation

Run this once, from any node, before starting the upgrade process:

curl -u $USER:$PASSWORD -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

curl -u $USER:$PASSWORD "localhost:9200/_flush?pretty"

2. Upgrade each node

Repeat the following steps for each node in the cluster.

Warning

Do not stop logserver, logserver-gui, or other services managed by the installer before running ./install.sh -u. The script stops and replaces these services itself; pre-stopping the database (in particular) prevents the installer from completing the upgrade.

a) Stop the collector on the node so no new data is ingested during the upgrade:

systemctl stop logserver-probe

b) Run the upgrade: install.sh stops the services it manages, replaces the packages, and waits for confirmation before restarting them:

cd install
./install.sh -u

When the script asks whether to restart services, answer no: services will be started manually in the correct order.

c) Start the database service first and wait for the node to rejoin the cluster:

systemctl start logserver

Wait until the node appears in the cluster:

curl -u $USER:$PASSWORD "localhost:9200/_cat/nodes?v"

d) Wait for cluster health to reach at least yellow before moving to the next node:

curl -u $USER:$PASSWORD "http://localhost:9200/_cluster/health?pretty"

3. Re-enable shard allocation

After all nodes have been upgraded:

curl -u $USER:$PASSWORD -X PUT \
  "http://localhost:9200/_cluster/settings?pretty" \
  -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'

4. Start remaining services

On each node that runs client services, start services in the following order: the database service (logserver) must be running and healthy before starting the remaining services:

systemctl start logserver-gui alert cerebro e-doc skimmer \
  intelligence intelligence-scheduler license-service logserver-probe

5. Verify

curl -u $USER:$PASSWORD "http://localhost:9200/_cluster/health?pretty"
curl -u $USER:$PASSWORD "localhost:9200/_cat/nodes?v"

After the upgrade, follow any version-specific post-upgrade steps described in Version-Specific Upgrades.

Upgrading a multi-node cluster from 7.x

Complete the steps before the upgrade once for the cluster, then upgrade the nodes as follows.

Warning

install.sh -u connects to the data node at http://127.0.0.1:9200 (hardcoded default; the URL prompt is skipped in upgrade mode). If network.host in logserver.yml is set to a specific IP without including 127.0.0.1 or _local_, the installer cannot reach OpenSearch and fails with Connection refused.

Before upgrading each node, verify that network.host includes the loopback address:

# /etc/logserver/logserver.yml
network.host: ["10.x.x.x", "127.0.0.1"]

Restart logserver.service on that node if you change this setting.

For multi-node deployments, run the upgrade in this order:

  1. Start with the client node (the node that serves the GUI and coordinates queries).

  2. When install.sh asks whether to restart services, answer No. Services on each upgraded host stay stopped until every node has finished the upgrade.

  3. Upgrade the remaining data nodes one by one using the same procedure.

  4. Once every node has completed install.sh -u, start the services manually across the cluster, logserver first on each data node, then the remaining services.

  5. After the data nodes are running and the cluster reports at least yellow status, finalize the built-in user initialization on the client node. The endpoint returns the generated passwords of the built-in users, so pass its output to the password utility, which stores them in the keystore the same way the installer does:

curl -sS -u $USER:$PASSWORD -X POST "127.0.0.1:9200/_logserver/init" | /usr/share/logserver/utils/logserver-password-util.sh - --force --quiet

Warning

This manual step is required when upgrading from 7.9.0 or older to 8.x. On 8.x the built-in users (admin and others) are created through _logserver/init. The installer runs this initialization automatically only when it restarts logserver.service itself, which happens during a data-node upgrade if you answer Yes to the “Some system services require to be restarted” prompt. In the multi-node procedure above you answer No to that prompt and restart the services by hand, so the automatic initialization is skipped and the cluster stays uninitialized until you run _logserver/init yourself.

To check whether a node still needs initialization, query its health endpoint. An uninitialized node responds with Node uninitialized. To finalize init process please run ...:

curl -X GET "127.0.0.1:9200/_logserver/health"

Warning

The data-node RPM stops logserver.service on the host being upgraded, so the cluster runs degraded from the first node onwards. Do not let the installer restart services mid-upgrade; restart them manually only after every node has been upgraded.

When the cluster is running again, complete the steps after the upgrade, including enabling shard allocation.