Backup and Disaster Recovery

Energy Logserver provides two backup mechanisms:

Configuration backup: the configuration-backup.sh script saves configuration files, system indices, and templates to a local archive. This script is included with the installation and is offered automatically before each upgrade. See Backup and Restore for details.
Snapshot backup: the snapshot API for full index data backup and disaster recovery. This requires a configured snapshot repository (described below).

Configuration Backup

The built-in configuration-backup.sh script creates a timestamped archive in /root/:

bash /usr/share/elasticsearch/utils/configuration-backup.sh

The archive includes: Logserver config, GUI config, Network Probe config, alert config, cerebro config, skimmer config, intelligence config, license-service config, e-doc config, credential keystore, system indices data, and templates.

To schedule automatic backups, add a cron entry:

0 1 * * * root /usr/share/elasticsearch/utils/configuration-backup.sh

Snapshot Backup

For full index data backup, use the snapshot API. This requires configuring a shared filesystem repository.

1. Configure snapshot repository path

Add the repository path to the Logserver configuration on every data node:

# /etc/elasticsearch/logserver.yml
path.repo: ["/mnt/snapshots"]

Restart the data node service after changing the configuration:

systemctl restart logserver

2. Register the repository

curl -u $USER:$PASSWORD -X PUT "localhost:9200/_snapshot/backup_repo" \
  -H 'Content-Type: application/json' -d '{
  "type": "fs",
  "settings": {
    "location": "/mnt/snapshots"
  }
}'

3. Create a snapshot

curl -u $USER:$PASSWORD -X PUT "localhost:9200/_snapshot/backup_repo/snapshot_$(date +%Y%m%d)" \
  -H 'Content-Type: application/json' -d '{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}'

4. List existing snapshots

curl -u $USER:$PASSWORD "localhost:9200/_snapshot/backup_repo/_all?pretty"

Disaster Recovery from Snapshot

1. Stop services

systemctl stop logserver-gui logserver-probe alert cerebro e-doc skimmer \
  intelligence intelligence-scheduler license-service

2. Restore from snapshot

curl -u $USER:$PASSWORD -X POST \
  "localhost:9200/_snapshot/backup_repo/SNAPSHOT_NAME/_restore" \
  -H 'Content-Type: application/json' -d '{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}'

3. Monitor restore progress

curl -u $USER:$PASSWORD "localhost:9200/_cat/recovery?v&active_only=true"

4. Start services

systemctl start logserver-gui alert cerebro e-doc skimmer \
  intelligence intelligence-scheduler license-service logserver-probe

Scheduling Snapshots

Run the snapshot call on a schedule with cron, the same way as the configuration backup. Escape the % characters for crontab:

0 2 * * * root curl -u $USER:$PASSWORD -X PUT "localhost:9200/_snapshot/backup_repo/snapshot_$(date +\%Y\%m\%d)" -H 'Content-Type: application/json' -d '{"indices":"*","ignore_unavailable":true,"include_global_state":true}'

Prune old snapshots in the same routine so the repository does not grow without bound.

High Availability within a Site

Keep every index replicated with at least one replica, and let the cluster place a primary and its replica on different physical hosts, racks, or rooms, so no single hardware failure loses data. Tag each node with a location attribute and enable allocation awareness:

# /etc/elasticsearch/logserver.yml (set a distinct value per node group)
node.attr.zone: zone-a
cluster.routing.allocation.awareness.attributes: zone

With awareness enabled, the cluster keeps a primary and its replica in different zones and spreads shards across zones. Running the nodes as virtual machines on separate hypervisors with this setting keeps a primary and its replica off the same physical host.

Multi-Site Recovery

Energy Logserver recovers a second site from snapshots rather than replicating a live cluster between sites.

Shared or replicated repository. Place the snapshot repository on storage the recovery site can reach: a shared filesystem mounted on both sites, or a directory whose contents you replicate to the recovery site. Register the repository on the recovery cluster as read-only, so it can restore and search without writing to it:

curl -u $USER:$PASSWORD -X PUT "localhost:9200/_snapshot/backup_repo" \
  -H 'Content-Type: application/json' -d '{
  "type": "fs",
  "settings": {
    "location": "/mnt/snapshots",
    "readonly": true
  }
}'

Only one cluster writes to a repository. Registering every other cluster as readonly avoids repository corruption from two writers.

Querying both sites. To search data held on separate clusters in different data centers from a single place, use Cross-cluster Search.

Moving selected data. To copy or migrate specific indices between clusters, use elasticdump, the same tool the bundled small_backup.sh uses, which exports and imports index data over the API.