Troubleshooting

Diagnostic tools, common issues, and resolution procedures.

Too many open files

If you have a problem with too many open files by the Energy Logserver process, check the values in the following configuration files:

  • /usr/lib/systemd/system/logserver.service, LimitNOFILE=262144

  • /etc/security/limits.d/30-logserver.conf, logserver - nofile 262144

  • /etc/sysconfig/logserver, MAX_OPEN_FILES=262144

If you need to increase these values, edit the files and then reload systemd:

systemctl daemon-reload
systemctl restart logserver

The Logserver-GUI status code 500

If the login page is displayed in Logserver-GUI, but after the attempt to login, the browser displays “error: 500”, and the logs will show entries:

Error: Failed to encode cookie (sid-auth) value: Password string too short (min 32 characters required).

Generate a new login.ironsecret with the following command:

echo "login.ironsecret: \"$(</dev/urandom tr -dc _A-Z-a-z-0-9 | head -c32)\"" >> /etc/logserver-gui/logserver-gui.yml

Support tool

Energy Logserver includes a support tool (formerly diagnostic tool) that collects system data necessary for problem analysis by the support team.

The tool is located at: /usr/share/logserver/utils/support-tool.sh (a diagnostic-tool.sh symlink is also available for backward compatibility).

The support tool collects the following information:

  • Configuration files: Data Node (/etc/logserver/), GUI (/etc/logserver-gui/), Network Probe (/etc/logserver-probe/conf.d/), Alert (/opt/alert/config.yaml), Cerebro (/opt/cerebro/), Skimmer (/opt/skimmer/skimmer.conf), E-Doc (/opt/e-doc/config.yml), SIEM/Endpoint Security (/var/ossec/etc/)

  • Service logs (via journalctl): logserver, logserver-gui, logserver-probe, alert, cerebro, skimmer, e-doc

  • Cluster information: cluster health, cluster settings, indices summary, shard details

Running the support tool

The tool reads credentials automatically from /etc/logserver-gui/logserver-gui.yml. No command-line arguments are needed:

/usr/share/logserver/utils/support-tool.sh

Warning

If you have changed the default password or protocol, edit the script and update the credentials manually (see the read_config section), or disable read_config and set esUsername, esPass, and protocol variables directly.

The tool saves the results as an AES-256-CBC encrypted archive at /root/diagnostics-logs-<hostname>-<date>.tar.gz.enc.

To decrypt the archive:

openssl enc -aes-256-cbc -md sha512 -base64 -d -in diagnostics-logs-*.tar.gz.enc -out diagnostics-logs.tar.gz

Verification steps and logs

Verification of Energy Logserver service

To verify the Energy Logserver service you can use the following commands:

  • Control of the Energy Logserver system service via systemd:

systemctl status logserver

output:

* logserver.service - Logserver
   Loaded: loaded (/usr/lib/systemd/system/logserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-11-14 15:17:16 CET; 5 days ago
 Main PID: 58816 (java)
   CGroup: /system.slice/logserver.service
           └─58816 /etc/alternatives/jre/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.n.
  • Control of Logserver instance via tcp port:

curl -XGET '127.0.0.1:9200/'

output:

{
  "name" : "node-1",
  "cluster_name" : "logserver",
  "cluster_uuid" : "B5SDCaaKQU2JdJpsKy6quQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.3.2",
    "build_type" : "tar",
    "build_snapshot" : false
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

The version.number field reports the search engine build, 3.3.2 on Energy Logserver 8.1, not the product version. The response also carries build_hash, build_date, and the Lucene and compatibility versions, omitted above.

  • Control of Logserver instance via log file:

      tail -f /var/log/logserver/logserver.log
    
  • other control commands via curl application:

      curl -XGET "http://localhost:9200/_cat/health?v"
      curl -XGET "http://localhost:9200/_cat/nodes?v"
      curl -XGET "http://localhost:9200/_cat/indices"
    

Verification of Logserver-Probe service

To verify the Logserver-Probe service you can use the following command:

  • control Logserver-Probe service via systemd:

      systemctl status logserver-probe
    

output:

    logserver-probe.service - logserver-probe
       Loaded: loaded (/usr/lib/systemd/system/logserver-probe.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2025-01-06 09:15:32 CET; 2 months ago
     Main PID: 87818 (java)
       CGroup: /system.slice/logserver-probe.service
              └─87818 /usr/bin/java -Xms1g -Xmx1g
  • control Logserver-Probe service via port tcp:

      curl -XGET '127.0.0.1:9600'
    

    output:

      {
         "host": "logserver-node",
         "version": "7.17.29",
         "http_address": "127.0.0.1:9600"
      }
    

    The version field reports the pipeline engine bundled with the probe, not the Energy Logserver release.

  • control Logserver-Probe service via log file:

      tail -f /var/log/logserver-probe/logserver-probe-plain.log
    

Debugging

  • dynamically update logging levels through the logging API (service restart not needed):

       curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
       {
           "logger.logserver-probe.outputs.logserver" : "DEBUG"
       }
       '
    
  • permanent change of logging level (service need to be restarted):

    • edit file /etc/logserver-probe/logserver-probe.yml and set the following parameter:

         log.level: debug
      
    • restart logserver-probe service:

          systemctl restart logserver-probe
      
  • checking the configuration of one pipeline. Run the test as the logserver-probe user with a temporary data directory, so it does not collide with the running service:

      mkdir -p /tmp/probe-test && chown logserver-probe /tmp/probe-test
      sudo -u logserver-probe /usr/share/logserver-probe/bin/logstash --path.settings /etc/logserver-probe \
        --path.data /tmp/probe-test -t -f /etc/logserver-probe/conf.d/<pipeline>
      rm -rf /tmp/probe-test
    

    Exit code 0 and Config Validation Result: OK in the log mean the configuration is valid. The test writes the details of an error to /var/log/logserver-probe/logserver-probe-plain.log, not to the console. Point -f at the directory of one pipeline: with -f /etc/logserver-probe/conf.d the test finds no files, because the configurations are in subdirectories, and still reports OK.

  • get information about load of the Logserver-Probe:

      curl -XGET '127.0.0.1:9600/_node/jvm?pretty=true'
    

output:

 {
  "host" : "logserver-node",
  "version" : "7.17.29",
  "http_address" : "127.0.0.1:9600",
  "id" : "5a440edc-1298-4205-a524-68d0d212cd55",
  "name" : "logserver-node",
  "jvm" : {
    "pid" : 14705,
    "version" : "21",
    "vm_version" : "21",
    "vm_vendor" : "Eclipse Adoptium",
    "vm_name" : "OpenJDK 64-Bit Server VM",
    "start_time_in_millis" : 1736150132000,
    "mem" : {
      "heap_init_in_bytes" : 268435456,
      "heap_max_in_bytes" : 1056309248,
      "non_heap_init_in_bytes" : 2555904,
      "non_heap_max_in_bytes" : 0
    },
    "gc_collectors" : [ "G1 Young Generation", "G1 Old Generation" ]
  }
 }

All pipelines stop and the probe restarts in a loop

If one pipeline fails to start, for example because a translate filter points to a missing dictionary file, the Logserver-Probe process exits and stops all other pipelines. systemd starts it again, and the cycle repeats. systemctl status logserver-probe can still show active (running), while:

  • journalctl -u logserver-probe repeats Main process exited, code=exited, status=1/FAILURE and Scheduled restart job, restart counter is at <N>,

  • the API on port 9600 does not respond.

Find the failing pipeline in the log:

grep -E "FATAL|Failed to execute action" /var/log/logserver-probe/logserver-probe-plain.log | tail

The line Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:<pipeline> names the pipeline, and the lines above it describe the error. Fix the configuration of that pipeline, test it as described in Debugging, and restart the service.

Verification of Energy Logserver GUI service

To verify the Energy Logserver GUI service you can use the following commands:

  • Control the Energy Logserver GUI service via systemd:

      systemctl status logserver-gui
    

output:

    * logserver-gui.service - Logserver-GUI
       Loaded: loaded (/usr/lib/systemd/system/logserver-gui.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2025-01-06 09:16:45 CET; 2 months ago
     Main PID: 1330 (node)
       CGroup: /system.slice/logserver-gui.service
               └─1330 /usr/share/logserver-gui/bin/../node/bin/node --no-warnings /usr/share/logserver-gui/bin/../src/cli -c /etc/logserver-gui/logserver-gui.yml
  • Control the Energy Logserver GUI via port tcp/http:

      curl -XGET '127.0.0.1:5601/'
    

output:

      <script>var hashRoute = '/app/logserver-gui';
      var defaultRoute = '/app/logserver-gui';
      var hash = window.location.hash;
      if (hash.length) {
        window.location = hashRoute + hash;
      } else {
        window.location = defaultRoute;
      }</script>
  • Control the Energy Logserver GUI via log file:

      journalctl -fu logserver-gui
    

SIEM Agent - Windows CP1250 decoding problem

If the SIEM Agent works on an operating system which uses a non-Latin-script alphabet, the encoding could cause dropping documents by logserver-probe. In the logserver-probe log you can notice lines like the one below.

[2023-06-01T15:36:02,091][WARN ][logserver-probe.codecs.json     ] Received an event that has a different character encoding than you configured. {:text=>"{\\\"timestamp\\\":\\\"2023-06-01T15:36:01.214+0000\\\",\\\"agent\\\":{\\\"id\\\":\\\"002\\\",\\\"name\\\":\\\"win10_laptop\\\"},\\\"manager\\\":{\\\"name\\\":\\\"SiemPlan.local\\\"},\\\"id\\\":\\\"1549035361.0\\\",\\\"full_log\\\":\\\"{\\\\\\\"type\\\\\\\":\\\\\\\"program\\\\\\\",\\\\\\\"ID\\\\\\\":78741874,\\\\\\\"timestamp\\\\\\\":\\\\\\\"2023/06/01 15:36:00\\\\\\\",\\\\\\\"program\\\\\\\":{\\\\\\\"format\\\\\\\":\\\\\\\"win\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"Skype\\x99 7.34\\\\\\\",\\\\\\\"architecture\\\\\\\":\\\\\\\"i686\\\\\\\",\\\\\\\"version\\\\\\\":\\\\\\\"7.34.102\\\\\\\",\\\\\\\"vendor\\\\\\\":\\\\\\\"Skype Technologies S.A.\\\\\\\",\\\\\\\"install_time\\\\\\\":\\\\\\\"20180212\\\\\\\",\\\\\\\"location\\\\\\\":\\\\\\\"C:\\\\\\\\\\\\\\\\Program Files (x86)\\\\\\\\\\\\\\\\Skype\\\\\\\\\\\\\\\\\\\\\\\"}}\\\",\\\"decoder\\\":{\\\"name\\\":\\\"syscollector\\\"},\\\"location\\\":\\\"syscollector\\\"}", :expected_charset=>"UTF-8"}

This is caused by default Windows encoding CP1250. You can change default encoding to UTF-8 by following this steps:

  1. Go to Language settings

    Language settings

  2. Open Administrative language settings

    Administrative language settings

  3. Click on Change system locale... button

    Region settings

  4. Tick the checkbox Use Unicode UTF-8..

    UTF-8 setting

  5. To make this change active you have to reboot system.

Wazuh manager

The start exceeds the systemd timeout

The wazuh-manager unit allows 45 seconds for the start (TimeoutSec=45). When the start takes longer, systemctl restart wazuh-manager ends with Job for wazuh-manager.service failed because a timeout was exceeded, and systemctl show wazuh-manager -p Result returns Result=timeout. Check journalctl -u wazuh-manager for the cause first. If the start is only slow, raise the limit with a drop-in file instead of editing the unit file:

mkdir -p /etc/systemd/system/wazuh-manager.service.d
printf "[Service]\nTimeoutStartSec=600\n" > /etc/systemd/system/wazuh-manager.service.d/timeout.conf
systemctl daemon-reload

Events are not analyzed after log directories were deleted

If a subdirectory of /var/ossec/logs, for example alerts, is deleted to free disk space, the analysis daemon wazuh-analysisd does not start, while systemctl status wazuh-manager still shows the service as active. Check the daemons with:

/var/ossec/bin/wazuh-control status

The output contains wazuh-analysisd not running..., and /var/ossec/logs/ossec.log shows a line like wazuh-analysisd: CRITICAL: (1107): Could not create directory 'logs/alerts/2026/' due to [(2)-(No such file or directory)]. Recreate the directory and restart the service:

mkdir /var/ossec/logs/alerts
chown wazuh:wazuh /var/ossec/logs/alerts
chmod 750 /var/ossec/logs/alerts
systemctl restart wazuh-manager

wazuh-analysisd creates the date subdirectories by itself.

My disk is full

The cluster has reached the flood stage watermark and switched to read-only mode. Non-administrative users cannot log in until the issue is resolved. For background on watermark levels see Disk-based shard allocation.

Step 1: free disk space

Check for heap dumps first. After an out-of-memory error the ELS Data Node writes a .hprof file that can be as large as its heap to /var/lib/logserver, as set by -XX:HeapDumpPath in /etc/logserver/jvm.options. The Network Probe also writes heap dumps on out-of-memory errors. List them with:

find / /var/lib/logserver -xdev -name '*.hprof' -exec ls -lh {} +

Delete the dumps you do not need for analysis.

Remove old indices via Index Management in the GUI, or via the API:

curl -u $USER:$PASSWORD -XDELETE \
  '127.0.0.1:9200/<index-name>?pretty'

To trigger immediate log rotation (/etc/logrotate.d/logtrail, 14-day retention):

logrotate -f /etc/logrotate.d/logtrail

Step 2: unblock indices

When disk usage drops below the high watermark the block releases automatically. To release it immediately without waiting:

curl -u $USER:$PASSWORD -XPUT '127.0.0.1:9200/_all/_settings' \
  -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

Step 3: verify

curl -u $USER:$PASSWORD '127.0.0.1:9200/_cluster/health?pretty'
curl -u $USER:$PASSWORD '127.0.0.1:9200/_cat/nodes?v'

Status should return green or yellow.