Agents

Overview

The Agents module lets you manage Beats agents on monitored hosts from the Logserver GUI. It uses the same MasterBeatAgent.jar executable in two roles:

  • MasterAgent (server-side tool), installed at /opt/agents/ on the Logserver host. The Agents GUI plugin invokes it on demand to send management commands to registered agents. It does not run as a persistent service.

  • Agent (client service), installed on each monitored host as a systemd service on Linux or as a Windows service (WinSW wrapper agents.exe) on Windows. It connects to Network Probe on TCP port 8080 to register itself and report status, and listens on TCP port 8081 for incoming management commands.

Agent state is stored in the .agents OpenSearch index, written by the Network Probe pipeline when agents connect.

From the Agents plugin in the Logserver GUI you can view registered agents with their detected Beats processes, start, stop, and restart individual agents or Beats, push configuration file templates to monitored hosts, and reindex the .agents index. The plugin is available only to users with the admin role.

When you apply a template or reindex, the Logserver GUI process connects to each agent directly over HTTPS on the agent port (8081 by default), using the KeyStore and TrustStore from /opt/agents/agent.conf and the openssl command to read the .p12 file. The Logserver GUI host must therefore reach port 8081 on the monitored hosts and have openssl installed.

Both roles require Java 8 or later. Communication uses mutual TLS; certificates are generated with the bundled tlstool.sh script.

Agents module

Before use ensure that you have all the required files

  • Tool for generating the necessary certificates: /usr/share/logserver/utils/tlstool/tlstool.sh;

  • Network Probe pipeline files, installed with the Network Probe and disabled by default:

      /etc/logserver-probe/conf.d/masteragent/ {01-input-agents.conf, 050-filter-agents.conf, 100-output-agents.conf}
      /etc/logserver-probe/pipelines.d/masteragent.yml.off
    
  • Linux Agent files, delivered as an archive at ./agents/masteragent/linux/masteragent-linux-<version>.zip. Unpack it in place and the files appear in a masteragent directory beside it:

    cd ./agents/masteragent/linux
    unzip masteragent-linux-*.zip
    
    • Executable: masteragent/MasterBeatAgent.jar

    • Configuration File for MasterAgent (server): masteragent/MasterBeatAgent.conf

    • Configuration File for Agent (client): masteragent/agent.conf

    • Service file: masteragent/masteragent.service

    The Windows archive at ./agents/masteragent/windows/masteragent-windows-<version>.zip has the same structure and contains Agents.jar, agents.exe, agents.xml and agent.conf.

    The same archives are available in the GUI under Agents > Agents download. Choose Latest or 8.1.0. The list also offers 7.9.0, which returns an error, and after an upgrade 8.0.0, which downloads an empty archive.

Preparations

Note

All commands below must be executed from the /INSTALL directory.

  1. Generate the certificates using tlstool.sh script from /usr/share/logserver/utils/tlstool/.

    • Update the IP of the node directive, by replacing <logstash ip address> with the Network Probe server ip in the provided masteragent.yml config (/usr/share/logserver/utils/tlstool/config/masteragent.yml).

    • Generate certs using masteragent configuration. The target directory must exist, otherwise tlstool.sh stops with Target directory does not exist:

      mkdir -p agents/masteragent/certificates
      /usr/share/logserver/utils/tlstool/tlstool.sh -c /usr/share/logserver/utils/tlstool/config/masteragent.yml -ca -crt -t agents/masteragent/certificates
      
    • Create KeyStore and TrustStore:

      cd agents/masteragent/certificates
      keytool -import -file rootCA.crt -alias root -keystore root.jks -storetype jks
      openssl pkcs12 -export -in localhost.crt -inkey localhost.key -out node_name.p12 -name localhost -certfile rootCA.crt
      cd -
      
    • At the keytool prompt, set the TrustStore password of your choice that is used to secure CAs, and type ‘yes’ when the “Trust this certificate?” prompt is shown.

    • At the openssl prompt, set the KeyStore password of your choice that is utilized to securely store certificates. Remember both passwords, you will need them later.

  2. Configure firewall to enable communication on used ports (defaults: TCP 8080 -> Network Probe, TCP 8081 -> agent’s server).

    • These ports can be changed but must reflect “port” and “logstash” directives from an agent.conf file to ensure a connection with the agent.

    • Port 8080 must be open on the Network Probe host. The installer opens it when it installs the Network Probe; otherwise run:

      firewall-cmd --permanent --zone public --add-port 8080/tcp
      
    • Port 8081 must be open on each monitored host, where the agent listens, and reachable from the Logserver GUI host. On a Linux monitored host with firewalld, run:

      firewall-cmd --permanent --zone public --add-port 8081/tcp
      
  3. Configure Network Probe:

    • Enable the masteragent pipeline. In the GUI, open Integrations, select Master Agent and activate its pipeline on the Network Probe in the Pipelines step, as described in Integrations. To enable it from the command line instead, rename the definition to .yml, so that the installer keeps it when it rebuilds pipelines.yml, and append it to pipelines.yml:

      cd /etc/logserver-probe/pipelines.d
      mv masteragent.yml.off masteragent.yml
      cat masteragent.yml >> /etc/logserver-probe/pipelines.yml
      cd -
      

      After an upgrade from 7.x, /etc/logserver-probe/conf.d/masteragent/ and pipelines.d/masteragent.yml.off still hold the 7.x files, which point to /etc/logstash/ paths. Replace them with the files from install/logstash/logstash/conf.d/masteragent/ and install/logstash/logstash/pipelines.d/masteragent.yml.off in the 8.1 installation package before you enable the pipeline.

    • Configure SSL connection, by copying previously generated certificates:

      mkdir -p /etc/logserver-probe/conf.d/masteragent/ssl
      /bin/cp -rf ./agents/masteragent/certificates/localhost.* ./agents/masteragent/certificates/rootCA.crt /etc/logserver-probe/conf.d/masteragent/ssl/
      
    • Enable SSL on the input. The agent connects over https:// by default, but the ssl lines in /etc/logserver-probe/conf.d/masteragent/01-input-agents.conf ship commented out. Uncomment ssl, ssl_key, ssl_certificate, ssl_certificate_authorities and ssl_verify_mode; leave ssl_key_passphrase commented, because tlstool.sh generates the key without a password.

    • The output in 100-output-agents.conf writes to 127.0.0.1:9200 over HTTP. If the Logserver API on that host uses HTTPS, set in each logserver block the same connection settings as other outputs of the Network Probe, for example:

      hosts => "${HOSTS}"
      ssl => "${SSL}"
      ssl_certificate_verification => false
      
    • Set permissions:

      chown -R logserver-probe:logserver-probe /etc/logserver-probe/conf.d/masteragent
      
    • Restart the Network Probe service:

      systemctl restart logserver-probe
      

Installation of MasterAgent - Server Side

  • Copy executable and config:

    mkdir -p /opt/agents
    /bin/cp -rf ./agents/masteragent/linux/masteragent/MasterBeatAgent.jar /opt/agents
    /bin/cp -rf ./agents/masteragent/linux/masteragent/MasterBeatAgent.conf /opt/agents/agent.conf
    
  • Copy certificates:

    /bin/cp -rf ./agents/masteragent/certificates/node_name.p12 ./agents/masteragent/certificates/root.jks /opt/agents/
    
  • Set permissions:

    chown -R logserver-probe:logserver-probe /opt/agents
    
  • Update the configuration file with KeyStore/TrustStore paths and passwords. Use your preferred editor eg. vim:

    vim /opt/agents/agent.conf
    

Installation of Agent - Client Side

Linux

Requires Java 8 or later.

Linux Agent - software installed on clients running on Linux OS:

  1. Install net-tools package to use Agent on Linux RH / Centos:

    yum install net-tools
    
  2. Copy executable and config:

    mkdir -p /opt/masteragent
    /bin/cp -rf ./agents/masteragent/linux/masteragent/agent.conf ./agents/masteragent/linux/masteragent/MasterBeatAgent.jar /opt/masteragent
    /bin/cp -rf ./agents/masteragent/linux/masteragent/masteragent.service /usr/lib/systemd/system/masteragent.service
    
  3. Copy certificates:

    /bin/cp -rf ./agents/masteragent/certificates/node_name.p12 ./agents/masteragent/certificates/root.jks /opt/masteragent/
    
  4. Update the configuration file with KeyStore/TrustStore paths and passwords. Also, set the logstash directive to the Network Probe URL and port. Use your preferred editor eg. vim:

    vim /opt/masteragent/agent.conf
    

    Key directives to set:

    Directive

    Description

    logstash

    Network Probe endpoint URL, e.g. https://192.168.1.10:8080

    port

    Port the agent listens on for management commands (default: 8081)

    https_keystore

    Path to the .p12 KeyStore file

    https_keystore_pass

    KeyStore password

    https_truststore

    Path to the .jks TrustStore file

    https_truststore_pass

    TrustStore password

    http_enabled

    Enable the agent’s embedded HTTP server for management commands (true by default)

    filebeat_path, metricbeat_path, packetbeat_path, winlogbeat_path

    Directories with the Beats configuration files that the agent reports and manages. agent.conf has no such directive for Auditbeat and Heartbeat; custom_list adds extra directories and files to monitor

    createfile_folder

    Directories where the agent may create new files from templates, separated by semicolons

  5. Enable masteragent service:

    systemctl daemon-reload
    systemctl enable masteragent
    systemctl start masteragent
    
  6. Finally, verify in the Logserver GUI ‘Agents’ plugin if a newly added agent is present. Check masteragent logs executing:

    journalctl -fu masteragent
    

Windows

Requires Java 8 or later.

  1. Unpack the Windows archive in place, the same way as the Linux one:

    cd ./agents/masteragent/windows
    unzip masteragent-windows-*.zip
    

    Ensure that you have all required files in ./agents/masteragent/windows/masteragent:

    • Installer and manifest: agents.exe, agents.xml

    • Client: Agents.jar

    • Configuration File: agent.conf

  2. Configure firewall:

    Add an exception to the firewall to allow outgoing connection to TCP port 8080 (Network Probe).
    Add an exception to the firewall to listen on TCP port 8081, required only when http_enabled = true in agent.conf (the default). Disabling http_enabled prevents the agent from receiving management commands from the Logserver GUI.

  3. Create C:\Program Files\MasterAgent directory.

  4. Copy the contents of the ./agents/masteragent/windows/masteragent directory to the C:\Program Files\MasterAgent.

  5. Copy node_name.p12 and root.jks files from the ./agents/masteragent/certificates to desired directory.

  6. Update the C:\Program Files\MasterAgent\agent.conf file with KeyStore/TrustStore paths from the previous step and passwords. Also, set the logstash directive to the Network Probe URL and port. See the directive table in the Linux section above.

    The default winlogbeat_path, filebeat_path, metricbeat_path and packetbeat_path point to C:\ProgramData\Elastic\Beats\<beat>. If you install the Beats as described in Windows - Beats agents installation, set them to the installation directories, for example winlogbeat_path = C:\Program Files\Winlogbeat, otherwise the agent does not find the Beats configuration files.

  7. Start PowerShell as an administrator:

    To install an agent you can use interchangeably the following methods:

    • Method 1 - use installer:

      cd "C:\Program Files\MasterAgent"
      .\agents.exe install
      .\agents.exe start
      
    • Method 2 - manually creating service:

      New-Service -name masteragent -displayName masteragent -binaryPathName "C:\Program Files\MasterAgent\agents.exe"
      
  8. Finally, verify in the Logserver GUI ‘Agents’ plugin if a newly added agent is present. To check out logs and errors, look for ‘agents.out.log’ and ‘agents.err.log’ files in the C:\Program Files\MasterAgent directory after the service starts. Also, check the service status:

    .\agents.exe status

Beats - configuration templates

  1. Go to the Agents which is located in the main menu. Then go to Templates and click the Add template button.

  2. Click the Create new file button at the bottom.

  3. You will see the form to create a file that will be on a client system.
    There are inputs such as:

    • Destination Path,

    • File name,

    • Description,

    • Upload file,

    • Content.

  4. Remember that you must provide the exact path to your directory in the Destination Path field. The agent creates new files only in the directories listed in createfile_folder in its agent.conf.

    The GUI accepts only files with the .conf, .cfg, .yml or .yaml extension (the agents.templateFilesExtensions setting in /etc/logserver-gui/logserver-gui.yml) and smaller than 1 MB. Templates are stored in the .agentsconfig index.

  5. After that add your file to the template by checking it from the Available files list and clicking Add. Enter a name in the Template name field and click Save template.

  6. You can now see your template in the Template tab

  7. The next step will be to add the template to the agent by checking the agent’s form list and clicking Apply Template.

  8. The last step is to apply the template by checking it from the list and clicking the Apply button.

    You can also select multiple agents. Remember, if your file path is Windows type You can only select Windows agents. You can check the Logs by clicking the icon in the logs column.

Agent module compatibility

The Agents module among others is compatible with Beats agents in the following versions:

No

Agent Name

Beats Version

1

Filebeat

OSS 7.17.29

2

Packetbeat

OSS 7.17.29

3

Winlogbeat

OSS 7.17.29

4

Metricbeat

OSS 7.17.29

5

Heartbeat

OSS 7.17.29

6

Auditbeat

OSS 7.17.29

Windows - Beats agents installation

Note

Beats for Windows are shipped as ZIP archives containing the executable, a default .yml, and the install-service-<beat>.ps1 script that registers the Windows service. There is no MSI, so rolling them out through Group Policy or a distribution tool means wrapping the unpack, the configuration file, and that script into a package of your own.

MasterAgent does not change this. It distributes configuration and restarts Beats on hosts that already run them, so install the Beat first and let MasterAgent manage it afterwards. If you need an agent that installs unattended out of the box, the Endpoint Security agent ships as an MSI, see SIEM Agents Configuration.

Winlogbeat

Installation
  1. Copy the Winlogbeat installer from the installation directory install/agents/beats/windows/winlogbeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Winlogbeat directory

Configuration

Editing the file: C:\Program Files\Winlogbeat\winlogbeat.yml:

  1. In section:

    winlogbeat.event_logs:
      - name: Application
        ignore_older: 72h
    
      - name: System
    
      - name: Security
    
      - name: ForwardedEvents
        tags: [forwarded]
    
      - name: Windows PowerShell
        event_id: 400, 403, 600, 800
    
      - name: Microsoft-Windows-PowerShell/Operational
        event_id: 4103, 4104, 4105, 4106
    

    change to:

    winlogbeat.event_logs:
      - name: Application
        ignore_older: 72h
    
      - name: System
        ignore_older: 72h
    
      - name: Security
        ignore_older: 72h
    
      - name: ForwardedEvents
        tags: [forwarded]
    
      - name: Windows PowerShell
        event_id: 400, 403, 600, 800
    
      - name: Microsoft-Windows-PowerShell/Operational
        event_id: 4103, 4104, 4105, 4106
    
  2. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  3. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  4. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  5. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  6. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["winlogbeat"]
    
  7. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Winlogbeat'
    .\install-service-winlogbeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Winlogbeat\install-service-winlogbeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped    Winlogbeat      Winlogbeat
    
  8. Start Winlogbeat service:

    Start-Service winlogbeat
    

Test configuration:

cd 'C:\Program Files\Winlogbeat'
.\winlogbeat.exe test config
.\winlogbeat.exe test output
Drop event

We can also drop events on the agent side. To do this we need to use the drop_event processor

processors:
  - drop_event:
      when:
        condition

Each condition receives a field to compare. You can specify multiple fields under the same condition by using AND between the fields (for example, field1 AND field2).

For each field, you can specify a simple field name or a nested map, for example, dns.question.name.

See Exported Fields for a list of all the fields that are exported by Winlogbeat.

The supported conditions are:

  • equals

  • contains

  • regexp

  • range

  • network

  • has_fields

  • or

  • and

  • not

equals

With the equals condition, you can compare if a field has a certain value. The condition accepts only an integer or a string value.

For example, the following condition checks if the response code of the HTTP transaction is 200:

equals:
  http.response.code: 200
contains

The contains condition checks if a value is part of a field. The field can be a string or an array of strings. The condition accepts only a string value.

For example, the following condition checks if an error is part of the transaction status:

contains:
  status: "Specific error"
regexp

The regexp condition checks the field against a regular expression. The condition accepts only strings.

For example, the following condition checks if the process name starts with foo:

regexp:
  system.process.name: "^foo.*"
range

The range condition checks if the field is in a certain range of values. The condition supports lt, lte, gt, and gte. The condition accepts only integer or float values.

For example, the following condition checks for failed HTTP transactions by comparing the http.response.code field with 400.

range:
  http.response.code:
    gte: 400

This can also be written as:

range:
  http.response.code.gte: 400

The following condition checks if the CPU usage in percentage has a value between 0.5 and 0.8.

range:
  system.cpu.user.pct.gte: 0.5
  system.cpu.user.pct.lt: 0.8
network

The network condition checks if the field is in a certain IP network range. Both IPv4 and IPv6 addresses are supported. The network range may be specified using CIDR notation, like “192.0.2.0/24” or “2001:db8::/32”, or by using one of these named ranges:

  • loopback - Matches loopback addresses in the range of 127.0.0.0/8 or ::1/128.

  • unicast - Matches global unicast addresses defined in RFC 1122, RFC 4632, and RFC 4291 with the exception of the IPv4 broadcast address (255.255.255.255). This includes private address ranges.

  • multicast - Matches multicast addresses.

  • interface_local_multicast - Matches IPv6 interface-local multicast addresses.

  • link_local_unicast - Matches link-local unicast addresses.

  • link_local_multicast - Matches link-local multicast addresses.

  • private - Matches private address ranges defined in RFC 1918 (IPv4) and RFC 4193 (IPv6).

  • public - Matches addresses that are not loopback, unspecified, IPv4 broadcast, link-local unicast, link-local multicast, interface local multicast, or private.

  • unspecified - Matches unspecified addresses (either the IPv4 address “0.0.0.0” or the IPv6 address “::”).

The following condition returns true if the source.ip value is within the private address space.

network:
  source.ip: private

This condition returns true if the destination.ip value is within the IPv4 range of 192.168.1.0 - 192.168.1.255.

network:
  destination.ip: '192.168.1.0/24'

This condition returns true when destination.ip is within any of the given subnets.

network:
  destination.ip: ['192.168.1.0/24', '10.0.0.0/8', loopback]
has_fields

The has_fields condition checks if all the given fields exist in the event. The condition accepts a list of string values denoting the field names.

For example, the following condition checks if the http.response.code field is present in the event.

has_fields: ['http.response.code']
or

The or operator receives a list of conditions.

or:
  - <condition1>
  - <condition2>
  - <condition3>
  ...

For example, to configure the condition http.response.code = 304 OR http.response.code = 404:

or:
  - equals:
      http.response.code: 304
  - equals:
      http.response.code: 404
and

The and operator receives a list of conditions.

and:
  - <condition1>
  - <condition2>
  - <condition3>
  ...

For example, to configure the condition http.response.code = 200 AND status = OK:

and:
  - equals:
      http.response.code: 200
  - equals:
      status: OK

To combine operators, for example <condition1> OR (<condition2> AND <condition3>):

or:
  - <condition1>
  - and:
    - <condition2>
    - <condition3>
not

The not operator receives the condition to negate.

not:
  <condition>

For example, to configure the condition NOT status = OK:

not:
  equals:
    status: OK
Internal queue

Winlogbeat uses an internal queue to store events before publishing them. The queue is responsible for buffering and combining events into batches that can be consumed by the outputs. The outputs will use bulk operations to send a batch of events in one transaction.

You can configure the type and behavior of the internal queue by setting options in the queue section of the winlogbeat.yml config file. Only one queue type can be configured.

This sample configuration sets the memory queue to buffer up to 4096 events:

queue.mem:
  events: 4096

Configure the memory queue The memory queue keeps all events in memory.

If no flush interval and no number of events to flush is configured, all events published to this queue will be directly consumed by the outputs. To enforce spooling in the queue, set the flush.min_events and flush.timeout options.

By default flush.min_events is set to 2048 and flush.timeout is set to 1s.

The output’s bulk_max_size setting limits the number of events being processed at once.

The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memory queue. Only after the signal from the output will the queue free up space for more events to be accepted.

This sample configuration forwards events to the output if 512 events are available or the oldest available event has been waiting for 5s in the queue:

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

Configuration options

You can specify the following options in the queue.mem section of the winlogbeat.yml config file: events Number of events the queue can store. The default value is 4096 events.

flush.min_events Minimum number of events required for publishing. If this value is set to 0, the output can start publishing events without additional waiting times. Otherwise, the output has to wait for more events to become available.

The default value is 2048.

flush.timeout Maximum wait time for flush.min_events to be fulfilled. If set to 0s, events will be immediately available for consumption. The default value is 1s.

Configure disk queue The disk queue stores pending events on the disk rather than the main memory. This allows Beats to queue a larger number of events than is possible with the memory queue, and to save events when a Beat or device is restarted. This increased reliability comes with a performance tradeoff, as every incoming event must be written and read from the device’s disk. However, for setups where the disk is not the main bottleneck, the disk queue gives a simple and relatively low-overhead way to add a layer of robustness to incoming event data.

The disk queue is expected to replace the file spool in a future release.

To enable the disk queue with default settings, specify a maximum size:

queue.disk:
  max_size: 10GB

The queue will be used up to the specified maximum size on the disk. It will only use as much space as required. For example, if the queue is only storing 1GB of events, then it will only occupy 1GB on disk no matter how high the maximum is. Queue data is deleted from the disk after it has been successfully sent to the output.

Configuration options

You can specify the following options in the queue.disk section of the winlogbeat.yml config file:

path The path to the directory where the disk queue should store its data files. The directory is created on startup if it doesn’t exist.

The default value is "${path.data}/diskqueue".

max_size (required) The maximum size the queue should use on disk. Events that exceed this maximum will either pause their input or be discarded, depending on the input’s configuration.

A value of 0 means that no maximum size is enforced, and the queue can grow up to the amount of free space on the disk. This value should be used with caution, as filling a system’s main disk can make it inoperable. It is best to use this setting only with a dedicated data or backup partition that will not interfere with Winlogbeat or the rest of the host system.

The default value is 10GB.

segment_size Data added to the queue is stored in segment files. Each segment contains some number of events waiting to be sent to the outputs and is deleted when all its events are sent. By default, segment size is limited to 1/10 of the maximum queue size. Using a smaller size means that the queue will use more data files, but they will be deleted more quickly after use. Using a larger size means some data will take longer to delete, but the queue will use fewer auxiliary files. It is usually fine to leave this value unchanged.

The default value is max_size / 10.

read_ahead The number of events that should be read from disk into memory while waiting for an output to request them. If you find outputs are slowing down because they can’t read as many events at a time, adjusting this setting upward may help, at the cost of higher memory usage.

The default value is 512.

write_ahead The number of events the queue should accept and store in memory while waiting for them to be written to disk. If you find the queue’s memory use is too high because events are waiting too long to be written to disk, adjusting this setting downward may help, at the cost of reduced event throughput. On the other hand, if inputs are waiting or discarding events because they are being produced faster than the disk can handle, adjusting this setting upward may help, at the cost of higher memory usage.

The default value is 2048.

retry_interval Some disk errors may block the operation of the queue, for example, a permission error writing to the data directory, or a disk full error while writing an event. In this case, the queue reports the error and retries after pausing for the time specified in retry_interval.

The default value is 1s (one second).

max_retry_interval When multiple consecutive errors are written to the disk, the queue increases the retry interval by factors of 2 up to a maximum of max_retry_interval. Increase this value if you are concerned about logging too many errors or overloading the host system if the target disk becomes unavailable for an extended time.

The default value is 30s (thirty seconds).

Filebeat

Filebeat Installation
  1. Copy the Filebeat installer from the installation directory install/agents/beats/windows/filebeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Filebeat directory

Configuration

Editing the file: C:\Program Files\Filebeat\filebeat.yml:

  1. In section:

    - type: filestream
    
      # Unique ID among all inputs, an ID is required.
      id: my-filestream-id
    
      # Change to true to enable this input configuration.
      enabled: false
    

    change to:

    - type: filestream
    
      # Unique ID among all inputs, an ID is required.
      id: my-filestream-id
    
      # Change to true to enable this input configuration.
      enabled: true
    
  2. In section:

      paths:
        - /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*
    

    change to:

      paths:
        #- /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*
        - 'C:\Program Files\Microsoft SQL Server\*\MSSQL\Log\*'
        - 'C:\inetpub\logs\*'
    
  3. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  4. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  5. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  6. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  7. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["filebeat"]
    
  8. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Filebeat'
    .\install-service-filebeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Filebeat\install-service-filebeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped  Filebeat        Filebeat
    
  9. Start Filebeat service:

    Start-Service filebeat
    

You can enable, disable, and list Filebeat modules using the following command:

cd 'C:\Program Files\Filebeat'
.\filebeat.exe modules list
.\filebeat.exe modules enable apache
.\filebeat.exe modules disable apache

Test configuration:

cd 'C:\Program Files\Filebeat'
.\filebeat.exe test config
.\filebeat.exe test output

Metricbeat

Metricbeat Installation
  1. Copy the Metricbeat installer from the installation directory install/agents/beats/windows/metricbeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Metricbeat directory

Configuration

Editing the file: C:\Program Files\Metricbeat\metricbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
      #index.codec: best_compression
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["metricbeat"]
    
  6. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Metricbeat'
    .\install-service-metricbeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Metricbeat\install-service-metricbeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped  Metricbeat        Metricbeat
    
  7. Start Metricbeat service:

    Start-Service metricbeat
    

You can enable, disable, and list Metricbeat modules using the following command:

cd 'C:\Program Files\Metricbeat'
.\metricbeat.exe modules list
.\metricbeat.exe modules enable apache
.\metricbeat.exe modules disable apache

Test configuration:

cd 'C:\Program Files\Metricbeat'
.\metricbeat.exe test config
.\metricbeat.exe test output

Packetbeat

Packetbeat Installation
  1. Copy the Packetbeat installer from the installation directory install/agents/beats/windows/packetbeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Packetbeat directory

Configuration

Editing the file: C:\Program Files\Packetbeat\packetbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: [forwarded]
    

    change to:

    tags: ["packetbeat"]
    
  6. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Packetbeat'
    .\install-service-packetbeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Packetbeat\install-service-packetbeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped  Packetbeat        Packetbeat
    
  7. Start Packetbeat service:

    Start-Service packetbeat
    

Test configuration:

cd 'C:\Program Files\Packetbeat'
.\packetbeat.exe test config
.\packetbeat.exe test output

Auditbeat

Auditbeat Installation
  1. Copy the Auditbeat installer from the installation directory install/agents/beats/windows/auditbeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Auditbeat directory

Configuration

Editing the file: C:\Program Files\Auditbeat\auditbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["auditbeat"]
    
  6. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Auditbeat'
    .\install-service-auditbeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Auditbeat\install-service-auditbeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped  Auditbeat        Auditbeat
    
  7. Start Auditbeat service:

    Start-Service auditbeat
    

Test configuration:

cd 'C:\Program Files\Auditbeat'
.\auditbeat.exe test config
.\auditbeat.exe test output

Heartbeat

Heartbeat Installation
  1. Copy the Heartbeat installer from the installation directory install/agents/beats/windows/heartbeat-oss-7.17.29-windows-x86_64.zip and unpack

  2. Copy the installation files to the C:\Program Files\Heartbeat directory

Configuration

Editing the file: C:\Program Files\Heartbeat\heartbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
      #index.codec: best_compression
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["heartbeat"]
    
  6. Run the PowerShell console as Administrator and execute the following commands:

    cd 'C:\Program Files\Heartbeat'
    .\install-service-heartbeat.ps1
    
    Security warning
    Run only scripts that you trust. While scripts from the internet can be useful,
    this script can potentially harm your computer. If you trust this script, use
    the Unblock-File cmdlet to allow the script to run without this warning message.
    Do you want to run C:\Program Files\Heartbeat\install-service-heartbeat.ps1?
    [D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
    

    Output:

    Status   Name               DisplayName
    ------   ----               -----------
    Stopped  Heartbeat        Heartbeat
    
  7. Start Heartbeat service:

    Start-Service heartbeat
    

Test configuration:

cd 'C:\Program Files\Heartbeat'
.\heartbeat.exe test config
.\heartbeat.exe test output

Linux - Beats agents installation

Filebeat (Linux)

Filebeat (Linux) Installation
  1. Copy the Filebeat installer from the installation directory install/agents/beats/linux/filebeat-oss-7.17.29-x86_64.rpm

  2. Install filebeat with the following command:

    yum install -y filebeat-oss-7.17.29-x86_64.rpm
    
Configuration

Editing the file: /etc/filebeat/filebeat.yml:

  1. In section:

    - type: filestream
    
      # Unique ID among all inputs, an ID is required.
      id: my-filestream-id
    
      # Change to true to enable this input configuration.
      enabled: false
    

    change to:

    - type: filestream
    
      # Unique ID among all inputs, an ID is required.
      id: my-filestream-id
    
      # Change to true to enable this input configuration.
      enabled: true
    
  2. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  3. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  4. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  5. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  6. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["filebeat"]
    
  7. Enable and start Filebeat service:

    systemctl enable filebeat
    systemctl start filebeat
    

You can enable, disable, and list Filebeat modules using the following command:

filebeat modules list
filebeat modules enable apache
filebeat modules disable apache

Test configuration:

filebeat test config
filebeat test output

Metricbeat (Linux)

Metricbeat (Linux) Installation
  1. Copy the Metricbeat installer from the installation directory install/agents/beats/linux/metricbeat-oss-7.17.29-x86_64.rpm

  2. Install Metricbeat with the following command:

    yum install -y metricbeat-oss-7.17.29-x86_64.rpm
    
Configuration

Editing the file: /etc/metricbeat/metricbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
      #index.codec: best_compression
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["metricbeat"]
    

Enable and start Metricbeat service:

systemctl enable metricbeat
systemctl start metricbeat

You can enable, disable, and list Metricbeat modules using the following command:

metricbeat modules list
metricbeat modules enable apache
metricbeat modules disable apache

Test configuration:

metricbeat test config
metricbeat test output

Packetbeat (Linux)

Packetbeat (Linux) Installation
  1. Copy the Packetbeat installer from the installation directory install/agents/beats/linux/packetbeat-oss-7.17.29-x86_64.rpm

  2. Install Packetbeat with the following command:

    yum install -y packetbeat-oss-7.17.29-x86_64.rpm
    
Configuration

Editing the file: /etc/packetbeat/packetbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: [forwarded]
    

    change to:

    tags: ["packetbeat"]
    

Enable and start Packetbeat service:

systemctl enable packetbeat
systemctl start packetbeat

Test configuration:

packetbeat test config
packetbeat test output

Auditbeat (Linux)

Auditbeat (Linux) Installation
  1. Copy the Auditbeat installer from the installation directory install/agents/beats/linux/auditbeat-oss-7.17.29-x86_64.rpm

  2. Install Auditbeat with the following command:

    yum install -y auditbeat-oss-7.17.29-x86_64.rpm
    
Configuration

Editing the file: /etc/auditbeat/auditbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["auditbeat"]
    

Enable and start Auditbeat service:

systemctl enable auditbeat
systemctl start auditbeat

Test configuration:

auditbeat test config
auditbeat test output

Heartbeat (Linux)

Heartbeat (Linux) Installation
  1. Copy the Heartbeat installer from the installation directory install/agents/beats/linux/heartbeat-oss-7.17.29-x86_64.rpm

  2. Install Heartbeat with the following command:

    yum install -y heartbeat-oss-7.17.29-x86_64.rpm
    
Configuration

Editing the file: /etc/heartbeat/heartbeat.yml:

  1. In section:

    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
    

    change to:

    #setup.template.settings:
      #index.number_of_shards: 1
      #index.codec: best_compression
    
  2. In section:

    setup.kibana:
    

    change to:

    #setup.kibana:
    
  3. In section:

    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
    

    change to:

    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
  4. In section:

    #output.logstash:
      # The Logstash hosts
      #hosts: ["localhost:5044"]
    

    change to:

    output.logstash:
      # The Logstash hosts
      hosts: ["logserver-probe-ip:5044"]
    
  5. In section:

    #tags: ["service-X", "web-tier"]
    

    change to:

    tags: ["heartbeat"]
    

Enable and start Heartbeat service:

systemctl enable heartbeat-elastic
systemctl start heartbeat-elastic

Test configuration:

heartbeat test config
heartbeat test output