Agents module¶
The Agents module is used for the central management of agents used in Energy Logserver such as Filebeat, Winlogbeat, Packetbeat, Metricbeat, Logstash and all other configuration files.
Component modules¶
The software consists of two modules:
- Agent Module - installation just like any standard Kibana plugin.
- Agent software - installed on host with agent (like beats);
Agent Module installation¶
All necessary components can be found in the installation folder ./install/Agents/masteragent
.
- Go to installation directory:
cd ./install/Agents/masteragent
Generating the certificates:
cd certificates/
set DOMAIN and DOMAIN_IP in scripts from
./certificates
directory:#!/bin/bash DOMAIN="localhost" DOMAIN_IP="10.4.3.185"
execute the scripts in the following order:
./1_rootca.sh ./2_clientcrt.sh ./3_createstore.sh
Install the required packages:
yum install net-tools
Add an exception to the firewall to listen on TCP 8080 and 8081:
firewall-cmd --permanent --zone public --add-port 8080/tcp firewall-cmd --permanent --zone public --add-port 8081/tcp
Logstash pipeline configuration:
/bin/cp -rf ./logstash/agents_template.json /etc/logstash/templates.d/ mkdir /etc/logstash/conf.d/masteragent /bin/cp -rf ./logstash/*.conf /etc/logstash/conf.d/masteragent/
- Edit file
/etc/logstash/pipelines.yml
by uncomment this line(be awer to this lines looks likes other uncomment lines. It’s yml file.):
- pipeline.id: masteragent path.config: "/etc/logstash/conf.d/masteragent/*.conf
- Logstash SSL configuration:
mkdir /etc/logstash/conf.d/masteragent/ssl /bin/cp -rf ./certificates/domain.key /etc/logstash/conf.d/masteragent/ssl/ /bin/cp -rf ./certificates/domain.crt /etc/logstash/conf.d/masteragent/ssl/ /bin/cp -rf ./certificates/rootCA.crt /etc/logstash/conf.d/masteragent/ssl/ chown -R logstash:logstash /etc/logstash
- Edit file
Linux Agent installation¶
Copy necessary files to destination host:
/bin/cp -rf ./install/Agents/masteragent/agents/linux/masteragent /opt/masteragent /bin/cp -rf ./install/Agents/masteragent/certificates/node_name.p12 /opt/masteragent /bin/cp -rf ./install/Agents/masteragent/certificates/root.jks /opt/masteragent /bin/cp -rf ./install/Agents/masteragent/agents/linux/masteragent/masteragent.service /usr/lib/systemd/system/masteragent.service
Set correct IP address of Logstash and Kibana in /opt/masteragent/agent.conf and verify paths for Filebeat, Metricbeat, etc. are correct.
systemctl daemon-reload systemctl enable masteragent systemctl start masteragent
Restart logstash:
systemctl restart logstash
In the GUI, in the Agents tab, you can check the status of the newly connected host.
Windows Agent installation¶
Add an exception to the firewall to listen on TCP port 8081.
Add an exception to the firewall enabling connection on TCP LOGSTASH_IP:8080 port.
Copy content of the ./agents/windows from installation directory to “C:\Program Files\MasterAgnet”
Change IP address of the Kibana GUI server and Logstash server in “C:\Program Files\MasterAgnet\agent.conf” file.
In order to install the service, start the console as an administrator and execute the following commands:
cd "C:\Program Files\MasterAgent" agents.exe install agents.exe start
An alternative method of installing the service, run the PowerShell console as administrator and execute the following commands:
New-Service -name masteragent -displayName masteragent - binaryPathName "C:\Program Files\MasterAgent\agents.exe"
Check status of service via services.msc (if stoped, try start it agian).
In the GUI, in the Agents tab, you can check the status of the newly connected host.
Agent module compatibility¶
The Agents module works with Beats agents in the following versions:
Nr | Agent Name | Beats Version | Link to download |
---|---|---|---|
1 |
Filebeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/filebeat-oss-6-8-13 |
2 |
Packetbeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/packetbeat-oss-6-8-13 |
3 |
Winlogbeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/winlogbeat-oss-6-8-13 |
4 |
Metricbeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/metricbeat-oss-6-8-13 |
5 |
Heartbeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/heartbeat-oss-6-8-13 |
6 |
Auditbeat |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/auditbeat-oss-6-8-13 |
7 |
Logstash |
OSS 6.8.14 |
https://www.elastic.co/downloads/past-releases/logstash-oss-6-8-13 |
Beats agents installation¶
Windows¶
Winlogbeat¶
Installation¶
- Copy the Winlogbeat installer from the installation directory
install/Agents/beats/windows/winlogbeat-oss-6.8.14-windows-x86_64.zip
and unpack - Copy the installation files to the
C:\Program Files\Winlogbeat
directory
Configuration¶
Editing the file: C:\Program Files\Winlogbeat\winlogbeat.yml
:
In section:
winlogbeat.event_logs: - name: Application ignore_older: 72h - name: Security - name: System
change to:
winlogbeat.event_logs: - name: Application ignore_older: 72h - name: Security ignore_older: 72h - name: System ignore_older: 72h
In section:
setup.template.settings: index.number_of_shards: 1
change to:
#setup.template.settings: #index.number_of_shards: 1
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["winlogbeat"]
Run the PowerShell
console as Administrator and execute the following commands:
cd 'C:\Program Files\Winlogbeat'
.\install-service-winlogbeat.ps1
Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Winlogbeat\install-service-winlogbeat.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R
Output:
Status Name DisplayName
------ ---- -----------
Stopped Winlogbeat Winlogbeat
Start Winlogbeat service:
sc start Winlogbeat
Test configuration:
cd 'C:\Program Files\Winlogbeat'
winlogbeat.exe test config
winlogbeat.exe test output
Filebeat¶
Installation¶
- Copy the Filebeat installer from the installation directory
install/Agents/beats/windows/filebeat-oss-6.8.14-windows-x86_64.zip
and unpack - Copy the installation files to the
C:\Program Files\Filebeat
directory
Configuration¶
Editing the file: C:\Program Files\Filebeat\filebeat.yml
:
In section:
- type: log # Change to true to enable this input configuration. enabled: false
change to:
- type: log # Change to true to enable this input configuration. enabled: true
In section:
paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs\*
change to:
paths: #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* - "C:\Program Files\Microsoft SQL Server\*\MSSQL\Log\*" - "C:\inetpub\logs\*""
In section:
setup.template.settings: index.number_of_shards: 1
change to:
#setup.template.settings: #index.number_of_shards: 1
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["filebeat"]
Run the PowerShell
console as Administrator and execute the following commands:
cd 'C:\Program Files\Filebeat'
.\install-service-filebeat.ps1
Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Filebeat\install-service-filebeat.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R
Output:
Status Name DisplayName
------ ---- -----------
Stopped Filebeat Filebeat
Start Filebeat service:
sc start filebeat
You can enable, disable and list Filebeat modules using the following command:
cd 'C:\Program Files\Filebeat'
filebeat.exe modules list
filebeat.exe modules apache enable
filebeat.exe modules apache disable
Test configuration:
cd 'C:\Program Files\Filebeat'
filebeat.exe test config
filebeat.exe test output
Merticbeat¶
Installation¶
- Copy the Merticbeat installer from the installation directory
install/Agents/beats/windows/merticbeat-oss-6.8.14-windows-x86_64.zip
and unpack - Copy the installation files to the
C:\Program Files\Merticbeat
directory
Configuration¶
Editing the file: C:\Program Files\Merticbeat\metricbeat.yml
:
In section:
setup.template.settings: index.number_of_shards: 1 index.codec: best_compression
change to:
#setup.template.settings: #index.number_of_shards: 1 #index.codec: best_compression
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["metricbeat"]
Run the PowerShell
console as Administrator and execute the following commands:
cd 'C:\Program Files\Metricbeat'
.\install-service-metricbeat.ps1
Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Metricbeat\install-service-metricbeat.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R
Output:
Status Name DisplayName
------ ---- -----------
Stopped Metricbeat Metricbeat
Start Filebeat service:
sc start metricbeat
You can enable, disable and list Metricbeat modules using the following command:
cd 'C:\Program Files\Metricbeat'
metricbeat.exe modules list
metricbeat.exe modules apache enable
metricbeat.exe modules apache disable
Test configuration:
cd 'C:\Program Files\Metricbeat'
metricbeat.exe test config
metricbeat.exe test output
Packetbeat¶
Installation¶
- Copy the Packetbeatinstaller from the installation directory
install/Agents/beats/windows/packetbeat-oss-6.8.14-windows-x86_64.zip
and unpack - Copy the installation files to the
C:\Program Files\Packetbeat
directory
Configuration¶
Editing the file: C:\Program Files\Packetbeat\packetbeat.yml
:
In section:
setup.template.settings: index.number_of_shards: 3
change to:
#setup.template.settings: #index.number_of_shards: 3
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["packetbeat"]
Run the PowerShell
console as Administrator and execute the following commands:
cd 'C:\Program Files\\Packetbeat'
.\install-service-packetbeat.ps1
Security warning
Run only scripts that you trust. While scripts from the internet can be useful,
this script can potentially harm your computer. If you trust this script, use
the Unblock-File cmdlet to allow the script to run without this warning message.
Do you want to run C:\Program Files\Packetbeat\install-service-packetbeat.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R
Output:
Status Name DisplayName
------ ---- -----------
Stopped Packetbeat Packetbeat
Start Packetbeat service:
sc start packetbeat
Test configuration:
cd 'C:\Program Files\Packetbeat'
packetbeat.exe test config
packetbeat.exe test output
Linux¶
Filebeat¶
Installation¶
Copy the Filebeat installer from the installation directory
install/Agents/beats/linux/filebeat-oss-6.8.14-x86_64.rpm
Install filebeat with following commadn:
yum install -y filebeat-oss-6.8.14-x86_64.rpm
Configuration¶
Editing the file: /etc/filebeat/filebeat.yml
:
In section:
- type: log # Change to true to enable this input configuration. enabled: false
change to:
- type: log # Change to true to enable this input configuration. enabled: true
In section:
setup.template.settings: index.number_of_shards: 1
change to:
#setup.template.settings: #index.number_of_shards: 1
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["filebeat"]
Start Filebeat service:
systemctl start filebeat
You can enable, disable and list Filebeat modules using the following command:
filebeat modules list
filebeat modules apache enable
filebeat modules apache disable
Test configuration:
filebeat test config
filebeat test output
Merticbeat¶
Installation¶
Copy the Merticbeatinstaller from the installation directory
install/Agents/beats/linux/metricbeat-oss-6.8.14-x86_64.rpm
Install Merticbeat with following command:
yum install -y metricbeat-oss-6.8.14-x86_64.rpm
Configuration¶
Editing the file: /etc/metricbeat/metricbeat.yml
:
In section:
setup.template.settings: index.number_of_shards: 1 index.codec: best_compression
change to:
#setup.template.settings: #index.number_of_shards: 1 #index.codec: best_compression
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["metricbeat"]
Start Filebeat service:
systemctl start metricbeat
You can enable, disable and list Metricbeat modules using the following command:
metricbeat modules list
metricbeat modules apache enable
metricbeat modules apache disable
Test configuration:
metricbeat test config
metricbeat test output
Packetbeat¶
Installation¶
Copy the Packetbeat installer from the installation directory
install/Agents/beats/linux/packetbeat-oss-6.8.14-x86_64.rpm
Install Packetbeatwith following command:
yum install -y packetbeat-oss-6.8.14-x86_64.rpm
Configuration¶
Editing the file: /etc/packetbeat/packetbeat.yml
:
In section:
setup.template.settings: index.number_of_shards: 3
change to:
#setup.template.settings: #index.number_of_shards: 3
In section:
setup.kibana:
change to:
#setup.kibana:
In section:
output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"]
change to:
#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"]
In section:
#output.logstash: # The Logstash hosts #hosts: ["localhost:5044"]
change to:
output.logstash: # The Logstash hosts hosts: ["LOGSTASH_IP:5044"]
In section:
#tags: ["service-X", "web-tier"]
change to:
tags: ["packetbeat"]
Start Packetbeat service:
servicectl start packetbeat
Test configuration:
packetbeat test config
packetbeat test output