Connecting Data Sources

Every source reaches Energy Logserver in one of two ways. Either you install software on the source and it ships data out, or the Network Probe receives or fetches the data without anything being installed on the source at all. The first is agent-based collection, the second is agentless.

This page helps you pick between them and points to the procedure for each method. It does not repeat those procedures.

How data reaches the system 

Whichever method you choose, the data lands in a Network Probe pipeline, which receives it, parses it into fields, and writes it to the Data Node. The three pipeline stages and the files that hold them are described in How data flows through a probe.

What the method decides is only the first stage, the input. An agent connects to a listener the probe already runs. An agentless source either sends to a listener or is polled by the probe on a schedule.

Choosing between agent and agentless 

Method	Runs on the source	Direction	Choose it when	Trade-off
Beats agent	Beat service	Source connects to probe	You control the host and need file, event log, metric, or packet data	One service to deploy and update per host
Beats managed by MasterAgent	Beat service plus MasterAgent	Source connects to probe	You run many hosts and want to push configuration and restart Beats from the GUI	Adds a second service and a certificate to manage
Endpoint Security agent	Endpoint Security agent	Source connects to manager	You need file integrity monitoring and security configuration assessment	Heavier than a Beat, separate enrollment
Syslog	Nothing	Source sends to probe	The device can only emit syslog, which covers most network and security appliances	No delivery guarantee over UDP, parsing depends on the vendor format
Windows Event Collector	Nothing	Source pushes to collector	You cannot install software on Windows hosts	Requires certificates, group policy, and a WS-Man subscription
WinRM polling	Nothing	Probe polls the source	You need Windows data from a small number of hosts without agents or subscriptions	Polling interval limits freshness, credentials stored on the probe
Flow export	Nothing	Source sends to probe	You need traffic visibility across a network rather than logs from a host	Flow records describe connections, not payloads
SPAN through Kafka	Nothing	Probe reads the mirror	You need full traffic analysis from a mirrored port	Requires switch configuration and the bundled Kafka broker
Database polling	Nothing	Probe polls the source	The application logs to a table rather than a file	Adds query load on the database, needs a read account
API or script polling	Nothing	Probe polls the source	The vendor exposes an API, as cloud and vulnerability scanners do	Credentials live on the probe, rate limits apply
File or share	Nothing	Probe reads the path	The source already writes to a share the probe can mount	Delivery depends on the share staying available

Agent-based collection 

Path	Where the procedure lives
MasterAgent and Beats	Agents
Endpoint Security agent	SIEM Agents Configuration and Endpoint Security Capabilities
Collector shipped with an integration	Advanced Installation

Beats ship with the installer at version 7.17.29 for both Windows and Linux. Winlogbeat is Windows only. The installer copies the Linux packages and the Windows archives into the agent repository that the GUI serves, so you can download them from the Console rather than from the installation media.

Only a small number of integrations bundle a collector. For every other integration the wizard has no agent step, because the source sends data on its own or the probe fetches it.

Agentless collection 

Syslog 

The probe listens on TCP and UDP 5514, and the installer forwards the standard port 514 to it. This is the path for firewalls, routers, switches, and any appliance that emits syslog. See Input “network” for the listener and the firewall rules, and Custom Integrations for writing a parser when the vendor is not in the catalog.

Windows Event Collector 

Windows hosts forward their event logs to a collector that subscribes to them over WS-Man, with nothing installed on the source. The collector listens on TCP 5986 and forwards the events to a probe pipeline. Certificates and a group policy on the source side are required. See Event Collector.

WinRM polling 

The probe runs a script that queries Windows hosts over WinRM on a schedule and turns the results into events. This suits a handful of hosts where neither an agent nor a subscription is practical. The integration is installed from the catalog, see Integrations.

Flow export 

Network devices export flow records to the probe: NetFlow on UDP 2055, IPFIX on 4739, and sFlow on UDP 6343. To account for traffic that the probe host itself sees, rather than traffic reported by a device, see pmacct Local Traffic Monitoring.

SPAN through Kafka 

Traffic mirrored from a switch port is analysed by the bundled sensors and carried to the pipeline through Kafka. Kafka and Zookeeper are installed but masked, and must be unmasked before use, see Kafka and Zookeeper. The sensor side is covered in Suricata IDS and Zeek.

grep -r "port =>" /etc/logserver-probe/conf.d/<integration>/
firewall-cmd --zone=public --add-port=<port>/tcp --permanent
firewall-cmd --zone=public --add-port=<port>/udp --permanent
firewall-cmd --reload

Note

Two integrations cannot share a listener port. If a pipeline fails to start with an address already in use error, check whether another enabled integration binds the same port and change one of them.

Verify the source is delivering 

Confirm the probe is running: systemctl status logserver-probe.
Confirm the pipeline is registered in /etc/logserver-probe/pipelines.yml.
Confirm the listener is bound, for a source that sends data: ss -lnp | grep <port>.
Open Discover in the Console and select the index the pipeline writes to.
If no documents arrive, work through Custom Integrations Troubleshooting.