Custom Integrations: Parsers and Pipelines

Built-in Integrations cover common vendors with one-click pipelines, dashboards, and alerts. When a device or application is not on that list (a niche appliance, an in-house system, or hardware that emits logs in its own format), you connect it by building a custom pipeline: an input that receives the data, a parser that turns each raw line into structured fields, and an output that writes the result to the Data Node.

This chapter walks through that process end to end on a single example: a custom syslog device. The same three stages apply to any source; only the input and the parser change.

How data flows through a probe 

A Network Probe pipeline is a Logstash configuration with three stages:

Stage	Purpose	Reference
input	Receives raw events (syslog, files, HTTP, database, Kafka, and more) and tags them with a `type`.	Technical Configuration, the `Input *` sections
filter (parser)	Matches the raw `message` against patterns and extracts named fields. This is the parser.	Filter “network”, Filter “beats syslog”
output	Sends the structured event to the Data Node and creates the index.	Output to Data Node

Pipeline configuration files live on the probe in /etc/logserver-probe/conf.d/ and use the .conf extension. The probe runs them through its bundled Logstash engine at /usr/share/logserver-probe/bin/logstash. Built-in integrations keep their files in per-vendor subdirectories and split the stages across numbered files (for example 01.input.conf, 20.filter.conf, 30.output.conf) so that they load in order. A custom pipeline can follow the same convention or use a single combined .conf file.

Before you start 

Check the Integrations catalog first. If your device is listed, use One Click Installation instead of building a pipeline by hand.
Confirm the probe can receive the data (network path, firewall, credentials).
Capture a sample of the raw logs exactly as the device sends them. You cannot write a parser without a representative line.

Worked example: onboarding a custom syslog device 

The example device is a firewall appliance (acme-fw01) that sends syslog in a custom key-value format that no built-in integration recognises:

<134>Jul  1 10:15:32 acme-fw01 ACME: user=jsmith action=BLOCK src=10.0.0.5 dst=8.8.8.8 proto=TCP

The goal: receive these lines, extract the fields (user, action, src_ip, dst_ip, proto), and index them so they are searchable in Discover.

1. Receive the logs (input)

Use the tcp and udp inputs to listen for syslog, and tag the events with a type that identifies this device. The type is what the parser and output key off later.

input {
  tcp { port => 5514 type => "acme" }
  udp { port => 5514 type => "acme" }
}

If the device sends to the standard syslog port 514, redirect it to the collector port with the firewall rules shown in Input “network”.

2. Write the parser (grok filter)

The parser is a grok filter guarded by if [type] == "acme", so it only runs on events from this device. Each %{PATTERN:field} extracts one value into a named field:

filter {
  if [type] == "acme" {
    grok {
      match => {
        "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} ACME: user=%{USERNAME:user} action=%{WORD:action} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} proto=%{WORD:proto}"
      }
    }
    date {
      match  => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      target => "@timestamp"
    }
  }
}

The pattern names used here (SYSLOGTIMESTAMP, SYSLOGHOST, IPORHOST, WORD, USERNAME) are standard grok patterns. For more complex formats you can define your own with pattern_definitions, set named_captures_only => true, or map values through a dictionary. See the worked filters in Filter “beats syslog” and Filter “network”.

3. Test the parser before deploying 

There is no separate grok-testing UI, but the probe already ships Logstash, so you can test a parser against your sample line in seconds before touching any probe. Create a throwaway config that reads from the keyboard and prints the parsed result:

# /tmp/acme-test.conf
input  { stdin { type => "acme" } }
filter {
  grok {
    match => { "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:device} ACME: user=%{USERNAME:user} action=%{WORD:action} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} proto=%{WORD:proto}" }
  }
}
output { stdout { codec => rubydebug } }

Run it with a temporary data directory so it does not clash with the running probe service, paste your sample line, and press Ctrl+D:

/usr/share/logserver-probe/bin/logstash \
  -f /tmp/acme-test.conf \
  --path.data /tmp/ls-grok-test

If the pattern matches, Logstash prints the event with the extracted fields (user, action, src_ip, …). If you see a _grokparsefailure tag, adjust the pattern and run again. When the fields look right, validate the full configuration syntax without starting the pipeline:

/usr/share/logserver-probe/bin/logstash --config.test_and_exit -f /tmp/acme-test.conf

Note

This is a local test only. It never writes to the Data Node and does not change any probe pipeline. Remove /tmp/acme-test.conf and /tmp/ls-grok-test when you are done.

4. Send to the Data Node (output)

Add the logserver output. index => "%{type}-%{+YYYY.MM.dd}" creates a daily index named after the type, so this device lands in acme-*:

output {
  logserver {
    hosts    => [ "127.0.0.1:9200" ]
    index    => "%{type}-%{+YYYY.MM.dd}"
    user     => "user"
    password => "password"
  }
}

5. Assemble and deploy the pipeline 

Combine the input, filter, and output into one .conf file (or keep them as separate numbered files) and deploy it to the probes. There are two ways:

GUI: New Pipeline Creator. In the Network Probe tab, click New pipeline, name it, and drag in your .conf file(s). Each must be .conf and no larger than 1 MB. Select the probes to enable it on and Submit. See New Pipeline Creator.
GUI: Files Section. Create or upload the file directly under /etc/logserver-probe/conf.d/ on a probe, then enable it. The editor validates the file: a .conf that cannot be parsed is flagged with a warning icon you can click to read the error. See Files Section.

6. Verify 

In Pipelines Section, confirm the pipeline shows a green, enabled status on each assigned probe.
In Files parsing, confirm the .conf has no warning icon.
In Discover, open the acme-* index pattern and confirm events arrive with the extracted fields populated.

Common patterns and pitfalls 

Guard every parser with its type. if [type] == "acme" keeps this device’s grok from running on unrelated events and prevents _grokparsefailure noise.
Reuse patterns and dictionaries. Translate raw codes to readable values with the translate filter and dictionaries in /etc/logserver-probe/dictionaries/, as shown in Filter “network”.
Multiline events. Stack traces or multi-line records need the multiline codec. See Multiline codec.
Duplicate documents. If events are indexed twice, review Avoiding duplicate documents.

Troubleshooting 

Parser produces no fields / _grokparsefailure: re-run the local test in step 3 against the exact failing line; the pattern does not match the real input.
Pipeline will not start after deploy: run --config.test_and_exit on the deployed file, or read the warning in Files parsing.
Wider probe issues: see Known issues and troubleshooting.