Integrations

OP5 - Naemon logs

Logstash

  1. In Energy Logserver naemon_beat.conf set up ELASTICSEARCH_HOST, ES_PORT, FILEBEAT_PORT

  2. Copy Energy Logserver naemon_beat.conf to /etc/logstash/conf.d

  3. Based on “FILEBEAT_PORT” if firewall is running:

    sudo firewall-cmd --zone=public --permanent --add-port=FILEBEAT_PORT/tcp
    sudo firewall-cmd --reload
    
  4. Based on amount of data that elasticsearch will receive you can also choose whether you want index creation to be based on moths or days:

    index => "Energy Logserver-naemon-%{+YYYY.MM}"
    or
    index => "Energy Logserver-naemon-%{+YYYY.MM.dd}"
    
  5. Copy naemon file to /etc/logstash/patterns and make sure it is readable by logstash process

  6. Restart logstash configuration e.g.:

    sudo systemct restart logstash
    

Elasticsearch

  • Connect to Elasticsearch node via SSH and Install index pattern for naemon logs. Note that if you have a default pattern covering settings section you should delete/modify that in naemon_template.sh:

    "settings": {
        "number_of_shards": 5,
        "auto_expand_replicas": "0-1"
      },
    
  • Install template by running: ./naemon_template.sh
  • Energy Logserver Monitor

    1. On Energy Logserver Monitor host install filebeat (for instance via rpm https://www.elastic.co/downloads/beats/filebeat)

    2. In /etc/filebeat/filebeat.yml add:

       #=========================== Filebeat inputs =============================
       filebeat.config.inputs:
         enabled: true
         path: configs/*.yml
      
    3. You also will have to configure the output section in filebeat.yml. You should have one logstash output:

       #----------------------------- Logstash output --------------------------------
       output.logstash:
         # The Logstash hosts
         hosts: ["LOGSTASH_IP:FILEBEAT_PORT"]
      

      If you have few logstash instances - Logstash section has to be repeated on every node and hosts: should point to all of them:

       hosts: ["LOGSTASH_IP:FILEBEAT_PORT", "LOGSTASH_IP:FILEBEAT_PORT", "LOGSTASH_IP:FILEBEAT_PORT" ]
      
    4. Create /etc/filebeat/configs catalog.

    5. Copy naemon_logs.yml to a newly created catalog.

    6. Check the newly added configuration and connection to logstash. Location of executable might vary based on os:

       /usr/share/filebeat/bin/filebeat --path.config /etc/filebeat/ test config
       /usr/share/filebeat/bin/filebeat --path.config /etc/filebeat/ test output
      
    7. Restart filebeat:

       sudo systemctl restart filebeat # RHEL/CentOS 7
       sudo service filebeat restart # RHEL/CentOS 6
      

    Elasticsearch

    At this moment there should be a new index on the Elasticsearch node:

    curl -XGET '127.0.0.1:9200/_cat/indices?v'
    

    Example output:

    	health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    	green  open   Energy Logserver-naemon-2018.11    gO8XRsHiTNm63nI_RVCy8w   1   0      23176            0      8.3mb          8.3mb
    

    If the index has been created, in order to browse and visualise the data, “index pattern” needs to be added in Kibana.

    OP5 - Performance data

    Below instruction requires that between Energy Logserver node and Elasticsearch node is working Logstash instance.

    Elasticsearch

    1. First, settings section in Energy Logservertemplate.sh should be adjusted, either:

      • there is a default template present on Elasticsearch that already covers shards and replicas then settings sections should be removed from the Energy Logservertemplate.sh before executing

      • there is no default template - shards and replicas should be adjusted for you environment (keep in mind replicas can be added later, while changing shards count on existing index requires reindexing it)

          "settings": {
            "number_of_shards": 5,
            "number_of_replicas": 0
          }
        
    2. In URL Energy Logserverperfdata is a name for the template - later it can be search for or modify with it.

    3. The “template” is an index pattern. New indices matching it will have the settings and mapping applied automatically (change it if you index name for Energy Logserver perfdata is different).

    4. Mapping name should match documents type:

      "mappings": {
      	  "Energy Logserverperflogs"
      

      Running Energy Logservertemplate.sh will create a template (not index) for Energy Logserver perf data documents.

    Logstash

    1. The Energy Logserverperflogs.conf contains example of input/filter/output configuration. It has to be copied to /etc/logstash/conf.d/. Make sure that the logstash has permissions to read the configuration files:

      chmod 664 /etc/logstash/conf.d/Energy Logserverperflogs.conf
      
    2. In the input section comment/uncomment “beats” or “tcp” depending on preference (beats if Filebeat will be used and tcp if NetCat). The port and the type has to be adjusted as well:

      port => PORT_NUMBER
      type => "Energy Logserverperflogs"
      
    3. In a filter section type has to be changed if needed to match the input section and Elasticsearch mapping.

    4. In an output section type should match with the rest of a config. host should point to your elasticsearch node. index name should correspond with what has been set in elasticsearch template to allow mapping application. The date for index rotation in its name is recommended and depending on the amount of data expecting to be transferred should be set to daily (+YYYY.MM.dd) or monthly (+YYYY.MM) rotation:

      hosts => ["127.0.0.1:9200"]
      index => "Energy Logserver-perflogs-%{+YYYY.MM.dd}"
      
    5. Port has to be opened on a firewall:

      sudo firewall-cmd --zone=public --permanent --add-port=PORT_NUMBER/tcp
      sudo firewall-cmd --reload
      
    6. Logstash has to be reloaded:

      sudo systemctl restart logstash
      

      or

      sudo kill -1 LOGSTASH_PID
      

    Energy Logserver Monitor

    1. You have to decide wether FileBeat or NetCat will be used. In case of Filebeat - skip to the second step. Otherwise:

      • Comment line:

        54    open(my $logFileHandler, '>>', $hostPerfLogs) or die "Could not open $hostPerfLogs"; #FileBeat
        •	Uncomment lines:
        55 #    open(my $logFileHandler, '>', $hostPerfLogs) or die "Could not open $hostPerfLogs"; #NetCat
        ...
        88 #    my $logstashIP = "LOGSTASH_IP";
        89 #    my $logstashPORT = "LOGSTASH_PORT";
        90 #    if (-e $hostPerfLogs) {
        91 #        my $pid1 = fork();
        92 #        if ($pid1 == 0) {
        93 #            exec("/bin/cat $hostPerfLogs | /usr/bin/nc -w 30 $logstashIP $logstashPORT");
        94 #        }
        95 #    }
        
      • In process-service-perfdata-log.pl and process-host-perfdata-log.pl: change logstash IP and port:

        92 my $logstashIP = "LOGSTASH_IP";
        93 my $logstashPORT = "LOGSTASH_PORT";
        
    2. In case of running single Energy Logserver node, there is no problem with the setup. In case of a peered environment $do_on_host variable has to be set up and the script process-service-perfdata-log.pl/process-host-perfdata-log.pl has to be propagated on all of Energy Logserver nodes:

      16 $do_on_host = "EXAMPLE_HOSTNAME"; # Energy Logserver node name to run the script on
      17 $hostName = hostname; # will read hostname of a node running the script
      
    3. Example of command definition (/opt/monitor/etc/checkcommands.cfg) if scripts have been copied to /opt/plugins/custom/:

      # command 'process-service-perfdata-log'
      define command{
          command_name                   process-service-perfdata-log
          command_line                   /opt/plugins/custom/process-service-perfdata-log.pl $TIMET$
          }
      # command 'process-host-perfdata-log'
      define command{
          command_name                   process-host-perfdata-log
          command_line                   /opt/plugins/custom/process-host-perfdata-log.pl $TIMET$
          }
      
    4. In /opt/monitor/etc/naemon.cfg service_perfdata_file_processing_command and host_perfdata_file_processing_command has to be changed to run those custom scripts:

      service_perfdata_file_processing_command=process-service-perfdata-log
      host_perfdata_file_processing_command=process-host-perfdata-log
      
    5. In addition service_perfdata_file_template and host_perfdata_file_template can be changed to support sending more data to Elasticsearch. For instance, by adding $HOSTGROUPNAMES$ and $SERVICEGROUPNAMES$ macros logs can be separated better (it requires changes to Logstash filter config as well)

    6. Restart naemon service:

      sudo systemctl restart naemon # CentOS/RHEL 7.x
      sudo service naemon restart # CentOS/RHEL 7.x
      
    7. If FileBeat has been chosen, append below to filebeat.conf (adjust IP and PORT):

      filebeat.inputs:
      type: log
      enabled: true
      paths:
        - /opt/monitor/var/service_performance.log
        - /opt/monitor/var/host_performance.log
      tags: ["Energy Logserverperflogs"]
        output.logstash:
      # The Logstash hosts
        hosts: ["LOGSTASH_IP:LOGSTASH_PORT"]
      
    8. Restart FileBeat service:

      sudo systemctl restart filebeat # CentOS/RHEL 7.x
      sudo service filebeat restart # CentOS/RHEL 7.x
      

    Kibana

    At this moment there should be new index on the Elasticsearch node with performance data documents from Energy Logserver Monitor. Login to an Elasticsearch node and run: curl -XGET '127.0.0.1:9200/_cat/indices?v' Example output:

    health status index                      pri rep docs.count docs.deleted store.size pri.store.size
    green  open   auth                       5   0          7         6230      1.8mb          1.8mb
    green  open   Energy Logserver-perflogs-2018.09.14    5   0      72109            0     24.7mb         24.7mb
    

    After a while, if there is no new index make sure that:

    • Naemon is runnig on Energy Logserver node
    • Logstash service is running and there are no errors in: /var/log/logstash/logstash-plain.log
    • Elasticsearch service is running an there are no errors in: /var/log/elasticsearch/elasticsearch.log

    If the index has been created, in order to browse and visualize the data “index pattern” needs to be added to Kibana.

    1. After logging in to Kibana GUI go to Settings tab and add Energy Logserver-perflogs-* pattern. Chose @timestamp time field and click Create.
    2. Performance data logs should be now accessible from Kibana GUI Discovery tab ready to be visualize.

    OP5 Beat

    The op5beat is small agent for collecting metrics from op5 Monitor.

    The op5beat is located in the installation directory: utils/op5integration/op5beat

    Installation for Centos7 and newer

    1. Copy the necessary files to the appropriate directories:

      cp -rf etc/* /etc/
      cp -rf usr/* /usr/
      cp -rf var/* /var/
      
    2. Configure and start op5beat service (systemd):

      cp -rf op5beat.service /usr/lib/systemd/system/
      systemctl daemon-reload
      systemctl enable op5beat
      systemctl start op5beat
      

    Installation for Centos6 and older

    1. Copy the necessary files to the appropriate directories:

      cp -rf etc/* /etc/
      cp -rf usr/* /usr/
      cp -rf var/* /var/
      
    2. Configure and start op5beat service:

      • sysV init:

        cp -rf op5beat.service /etc/rc.d/init.d/op5beat
        chkconfig op5beat on
        service op5beat start
        
      • supervisord (optional):

        yum install supervisor
        cp -rf supervisord.conf /etc/supervisord.conf
        

    The Grafana instalation

    1. To install the Grafana application you should:

      • add necessary repository to operating system:

        [root@localhost ~]# cat /etc/yum.repos.d/grafan.repo
          [grafana]
          name=grafana
          baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
          repo_gpgcheck=1
          enabled=1
          gpgcheck=1
          gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
          sslverify=1
          sslcacert=/etc/pki/tls/certs/ca-bundle.crt
          [root@localhost ~]#
        
      • install the Grafana with following commands:

         [root@localhost ~]# yum search grafana
          Loaded plugins: fastestmirror
          Loading mirror speeds from cached hostfile
           * base: ftp.man.szczecin.pl
           * extras: centos.slaskdatacenter.com
           * updates: centos.slaskdatacenter.com
          =========================================================================================================== N/S matched: grafana ===========================================================================================================
          grafana.x86_64 : Grafana
          pcp-webapp-grafana.noarch : Grafana web application for Performance Co-Pilot (PCP)
        
            Name and summary matches only, use "search all" for everything.
        
          [root@localhost ~]# yum install grafana
        
      • to run application use following commands:

         [root@localhost ~]# systemctl enable grafana-server
          Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
          [root@localhost ~]#
          [root@localhost ~]# systemctl start grafana-server
          [root@localhost ~]# systemctl status grafana-server
          ● grafana-server.service - Grafana instance
             Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
             Active: active (running) since Thu 2018-10-18 10:41:48 CEST; 5s ago
               Docs: http://docs.grafana.org
           Main PID: 1757 (grafana-server)
             CGroup: /system.slice/grafana-server.service
                     └─1757 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var...
        
           [root@localhost ~]#
        
    2. To connect the Grafana application you should:

      • define the default login/password (line 151;154 in config file):

        [root@localhost ~]# cat /etc/grafana/grafana.ini
        148 #################################### Security ####################################
        149 [security]
        150 # default admin user, created on startup
        151 admin_user = admin
        152
        153 # default admin password, can be changed before first start of grafana,  or in profile settings
        154 admin_password = admin
        155
        
      • restart grafana-server service:

        systemctl restart grafana-server
        
      • Login to Grafana user interface using web browser: http://ip:3000

        ../_images/image112.png

      • use login and password that you set in the config file.

      • Use below example to set conection to Elasticsearch server:

        ../_images/image113.png

    The Beats configuration

    Kibana API

    Reference link: https://www.elastic.co/guide/en/kibana/master/api.html

    After installing any of beats package you can use ready to use dashboard related to this beat package. For instance dashboard and index pattern are available in /usr/share/filebeat/kibana/6/ directory on Linux.

    Before uploading index-pattern or dashboard you have to authorize yourself:

    1. Set up login/password/kibana_ip variables, e.g.:

       login=my_user
       password=my_password
       kibana_ip=10.4.11.243
      
    2. Execute command which will save authorization cookie:

       curl -c authorization.txt -XPOST -k "https://${kibana_ip}:5601/login" -d "username=${username}&password=${password}&version=6.2.3&location=https%3A%2F%2F${kibana_ip}%3A5601%2Flogin"
      
    3. Upload index-pattern and dashboard to Kibana, e.g.:

      curl -b authorization.txt -XPOST -k "https://${kibana_ip}:5601/api/kibana/dashboards/import" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d@/usr/share/filebeat/kibana/6/index-pattern/filebeat.json
      curl -b authorization.txt -XPOST -k "https://${kibana_ip}:5601/api/kibana/dashboards/import" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d@/usr/share/filebeat/kibana/6/dashboard/Filebeat-mysql.json
      
    4. When you want to upload beats index template to Ealsticsearch you have to recover it first (usually you do not send logs directly to Es rather than to Logstash first):

      /usr/bin/filebeat export template --es.version 6.2.3 >> /path/to/beats_template.json
      
    5. After that you can upload it as any other template (Access Es node with SSH):

      curl -XPUT "localhost:9200/_template/Energy Logserverperfdata" -H'Content-Type: application/json' -d@beats_template.json
      

    Wazuh integration

    Energy Logserver can integrate with the Wazuh, which is lightweight agent is designed to perform a number of tasks with the objective of detecting threats and, when necessary, trigger automatic responses. The agent core capabilities are:

    • Log and events data collection
    • File and registry keys integrity monitoring
    • Inventory of running processes and installed applications
    • Monitoring of open ports and network configuration
    • Detection of rootkits or malware artifacts
    • Configuration assessment and policy monitoring
    • Execution of active responses

    The Wazuh agents run on many different platforms, including Windows, Linux, Mac OS X, AIX, Solaris and HP-UX. They can be configured and managed from the Wazuh server.

    Deploying Wazuh Server

    https://documentation.wazuh.com/3.13/installation-guide/installing-wazuh-manager/linux/centos/index.html

    Deploing Wazuh Agent

    https://documentation.wazuh.com/3.13/installation-guide/installing-wazuh-agent/index.html

    Filebeat configuration

    2FA authorization with Google Auth Provider (example)

    Software used (tested versions):

    The NGiNX configuration:

    1. Copy the ng_oauth2_proxy.conf to /etc/nginx/conf.d/;

      server {
          listen 443 default ssl;
          server_name logserver.local;
          ssl_certificate /etc/kibana/ssl/logserver.org.crt;
          ssl_certificate_key /etc/kibana/ssl/logserver.org.key;
          ssl_session_cache   builtin:1000  shared:SSL:10m;
          add_header Strict-Transport-Security max-age=2592000;
      
        location /oauth2/ {
          proxy_pass       http://127.0.0.1:4180;
          proxy_set_header Host                    $host;
          proxy_set_header X-Real-IP               $remote_addr;
          proxy_set_header X-Scheme                $scheme;
          proxy_set_header X-Auth-Request-Redirect $request_uri;
          # or, if you are handling multiple domains:
          # proxy_set_header X-Auth-Request-Redirect $scheme://$host$request_uri;
        }
        location = /oauth2/auth {
          proxy_pass       http://127.0.0.1:4180;
          proxy_set_header Host             $host;
          proxy_set_header X-Real-IP        $remote_addr;
          proxy_set_header X-Scheme         $scheme;
          # nginx auth_request includes headers but not body
          proxy_set_header Content-Length   "";
          proxy_pass_request_body           off;
        }
      
        location / {
          auth_request /oauth2/auth;
          error_page 401 = /oauth2/sign_in;
      
          # pass information via X-User and X-Email headers to backend,
          # requires running with --set-xauthrequest flag
          auth_request_set $user   $upstream_http_x_auth_request_user;
          auth_request_set $email  $upstream_http_x_auth_request_email;
          proxy_set_header X-User  $user;
          proxy_set_header X-Email $email;
      
          # if you enabled --pass-access-token, this will pass the token to the backend
          auth_request_set $token  $upstream_http_x_auth_request_access_token;
          proxy_set_header X-Access-Token $token;
      
          # if you enabled --cookie-refresh, this is needed for it to work with auth_request
          auth_request_set $auth_cookie $upstream_http_set_cookie;
          add_header Set-Cookie $auth_cookie;
      
          # When using the --set-authorization-header flag, some provider's cookies can exceed the 4kb
          # limit and so the OAuth2 Proxy splits these into multiple parts.
          # Nginx normally only copies the first `Set-Cookie` header from the auth_request to the response,
          # so if your cookies are larger than 4kb, you will need to extract additional cookies manually.
          auth_request_set $auth_cookie_name_upstream_1 $upstream_cookie_auth_cookie_name_1;
      
          # Extract the Cookie attributes from the first Set-Cookie header and append them
          # to the second part ($upstream_cookie_* variables only contain the raw cookie content)
          if ($auth_cookie ~* "(; .*)") {
              set $auth_cookie_name_0 $auth_cookie;
              set $auth_cookie_name_1 "auth_cookie__oauth2_proxy_1=$auth_cookie_name_upstream_1$1";
          }
      
          # Send both Set-Cookie headers now if there was a second part
          if ($auth_cookie_name_upstream_1) {
              add_header Set-Cookie $auth_cookie_name_0;
              add_header Set-Cookie $auth_cookie_name_1;
          }
      
          proxy_pass https://127.0.0.1:5601;
          # or "root /path/to/site;" or "fastcgi_pass ..." etc
        }
      }
      
    2. Set ssl_certificate and ssl_certificate_key path in ng_oauth2_proxy.conf

    When SSL is set using nginx proxy, Kibana can be started with http. However, if it is to be run with encryption, you also need to change proxy_pass to the appropriate one.

    The oauth2_proxy configuration:

    1. Create a directory in which the program will be located and its configuration:

      mkdir -p /usr/share/oauth2_proxy/
      mkdir -p /etc/oauth2_proxy/
      
    2. Copy files to directories:

      cp oauth2_proxy /usr/share/oauth2_proxy/
      cp oauth2_proxy.cfg /etc/oauth2_proxy/
      
    3. Set directives according to OAuth configuration in Google Cloud project

              cfg
              client_id =
              client_secret =
              # the following limits domains for authorization (* - all)email_domains = ["*"]
      
    4. Set the following according to the public hostname:

      cookie_domain = "kibana-host.org"
      
    5. In case og-in restrictions for a specific group defined on the Google side:

      • Create administrative account: https://developers.google.com/identity/protocols/OAuth2ServiceAccount ;

      • Get configuration to JSON file and copy Client ID;

      • On the dashboard of the Google Cloud select “APIs & Auth” -> “APIs”;

      • Click on “Admin SDK” and “Enable API”;

      • Follow the instruction at https://developers.google.com/admin-sdk/directory/v1/guides/delegation#delegate_domain-wide_authority_to_your_service_account and give the service account the following permissions:

          https://www.googleapis.com/auth/admin.directory.group.readonly
          https://www.googleapis.com/auth/admin.directory.user.readonly
        
      • Follow the instructions to grant access to the Admin API https://support.google.com/a/answer/60757

      • Create or select an existing administrative email in the Gmail domain to flag it google-admin-email

      • Create or select an existing group to flag it google-group

      • Copy the previously downloaded JSON file to /etc/oauth2_proxy/.

      • In file oauth2_proxy set the appropriate path:

      google_service_account_json =
      

    Service start up

    • Start the NGiNX service
    • Start the oauth2_proxy service
    /usr/share/oauth2_proxy/oauth2_proxy -config="/etc/oauth2_proxy/oauth2_proxy.cfg"
    

    In the browser enter the address pointing to the server with the Energy Logserver installation

    –type=alias

    
    #### Import aliases into ES
    
    ```bash
    elasticdump \
      --input=./alias.json \
      --output=http://es.com:9200 \
      --type=alias
    

    Backup templates to a file

    elasticdump \
      --input=http://es.com:9200/template-filter \
      --output=templates.json \
      --type=template
    

    Import templates into ES

    elasticdump \
      --input=./templates.json \
      --output=http://es.com:9200 \
      --type=template
    

    Split files into multiple parts

    elasticdump \
      --input=http://production.es.com:9200/my_index \
      --output=/data/my_index.json \
      --fileSize=10mb
    

    Import data from S3 into ES (using s3urls)

    elasticdump \
      --s3AccessKeyId "${access_key_id}" \
      --s3SecretAccessKey "${access_key_secret}" \
      --input "s3://${bucket_name}/${file_name}.json" \
      --output=http://production.es.com:9200/my_index
    

    Export ES data to S3 (using s3urls)

    elasticdump \
      --s3AccessKeyId "${access_key_id}" \
      --s3SecretAccessKey "${access_key_secret}" \
      --input=http://production.es.com:9200/my_index \
      --output "s3://${bucket_name}/${file_name}.json"
    

    Import data from MINIO (s3 compatible) into ES (using s3urls)

    elasticdump \
      --s3AccessKeyId "${access_key_id}" \
      --s3SecretAccessKey "${access_key_secret}" \
      --input "s3://${bucket_name}/${file_name}.json" \
      --output=http://production.es.com:9200/my_index
      --s3ForcePathStyle true
      --s3Endpoint https://production.minio.co
    

    Export ES data to MINIO (s3 compatible) (using s3urls)

    elasticdump \
      --s3AccessKeyId "${access_key_id}" \
      --s3SecretAccessKey "${access_key_secret}" \
      --input=http://production.es.com:9200/my_index \
      --output "s3://${bucket_name}/${file_name}.json"
      --s3ForcePathStyle true
      --s3Endpoint https://production.minio.co
    

    Import data from CSV file into ES (using csvurls)

    elasticdump \
    
      # csv:// prefix must be included to allow parsing of csv files
    
      # --input "csv://${file_path}.csv" \
    
      --input "csv:///data/cars.csv"
      --output=http://production.es.com:9200/my_index \
      --csvSkipRows 1    # used to skip parsed rows (this does not include the headers row)
      --csvDelimiter ";" # default csvDelimiter is ','
    

    Copy a single index from a elasticsearch:

    elasticdump \
      --input=http://es.com:9200/api/search \
      --input-index=my_index \
      --output=http://es.com:9200/api/search \
      --output-index=my_index \
      --type=mapping
    

    Embedding dashboard in iframe

    It is possible to send alerts containing HTML iframe as notification content. For example:

    <a href="https://siem-vip:5601/app/kibana#/discover/72503360-1b25-11ea-bbe4-d7be84731d2c?_g=%28refreshInterval%3A%28display%3AOff%2Csection%3A0%2Cvalue%3A0%29%2Ctime%3A%28from%3A%272021-03-03T08%3A36%3A50Z%27%2Cmode%3Aabsolute%2Cto%3A%272021-03-04T08%3A36%3A50Z%27%29%29" target="_blank" rel="noreferrer">https://siem-vip:5601/app/kibana#/discover/72503360-1b25-11ea-bbe4-d7be84731d2c?_g=%28refreshInterval%3A%28display%3AOff%2Csection%3A0%2Cvalue%3A0%29%2Ctime%3A%28from%3A%272021-03-03T08%3A36%3A50Z%27%2Cmode%3Aabsolute%2Cto%3A%272021-03-04T08%3A36%3A50Z%27%29%29</a>
    

    If you want an existing HTTP session to be used to display the iframe content, you need to set the following parameters in the /etc/kibana/kibana.yml file:

    login.isSameSite: "Lax"
    login.isSecure: true
    

    Possible values for isSameSite are: “None”, “Lax”, “Strict”, false

    For isSecure: false or true

    Integration with AWS service

    The scope of integration

    The integration of Energy Logserver with the AWS cloud environment was prepared based on the following requirements:

    1. General information of the EC2 area, i.e .:
      • number of machines
      • number of CPUs
      • amount of RAM
    2. General information of the RDS area, i.e.:
      • Number of RDS instances
      • The number of RDS CPUs
      • Amount of RDS RAM
    3. EC2 area information containing information for each machine i.e .:
      • list of tags;
      • cloudwatch alarms configured;
      • basic information (e.g. imageID, reservtionid, accountid, launch date, private and public address, last backup, etc.);
      • list of available metrics in cloudwatch;
      • list of snapshots;
      • AMI list;
      • cloudtrail (all records, with detailed details).
    4. Information on Backups of EC2 and RDS instances
    5. Search for S3 objects, shoes, AMI images
    6. Downloading additional information about other resources, ie IG, NAT Gateway, Transit Gateway.
    7. Monitoring changes in the infrastructure based on Cloudtrail logs;
    8. Monitoring costs based on billing and usage reports.
    9. Monitoring the Security Group and resources connected to them and resources not connected to the Security Group
    10. Monitoring user activity and inactivity.
    11. Integration supports service for multiple member accounts in AWS organization

    The integration uses a Data Collector, i.e. the Energy Logserver host, which is responsible for receiving data from external sources.

    Data download mechanism

    The integration was prepared based on AWS (CLI), a unified tool for managing AWS services, with which it is possible to download and monitor many AWS services from the command line. The AWS (CLI) tool is controlled by the Energy Logserver data collector, which execute commands at specified intervals and captures the results of data received from the AWS service. The obtained data is processed and enriched and, as a result, saved to the Energy Logserver indexes.

    AWS Cost & Usage Report

    The integration of Energy Logserver with the AWS billing environment requires access to AWS Cost & Usage reports, which generated in accordance with the agreed schedule constitute the basic source of data for cost analysis in Energy Logserver. The generated report is stored on S3 in the bucket defined for this purpose and cyclically downloaded from it by the Energy Logserver collector. After the report is downloaded, it is processed and saved to a dedicated Elasticsearch index. The configuration of generating and saving a report to S3 is described in the AWS documentation: https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/.

    Cloud Trail

    The integration of the Energy Logserver with the AWS environment in order to receive events from the AWS environment requires access to the S3 bucket, on which the so-called AWS Trails. The operation of the Energy Logserver collector is based on periodical checking of the “cloudtraillogs” bucket and downloading new events from it. After the events are retrieved, they are processed so that the date the event occurred matches the date the document was indexed. The AWS Trail creation configuration is described in the AWS documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html#creating-a-trail-in-the-console.

    Configuration

    Configuration of access to the AWS account

    Configuration of access to AWS is in the configuration file of the AWS service (CLI), which was placed in the home directory of the Logstash user:

    /home/logstash/.aws/config
     [default]
    aws_access_key_id=A************************4
    aws_secret_access_key=*******************************************u
    

    The “default” section contains aws_access_key_id and aws_secret_access_key. Configuration file containing the list of AWS accounts that are included in the integration:

    /etc/logstash/lists/account.txt

    Configuration of AWS profiles

    AWS profiles allow you to navigate to different AWS accounts using the defined AWS role for example : “LogserverReadOnly”. Profiles are defined in the configuration file:

    /home/logstash/.aws/config

    Profile configuration example:
    [profile 111111111222]
    role_arn = arn: aws: iam :: 111111111222: role / LogserverReadOnly
    source_profile = default
    region = eu-west-1
    output = json
    

    The above section includes

    • profile name;
    • role_arn - definition of the account and the role assigned to the account;
    • source_profile - definition of the source profile;
    • region - AWS region;
    • output - the default format of the output data.

    Configure S3 buckets scanning

    The configuration of scanning buckets and S3 objects for the “s3” dashboard was placed in the following configuration files:

    • /etc/logstash/lists/bucket_s3.txt - configuration of buckets that are included in the scan;
    • /etc/logstash/lists/account_s3.txt - configuration of accounts that are included in the scan;

    Configuration of AWS Cost & Usage reports

    Downloading AWS Cost & Usage reports is done using the script: “/etc/logstash/lists/bin/aws_get_billing.sh”

    In which the following parameters should be set:

    • BUCKET = bucket_bame - bucket containing packed rarpotes;
    • PROFILE = profile_name - a profile authorized to download reports from the bucket.

    Logstash Pipelines

    Integration mechanisms are managed by the Logstash process, which is responsible for executing scripts, querying AWS, receiving data, reading data from files, processing the received data and enriching it and, as a result, submitting it to the Energy Logserver index. These processes were set up under the following Logstash pipelines:

    - pipeline.id: aws
      path.config: "/etc/logstash/aws/conf.d/*.conf"
      pipeline.workers: 1
     
    - pipeline.id: awstrails
      path.config: "/etc/logstash/awstrails/conf.d/*.conf"
      pipeline.workers: 1
     
    - pipeline.id: awss3
      path.config: "/etc/logstash/awss3/conf.d/*.conf"
      pipeline.workers: 1
     
    - pipeline.id: awsbilling
      path.config: "/etc/logstash/awsbilling/conf.d/*.conf"
      pipeline.workers: 1 
    

    Configuration of AWS permissions and access

    To enable the correct implementation of the integration assumptions in the configuration of the IAM area, an Logserver-ReadOnly account was created with programming access with the following policies assigned:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "backup:Describe*",
                    "backup:Get*",
                    "backup:List*",
                    "cloudwatch:Describe*",
                    "cloudwatch:Get*",
                    "cloudwatch:List*",
                    "ec2:Describe*",
                    "iam:GenerateCredentialReport",
                    "iam:GetCredentialReport",
                    "logs:Describe*",
                    "logs:Get*",
                    "rds:Describe*",
                    "rds:List*",
                    "tag:Get*"
                ],
                "Resource": "*"
            },
            {
                "Sid": "AllowSpecificS3ForLogServer",
                "Effect": "Allow",
                "Action": [
                    "s3:Get*",
                    "s3:List*"
                ],
                "Resource": [
                    "arn:aws:s3:::veoliaplcloudtraillogs",
                    "arn:aws:s3:::veoliaplcloudtraillogs/*"
                ]
            }
        ]
    }
    

    Data indexing

    The data in the indexes has been divided into the following types:

    • awscli-* - storing volumetric data about AWS infrastructure;
    • awsbilling-* - storing billing data from billing reports;
    • awscli-trail-* - storing AWS environment events / logs from CloudTrail;
    • awsusers-000001 - storing data about users and administrators of the AWS service.

    Dashboards

    The data collected in the integration process has been visualized and divided into the following sections (dashboards):

    • Overview - The section provides an overview of the quantitative state of the environment
    • EC2 - the section contains details about the EC2 instance;
    • RDS - the section contains details about RDS instances;
    • AMI - the section contains details about Images;
    • S3 - section for searching for objects and buckets S3;
    • Snapshots - section for reviewing snapshots taken;
    • Backups - section to review the backups made;
    • CloudTrail - a section for analyzing logs downloaded from CloudTrail;
    • IAM - a section containing user and administrator activity and configuration of AWS environment access accounts;
    • Billing - AWS service billing section;
    • Gateways - section containing details and configuration of AWS Gateways.
    Overview

    The following views are included in the “Overview” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Overview Selector - active selector used to filter sections;
    • [AWS] Total Instances - metric indicator of the number of EC2 instances;
    • [AWS] Total CPU Running Instances - metric indicator of the number of CPUs running EC2 instances;
    • [AWS] Total Memory Running Instances - metric indicator of RAM [MB] amount of running EC2 instances;
    • [AWS] Total RDS Instances - metric indicator of the number of RDS instances;
    • [AWS] Total CPU Running RDS - metric indicator of the number of CPUs running RDS instances;
    • [AWS] Total Memory Running RDS - metric indicator of the amount of RAM [GB] of running RDS instances;
    • [AWS] Instance List - an array containing aggregated details about an EC2 instance;
    • [AWS] RDS Instance List - an array containing aggregated details about an EC2 instance;
    • [AWS] Alarm List - table containing the list of AWS environment alarms;
    • [AWS] Tags List - an array containing a list of AWS tags;
    • [AWS] CloudWatch Metrics - table containing a list of AWS metrics;
    EC2

    The following views have been placed in the “EC2” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] State Selector - active selector used to filter sections;
    • [AWS] Total Instances - metric indicator of the number of EC2 instances;
    • [AWS] Total CPU Running Instances - metric indicator of the number of CPUs running EC2 instances;
    • [AWS] Running histogram - graphical interpretation of the instance status in the timeline;
    • [AWS] Total Memory Running Instances - metric indicator of RAM [MB] amount of running EC2 instances;
    • [AWS] OP5 Monitored Count - metric indicator of monitored instances in the OP5 Monitor system;
    • [AWS] OP5 NOT Monitored Count - metric indicator of unmonitored instances in the OP5 Monitor system;
    • [AWS] OP5 Monitored Details - a table containing a list of instances with monitoring details in the OP5 Monitoring system;
    • [AWS] Instance Details List - table containing details of the EC2 instance;
    • [AWS] CloudWatch Metrics - table containing details of EC2 metrics downloaded from AWS service;
    RDS

    The following views have been placed in the “RDS” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] RDS State Selector - active selector used for section filtering;
    • [AWS] Total RDS Instances - metric indicator of the number of RDS instances;
    • [AWS] Total CPU Running RDS - metric indicator of the number of CPUs running RDS instances;
    • [AWS] RDS Running histogram - graphical interpretation of the instance status in the timeline;
    • [AWS] RDS Instance Details - a table containing aggregated details of a RDS instance;
    • [AWS] RDS Details - table containing full details of the RDS instance;
    • [AWS] CloudWatch Metrics - table containing details of EC2 metrics downloaded from AWS service;
    AMI

    The following views have been placed in the “AMI” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Image Selector - active selector used to filter sections;
    • [AWS] Image Details - a table containing full details of the images taken;
    • [AWS] Image by Admin Details - a table containing full details of images made by the administrator;
    • [AWS] AMI type by time - graphical interpretation of image creation presented in time;
    Security

    The following views have been placed in the “Security” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Security Selector - active selector used to filter sections;
    • [AWS] Security Group ID by InstanceID - a table containing Security Groups with assigned Instances;
    • [AWS] Instance by Security Group - a table containing Instances with assigned Security Groups and details;
    • [AWS] Security Group connect state - table containing the status of connecting the Security Groups to the EC2 and RDS instances.
    Snapshots

    The following views have been placed in the “Snapshots” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Snapshot Selector - active selector used to filter sections;
    • [AWS] Snapshots List - a view containing a list of snapshots made with details;
    • [AWS] Snapshots by time - graphical interpretation of creating snapshots over time;
    Backups

    The following views have been placed in the “Backup” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Backup Selector - active selector used to filter sections;
    • [AWS] Backup List - view containing the list of completed Backup with details;
    • [AWS] Backup by time - graphical interpretation of backups presented in time;
    CloudTrail

    The following views have been placed in the “CloudTrail” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Event Selector - active selector used to filter sections;
    • [AWS] Events Name Activity - event activity table with event details;
    • [AWS] CloudTrail - graphical interpretation of generating events in the AWS service presented over time;
    IAM

    The following views have been placed in the “IAM” section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] IAM Selector - active selector used to filter sections;
    • [AWS] IAM Details - the table contains AWS service users, configured login methods, account creation time and account assignment;
    • [AWS] User last login - user activity table containing the period from the last login depending on the login method;
    Gateways

    The following views have been placed in the Gateways section:

    • [AWS] Navigation - navigation between sections;
    • [AWS] Gateways Selector - active selector used to filter sections;
    • [AWS] Internet Gateway - details table of configured AWS Internet Gateways;
    • [AWS] Transit Gateways - details table of configured AWS Transit Gateways;
    • [AWS] Nat Gateway - details table of configured AWS Nat Gateways;

    Integration with Azure / o365

    Introduction

    The goal of the integration is to create a single repository with aggregated information from multiple Azure / o365 accounts or subscriptions and presented in a readable way with the ability to search, analyze and generate reports.

    Scope of Integration

    The scope of integration include:

    1. User activity:
      • Event category,
      • Login status,
      • Client application,
      • Location,
      • Type of activity,
      • Login problems and their reasons.
    2. Infrastructure Metrics:
      • Azure Monitor Metrics (or Metrics) is a platform service that provides a single source for monitoring Azure resources.
      • Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms and can be used for live web application monitoring - it automatically detects performance anomalies.

    System components

    Logstash

    Logstash is an event collector and executor of queries which, upon receipt, are initially processed and sent to the event buffer.

    Kafka

    Component that enables buffering of events before they are saved on Energy Logserver Data servers. Kafka also has the task of storing data when the Energy Logserver Data nodes are unavailable.

    Energy Logserver Data

    The Energy Logserver cluster is responsible for storing and sharing data.

    Energy Logserver GUI

    Energy Logserver GUI is a graphical tool for searching, analyzing and visualizing data. It has an alert module that can monitor the collected metrics and take action in the event of a breach of the permitted thresholds.

    Data sources

    Energy Logserver can access metrics from the Azure services via API. Service access can be configured with the same credentials if the account was configured with Azure AD. Configuration procedures:

    • https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal
    • https://dev.loganalytics.io/documentation/Authorization/AAD-Setup
    • https://dev.applicationinsights.io/quickstart/

    Azure Monitor datasource configuration

    To enable an Azure Monitor data source, the following information from the Azure portal is required:

    • Tenant Id (Azure Active Directory -> Properties -> Directory ID)
    • Client Id (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
    • Client Secret (Azure Active Directory -> App Registrations -> Choose your app -> Keys)
    • Default Subscription Id (Subscriptions -> Choose subscription -> Overview -> Subscription ID)

    Azure Insights datasource configuration

    To enable an Azure Insights data source, the following information is required from the Azure portal:

    • Application ID
    • API Key

    Azure Command-Line Interface

    To verify the configuration and connect Energy Logserver to the Azure cloud, it is recommended to use the Azure command line interface:

    • https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest

    This tool deliver a set of commands for creating and managing Azure resources. Azure CLI is available in Azure services and is designed to allow you to work quickly with Azure with an emphasis on automation. Example command:

    • Login to the Azure platform using azure-cli:
    az login --service-principal -u $ (client_id) -p $ (client_secret) --tenant $ (tenant_id)
    

    Permission

    The following permissions are required to access the metrics:

    • Logon,
    • Geting a resource list with an ID (az resource list),
    • Geting a list of metrics for a given resource (az monitor metrics list-definitions),
    • Listing of metric values for a given resource and metric (az monitor metrics list).

    Service selection

    The service is selected by launching the appropriate pipeline in Logstash collectors:

    • Azure Meters
    • Azure Application Insights The collector’s queries will then be properly adapted to the chosen service.

    Azure Monitor metrics

    Sample metrics:

    • Microsoft.Compute / virtualMachines - Percentage CPU
    • Microsoft.Network/networkInterfaces - Bytes sent
    • Microsoft.Storage/storageAccounts - Used Capacity

    The Logstash collector gets the metrics through the following commands:

    • downloading a list of resources for a given account: /usr/bin/az resource list
    • downloading a list of resource-specific metrics: /usr/bin/az monitor metrics list-definitions –resource $ (resource_id)
    • for a given resource, downloading the metric value in the 1-minute interval /usr/bin/az monitor metrics list –resource “$ (resource_id)” –metric “$ (metric_name)”

    Azure Monitor metric list:

    • https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported

    The downloaded data is decoded by the filter logstash:

    filter {
            ruby {
                code => "
                  e = event.to_hash
                  data = e['value'][0]['timeseries'][0]['data']
                    for d in Array(data) do
                      new_event = LogStash::Event.new()
                      new_event.set('@timestamp', e['@timestamp'])
                      new_event.set('data', d)
                      new_event.set('namespace', e['namespace'])
                      new_event.set('resourceregion', e['resourceregion'])
                      new_event.set('resourceGroup', e['value'][0]['resourceGroup'])
                      new_event.set('valueUnit', e['value'][0]['unit'])
                      new_event.set('valueType', e['value'][0]['type'])
                      new_event.set('id', e['value'][0]['id'])
                      new_event.set('errorCode', e['value'][0]['errorCode'])
                      new_event.set('displayDescription', e['value'][0]['displayDescription'])
                      new_event.set('localizedValue', e['value'][0]['name']['localizedValue'])
                      new_event.set('valueName', e['value'][0]['name']['@value'])
                      new_event_block.call(new_event)
                    end
                  event.cancel()
               "
            }
            if "_rubyexception" in [tags] {
                    drop {}
            }
            date {
                    match => [ "[data][timeStamp]", "yyyy-MM-dd'T'HH:mm:ssZZ" ]
            }
            mutate {
                    convert => {
                            "[data][count]" => "integer"
                            "[data][minimum]" => "integer"
                            "[data][total]" => "integer"
                            "[data][maximum]" => "integer"
                            "[data][average]" => "integer"
                    }
            }
    }
    

    After processing, the obtained documents are saved to the Kafka topic using Logstash output:

    output {
            kafka {
                    bootstrap_servers => "localhost:9092"
                    client_id => "gk-eslapp01v"
                    topic_id => "azurelogs"
                    codec => json
            }
    }
    

    Azure Application Insights metrics

    Sample metrics:

    • performanceCounters / exceptionsPerSecond
    • performanceCounters / memoryAvailableBytes
    • performanceCounters / processCpuPercentage
    • performanceCounters / processIOBytesPerSecond
    • performanceCounters / processPrivateBytes

    Sample query:

    GET https://api.applicationinsights.io/v1/apps/[appIdarówka/metrics/ nutsmetricId}
    

    Metrics List:

    • https://docs.microsoft.com/en-us/rest/api/application-insights/metrics/get

    Energy Logserver GUI

    Metrics

    Metric data is recorded in the monthly indexes:

    azure-metrics -% {YYYY.MM}

    The pattern index in Energy Logserver GUI is:

    azure-metrics *

    Energy Logserver Discover data is available using the saved search: “[Azure Metrics] Metrics Details”

    ../_images/image232.png

    The analysis of the collected metrics is possible using the provided dashboard:

    ../_images/image233.png

    On which the following views have been placed:

    • [Azure Metrics] Main Selector - a selector that allows you to search by name and select a resource group, metric or namespace for a filter.
    • [Azure Metrics] Main Average - a numeric field that calculates the average value of a selected metric;
    • [Azure Metrics] Main Median - numeric field that calculates the median of the selected metric;
    • [Azure Metrics] Average Line - a line chart of the value of the selected metric over time;
    • [Azure Metrics] Top Resource Group - horizontal bar chart of resource groups with the most metrics
    • [Azure Metrics] Top Metrics - horizontal bar chart, metrics with the largest amount of data
    • [Azure Metrics] Top Namespace - horizontal bar chart, namespace with the most metrics
    • [Azure Metrics] Metrics Details - table containing details / raw data;

    Dashboard with an active filter:

    ../_images/image234.png

    Events

    Events are stored in the monthly indexes:

    azure_events -% {YYYY.MM}

    The index pattern in Energy Logserver GUI is:

    azure_events *

    ../_images/image235.png

    Examples of fields decoded in the event:

    The analysis of the collected events is possible using the provided dashboard:

    ../_images/image236.png

    Componens:

    • [AZURE] Event category - pie chart, division into event categories,
    • [AZURE] Login Status - pie chart, login status breakdown,
    • [AZURE] User localtion - map, location of logging in users,
    • [AZURE] Client App Type - pie chart, division into client application type,
    • [AZURE] Client APP - bar chart, the most used client application,
    • [AZURE] Top activity type - pie chart, division into user activity type,
    • [AZURE] Client Top App - table, the most frequently used client application,
    • [AZURE] Failed login reason - save search, user access problems, raw data.

    Google Cloud Platform

    The Energy Logserver accepts data from the Google Cloud Platform using the Pub/Sub service. Pub/Sub is used for streaming analytics and data integration pipelines to ingest and distribute data. It’s equally effective as a messaging-oriented middleware for service integration or as a queue to parallelize tasks. https://cloud.google.com/pubsub/docs/overview

    To fetch events from the GCP service add the following condition to the Logstash configuration file:

      input {
              google_pubsub {
                      # Your GCP project id (name)
                      project_id => "augmented-form-349311"
    
                      # The topic name below is currently hard-coded in the plugin. You
                      # must first create this topic by hand and ensure you are exporting
                      # logging to this pubsub topic.
                      topic => "topic_1"
    
                      # The subscription name is customizeable. The plugin will attempt to
                      # create the subscription (but use the hard-coded topic name above).
                      subscription => "sub_1"
    
                      # If you are running logstash within GCE, it will use
                      # Application Default Credentials and use GCE's metadata
                      # service to fetch tokens.  However, if you are running logstash
                      # outside of GCE, you will need to specify the service account's
                      # JSON key file below.
                      json_key_file => "/etc/logstash/conf.d/tests/09_GCP/pkey.json"
    
                      # Should the plugin attempt to create the subscription on startup?
                      # This is not recommended for security reasons but may be useful in
                      # some cases.
                      #create_subscription => true
              }
      }
      filter {}
      output {
              elasticsearch {
                      hosts => ["127.0.0.1:9200"]
                      index => "gcp-%{+YYYY.MM}"
                      user => "logstash"
                      password => "logstash"
                      ilm_enabled => false
              }
      }
    

    Authentication to the Pub/Sub service must be done with a private key: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating

    F5

    The Energy Logserver accepts data from the F5 system using the SYSLOG protocol. The F5 configuration procedure is as follows: https://support.f5.com/csp/article/K13080

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["F5"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘F5’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "F5" in [tags] {
        elasticsearch {
          hosts => "https://localhost:9200"
          ssl => true
          ssl_certificate_verification => false
          index => "F5-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Aruba Devices

    The Energy Logserver accepts data from the Aruba Devices system using the SYSLOG protocol. The Aruba Switches configuration procedure is as follows: https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=80765a47-fe42-4d69-b500-277217f5312e

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["ArubaSW"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘ArubaSW’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "ArubaSW" in [tags] {
        elasticsearch {
          hosts => "https://localhost:9200"
          ssl => true
          ssl_certificate_verification => false
          index => "ArubaSW-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Sophos Central

    The Energy Logserver accepts data from the Sophos Central system using the API interface. The Sophos Central configuration procedure is as follows: https://github.com/sophos/Sophos-Central-SIEM-Integration

    Pipeline configuration in Logstash collector:

       input {
          exec {
              command => "/etc/lists/bin/Sophos-Central/siem.py -c /usr/local/Sophos-Central/config.ini -q"
              interval => 60
              codec => "json_lines"
          }
       }
       filter {
          date {
                  match => [ "[data][created_at]", "UNIX_MS" ]
          }
       }
       output {
          elasticsearch {
              hosts => "http://localhost:9200"
              index => "sophos-central-%{+YYYY.MM}"
              user => "logstash"
              password => "logstash"
          }
       }
    

    Example of config.ini file:

       /usr/local/Sophos-Central/config.ini
       [login]
       token_info = 'url: https://api4.central.sophos.com/gateway, x-api-key: dcaz, Authorization: Basic abdc'
       client_id = UUID
       client_secret = client-secrter
       tenant_id =
       auth_url = https://id.sophos.com/api/v2/oauth2/token
       api_host = api.central.sophos.com
       format = json
       filename = stdout
       endpoint = all
       address = /var/run/syslog
       facility = daemon
       socktype = udp
       state_file_path = siem_sophos.json
    

    The Energy Logserver can make automatic configuration changes via the API in Sophos E-mail Appliance, such as: adding a domain to the blocked domain list. This is done by using the command alert method and entering the correct API request in the Path to script/command field.

    ../_images/image241.png

    FreeRadius

    The Energy Logserver accepts data from the FreeRadius system using the SYSLOG protocol. The FreeRadius configuration procedure is as follows: https://wiki.freeradius.org/config/Logging

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["FreeRadius"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘FreeRadius’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "FreeRadius" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "FreeRadius-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Microsoft Advanced Threat Analytics

    The Energy Logserver accepts data from the Advanced Threat Analytics system using the SYSLOG protocol with message in CEF format. The Advanced Threat Analytics configuration procedure is as follows: https://docs.microsoft.com/pl-pl/advanced-threat-analytics/cef-format-sa

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["ATA"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘ATA’

    The event is recognized and decoded:

    filter {
       if [msg] =~ /CEF:/ {
           grok {
             keep_empty_captures => true
             named_captures_only => true
             remove_field => [
               "msg",
               "[cef][version]"
             ]
             match => {
               "msg" => [
                 "^%{DATA} CEF:%{NUMBER:[cef][version]}\|%{DATA:[cef][device][vendor]}\|%{DATA:[cef][device][product]}\|%{DATA:[cef] [device][version]}\|%{DATA:[cef][sig][id]}\|%{DATA:[cef][sig][name]}\|%{DATA:[cef][sig][severity]}\|%{GREEDYDATA:[cef]   [extensions]}"
               ]
             }
           }
         }
       if "ATA" in [tags] {
           if [cef][extensions] {
    
             kv {
               source => "[cef][extensions]"
               remove_field => [
                 "[cef][extensions]",
                 "device_time"
               ]
               field_split_pattern => "\s(?=\w+=[^\s])"
               include_brackets => true
               transform_key => "lowercase"
               trim_value => "\s"
               allow_duplicate_values => true
             }
            if [json] {
    
               mutate {
                 gsub => [
                   "json", "null", '""',
                   "json", ":,", ':"",'
                 ]
               }
    
               json {
                 skip_on_invalid_json => true
                 source => "json"
                 remove_field => [
                   "json"
                 ]
               }
    
             }
             mutate {
               rename => { "device_ip" => "[device][ip]" }
               rename => { "device_uid" => "[device][uid]" }
               rename => { "internalhost" => "[internal][host]" }
               rename => { "external_ip" => "[external][ip]" }
               rename => { "internalip" => "[internal][ip]" }
             }
           }
         }
       }
    }
    

    Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "ATA" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "ATA-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    CheckPoint Firewalls

    The Energy Logserver accepts data from the CheckPoint Firewalls system using the SYSLOG protocol. The CheckPoint Firewalls configuration procedure is as follows: https://sc1.checkpoint.com/documents/SMB_R80.20/AdminGuides/Locally_Managed/EN/Content/Topics/Configuring-External-Log-Servers.htm?TocPath=Appliance%20Configuration%7CLogs%20and%20Monitoring%7C_____3

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["CheckPoint"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘CheckPoint’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "F5BIGIP" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "CheckPoint-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    The Energy Logserver can make automatic configuration changes via the API in Checkpoint firewalls such as adding a rule in the firewall. This is done using the command alert method and entering the correct API request in the Path to script/command field.

    ../_images/image241.png

    WAF F5 Networks Big-IP

    The Energy Logserver accepts data from the F5 system using the SYSLOG protocol. The F5 configuration procedure is as follows: https://support.f5.com/csp/article/K13080

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["F5BIGIP"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘F5’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "F5BIGIP" in [tags] {
        elasticsearch {
          hosts => "https://localhost:9200"
          ssl => true
          ssl_certificate_verification => false
          index => "F5BIGIP-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Infoblox DNS Firewall

    The Energy Logserver accepts data from the Infoblox system using the SYSLOG protocol. The Infoblox configuration procedure is as follows: https://docs.infoblox.com/space/NAG8/22252249/Using+a+Syslog+Server#Specifying-Syslog-Servers

    To identify and collect events from a Infoblox, is nessery to use Filebeat with infoblox module. To run Filebeat with infoblox moduel run following commnds:

    filebeat modules enable infoblox
    

    Configure output section in /etc/filebat/filebeat.yml file:

    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver can make automatic configuration changes via an API in the Infoblox DNS Firewall, e.g.: automatic domain locking. This is done using the command alert method and entering the correct API request in the Path to script/command field.

    ../_images/image241.png

    CISCO Devices

    The Energy Logserver accepts data from the Cisco devices - router, switch, firewall and access point using the SYSLOG protocol. The Cisco devices configuration procedure is as follows: https://www.ciscopress.com/articles/article.asp?p=426638&seqNum=3

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["CISCO"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘CISCO’. Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "CISCO" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "CISCO-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Microsoft Windows Systems

    The Energy Logserver getting events from Microsoft Systems using the Winlogbeat agent.

    To identify and collect events from a Windows eventchannel, it is nessery to setup following parameters in winlobeat.yml configuration file.

       winlogbeat.event_logs:
       - name: Application
          ignore_older: 72h
       - name: Security
       - name: System
    
       #output.elasticsearch:
       # Array of hosts to connect to.
       #hosts: ["localhost:9200"]
    
       output.logstash:
       # The Logstash hosts
      hosts: ["$IP:5044"]
    

    Where $IP is IP address of Energy Logserver datanode.

    Linux Systems

    The Energy Logserver accepts data from the Linux systems using the SYSLOG protocol.

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["LINUX"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘LINUX’. Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "LINUX" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "LINUX-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    AIX Systems

    The Energy Logserver accepts data from the AIX systems using the SYSLOG protocol.

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["AIX"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘AIX’. Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "AIX" in [tags] {
        elasticsearch {
          hosts => "http://localhost:9200"
          index => "AIX-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Microsoft Windows DNS, DHCP Service

    The Energy Logserver accepts data from the Microsoft DNS and DHCP services using the Filebeat agent.

    To identify and collect events from Microsoft DNS and DHCP services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - c:\\Path_to_DNS_logs\*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Microsoft IIS Service

    The Energy Logserver accepts data from the Microsoft IIS services using the Filebeat agent.

    To identify and collect events from Microsoft IIS services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - c:\\Path_to_IIS_logs\*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Apache Service

    The Energy Logserver accepts data from the Linux Apache services using the Filebeat agent.

    To identify and collect events from Linux Apache services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in /etc/filebat/filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/apache/*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Microsoft Exchange

    The Energy Logserver accepts data from the Microsoft Exchange services using the Filebeat agent.

    To identify and collect events from Microsoft Exchange services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - c:\\Path_to_Exchange_logs\*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Microsoft Exchange message tracking

    The message tracking log is a detailed record of all activity as mail flows through the transport pipeline on Mailbox servers and Edge Transport servers. You can use message tracking for message forensics, mail flow analysis, reporting, and troubleshooting.

    By default, Exchange uses circular logging to limit the message tracking log based on file size and file age to help control the hard disk space that’s used by the log files. To configure the message tracking log, see the documentation: https://docs.microsoft.com/en-us/exchange/mail-flow/transport-logs/configure-message-tracking?view=exchserver-2019

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - "%ExchangeInstallPath%TransportRoles\Logs\MessageTracking\*"
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Microsoft AD, Radius, Network Policy Server

    The Energy Logserver accepts data from the Active Directory, Radius, Network Policy Server services using the Winlogbeat agent.

    To identify and collect events from Active Directory, Radius, Network Policy Server services, is nessery to set correct path do logs in Winlogbeat configuration file.

    Configure output section in C:\Program Files (x86)\winlogbeat\winlogbeat.yml file:

    winlogbeat.event_logs:
      - name: Application
    
      - name: System
    
      - name: Security
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    winlogbeat test config
    

    and:

    winlogbeat test output
    

    The Energy Logserver save collected data in winlogbeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Microsoft MS SQL Server

    The Energy Logserver accepts data from the Microsoft MS SQL Server services using the Filebeat agent.

    To identify and collect events from Microsoft MS SQL Server services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - "C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL\MSSQL\Log\*LOG*"
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    MySQL Server

    The Energy Logserver accepts data from the MySQL Server services using the Filebeat agent.

    To identify and collect events from MySQL Server services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in /etc/filebeat/filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/mysql/*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Oracle Database Server

    The Energy Logserver accepts data from the Oracle Database Server services using the Filebeat agent.

    To identify and collect events from Oracle Database Server services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in /etc/filebeat/filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/oracle/*.xml
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Postgres Database Server

    The Energy Logserver accepts data from the Postgres Database Server services using the Filebeat agent.

    To identify and collect events from Oracle Postgres Server services, is nessery to set correct path do logs in Filebeat configuration file.

    Configure output section in /etc/filebeat/filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - //opt/postgresql/9.3/data/pg_log/*.log
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    VMware Platform

    The Energy Logserver accepts data from the VMware platform using the SYSLOG protocol. The VMware vCenter Server configuration procedure is as follows: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-FD51CE83-8B2A-4EBA-A16C-75DB2E384E95.html

    To identify events from a specific source, add the following condition to the Logstash configuration file:

    filter {
         if "syslog" in [tags] {
                 if [host] == "$IP" {
                     mutate {
                         add_tag => ["vmware"]
                     }
                 }
         }
    }
    

    Where $IP is IP address of source system and each document coming from the address will be tagged with ‘VMware vCenter Server’ Using the assigned tag, the documents is send to the appropriate index:

    output {
      if "vmware" in [tags] {
        elasticsearch {
          hosts => "https://localhost:9200"
          ssl => true
          ssl_certificate_verification => false
          index => "vmware-%{+YYYY.MM.dd}"
          user => "logstash"
          password => "logstash"
        }
      }
    }
    

    Nerwork Flows

    The Energy Logserver has the ability to receive and process various types of network flows. For this purpose, the following input ports have been prepared:

    • IPFIX, Netflow v10 - 4739/TCP, 4739/UDP
    • NetFlow v5,9 - 2055/UDP
    • Sflow - 6343/UDP

    Example of inputs configuration:

      input {
      udp {
          port => 4739
          codec => netflow {
            ipfix_definitions => "/etc/logstash/netflow/definitions/ipfix.yaml"
            versions => [10]
            target => ipfix
            include_flowset_id => "true"
        }
        type => ipfix
        tags => ["ipfix", "v10", "udp"]
        }
        tcp {
          port => 4739
          codec => netflow {
            ipfix_definitions => "/etc/logstash/netflow/definitions/ipfix.yaml"
            versions => [10]
            target => ipfix
            include_flowset_id => "true"
          }
          type => ipfix
          tags => ["ipfix", "v10", "tcp"]
        }
      }
    
      input {
        udp {
          port => 2055
          type => netflow
          codec => netflow {
            netflow_definitions => "/etc/logstash/netflow/definitions/netflow.yaml"
            versions => [5,9]
          }
          tags => ["netflow"]
        }
      }
    
      input {
        udp {
        port => 6343
        type => sflow
        codec => sflow
        tags => ["sflow"]
      }
      }
    

    Citrix XenApp and XenDesktop

    This Energy Logserver has the ability to acquire data from Citrix XenApp and XenDesktop.

    An example command to enable Citrix Broker Service log to a file is as follows:

    BrokerService.exe –Logfile "C:\XDLogs\Citrix Broker Service.log"
    

    Or there is the possibility of extracting results, data from a report generated using the console:

    https://docs.citrix.com/en-us/xenapp-and-xendesktop/7-15-ltsr/monitor/configuration-logging.html#generate-reports

    The Energy Logserver accepts data from Citrix XenApp and XenDesktop server using the Filebeat agent.

    To identify and collect events from Citrix XenApp and XenDesktop servers, you need to set the correct path to the logs in the Filebeat configuration file.

    Configure output section in C:\Program Files (x86)\filebeat\filebeat.yml file:

    filebeat.inputs:
    - type: log
      paths:
        - "C:\XDLogs\Citrix Broker Service.log"
    
    output.logstash:
      hosts: ["127.0.0.1:5044"]
    

    Test the configuration:

    filebeat test config
    

    and:

    filebeat test output
    

    The Energy Logserver save collected data in filebeat-* index pattern and its available to review in the Discover module.

    If additional agent data information is required, e.g.: IP address, add the following section in the agent configuration file:

    processors:
      - add_host_metadata: 
          netinfo.enabled: true
    

    Sumologic Cloud SOAR

    The Energy Logserver has the ability to forward detected alerts to Sumologic Cloud SOAR. To do this, select the “syslog” method in the alert definition and set the following parameters:

    • Host
    • Port
    • Protocol
    • Logging Level
    • Facility

    ../_images/image237.png

    Energy Logserver has the ability to create security dashboards from data found in SOAR, such as statistics. It has the ability to create and configure master views from extracted SOAR data.

    An example of an API request retrieving data:

      curl -X GET "https://10.4.3.202/incmansuite_ng/api/v2/kpi?output_set=Weekly%20summary&type=json" -H "accept: application/json" -H "Authorization: bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9. eyJpYXQiOjE2NTc3MTg2ODAsImp0aSI6IjdmMzg1ZDdhLTc1YjYtNGZmMC05YTdmLTVkMmNjYTjZjTQ0YiIsImlzcyI6IkluY01hbiA1LjMuMC4wIiwibmJmIjoxNjU3NzE4NjgwLCJleHAiOm51bGwsImRhdGEiOnsidXNlcklkIjoxfX0. pCJlM9hxj8VdavGuNfIuq1y5Dwd9kJT_UMyoRca_gUZjUXQ85nwEQZz_QEquE1rXTgVW9TO__gDNjY30r8yjoA" -k
    

    Example of request response:

      [{"[INCIDENT] Created by": "System","[INCIDENT] Owner": "IncMan Administrator","[INCIDENT] Kind": "Forensic - Incident response","[INCIDENT] Status": "Open","[INCIDENT] Incident ID": "2022","[INCIDENT] Opening time": "07/15/22 10:47:11","[INCIDENT] Closing time":"","[INCIDENT] Category": "General","[INCIDENT] Type": "General, Incident Response","[OBSERVABLES] EMAIL":["adam@it.emca.pl"]},{"[INCIDENT] Created by": "System","[INCIDENT] Owner": "IncMan Administrator","[INCIDENT] Kind": "Forensic - Incident response","[INCIDENT] Status": "Open","[INCIDENT] Incident ID": "ENE-LOGS EVENTS FROM LOGSERVER 2022-07-15 08:23:00","[INCIDENT] Opening time": "07/15/22 10:23:01","[INCIDENT] Closing time":"","[INCIDENT] Category": "General","[INCIDENT] Type": "General, Intrusion attempt"},[{"[INCIDENT] Created by": "System","[INCIDENT] Owner": "IncMan Administrator","[INCIDENT] Kind": "Forensic - Incident response","[INCIDENT] Status": "Open","[INCIDENT] Incident ID": "ENE-LOGS EVENTS FROM LOGSERVER 2022-07-15 08:20: 49","[INCIDENT] Opening time": "07/15/22 10:20:50","[INCIDENT] Closing time":"","[INCIDENT] Category": "General","[INCIDENT] Type": "General, Intrusion attempt"}]]
    

    Integration pipeline configuration:

    input {
      exec {
         command => ""https://10.4.3.202/incmansuite_ng/api/v2/kpi?output_set=Weekly%20summary&type=json" -H "accept: application/json" -H "Authorization: bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9. eyJpYXQiOjE2NTc3MTg2ODAsImp0aSI6IjdmMzg1ZDdhLTc1YjYtNGZmMC05YTdmLTVkMmNjYTjZjTQ0YiIsImlzcyI6IkluY01hbiA1LjMuMC4wIiwibmJmIjoxNjU3NzE4NjgwLCJleHAiOm51bGwsImRhdGEiOnsidXNlcklkIjoxfX0. pCJlM9hxj8VdavGuNfIuq1y5Dwd9kJT_UMyoRca_gUZjUXQ85nwEQZz_QEquE1rXTgVW9TO__gDNjY30r8yjoA" -k"
         interval => 86400
       }
    }
    
    # optional
    filter {}
    
    output {
      elasticsearch {
        hosts => [ "http://localhost:9200" ]
        index => "soar-%{+YYYY.MM}"
        user => "logserver"
        password => "logserver"
      }
    }
    

    Microsfort System Center Operations Manager

    The Energy Logserver has the ability to integrate with MS SCOM (System Center Operations Manager) monitoring systems to monitor metrics and service availability in the context of the end system user.

    An example of the integration pipeline configuration with SCOM:

    input {
            # scom
            jdbc {
                jdbc_driver_library => "/usr/share/logstash/jdbc/mssql-jdbc-6.2.2.jre8.jar"
                jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
                jdbc_connection_string => "jdbc:sqlserver://VB2010000302;databaseName=OperationsManagerDW2012;"
                jdbc_user => "PerfdataSCOM"
                jdbc_password => "${SCOM_PASSWORD}"
                jdbc_default_timezone => "UTC"
                statement_filepath => "/usr/share/logstash/plugin/query"
                schedule => "*/5 * * * *"
                sql_log_level => "warn"
                record_last_run => "false"
                clean_run => "true"
                tags => "scom"
                }
    }
    # optional filter section
    filter {}
    output {
      if "scom" in [tags] {
          elasticsearch {
            hosts => [ "http://localhost:9200" ]
            index => "scom-%{+YYYY.MM}"
            user => "logstash"
            password => "logstash"
        }
      }
    }
    

    The SLQ query stored in /usr/share/logstash/plugin/query file:

      #query
    
      SELECT
            Path,
            FullName,
            ObjectName,
            CounterName,
            InstanceName,
            SampleValue AS Value,
            DateTime
      FROM Perf.vPerfRaw pvpr WITH (NOLOCK)
      INNER JOIN vManagedEntity vme WITH (NOLOCK)
            ON pvpr.ManagedEntityRowId = vme.ManagedEntityRowId
      INNER JOIN vPerformanceRuleInstance vpri WITH (NOLOCK)
            ON pvpr.PerformanceRuleInstanceRowId = vpri.PerformanceRuleInstanceRowId
      INNER JOIN vPerformanceRule vpr WITH (NOLOCK)
            ON vpr.RuleRowId = vpri.RuleRowId
      WHERE ObjectName IN (
      'AD FS',
      'AD Replication',
      'Cluster Disk',
      'Cluster Shared Volume',
      'DirectoryServices',
      'General Response',
      'Health Service',
      'LogicalDisk',
      'Memory',
      'Network Adapter',
      'Network Interface',
      'Paging File',
      'Processor',
      'Processor Information',
      'Security System-Wide Statistics',
      'SQL Database',
      'System',
      'Web Service'
      )
      AND CounterName IN (
      'Artifact resolution Requests',
      'Artifact resolution Requests/sec',
      'Federation Metadata Requests',
      'Federation Metadata Requests/sec',
      'Token Requests',
      'Token Requests/sec',
      'AD Replication Queue',
      'Replication Latency',
      'Free space / MB',
      'Free space / Percent',
      'Total size / MB',
      'ATQ Outstanding Queued Requests',
      'ATQ Request Latency',
      'ATQ Threads LDAP',
      'ATQ Threads Total',
      'Active Directory Last Bind',
      'Global Catalog Search Time',
      'agent processor utilization',
      '% Free Space',
      'Avg. Disk Queue Length',
      'Avg. Disk sec/Read',
      'Avg. Disk sec/Write',
      'Current Disk Queue Length',
      'Disk Bytes/sec',
      'Disk Read Bytes/sec',
      'Disk Reads/sec',
      'Disk Write Bytes/sec',
      'Disk Writes/sec',
      'Free Megabytes',
      'Bytes Total/sec',
      'Bytes Received/sec',
      'Bytes Sent/sec',
      'Bytes Total/sec',
      'Current Bandwidth',
      '% Processor Time',
      '% Usage',
      '% Committed Bytes In Use',
      'Available Bytes',
      'Available MBytes',
      'Cache Bytes',
      'Cache Faults/sec',
      'Committed Bytes',
      'Free System Page Table Entries',
      'Page Reads/sec',
      'Page Writes/sec',
      'Pages/sec',
      'PercentMemoryUsed',
      'Pool Nonpaged Bytes',
      'Pool Paged Bytes',
      'KDC AS Requests',
      'KDC TGS Requests',
      'Kerberos Authentications',
      'NTLM Authentications',
      'DB Active Connections',
      'DB Active Sessions',
      'DB Active Transactions',
      'DB Allocated Free Space (MB)',
      'DB Allocated Size (MB)',
      'DB Allocated Space (MB)',
      'DB Allocated Space Used (MB)',
      'DB Available Space Total (%)',
      'DB Available Space Total (MB)',
      'DB Avg. Disk ms/Read',
      'DB Avg. Disk ms/Write',
      'DB Disk Free Space (MB)',
      'DB Disk Read Latency (ms)',
      'DB Disk Write Latency (ms)',
      'DB Total Free Space (%)',
      'DB Total Free Space (MB)',
      'DB Transaction Log Available Space Total (%)',
      'DB Transactions/sec',
      'DB Used Space (MB)',
      'Log Free Space (%)',
      'Log Free Space (MB)',
      'Log Size (MB)',
      'Processor Queue Length',
      'System Up Time',
      'Connection Attempts/sec',
      'Current Connections'
      )
    
      AND DateTime >= DATEADD(MI, -6, GETUTCDATE())
    

    MISP Integration

    Integration with MISP is divided into two parts, server side and client side.

    Requirements

    1. Server side
    • Prepare the MISP instance - you need an authorization key (go to Administration -> List Auth Keys -> Add authentication key)
    • Complete the necessary information in the misp_threat_lists_update.sh script, in particular the variables:
      • KEY (authentication key)
      • MISP_URL (misp instance to connect)
      • DEST (directory made available to the http server for output files)
    #!/bin/bash
    DEST=/etc/logstash/lists
    IOC_USER=''
    IOC_PASS=''
    LOGSERVER_USER='logserver'
    LOGSERVER_PASS='logserver'
    LOGSERVER_HOST=127.0.0.1
    LOGSERVER_PORT=9200
    LOGSERVER_SSL=false
    REPOSITORY_URL=https://repository.energylogserver.pl
    
    
    # FILTER
    function prepare_blacklist() {
    	blacklist=$1
    	/bin/mv -f ${DEST}/misp_${blacklist}.blacklist ${DEST}/misp_${blacklist}-$(date +%s).blacklist
    	#/usr/bin/echo "#"`/usr/bin/date` > ${DEST}/misp_${blacklist}.yml
    	#/usr/bin/awk '{print $1}' ${DEST}/misp_${blacklist}-*.blacklist | /usr/bin/sed -r "s/(.*)/\"\1\": \""bad_${blacklist}"\"/" >> ${DEST}/misp_${blacklist}.yml
    	#/usr/bin/sort -u ${DEST}/misp_${blacklist}.yml -o ${DEST}/misp_${blacklist}.yml
    }
    
    # INPUT
    function update_blacklist() {
    	blacklist=$1
    	local CURL_STATUS=""
    	CURL_STATUS=$(/usr/bin/curl -w "%{http_code}\n" -sS -u "${IOC_USER}":"${IOC_PASS}" ${REPOSITORY_URL}/ioc/misp_${blacklist}.blacklist -o ${DEST}/misp_${blacklist}.blacklist)
    	if [ ${CURL_STATUS} == 200 ]
    	then
    		prepare_blacklist ${blacklist}
    	fi
    }
    
    update_blacklist domain
    update_blacklist email
    update_blacklist filename
    update_blacklist ip
    update_blacklist url
    update_blacklist filehash
    update_blacklist certhash
    update_blacklist regkey
    
    ## OUTPUT
    if [ ${LOGSERVER_SSL} = true ]
    then
    	/usr/bin/curl -k -sS -u "${LOGSERVER_USER}":"${LOGSERVER_PASS}" -X POST "https://${LOGSERVER_HOST}:${LOGSERVER_PORT}/.blacklists/_delete_by_query" --connect-timeout 2 -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"range":{"@timestamp":{"lt":"now-5m/m"}}},{"term":{"tags":"misp_blacklist"}}]}}}' 2>&1 > /dev/null
    else
    	/usr/bin/curl -sS -u "${LOGSERVER_USER}":"${LOGSERVER_PASS}" -X POST "http://${LOGSERVER_HOST}:${LOGSERVER_PORT}/.blacklists/_delete_by_query" --connect-timeout 2 -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"range":{"@timestamp":{"lt":"now-5m/m"}}},{"term":{"tags":"misp_blacklist"}}]}}}' 2>&1 > /dev/null
    fi
    
    • Add the script to the schedule:
    # crontab -e
    # 0 1 * * * /path/to/misp_threat_lists_update.sh
    
    1. Clinet side
    • Has access to the server repository
    • Has logstash installed
    • Complete the necessary information in the misp_threat_lists.sh script, in particular the variables:
      • REPOSITORY_URL
      • IOC_USER (if needed)
      • IOC_PASS (if needed)
    #!/bin/bash
    DEST=/etc/logstash/lists
    IOC_USER=''
    IOC_PASS=''
    LOGSERVER_USER='logserver'
    LOGSERVER_PASS='logserver'
    LOGSERVER_HOST=127.0.0.1
    LOGSERVER_PORT=9200
    LOGSERVER_SSL=false
    REPOSITORY_URL=https://repository.energylogserver.pl
    
    
    # FILTER
    function prepare_blacklist() {
    	blacklist=$1
    	/bin/mv -f ${DEST}/misp_${blacklist}.blacklist ${DEST}/misp_${blacklist}-$(date +%s).blacklist
    	#/usr/bin/echo "#"`/usr/bin/date` > ${DEST}/misp_${blacklist}.yml
    	#/usr/bin/awk '{print $1}' ${DEST}/misp_${blacklist}-*.blacklist | /usr/bin/sed -r "s/(.*)/\"\1\": \""bad_${blacklist}"\"/" >> ${DEST}/misp_${blacklist}.yml
    	#/usr/bin/sort -u ${DEST}/misp_${blacklist}.yml -o ${DEST}/misp_${blacklist}.yml
    }
    
    # INPUT
    function update_blacklist() {
    	blacklist=$1
    	local CURL_STATUS=""
    	CURL_STATUS=$(/usr/bin/curl -w "%{http_code}\n" -sS -u "${IOC_USER}":"${IOC_PASS}" ${REPOSITORY_URL}/ioc/misp_${blacklist}.blacklist -o ${DEST}/misp_${blacklist}.blacklist)
    	if [ ${CURL_STATUS} == 200 ]
    	then
    		prepare_blacklist ${blacklist}
    	fi
    }
    
    update_blacklist domain
    update_blacklist email
    update_blacklist filename
    update_blacklist ip
    update_blacklist url
    update_blacklist filehash
    update_blacklist certhash
    update_blacklist regkey
    
    ## OUTPUT
    if [ ${LOGSERVER_SSL} = true ]
    then
    	/usr/bin/curl -k -sS -u "${LOGSERVER_USER}":"${LOGSERVER_PASS}" -X POST "https://${LOGSERVER_HOST}:${LOGSERVER_PORT}/.blacklists/_delete_by_query" --connect-timeout 2 -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"range":{"@timestamp":{"lt":"now-5m/m"}}},{"term":{"tags":"misp_blacklist"}}]}}}' 2>&1 > /dev/null
    else
    	/usr/bin/curl -sS -u "${LOGSERVER_USER}":"${LOGSERVER_PASS}" -X POST "http://${LOGSERVER_HOST}:${LOGSERVER_PORT}/.blacklists/_delete_by_query" --connect-timeout 2 -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"range":{"@timestamp":{"lt":"now-5m/m"}}},{"term":{"tags":"misp_blacklist"}}]}}}' 2>&1 > /dev/null
    fi
    
    • Activate pipeline blacklists in /etc/logstash/pipelnes.yml.