Archive

The Archive module allows you to create compressed data files (zstd) from Elasticsearch indexes. The archive checks the age of each document in the index and if it is older than defined in the job, it is copied to the archive file.

Configuration

Enabling module

To configure module edit kibana.yml configuration file end set path to the archive directory - location where the archive files will be stored:

vim /etc/kibana/kibana.yml

remove the comment from the following line and set the correct path to the archive directory:

archive.archivefolderpath: '/var/lib/elastic_archive_test'

Archive Task

Create Archive task

  1. From the main navigation go to the “Archvie” module.

    ../_images/image155.png

  2. On the “Archive” tab select “Create Task” and define the following parameters:

    • Index pattern- for the indexes that will be archive, for example syslog* ;
    • Older than (days) - number of days after which documents will be archived;
    • Schedule task (crontab format) - the work schedule of the ordered task.

    ../_images/image156.png

Task List

In the Task List you can follow the current status of ordered tasks. You can modify task scheduler or delete ordered task.

../_images/image157.png

If the archiving task finds an existing archive file that matches the data being archived, it will check the number of documents in the archive and the number of documents in the index. If there is a difference in the number of documents then new documents will be added to the archive file.

Archive Upload

The Archive Upload module move data from archive to Elasticsearch index and make it online.

Create Upload task

  1. From the main navigation go to the Archive module.
  2. On the Upload tab select Create Task and define the following parameters:
    • Destination index - If destination index does not exist it will be created. If exists data will append.
  • File name - list of archive file that will be recover to Elasticsearch index.

../_images/image160.png

Task List

The process will index data back into Elasticsearch. Depend on archive size the process can take long time. On the Task List you can follow the status of the recovery process. Also you can view result and delete tasks.

../_images/image161.png

Command Line tools

Archive files can be handled by the following commands zstd, zstdcat, zstdgrep, zstdless, zstdmt.

zstd

The command for decompress *.zstd the Archive files, for example:

zstd -d winlogbeat-2020.10_2020-10-23.json.zstd -o
 winlogbeat-2020.10_2020-10-23.json

zstdcat

The command for concatenate *.zstd Archive files and print content on the standard output, for example:

zstdcat winlogbeat-2020.10_2020-10-23.json.zstd

zstdgrep

The command for print lines matching a pattern from *.zstd Archive files, for example:

zstdgrep "optima" winlogbeat-2020.10_2020-10-23.json.zstd

Above example is searching documents contain the “optima” phrase in winlogbeat-2020.10_2020-10-23.json.zstd archive file.

zstdless

The command for viewing Archive * .zstd files, for example:

zstdless winlogbeat-2020.10_2020-10-23.json.zstd

zstdmt

The command for compress and decompress Archive *.zdtd file useing multiple CPU core (default is 1), for example:

zstdmt -d winlogbeat-2020.10_2020-10-23.json.zstd -o winlogbeat-2020.10_2020-10-23.json