Unified Logging

Unified logging with Elasticsearch and Filebeat

3 minute read

This topic shows you how to enable unified logging using the Elasticsearch ELK stack.

Prerequisites

Elasticsearch v7.8.1
Filebeat v7.8.1
Kibana v7.8.1
Nuix REST v9.0, installed at the following location:
- Windows: C:\Program Files\Nuix\Nuix RESTful Service
- Linux: /opt/nuix-restful-service

The installation and configuration of Elasticsearch and Kibana are outside the scope of this topic. Refer to the Elastic documentation site for more information.

Note

Before proceeding, verify that the installed Nuix-REST service is not running.

Windows Filebeat Installation and Configuration

To install and configure Filebeat on Windows:

Download Filebeat (https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-windows-x86_64.zip)
Unzip the Filebeat installer to C:\Program Files\Filebeat.
Navigate to the C:\Program Files\Filebeat installation directory.
Rename filebeat.yml to filebeat.yml.backup.

Create a new filebeat.yml file in C:\Program Files\Filebeat and include the following configuration.

Click to View Windows Filebeat Configuration

filebeat.yaml

filebeat.inputs:
  -
  type: log
  enabled: true
  paths:
    - 'C:\Program Files\Nuix\Nuix RESTful Service\logs\**\*.json'
      json.message_key: message
      json.keys_under_root: true
      json.overwrite_keys: true
      json.add_error_key: true
     
  processors:
    - rename:
      fields:
      - from: "file"
      to: "file.source"
      ignore_missing: true
     
  output.elasticsearch:
  hosts: 'masterdata-node-0.elasticsearch.aws.chdev.nuix.com'

Open a command prompt with elevated administrative priviledges and navigate to C:\Program Files\Filebeat.
Enter the following command to install Filebeat in UnRestricted mode. Choose [R] to Run Once.

PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1

Filebeat is now installed and configured as a service but is not yet running.

Linux Filebeat Installation and Configuration

To install and configure Filebeat on Linux:

Download Filebeat using the following command:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.1-amd64.deb

Install Filebeat using the following command:
```
sudo dpkg -i filebeat-7.8.1-amd64.deb
```
Navigate to the installation directory and create a backup of the Filebeat configuration file:
```
cd /etc/filebeat
sudo mv filebeat.yml filebeat.yml.backup
```

Within the same directory, create a new filebeat.yml file and include the following configuration.

Click to View Linux Filebeat Configuration

filebeat.yaml

filebeat.inputs:
  -
  type: log
  enabled: true
  paths:
    - '/opt/nuix-restful-service/logs/**/*.json'
      json.message_key: message
      json.keys_under_root: true
      json.overwrite_keys: true
      json.add_error_key: true
     
  processors:
    - rename:
      fields:
      - from: "file"
      to: "file.source"
      ignore_missing: true
     
  output.elasticsearch:
  hosts: 'masterdata-node-0.elasticsearch.aws.chdev.nuix.com'

Filebeat is now installed and configured as a service but is not yet running.

Nuix RESTful Service Configuration Changes

To configure the logging settings within your instance of Nuix RESTful Service:

Application Log Configuration

Navigate to the settings directory within your Nuix RESTful Service installation and locate application.properties.

Windows: C:\Program Files\Nuix\Nuix RESTful Service\settings\application.properties
Linux: /opt/nuix-restful-service/settings/application.properties

Using a text editor with elevated administrative priviledges, open application.properties.
Add the following property to the file to configure Nuix RESTful Service to generate JSON formatted log files.
```
logging.json.enabled=true
```

After adding this property and saving the file, the new defaut log becomes ${logging.path}\nuix-rest.log.json.

Worker Log Configuration

Navigate to the Nuix RESTful Service installation directory and locate the Nuix-REST.vmoptions file.

Windows: C:\Program Files\Nuix\Nuix RESTful Service\Nuix-REST.vmoptions
Linux: /opt/nuix-restful-service/nuix-rest.vmoptions

Using a text editor with elevated administrative priviledges, open Nuix-REST.vmoptions.
Locate the -Dnuix.worker.jvm.arguments property and make the following modification to change its value from log4j.properties to the log4j-logstash.xml configuration that ships with the application. This change configures workers to generate JSON formatted log files.

Windows:

-Dnuix.worker.jvm.arguments=-Dlog4j.configuration=file:/C:/Program%20Files/Nuix/Nuix%20RESTful%20Service/nuix-engine/config/log4j-logstash.xml

Linux:

-Dnuix.worker.jvm.arguments=-Dlog4j.configuration=file:/opt/nuix-restful-service/nuix-engine/config/log4j-logstash.xml

After modifying this property and saving the file, the new default worker log becomes ${logging.path}\job-${UUID}\${UUID2}.

Services

After you install and configure both Filebeat and Nuix RESTful Service, you must start up all installed services.

Start the Nuix REST service
- On Linux: sudo service nuix-rest start
Start the Filebeat service
- On Linux: sudo service filebeat start

Log Viewing

Logs can now be viewed using Kibana. The installation and configuration of Kibana is outside the scope of this article, however, you can quickly look at logs if you have Docker installed using the following Docker command:

docker run -it \
    -e ELASTICSEARCH_HOSTS=http://your.elasticsearch.node.com:9200 \
    -p 5601:5601 \
    -e XPACK_SECURITY_ENABLED=false \
    -e MONITORING_ENABLED=false \
    -e XPACK_ML_ENABLED=false \
    -e XPACK_GRAPH_ENABLED=false \
    -e XPACK_REPORTING_ENABLED=false \
    docker.elastic.co/kibana/kibana:7.8.1

In a browser, navigate to http://localhost:5601

Feedback

Was this page helpful?

Thank you for your feedback.