Unified Logging with Elasticsearch

Unified logging with Filebeat and Elasticsearch

Use the following topic to enable unified logging using the Elasticsearch ELK stack.

Prerequisites

  • Elasticsearch v7.8.1
  • Filebeat v7.8.1
  • Kibana v7.8.1
  • Nuix REST v9.0

This topic assumes that you already have a running instance of Nuix RESTful Service version 9.0 installed at the following location:

  • Windows: C:\Program Files\Nuix\Nuix RESTful Service
  • Linux: /opt/nuix-restful-service

The installation and configuration of Elasticsearch and Kibana are outside the scope of this topic.

Windows Filebeat Installation and Configuration

Use the following procedure to install Filebeat on Windows:

  1. Download Filebeat (https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-windows-x86_64.zip)

  2. Unzip the Filebeat installer to C:\Program Files\Filebeat.

  3. Navigate to the C:\Program Files\Filebeat installation directory.

  4. Rename filebeat.yml to filebeat.yml.backup.

  5. Create a new filebeat.yml file in C:\Program Files\Filebeat and include the following configuration.

    Click to View Windows Filebeat Configuration
    filebeat.yaml
    filebeat.inputs:
      -
      type: log
      enabled: true
      paths:
        - 'C:\Program Files\Nuix\Nuix RESTful Service\logs\**\*.json'
          json.message_key: message
          json.keys_under_root: true
          json.overwrite_keys: true
          json.add_error_key: true
         
      processors:
        - rename:
          fields:
          - from: "file"
          to: "file.source"
          ignore_missing: true
         
      output.elasticsearch:
      hosts: 'masterdata-node-0.elasticsearch.aws.chdev.nuix.com'
    
  6. Open a command prompt with elevated administrative priviledges and navigate to C:\Program Files\Filebeat.

  7. Enter the following command to install Filebeat in UnRestricted mode. Choose [R] to Run Once.

PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1

Filebeat is now installed and configured as a service but is not yet running.

Linux Filebeat Installation and Configuration

Use the following procedure to install Filebeat on Linux:

  1. Download Filebeat using the following command:

    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.1-amd64.deb
    
  2. Install Filebeat using the following command:

    sudo dpkg -i filebeat-7.8.1-amd64.deb
    
  3. Navigate to the installation directory and create a backup of the Filebeat configuration file:

    cd /etc/filebeat
    sudo mv filebeat.yml filebeat.yml.backup
    
  4. Within the same directory, create a new filebeat.yml file and include the following configuration.

    Click to View Linux Filebeat Configuration
    filebeat.yaml
    filebeat.inputs:
      -
      type: log
      enabled: true
      paths:
        - '/opt/nuix-restful-service/logs/**/*.json'
          json.message_key: message
          json.keys_under_root: true
          json.overwrite_keys: true
          json.add_error_key: true
         
      processors:
        - rename:
          fields:
          - from: "file"
          to: "file.source"
          ignore_missing: true
         
      output.elasticsearch:
      hosts: 'masterdata-node-0.elasticsearch.aws.chdev.nuix.com'
    

Filebeat is now installed and configured as a service but is not yet running.

Nuix RESTful Service Configuration Changes

Use the following procedures to configure the logging settings within your instance of Nuix RESTful Service.

Application Log Configuration

  1. Navigate to the settings directory within your Nuix RESTful Service installation and locate application.properties.
  • Windows: C:\Program Files\Nuix\Nuix RESTful Service\settings\application.properties
  • Linux: /opt/nuix-restful-service/settings/application.properties
  1. Using a text editor with elevated administrative priviledges, open application.properties.

  2. Add the following property to the file to configure Nuix RESTful Service to generate JSON formatted log files.

    logging.json.enabled=true
    

After adding this property and saving the file, the new defaut log becomes ${logging.path}\nuix-rest.log.json.

Worker Log Configuration

  1. Navigate to the Nuix RESTful Service installation directory and locate the Nuix-REST.vmoptions file.
  • Windows: C:\Program Files\Nuix\Nuix RESTful Service\Nuix-REST.vmoptions
  • Linux: /opt/nuix-restful-service/nuix-rest.vmoptions
  1. Using a text editor with elevated administrative priviledges, open Nuix-REST.vmoptions.
  2. Locate the -Dnuix.worker.jvm.arguments property and make the following modification to change its value from log4j.properties to the log4j-logstash.xml configuration that ships with the application. This change configures workers to generate JSON formatted log files.

Windows:

-Dnuix.worker.jvm.arguments=-Dlog4j.configuration=file:/C:/Program%20Files/Nuix/Nuix%20RESTful%20Service/nuix-engine/config/log4j-logstash.xml

Linux:

-Dnuix.worker.jvm.arguments=-Dlog4j.configuration=file:/opt/nuix-restful-service/nuix-engine/config/log4j-logstash.xml

After modifying this property and saving the file, the new default worker log becomes ${logging.path}\job-${UUID}\${UUID2}.

Services

Start up all installed services once Filebeat and Nuix RESTful Service have been installed and configured.

  • Start the Nuix REST service
    • On Linux: sudo service nuix-rest start
  • Start the Filebeat service
    • On Linux: sudo service filebeat start

Log Viewing

Logs can now be viewed using Kibana. The installation and configuration of Kibana is outside the scope of this article, however, you can quickly look at logs if you have Docker installed using the following Docker command:

docker run -it \
    -e ELASTICSEARCH_HOSTS=http://your.elasticsearch.node.com:9200 \
    -p 5601:5601 \
    -e XPACK_SECURITY_ENABLED=false \
    -e MONITORING_ENABLED=false \
    -e XPACK_ML_ENABLED=false \
    -e XPACK_GRAPH_ENABLED=false \
    -e XPACK_REPORTING_ENABLED=false \
    docker.elastic.co/kibana/kibana:7.8.1

In a browser, navigate to http://localhost:5601

Last modified January 11, 2021