Configure logs gathering

Configure logs gathering

You can configure Fluentd to gather logs from custom entities, remove the default entities from the existing Fluentd configuration, as well as to filter and route logs. During configuration, you can define the following parameters:

  • input, to gather logs from external sources such as a log file, TCP socket, and so on. For details, see Input plugin overview.
  • filter, to filter the log entries gathered by the Input plugin. For example, to add, change, or remove fields. For details, see Filter plugin overview.
  • match, to push final log entries to a given destination such as Elasticsearch, file, and so on. For details, see Output plugin overview.
  • label, to connect the inputs. Logs gathered by Fluentd are processed from top-to-bottom of a configuration file. The label parameter connects the inputs, filters, and matches into a single flow. Using the label parameter ensures that filters for a given label are defined after input and before match.

Note

Perform all changes in the Reclass model. Add the custom log parsing rules used by a single environment to the cluster model. Place the log parsing rules for all deployments to the /meta/ directory in the Reclass model for a particular node. For details, see the */meta/fluentd.yml file of the required Salt formula.

To configure logs gathering:

  1. Log in to the Salt Master node.

  2. On the cluster level, specify the following snippets in the Reclass model for a particular node as required:

    • To add a new input:

      fluentd:
        agent:
          config:
            input:
              file_name:
                input_name:
                  parameterA: 10
                  parameterB: C
                input_nameB:
                  parameterC: ABC
      
    • To add a new filter:

      fluentd:
        agent:
          config:
            filter:
              file_name:
                filter_name:
                  parameterA: 10
                  parameterB: C
                filter_nameB:
                  parameterC: ABC
      
    • To add a new match:

      fluentd:
        agent:
          config:
            match:
              file_name:
                match_name:
                  parameterA: 10
                  parameterB: C
                match_nameB:
                  parameterC: ABC
      
    • If the service requires a more advanced processing than gathering logs from an external source (input), add a label. For example, if you want to add filtering, use the label parameter that defines the whole flow. All entries in label are optional. So you can define filter and match but skip input.

      fluentd:
        agent:
          config:
            label:
              label_name:
                input:
                  input1:
                    parameter1: abc
                  input2:
                    parameter1: abc
                filter:
                  filter1:
                    parameter1: abc
                    parameter2: abc
                match:
                  match1:
                    parameter1: abc
      

      Example:

      fluentd:
        agent:
          config:
            label:
              docker:
                input:
                  container:
                    type: tail
                    tag: temp.docker.container.*
                    path: /var/lib/docker/containers/*/*-json.log
                    path_key: log_path
                    pos_file: {{ positiondb }}/docker.container.pos
                    parser:
                      type: json
                      time_format: '%Y-%m-%dT%H:%M:%S.%NZ'
                      keep_time_key: false
                filter:
                  enrich:
                    tag: 'temp.docker.container.**'
                    type: record_transformer
                    enable_ruby: true
                    record:
                      - name: severity_label
                        value: INFO
                      - name: Severity
                        value: 6
                      - name: programname
                        value: docker
                match:
                  cast_service_tag:
                    tag: 'temp.docker.container.**'
                    type: rewrite_tag_filter
                    rule:
                      - name: log_path
                        regexp: '^.*\/(.*)-json\.log$'
                        result: docker.container.$1
                  push_to_default:
                    tag: 'docker.container.*'
                    type: relabel
                    label: default_output
      
    • To forward the logs gathered from a custom service to the default output, change the final match statement to default_output.

      Example:

      fluentd:
        agent:
          config:
            label:
              custom_daemon:
                input:
                  
                match:
                  push_to_default:
                    tag: 'some.tag'
                    type: relabel
                    label: default_output
      

      Note

      The default output is defined in the system Reclass model. For details, see Default output. All Fluentd labels defined in /meta/ must use this mechanism to ensure log forwarding to the default output destination.

    • To disable input, filter, match, or label, specify enabled: false for the required Fluentd entity.

      Example:

      fluentd:
        agent:
          config:
            label:
              docker:
                enabled: false
      
  3. Apply the following state:

    salt -C 'node_name' state.sls fluentd