StackLight logging indices

StackLight logging indices are managed by OpenSearch data streams, which are introduced in OpenSearch 2.6. It is a convenient way to manage insert-only pipelines such as log message collection. The solution consists of the following elements:

Data stream objects that can be referred to as alias:
- Audit - dedicated for MOSK management, MKE, and host audit logs, ensuring data integrity and security.
- System - provides a streamlined approach to log management.
Write index - current index where ingestion can be performed without removing a data stream.
Read indices - indices created after the rollover mechanism is applied.
Rollover policy - creating new write index for data stream based on the size of shards

Example of an initial index list:

health status index               uuid                    pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-audit-000001    30q4HLGmR0KmpRR8Kvy5jw    1   1    2961719            0    496.3mb          248mb
green  open   .ds-system-000001   5_eFtMAFQa6aFB7nttHjkA    1   1       2476            0      6.1mb            3mb

Example of the index after the rollover is applied to the audit index:

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-audit-000001    30q4HLGmR0KmpRR8Kvy5jw   1   1    9819913            0      1.5gb        784.8mb
green  open   .ds-audit-000002    U1fbs0i9TJmOsAOoR7cERg   1   1    2961719            0    496.3mb          248mb
green  open   .ds-system-000001   5_eFtMAFQa6aFB7nttHjkA   1   1       2476            0      6.1mb            3mb

Audit and system index templates

The following table contains a simplified template of the audit and system indices. The user can perform aggregation queries over keyword fields.

Audit and system template
Field	Type	Description
`@timestamp`	date	Time when a log event was produced, if available in the parsed message. Otherwise time when the event was ingested.
`container.id`	keyword	Identifier of the Docker container that the application generating the event was running in.
`container.image`	text	Name of the Docker image defined as `<registry>/<repo>:<tag>`.
`container.name`	keyword	Name of the Docker container that the application generating the event was running in.
`event.source`	keyword	Source of the event: `"file"`, `"journal"`, or `"container"`.
`event.provider`	keyword	Name of the application that produced the message.
`host.hostname`	keyword	Name of the host that the message was collected from.
`log.file.path`	keyword	Path on the host to the source file for the message if the message was not produced by the application running in the container or system unit.
`log.level`	keyword	Severity level of the event taken from the parsed message content.
`message`	text	Unparsed content of the event message.
`orchestrator.labels`	flat_object	Kubernetes metadata labels of the pod that runs the Docker container of the application.
`orchestrator.namespace`	keyword	Kubernetes namespace where the application pod was running.
`orchestrator.pod`	keyword	Kubernetes pod name of the pod running the application Docker container.
`orchestrator.type`	keyword	Type of orchestrator: `"mke"` or `"kubernetes"`. Empty for host file logs and journal logs.

The following table contains a simplified template of extra fields for the system index that are not present in the audit template.

System template - extra fields
Field	Type	Description
`http.destination.address`	keyword	IP address of the HTTP request destination.
`http.destination.domain`	keyword	Name of the OpenStack service that the HTTP request was sent to.
`http.request.duration`	long	Request duration in nanoseconds.
`http.request.id`	keyword	Request ID generated by OpenStack.
`http.request.method`	keyword	HTTP request method.
`http.request.path`	keyword	Path of the HTTP URL request.
`http.response.status_code`	long	HTTP status code of the response.
`http.source.address`	keyword	IP address of the HTTP request source.