Concepts

This section describes key concepts and defines terminology specific to this solution:

Log Analytics Engine

A log analytics engine is a sophisticated tool designed to process, analyze, and visualize vast amounts of log data from diverse systems, applications, and devices. Our solution primarily uses the Amazon OpenSearch Service as the default log analytics engine, complemented by a Light Engine specifically optimized for structured, infrequent logs.

OpenSearch Engine

The OpenSearch Engine in this solution refers to the Amazon OpenSearch Service, a distributed, community-driven, Apache 2.0-licensed, 100% open-source search and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search.

Light Engine

The Light Engine is a serverless log analytics engine that utilizes AWS services like Athena, Glue, Lambda, and Step Functions. Designed to analyze structured and infrequent logs, it offers up to a 90% cost reduction compared to the OpenSearch Engine.

Log Config

A Log Config defines the metadata of your logs, specifying the log type, format, sample logs, filters, and the schema needed to map raw log data into the structured format used by the log analytics engine.

Log Source

A Log Source refers to the location where logs are generated or stored. Centralized Logging with OpenSearch supports ingesting logs from diverse sources, encompassing both application logs and logs from AWS services. For supported AWS service logs, refer to AWS Service Logs. For supported application logs, refer to Application Logs.

Log Agent

A log agent is a program that reads logs from one location and sends them to another location (for example, OpenSearch). Currently, Centralized Logging with OpenSearch only supports Fluent Bit 1.9 log agent which is installed automatically. The Fluent Bit agent has a dependency of OpenSSL 1.1. To learn how to install OpenSSL on Linux instances, refer to OpenSSL installation. To find the supported platforms by Fluent Bit, refer to this link.

Log Buffer

Log Buffer is a buffer layer between the Log Agent and OpenSearch clusters. The agent uploads logs into the buffer layer before being processed and delivered into the OpenSearch clusters. A buffer layer is a way to protect OpenSearch clusters from overwhelming. This solution provides the following types of buffer layers.

  • Amazon S3. Use this option if you can bear minutes-level latency for log ingestion. The log agent periodically uploads logs to an Amazon S3 bucket. The frequency of data delivery to Amazon S3 is determined by Buffer size (default value is 50 MiB) and Buffer interval (default value is 60 seconds) value that you configured when creating the application log analytics pipelines. The condition satisfied first triggers data delivery to Amazon S3.

  • Amazon Kinesis Data Streams. Use this option if you need real-time log ingestion. The log agent uploads logs to Amazon Kinesis Data Stream in seconds. The frequency of data delivery to Kinesis Data Streams is determined by Buffer size (10 MiB) and Buffer interval (5 seconds). The condition satisfied first triggers data delivery to Kinesis Data Streams.

Log Buffer is optional when creating an application log analytics pipeline. For all types of application logs, this solution allows you to ingest logs without any buffer layers. However, we only recommend this option when you have small log volume, and you are confident that the logs will not exceed the thresholds at the OpenSearch side.

Log Analytics Pipeline

A Log Analytics Pipeline, or Log Pipeline, represents the end-to-end data flow from the source to the log analytics engines. It typically encompasses the stages of shipping, buffering, processing, filtering, enriching, and storing logs. Centralized Logging with OpenSearch supports two types of Log Analytics Pipelines: the Service Log Pipeline, tailored for ingesting logs generated by AWS Services, and the Application Log Pipeline, designed for ingesting logs from custom applications.

Instance Group

An Instance Group represents a group of EC2 instances, which enables the solution to associate a Log Config with multiple EC2 instances quickly. Centralized Logging with OpenSearch uses Systems Manager Agent(SSM Agent) to install/configure Fluent Bit agent, and sends log data to Kinesis Data Streams. Instance Group is one of the supported Log Source in this solution.

Main Account

An AWS account where the Centralized Logging with OpenSearch console is deployed. The Log Analytics Engines must also reside in the same account.

Member Account

Another AWS account from which you wish to ingest AWS Service logs or application logs. Logs are sent from Member Accounts to Main Accounts, where they are analyzed using resources in the Main Account.

Access Proxy

An Access Proxy serves as an intermediary for accessing Amazon OpenSearch Service domains from the internet securely. By default, an Amazon OpenSearch Service domain within a VPC is not accessible from the internet. The Centralized Logging with OpenSearch solution implements an Nginx-based proxy stack architecture to enable internet access to OpenSearch Dashboards. This allows users to conveniently interact with the dashboards from anywhere with internet connectivity.