Architecture diagram

Deploying this solution with the default parameters builds the following environment in the AWS Cloud.

arch Centralized Logging with OpenSearch architecture

This solution deploys the AWS CloudFormation template in your AWS account and completes the following settings.

  1. Amazon CloudFront distributes the frontend web UI assets hosted in Amazon S3 bucket.

  2. Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.

  3. AWS AppSync provides the backend GraphQL APIs.

  4. Amazon DynamoDB stores the solution related information as backend database.

  5. AWS Lambda interacts with other AWS Services to process core logic of managing log pipelines or log agents, and obtains information updated in DynamoDB tables.

  6. AWS Step Functions orchestrates on-demand AWS CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate AWS resources and are used to collect and process logs and ingest them into Amazon OpenSearch Service for further analysis and visualization.

  7. Service Log Pipeline or Application Log Pipeline are provisioned on demand via Centralized Logging with OpenSearch console.

  8. AWS Systems Manager and Amazon EventBridge manage log agents for collecting logs from application servers, such as installing Fluent Bit log agents for application servers and monitoring the health status of the agents.

  9. Fluent Bit installed on Amazon EC2 or Amazon EKS, uploads log data to application log pipeline.

  10. Application Log Pipelines read, parse, process application logs and ingest them into Amazon OpenSearch domains or Light Engine.

  11. Service Log Pipelines read, parse, process AWS service logs and ingest them into Amazon OpenSearch domains or Light Engine.

Note

After deploying the solution, you can use AWS WAF to protect CloudFront or AppSync. Moreover, you can follow this guide to configure your WAF settings to prevent GraphQL schema introspection.

This solution supports two types of log pipelines: Service Log Analytics Pipeline and Application Log Analytics Pipeline, and two log analytics engines: OpenSearch Engine and Light Engine. Architecture details for pipelines and Light Engine are described in: