Skip to content

Build AWS Service Log Analytics Pipelines

Centralized Logging with OpenSearch supports ingesting AWS service logs into Amazon OpenSearch Service through log analytics pipelines, which you can build using the Centralized Logging with OpenSearch web console or via a standalone CloudFormation template.

Centralized Logging with OpenSearch reads the data source, parse, cleanup/enrich and ingest logs into Amazon OpenSearch Service domains for analysis. Moreover, the solution provides templated dashboards to facilitate log visualization.

Amazon OpenSearch Service is suitable for real-time log analytics and frequent queries and has full-text search capability.

As of release 2.1.0, the solution starts to support log ingestion into Light Engine, which is suitable for non real-time log analytics and infrequent queries and has SQL-like search capability. The feature is supported by Amazon CloudFront logs, Application Load Balancing logs, and AWS WAF logs.

Important

  • AWS managed services must be in the same region as Centralized Logging with OpenSearch. To ingest logs from different AWS regions, we recommend using S3 cross-region replication.
  • The solution will rotate the index on a daily basis, and cannot be adjusted.

Supported AWS Services

Most of AWS managed services output logs to Amazon CloudWatch Logs, Amazon S3, Amazon Kinesis Data Streams or Amazon Kinesis Firehose.

The following table lists the supported AWS services and the corresponding features.

AWS Service Log Type Log Location Automatic Ingestion Built-in Dashboard
AWS CloudTrail N/A S3 Yes Yes
Amazon S3 Access logs S3 Yes Yes
Amazon RDS/Aurora MySQL Logs CloudWatch Logs Yes Yes
Amazon CloudFront Standard access logs S3 Yes Yes
Application Load Balancer Access logs S3 Yes Yes
AWS WAF Web ACL logs S3 Yes Yes
AWS Lambda N/A CloudWatch Logs Yes Yes
Amazon VPC Flow logs S3 Yes Yes
AWS Config N/A S3 Yes Yes
  • Automatic Ingestion: The solution detects the log location of the resource automatically and then reads the logs.
  • Built-in Dashboard: An out-of-box dashboard for the specified AWS service. The solution will automatically ingest a dashboard into the Amazon OpenSearch Service.

Most of supported AWS services in Centralized Logging with OpenSearch offers built-in dashboard when creating the log analytics pipelines. You go to the OpenSearch Dashboards to view the dashboards after the pipeline being provisioned.

In this chapter, you will learn how to create log ingestion and dashboards for the following AWS services:

Cross-region log ingestion

When you deploy Centralized Logging with OpenSearch in one Region, the solution allows you to process service logs from another Region.

Note

For Amazon RDS/Aurora and AWS Lambda service logs, this feature is not supported.

Important

The Region where the service resides is referred to as “Source Region”, while the Region where the Centralized Logging with OpenSearch console is deployed as “Logging Region”.

For AWS CloudTrail, you can create a new trail which send logs into a S3 bucket in the Logging Region, and you can find the CloudTrail in the list. To learn how to create a new trail, please refer to Creating a trail.

For other services with logs located in S3 buckets, you can manually transfer logs (for example, using S3 Cross-Region Replication feature) to the Logging Region S3 bucket.

You can follow the steps below to implement Cross-Region Logging:

  1. Set the service log location in another Region to be the Logging Region (such as Amazon WAF), or automatically copy logs from the Source Region to the Logging Region using Cross-Region Replication (CRR).

  2. In the solution console, choose AWS Service Log in the left navigation pane, and choose Create a pipeline.

  3. In the Select an AWS Service area, choose a service in the list, and choose Next.

  4. In Creation Method, choose Manual, then enter the resource name and S3 log location parameter, and choose Next.

  5. Set OpenSearch domain and Log Lifecycle as needed, and choose Next.

  6. Add tags if you need, and choose Next to create the pipeline.

Then you can use the OpenSearch dashboard to discover logs and view dashboards.