Amazon Instance Group as log source

An instance group represents a group of EC2 Linux instances, which enables the solution to associate a Log Config with multiple EC2 instances quickly. Centralized Logging with OpenSearch uses Systems Manager Agent (SSM Agent) to install/configure Fluent Bit agent, and sends log data to Kinesis Data Streams.

This article guides you to create a log pipeline that ingests logs from an Instance Group.

Create a log analytics pipeline (OpenSearch Engine)

Prerequisites

Import an Amazon OpenSearch Service domain.

Create a log analytics pipeline

Sign in to the Centralized Logging with OpenSearch Console.
In the left sidebar, under Log Analytics Pipelines, choose Application Log.
Choose Create a pipeline
Choose Instance Group as Log Source, choose Amazon OpenSearch, and choose Next.
Select an instance group. If you have no instance group yet, choose Create Instance Group at the top right corner, and follow the Instance Group guide to create an instance group. After that, choose Refresh and then select the newly created instance group.
(Auto Scaling Group only) If your instance group is created based on an Auto Scaling Group, after ingestion status become "Created", then you can find the generated Shell Script in the instance group's detail page. Copy the shell script and update the User Data of the Auto Scaling Launch configurations or Launch template.
Keep the default Permission grant method.
(Optional) If you choose I will manually add the below required permissions after pipeline creation, continue to do the following:
1. Choose Expand to view required permissions and copy the provided JSON policy.
2. Go to AWS Management Console.
3. On the left navigation pane, choose IAM, and select Policies under Access management.
4. Choose Create Policy, choose JSON and replace all the content inside the text block. Make sure to substitute <YOUR ACCOUNT ID> with your account id.
5. Choose Next, and then enter a name for this policy.
6. Attach the policy to your EC2 instance profile to grant the log agent permissions to send logs to the application log pipeline. If you are using Auto Scaling group, you need to update the IAM instance profile associated with the Auto Scaling Group. If needed, you can follow the documentation to update your launch template or launch configuration.
Choose Next.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon Instance Group as log source.

Select a log config. If you do not find desired log config from the drop-down list, choose Create New, and follow instructions in Log Cong.
Enter a Log Path to specify the location of logs you want to collect. You can use , to separate multiple paths. Choose Next.
Specify Index name in lowercase.

In the Buffer section, choose S3 or Kinesis Data Streams. If you don't want the buffer layer, choose None. Refer to the Log Buffer for more information about choosing the appropriate buffer layer.

S3 buffer parameters

Parameter	Default	Description
S3 Bucket	A log bucket will be created by the solution.	You can also select a bucket to store the log data.
S3 Bucket Prefix	`AppLogs/<index-prefix>/year=%Y/month=%m/day=%d`	The log agent appends the prefix when delivering the log files to the S3 bucket.
Buffer size	50 MiB	The maximum size of log data cached at the log agent side before delivering to S3. For more information, see Data Delivery Frequency.
Buffer interval	60 seconds	The maximum interval of the log agent to deliver logs to S3. For more information, see Data Delivery Frequency.
Compression for data records	`Gzip`	The log agent compresses records before delivering them to the S3 bucket.

Kinesis Data Streams buffer parameters

Parameter	Default	Description
Shard number	`<Requires input>`	The number of shards of the Kinesis Data Streams. Each shard can have up to 1,000 records per second and total data write rate of 1MB per second.
Enable auto scaling	`No`	This solution monitors the utilization of Kinesis Data Streams every 5 minutes, and scale in/out the number of shards automatically. The solution will scale in/out for a maximum of 8 times within 24 hours.
Maximum Shard number	`<Requires input>`	Required if auto scaling is enabled. The maximum number of shards.

Important

You may observe duplicate logs in OpenSearch if threshold error occurs in Kinesis Data Streams (KDS). This is because the Fluent Bit log agent uploads logs in chunk (contains multiple records), and will retry the chunk if upload failed. Each KDS shard can support up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second. Please estimate your log volume and choose an appropriate shard number.

Choose Next.
In the Specify OpenSearch domain section, select an imported domain for Amazon OpenSearch domain.
In the Log Lifecycle section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated Index State Management (ISM) policy automatically for this pipeline.
In the Select log processor section, please choose the log processor.
- When selecting Lambda as log processor, you can configure the Lambda concurrency if needed.
- (Optional) OSI as log processor is now supported in these regions. When OSI is selected, please type in the minimum and maximum number of OCU. See more information here.
Choose Next.
Enable Alarms if needed and select an exiting SNS topic. If you choose Create a new SNS topic, please provide a name and an email address for the new SNS topic.
Add tags if needed.
Choose Create.
Wait for the application pipeline turning to "Active" state.

Create a log analytics pipeline (Light Engine)

Create a log analytics pipeline

Sign in to the Centralized Logging with OpenSearch Console.
In the left sidebar, under Log Analytics Pipelines, choose Application Log.
Choose Create a pipeline
Choose Instance Group as Log Source, choose Light Engine, and choose Next.
Select an instance group. If you have no instance group yet, choose Create Instance Group at the top right corner, and follow the Instance Group guide to create an instance group. After that, choose Refresh and then select the newly created instance group.
(Auto Scaling Group only) If your instance group is created based on an Auto Scaling Group, after ingestion status become "Created", then you can find the generated Shell Script in the instance group's detail page. Copy the shell script and update the User Data of the Auto Scaling Launch configurations or Launch template.
Keep the default Permission grant method.
(Optional) If you choose I will manually add the below required permissions after pipeline creation, continue to do the following:
1. Choose Expand to view required permissions and copy the provided JSON policy.
2. Go to AWS Management Console.
3. On the left navigation pane, choose IAM, and select Policies under Access management.
4. Choose Create Policy, choose JSON and replace all the content inside the text block. Make sure to substitute <YOUR ACCOUNT ID> with your account id.
5. Choose Next, and then enter a name for this policy.
6. Attach the policy to your EC2 instance profile to grant the log agent permissions to send logs to the application log pipeline. If you are using Auto Scaling group, you need to update the IAM instance profile associated with the Auto Scaling Group. If needed, you can follow the documentation to update your launch template or launch configuration.
Choose Next.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon Instance Group as log source.

Select a log config. If you do not find desired log config from the drop-down list, choose Create New, and follow instructions in Log Cong.
Enter a Log Path to specify the location of logs you want to collect. You can use , to separate multiple paths. Choose Next.

In the Buffer section,

S3 buffer parameters

Parameter	Default	Description
S3 Bucket	A log bucket will be created by the solution.	You can also select a bucket to store the log data.
Buffer size	50 MiB	The maximum size of log data cached at the log agent side before delivering to S3. For more information, see Data Delivery Frequency.
Buffer interval	60 seconds	The maximum interval of the log agent to deliver logs to S3. For more information, see Data Delivery Frequency.
Compression for data records	`Gzip`	The log agent compresses records before delivering them to the S3 bucket.

Choose Next.
In the Specify Light Engine Configuration section, if you want to ingest associated templated Grafana dashboards, select Yes for the sample dashboard.
You can choose an existing Grafana, or if you need to import a new one, you can go to Grafana for configuration.
Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.
The log processing frequency is set to 5 minutes by default, with a minimum processing frequency of 1 minute.
In the Log Lifecycle section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.
Select Next.
Enable Alarms if needed and select an exiting SNS topic. If you choose Create a new SNS topic, please provide a name and an email address for the new SNS topic.
If desired, add tags.
Select Create.
Wait for the application pipeline turning to "Active" state.