Skip to content

Amazon EKS cluster as log source

For Amazon Elastic Kubernetes Service (Amazon EKS) clusters, Centralized Logging with OpenSearch will generate all-in-one configuration file for customers to deploy the log agent (Fluent Bit 1.9) as a DaemonSet or Sidecar. After log agent is deployed, Centralized Logging with OpenSearch will start collecting pod logs and send to Amazon OpenSearch.

This article guides you to create a log pipeline that ingests logs from an EKS cluster.

Create log analytics pipeline (Amazon OpenSearch)

Prerequisites

  1. Import an Amazon OpenSearch Service domain.

Create log analytics pipeline

  1. Sign in to the Centralized Logging with OpenSearch Console.

  2. In the left sidebar, under Log Analytics Pipelines, choose Application Log.

  3. Choose Create a pipeline.

  4. Choose EKS Clusters as Log Source, and choose Next.

  5. Choose the AWS account in which the logs are stored.

  6. Choose an EKS Cluster. If you have no imported cluster yet, choose Import an EKS Cluster , and follow the EKS cluster guide to import an EKS cluster. After that, select the newly imported EKS cluster from the drop-down list.

  7. Choose Next.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon EKS cluster as log source.

  1. Enter a Log Path to specify the location of logs you want to collect.

  2. Select a log config and then choose Next. If you do not find desired log config from the drop-down list, choose Create New and follow instructions in Log Cong.

  3. Specify Index name in lowercase.

  4. In the Buffer section, choose S3 or Kinesis Data Streams. If you don't want the buffer layer, choose None. Refer to the Log Buffer for more information about choosing the appropriate buffer layer.

    • S3 buffer parameters
    Parameter Default Description
    S3 Bucket A log bucket will be created by the solution. You can also select a bucket to store the log data.
    S3 Bucket Prefix AppLogs/<index-prefix>/year=%Y/month=%m/day=%d The log agent appends the prefix when delivering the log files to the S3 bucket.
    Buffer size 50 MiB The maximum size of log data cached at the log agent side before delivering to S3. For more information, see Data Delivery Frequency.
    Buffer interval 60 seconds The maximum interval of the log agent to deliver logs to S3. For more information, see Data Delivery Frequency.
    Compression for data records Gzip The log agent compresses records before delivering them to the S3 bucket.
    • Kinesis Data Streams buffer parameters
    Parameter Default Description
    Shard number <Requires input> The number of shards of the Kinesis Data Streams. Each shard can have up to 1,000 records per second and total data write rate of 1MB per second.
    Enable auto scaling No This solution monitors the utilization of Kinesis Data Streams every 5 minutes, and scale in/out the number of shards automatically. The solution will scale in/out for a maximum of 8 times within 24 hours.
    Maximum Shard number <Requires input> Required if auto scaling is enabled. The maximum number of shards.

    Important

    You may observe duplicate logs in OpenSearch if threshold error occurs in Kinesis Data Streams (KDS). This is because the Fluent Bit log agent uploads logs in chunk (contains multiple records), and will retry the chunk if upload failed. Each KDS shard can support up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second. Please estimate your log volume and choose an appropriate shard number.

  5. Choose Next.

  6. In the Specify OpenSearch domain section, select an imported domain for Amazon OpenSearch domain.

  7. In the Log Lifecycle section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated Index State Management (ISM) policy automatically for this pipeline.

  8. In the Select log processor section, please choose the log processor.

    • (Optional) OSI as log processor is now supported in these regions. When OSI is selected, please type in the minimum and maximum number of OCU. See more information here.

  9. Choose Next.

  10. Enable Alarms if needed and select an exiting SNS topic. If you choose Create a new SNS topic, please provide a name and an email address for the new SNS topic.

  11. Add tags if needed.

  12. Choose Create.

  13. Wait for the application pipeline turning to "Active" state.

Create a log analytics pipeline (Light Engine)

Create a log analytics pipeline

  1. Sign in to the Centralized Logging with OpenSearch Console.

  2. In the left sidebar, under Log Analytics Pipelines, choose Application Log.

  3. Choose Create a pipeline.

  4. Choose Instance Group as Log Source, choose Light Engine, and choose Next.

  5. Select an instance group. If you have no instance group yet, choose Create Instance Group at the top right corner, and follow the Instance Group guide to create an instance group. After that, choose Refresh and then select the newly created instance group.

  6. (Auto Scaling Group only) If your instance group is created based on an Auto Scaling Group, after ingestion status become "Created", then you can find the generated Shell Script in the instance group's detail page. Copy the shell script and update the User Data of the Auto Scaling Launch configurations or Launch template.

  7. Keep the default Permission grant method.

  8. (Optional) If you choose I will manually add the below required permissions after pipeline creation, continue to do the following:

    1. Choose Expand to view required permissions and copy the provided JSON policy.
    2. Go to AWS Management Console.
    3. On the left navigation pane, choose IAM, and select Policies under Access management.
    4. Choose Create Policy, choose JSON and replace all the content inside the text block. Make sure to substitute <YOUR ACCOUNT ID> with your account id.
    5. Choose Next, and then enter a name for this policy.
    6. Attach the policy to your EC2 instance profile to grant the log agent permissions to send logs to the application log pipeline. If you are using Auto Scaling group, you need to update the IAM instance profile associated with the Auto Scaling Group. If needed, you can follow the documentation to update your [launch template][launch-template] or [launch configuration][launch-configuration].

  9. Choose Next.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon EC2 instance group as log source.

  1. Enter a Log Path to specify the location of logs you want to collect.

  2. Select a log config and then choose Next. If you do not find desired log config from the drop-down list, choose Create New, and follow instructions in Log Cong.

  3. In the Buffer section,

    • S3 buffer parameters
    Parameter Default Description
    S3 Bucket A log bucket will be created by the solution. You can also select a bucket to store the log data.
    Buffer size 50 MiB The maximum size of log data cached at the log agent side before delivering to S3. For more information, see Data Delivery Frequency.
    Buffer interval 60 seconds The maximum interval of the log agent to deliver logs to S3. For more information, see Data Delivery Frequency.
    Compression for data records Gzip The log agent compresses records before delivering them to the S3 bucket.

  4. Choose Next.

  5. In the Specify Light Engine Configuration section, if you want to ingest associated templated Grafana dashboards, select Yes for the sample dashboard.

  6. You can choose an existing Grafana, or if you need to import a new one, you can go to Grafana for configuration.

  7. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.

  8. The log processing frequency is set to 5 minutes by default, with a minimum processing frequency of 1 minute.

  9. In the Log Lifecycle section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.

  10. Select Next.

  11. Enable Alarms if needed and select an exiting SNS topic. If you choose Create a new SNS topic, please provide a name and an email address for the new SNS topic.

  12. If desired, add tags.

  13. Select Create.

  14. Wait for the application pipeline turning to "Active" state.