Skip to content

VPC Flow Logs

VPC Flow Logs enable you to capture information about the IP traffic going to and from network interfaces in your VPC.

Create log ingestion

You can create a log ingestion into Amazon OpenSearch Service either by using the Centralized Logging with OpenSearch console or by deploying a standalone CloudFormation stack.

Important

  • Centralized Logging with OpenSearch supports VPCs who publish the flow log data to an Amazon S3 bucket or a CloudWatch log group. When publishing to S3, The S3 Bucket region must be the same as the Centralized Logging with OpenSearch solution region.
  • The Amazon OpenSearch Service index is rotated on a daily basis by default, and you can adjust the index in the Additional Settings.

Using the Centralized Logging with OpenSearch Console

  1. Sign in to the Centralized Logging with OpenSearch Console.
  2. In the navigation pane, under Log Analytics Pipelines, choose Service Log.
  3. Choose the Create a log ingestion button.
  4. In the AWS Services section, choose VPC Flow Logs.
  5. Choose Next.
  6. Under Specify settings, choose Automatic or Manual for VPC Flow Log enabling. The automatic mode will enable the VPC Flow Log and save the logs to a centralized S3 bucket if logging is not enabled yet.
    • For Automatic mode, choose the VPC from the dropdown list.
    • For Manual mode, enter the VPC Name and VPC Flow Logs location.
    • (Optional) If you are ingesting VPC Flow logs from another account, select a linked account from the Account dropdown list first.
  7. Under Log Source, select S3 or CloudWatch as the source.
  8. Choose Next.
  9. In the Specify OpenSearch domain section, select an imported domain for Amazon OpenSearch domain.
  10. Choose Yes for Sample dashboard if you want to ingest an associated built-in Amazon OpenSearch Service dashboard.
  11. You can change the Index Prefix of the target Amazon OpenSearch Service index if needed. The default prefix is your VPC name.
  12. In the Log Lifecycle section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch will create the associated Index State Management (ISM) policy automatically for this pipeline.
  13. In the Select log processor section, please choose the log processor.
    • (Optional) OSI as log processor is now supported in these regions. When OSI is selected, please type in the minimum and maximum number of OCU. See more information here.
  14. Choose Next.
  15. Add tags if needed.
  16. Choose Create.

Using the standalone CloudFormation Stack

This automated AWS CloudFormation template deploys the Centralized Logging with OpenSearch - VPC Flow Logs Ingestion solution in the AWS Cloud.

Launch in AWS Console Download Template
AWS Standard Regions Launch Stack Template
AWS China Regions Launch Stack Template

  1. Log in to the AWS Management Console and select above button to launch the AWS CloudFormation template. You can also download the template as a starting point for your own implementation.

  2. To launch the stack in a different AWS Region, use the Region selector in the console navigation bar.

  3. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next.

  4. On the Specify stack details page, assign a name to your solution stack.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following parameters.

    Parameter Default Description
    Log Bucket Name <Requires input> The S3 bucket name which stores the logs.
    Log Bucket Prefix <Requires input> The S3 bucket path prefix which stores the logs.
    Log Source Account ID <Optional> The AWS Account ID of the S3 bucket. Required for cross-account log ingestion (Please add a member account first). By default, the Account ID you logged in at Step 1 will be used.
    Log Source Region <Optional> The AWS Region of the S3 bucket. By default, the Region you selected at Step 2 will be used.
    Log Source Account Assume Role <Optional> The IAM Role ARN used for cross-account log ingestion. Required for cross-account log ingestion (Please add a member account first).
    Engine Type OpenSearch The engine type of the OpenSearch. Select OpenSearch or Elasticsearch.
    OpenSearch Domain Name <Requires input> The domain name of the Amazon OpenSearch cluster.
    OpenSearch Endpoint <Requires input> The OpenSearch endpoint URL. For example, vpc-your_opensearch_domain_name-xcvgw6uu2o6zafsiefxubwuohe.us-east-1.es.amazonaws.com
    Index Prefix <Requires input> The common prefix of OpenSearch index for the log. The index name will be <Index Prefix>-<Log Type>-<Other Suffix>.
    Create Sample Dashboard Yes Whether to create a sample OpenSearch dashboard.
    VPC ID <Requires input> Select a VPC which has access to the OpenSearch domain. The log processing Lambda will reside in the selected VPC.
    Subnet IDs <Requires input> Select at least two subnets which have access to the OpenSearch domain. The log processing Lambda will reside in the subnets. Make sure the subnets have access to the Amazon S3 service.
    Security Group ID <Requires input> Select a Security Group which will be associated with the log processing Lambda. Make sure the Security Group has access to the OpenSearch domain.
    S3 Backup Bucket <Requires input> The S3 backup bucket name to store the failed ingestion logs.
    KMS-CMK ARN <Optional> The KMS-CMK ARN for encryption. Leave it blank to create a new KMS CMK.
    Number Of Shards 5 Number of shards to distribute the index evenly across all data nodes. Keep the size of each shard between 10-50 GB.
    Number of Replicas 1 Number of replicas for OpenSearch Index. Each replica is a full copy of an index.
    Age to Warm Storage <Optional> The age required to move the index into warm storage (e.g. 7d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). This is only effective when warm storage is enabled in OpenSearch.
    Age to Cold Storage <Optional> The age required to move the index into cold storage (e.g. 30d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). This is only effective when cold storage is enabled in OpenSearch.
    Age to Retain <Optional> The age to retain the index (e.g. 180d). Index age is the time between its creation and the present. Supported units are d (days) and h (hours). If value is "", the index will not be deleted.
    Rollover Index Size <Optional> The minimum size of the shard storage required to roll over the index (e.g. 30GB).
    Index Suffix yyyy-MM-dd The common suffix format of OpenSearch index for the log(Example: yyyy-MM-dd, yyyy-MM-dd-HH). The index name will be <Index Prefix>-<Log Type>-<Index Suffix>-000001.
    Compression type best_compression The compression type to use to compress stored data. Available values are best_compression and default.
    Refresh Interval 1s How often the index should refresh, which publishes its most recent changes and makes them available for searching. Can be set to -1 to disable refreshing. Default is 1s.
    EnableS3Notification True An option to enable or disable notifications for Amazon S3 buckets. The default option is recommended for most cases.
    LogProcessorRoleName <Optional> Specify a role name for the log processor. The name should NOT duplicate an existing role name. If no name is specified, a random name is generated.
    QueueName <Optional> Specify a queue name for an SQS. The name should NOT duplicate an existing queue name. If no name is given, a random name will be generated.

  6. Choose Next.

  7. On the Configure stack options page, choose Next.

  8. On the Review page, review and confirm the settings. Check the box acknowledging that the template creates AWS Identity and Access Management (IAM) resources.

  9. Choose Create stack to deploy the stack.

You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately 10 minutes.

View dashboard

The dashboard includes the following visualizations.

Visualization Name Source Field Description
Global Filters
  • account-id
  • region
  • vpc-id
  • subnet-id
  • action
  • flow-direction
  • log-status
  • protocol-code
  • type
The charts are filtered according to Account ID, Region, VPC ID and other conditions.
Total Requests
  • log event
Shows the total number of network requests logged by VPC Flow Logs during a selected time period.
Request History
  • log event
Presents a bar chart that displays the distribution of events over time.
Requests By VPC ID
  • vpc-id
Displays the proportional breakdown of network requests by source VPC using a pie chart.
Total Requests By Action
  • action
Displays the total volume of requests segmented by action over time.
Total Bytes
  • bytes
 Provides visibility into overall bandwidth usage and traffic patterns across the monitored VPCs, subnets, network interfaces and security groups.
Total Packets
  • packets
Displays total logged packets over time to visualize trends, surges and dips.
Bytes Metric
  • bytes
  • flow-direction
 Shows the distribution of incoming (Ingress) and outgoing (Egress) network traffic volumes in bytes across the range of flows logged by VPC Flow Logs over a time period.
Requests By Direction
  • flow-direction
 Provides visibility into the proportional composition of incoming versus outgoing requests.
Requests By Direction
  • flow-direction
Displays the total number of network flows logged by VPC Flow Logs segmented by traffic direction - Ingress vs Egress.
Requests By Type
  • type
Shows the volume of flows for each type. This provides visibility into the protocol composition of network requests traversing the environment.
Top Source Bytes
  • srcaddr
  • bytes
 Displays the source IP addresses transmitting the highest outbound volume of data during the selected time period.
Top Destination Bytes
  • dstaddr
  • bytes
 Enables you to monitor and analyze outbound traffic from your VPC to external destinations.
Top Source Requests
  • srcaddr
Allows you to see which resources inside your VPC are initiating external requests.
Top Destination Requests
  • dstaddr
 Allows you to see which external hosts are being contacted most by your VPC resources.
Requests by Protocol
  • protocol-code
 Displays network flows logged by VPC Flow Logs segmented by traffic type - TCP, UDP, ICMP etc.
Requests by Status
  • log-status
 Provides a breakdown of network flows by their traffic status - Accepted, Rejected or Other.
Top Sources AWS Services
  • pkt-src-aws-service
 Show the proportional distribution of flows originating from top AWS sources like S3, CloudFront, Lambda, etc. during the selected time period.
Top Destination AWS Services
  • pkt-dst-aws-service
 Provide visibility into IP traffic going to and from AWS services located outside your VPC. By enabling flow logs on VPC subnets/interfaces and filtering on traffic with an ACCEPT action, you can view outbound flows from your VPC to various AWS services.
Network Flow
  • srcaddr
  • dstaddr
 Allows you to view information about the IP traffic going to and from network interfaces in your VPC.
Heat Map
  • srcaddr
  • dstaddr
 Offers a visual summary of connections between source and destination IPs in your flow log data.
Egress Traffic Path
  • traffic-path
 Allows you to enable flow logging on VPC network interfaces to capture information about all IP traffic going to and from that interface.
Search
  • @timestamp
  • account-id
  • vpc-id
  • flow-direction
  • action
  • protocol-code
  • srcaddr
  • scaport
  • dstaddr
  • dstport
  • bytes
  • packets
  • log-status
 Searching through the detailed flow log data allows pinpoint analysis of traffic around security events, network issues, changes in usage patterns, and more.

Sample Dashboard

You can access the built-in dashboard in Amazon OpenSearch to view log data. For more information, see Access Dashboard.

You can click the below image to view the high-resolution sample dashboard.

vpcflow-db