AWS services in this solution
This section describes the components and AWS services that make up this solution and the architecture details on how these components work together.
AWS services in this solution
The following AWS services are included in this solution:
| AWS service | Description |
|---|---|
| Amazon Elastic Load Balancing | Core. To distribute network traffic to ingestion fleet. |
| Amazon ECS | Core. To run the ingestion module fleet. |
| Amazon EC2 | Core. To provide the underlying computing resources for ingestion fleet. |
| Amazon ECR | Core. To host the container images used by ingestion fleet. |
| Amazon S3 | Core. To store the ingested and processed Clickstream data. And it also stores the service logs and static web assets (frontend user interface). |
| AWS Global Accelerator | Supporting. To improve the availability, performance, and security of the ingestion service in AWS Regions. |
| AWS CloudWatch | Supporting. To monitor the metrics, logs and trace of data pipeline. |
| Amazon SNS | Supporting. To provide topic and email subscription notifications for the alarms of data pipeline. |
| Amazon Kinesis Data Streams | Supporting. To provide the ingestion buffer. |
| AWS Lambda | Supporting. To integrate with kinds of AWS services. For example, sink ingestion data to S3, manage the lifecycle of AWS resources. |
| Amazon Managed Streaming for Apache Kafka (MSK) | Supporting. To provide the ingestion buffer with Apache Kafka. |
| Amazon EMR Serverless | Supporting. To process the ingested data. |
| Amazon Glue | Supporting. To manage the data catalog of ingested data. |
| Amazon EventBridge | Supporting. To integrate with AWS services with events or schedule. |
| Amazon Redshift | Supporting. To analyze your Clickstream data in data warehouse. |
| Amazon Athena | Supporting. To analyze your Clickstream data in data lake. |
| AWS Step Functions | Supporting. To orchestrate the lifecycle management of project's pipeline. Also it manages the workflow to load data into data warehouse. |
| AWS Secrets Manager | Supporting. To store the credential for OIDC credentials and BI user in Redshift. |
| Amazon QuickSight | Supporting. Visual your analysis reporting of your Clickstream data. |
| Amazon CloudFront | Supporting. To made available the static web assets (frontend user interface) and proxy the backend in the same origin. |
| Amazon Cognito | Supporting. To authenticate users (in AWS Regions). |
| Amazon API Gateway | Supporting. To provide the backend APIs. |
| Amazon DynamoDB | Supporting. To store projects data. |
| AWS CloudFormation | Supporting. To provision the AWS resources for the modules of data pipeline. |