Create Amazon ECR transfer task

The solution allows you to create an Amazon ECR transfer task in the following ways:

• using the web console
• using the ECR plugin
• using AWS CLI

You can make your choice according to your needs.

• The web console provides an intuitive user interface where you can start, clone or stop a data transfer task with a simple click. The frontend also provides metric monitoring and logging view, so you do not need to switch between different pages.

• The ECR plugin is a standalone CloudFormation template, and you can easily integrate it into your workflows. Because this option allows deployment without the frontend, it is useful if you want to deploy in AWS China Regions but do not have an ICP licensed domain.

• AWS CLI can quickly initiate data transfer tasks. Select this option if you want to leverage Data Transfer Hub in your automation scripts.

Using the web console

You can use the web console to create an Amazon ECR transfer task. For more information about how to launch the web console, see deployment.

1. From the Create Transfer Task page, select Start a New Task, and then select Next.

2. From the Engine options page, under engine, select Amazon ECR, and then choose Next Step. You can also copy image from Docker Hub，GCR.io，Quay.io, and so on by choosing Public Container Registry.

3. Specify the transfer task details. In Source Type, select the container warehouse type.

4. In Source settings, enter Source Region and Amazon Web Services Account ID.

5. To create credential information, select Secrets Manager to jump to the AWS Secrets Manager console in the current region.

   • From the left menu, select Secrets, then choose Store a new secret and select the other type of secrets key type.
   • Fill in the access_key_id and secret_access_key information in the Plaintext input box according to the displayed format. For more information, refer to IAM features in the IAM User Guide. Choose Next.
   • (Optional) Enter the key name and description. Choose Next.
   • In the configuration of automatic rotation, select Disable automatic rotation. Choose Next.
   • Keep the default value and choose Save to complete the creation of the key.
   • Navigate back to the Data Transfer Hub task creation interface and refresh the interface. Your new secret is displayed in the drop-down list.

6. Select the certificate (Secret).

   Note

   If the source is in the same account with Data Transfer Hub deployment, you need to create/provide credential info for the destination. Otherwise, no credential information is needed.

7. Enter an email address in Alarm Email.

8. Choose Next and review your task parameter details.

9. Choose Create Task.

After the task is created successfully, it will appear on the Tasks page.

Using the ECR plugin

Note

This tutorial is a deployment guide for the backend-only version. For more details, please refer to ECR Plugin Introduction.

Step 1. Prepare VPC (optional)

This solution can be deployed in both public and private subnets. Using public subnets is recommended.

• If you want to use existing VPC, please make sure the VPC has at least 2 subnets, and both subnets must have public internet access (Either public subnets with internet gateway or private subnets with NAT gateway)

• If you want to create new default VPC for this solution, please go to Step 2 and make sure you have Create a new VPC for this cluster selected when you create the cluster.

Step 2. Set up ECS Cluster

An ECS Cluster is required for this solution to run Fargate task.

1. Sign in to AWS Management Console, and choose Elastic Container Service (ECS).
2. From ECS Cluster home page, click Create Cluster.
3. Select Cluster Template. Choose Network Only type.
4. Specify a cluster name and click Create to create a cluster. If you want to also create a new VPC (public subnets only), please also check the Create a new VPC for this cluster option.

Step 3. Configure credentials

If source (or destination) is NOT in current AWS account, you will need to provide AccessKeyID and SecretAccessKey (namely AK/SK) to pull from or push to Amazon ECR. And Secrets Manager is used to store the credentials in a secure manner.

If source type is Public, there is no need to provide the source credentials.

Go to AWS Management Console > Secrets Manager. From Secrets Manager home page, click Store a new secret. For secret type, please use Other type of secrets. For key/value paris, please copy and paste below JSON text into the Plaintext section, and change value to your AK/SK accordingly.

{
  "access_key_id": "<Your Access Key ID>",
  "secret_access_key": "<Your Access Key Secret>"
}

Click Next to specify a secret name, and click Create in teh last step.

Step 4. Launch AWS Cloudformation Stack

Please follow below steps to deploy this plugin via AWS Cloudformation.

1. Sign in to AWS Management Console, and switch to the Region where you want to deploy the CloudFormation Stack.

2. Click the following button to launch the CloudFormation Stack in that Region.

   • For AWS China Regions

   

   • For AWS Global Regions

   

   • For AWS GovCloud (US) Regions

   

3. Click Next. Specify values to parameters accordingly. Change the stack name if required.

4. Click Next. Configure additional stack options such as tags (Optional).

5. Click Next. Review and confirm acknowledgement, and then click Create Stack to start the deployment.

The deployment will take approximately 3 to 5 minutes.

Using AWS CLI

You can use the AWS CLI to create an Amazon ECR transfer task. Note that if you have deployed the DTH Portal at the same time, the tasks started through the CLI will not appear in the Task List on your Portal.

1. Create an Amazon VPC with two public subnets or two private subnets with NAT gateway.

2. Replace <CLOUDFORMATION_URL> as shown below.

   https://solutions-reference.s3.amazonaws.com/data-transfer-hub/latest/DataTransferECRStack.template
   

3. Go to your terminal and enter the following command. For the parameter details, refer to the Parameters table.

   aws cloudformation create-stack --stack-name dth-ecr-task --template-url CLOUDFORMATION_URL \
   --capabilities CAPABILITY_NAMED_IAM \
   --parameters \
   ParameterKey=sourceType,ParameterValue=Amazon_ECR \
   ParameterKey=srcRegion,ParameterValue=us-east-1 \
   ParameterKey=srcAccountId,ParameterValue=123456789012 \
   ParameterKey=srcList,ParameterValue=ALL \
   ParameterKey=includeUntagged,ParameterValue=false \
   ParameterKey=srcImageList,ParameterValue= \
   ParameterKey=srcCredential,ParameterValue=dev-us-credential \
   ParameterKey=destAccountId,ParameterValue= \
   ParameterKey=destRegion,ParameterValue=us-west-2 \
   ParameterKey=destCredential,ParameterValue= \
   ParameterKey=destPrefix,ParameterValue= \
   ParameterKey=alarmEmail,ParameterValue=your_email@example.com \
   ParameterKey=ecsVpcId,ParameterValue=vpc-07f56e8e21630a2a0 \
   ParameterKey=ecsClusterName,ParameterValue=dth-v22-01-TaskCluster-eHzKkHatj0tN \
   ParameterKey=ecsSubnetA,ParameterValue=subnet-034c58fe0e696eb0b \
   ParameterKey=ecsSubnetB,ParameterValue=subnet-0487ae5a1d3badde7
   

   Parameters

Parameter Name	Allowed Value	Default Value	Description
sourceType	Amazon_ECR Public	Amazon_ECR	Choose type of source container registry, for example Amazon_ECR, or Public from Docker Hub, gco.io, etc.
srcRegion			Source Region Name (only required if source type is Amazon ECR), for example, us-west-1
srcAccountId			Source AWS Account ID (only required if source type is Amazon ECR), leave it blank if source is in current account
srcList	ALL SELECTED	ALL	Type of Source Image List, either ALL or SELECTED, for public registry, please use SELECTED only
srcImageList			Selected Image List delimited by comma, for example, ubuntu:latest,alpine:latest..., leave it blank if Type is ALL. For ECR source, using ALL_TAGS tag to get all tags.
srcCredential			The secret name in Secrets Manager only when using AK/SK credentials to pull images from source Amazon ECR, leave it blank for public registry
destRegion			Destination Region Name, for example, cn-north-1
destAccountId			Destination AWS Account ID, leave it blank if destination is in current account
destPrefix			Destination Repo Prefix
destCredential			The secret name in Secrets Manager only when using AK/SK credentials to push images to destination Amazon ECR
includeUntagged	true false	true	Whether to include untagged images in the replication
ecsClusterName			ECS Cluster Name to run ECS task (Please make sure the cluster exists)
ecsVpcId			VPC ID to run ECS task, e.g. vpc-bef13dc7
ecsSubnetA			First Subnet ID to run ECS task, e.g. subnet-97bfc4cd
ecsSubnetB			Second Subnet ID to run ECS task, e.g. subnet-7ad7de32
alarmEmail			Alarm Email address to receive notification in case of any failure