object storage migration service - huawei cloud

Object Storage Migration Service

FAQs

Issue 18

Date 2021-10-27

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 18 (2021-10-27) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Question Summary................................................................................................................. 1

2 Product Consulting..................................................................................................................22.1 When Do I Need to Perform Pre-migration Evaluation Before Creating Migration Tasks?.......................... 22.2 How Do I Migrate OBS Bucket Data from One Region to Another?.................................................................... 22.3 How Do I Migrate the OBS Bucket Data from One Account to Another?.......................................................... 22.4 How Do I Migrate Data from Multiple Source Buckets to a Single OBS Bucket on HUAWEI CLOUD?.............................................................................................................................................................................................................. 32.5 How Do I Migrate Data in the Root Directory?............................................................................................................ 32.6 How Does Migration Affect My Source Data? What Are the Impacts on Migration If the Source DataChanges?........................................................................................................................................................................................... 42.7 How Do I Change the Storage Path of a Migrated Object?.....................................................................................42.8 How Does OMS Ensure Data Consistency If Data Is Continuously Written to OBS?...................................... 52.9 How Do I Obtain Required Permissions for the Source and Destination Platform Accounts?.....................62.10 How Do I Review the Objects Migrated in a Migration Task?............................................................................102.11 How Do I Review the Objects Migrated in a Migration Task Group?.............................................................. 112.12 How Do I Review Objects Migrated in a Synchronization Task?....................................................................... 132.13 What Is the Storage Class of Archive Data Migrated to the Archive Destination Bucket?.......................142.14 What Are the Application Scenarios of Migration Tasks and Migration Task Groups?............................. 142.15 If I Delete an Object from the Source Object, Will the Corresponding Object in the DestinationBucket Be Deleted?......................................................................................................................................................................152.16 How Does OMS Ensure Data Consistency Between the Source and Destination Buckets?..................... 152.17 How Does Migration Affect Data in the Destination Bucket?............................................................................15

3 Migration Constraints.......................................................................................................... 173.1 What Are the Constraints on Migrating Object Storage Data to a Parallel File System with OMS?...... 173.2 How Do I Fix the Error "Destination bucket must be in the same region as OMS. Select a bucket thatis located in region xxx, or change the OMS region to xxx" When I Create a Migration Task?.......................173.3 How Do I Migrate Data from a Cloud Service Provider Not Supported by OMS?.........................................18

4 Migration Duration............................................................................................................... 204.1 How Can I Learn the Progress of a Migration Task?................................................................................................ 204.2 Why Is the Migration Speed Slow Sometimes?..........................................................................................................20

5 Exception Recovery............................................................................................................... 225.1 What Can I Do If Migration Tasks Fail To Be Created After Evaluation?.......................................................... 22

Object Storage Migration ServiceFAQs Contents

Issue 18 (2021-10-27) Copyright © Huawei Technologies Co., Ltd. ii

5.2 What Can I Do If the Migration Fails Because the Frequency of Accessing the Source Object StorageExceeded the Upper Limit?....................................................................................................................................................... 225.3 What Can I Do If a Migration Task Fails?.....................................................................................................................225.4 What Can I Do If a Migration Task Group Fails to Be Executed?.........................................................................275.5 What Can I Do If SMN Is Not Working Properly?..................................................................................................... 275.6 What Can I Do If a Migration Task Fails Due to Inconsistent Encryption Status?......................................... 285.7 What Can I Do If the Migration Task Failed Due to an Archive Data Read Failure?.................................... 285.8 What Are the Common Questions and Their Solutions in the CDN Scenario?...............................................29

6 Change History...................................................................................................................... 33

Object Storage Migration ServiceFAQs Contents

Issue 18 (2021-10-27) Copyright © Huawei Technologies Co., Ltd. iii

1 Question Summary

Product Consulting● How Does Migration Affect My Source Data? What Are the Impacts on

Migration If the Source Data Changes?

● How Do I Change the Storage Path of a Migrated Object?

● How Does OMS Ensure Data Consistency If Data Is Continuously Writtento OBS?

● How Does OMS Ensure Data Consistency Between the Source andDestination Buckets?

● How Does Migration Affect Data in the Destination Bucket?

Migration Constraints● What Are the Constraints on Migrating Object Storage Data to a Parallel

File System with OMS?

● How Do I Migrate Data from a Cloud Service Provider Not Supported byOMS?

Migration Duration● How Can I Learn the Progress of a Migration Task?

Troubleshooting● How Do I Fix the Error "Destination bucket must be in the same region as

OMS. Select a bucket that is located in region xxx, or change the OMSregion to xxx" When I Create a Migration Task?

● What Can I Do If a Migration Task Fails?

● What Can I Do If a Migration Task Group Fails to Be Executed?

● What Can I Do If the Migration Task Failed Due to an Archive Data ReadFailure?

● What Are the Common Questions and Their Solutions in the CDNScenario?

Object Storage Migration ServiceFAQs 1 Question Summary

Issue 18 (2021-10-27) Copyright © Huawei Technologies Co., Ltd. 1

2 Product Consulting

2.1 When Do I Need to Perform Pre-migrationEvaluation Before Creating Migration Tasks?

If there is more than 3 TB of data or 5 million objects in the source bucket, youare advised to perform pre-migration evaluation before creating migration tasks.After the evaluation succeeds, you can create multiple migration tasks on the Pre-Migration Evaluation page. The system can arrange and create migration tasksbased on the evaluation results to maximize the concurrent migrationperformance and reduce migration time.

For details, see Evaluating Data in a Bucket and Creating Migration TasksBased on Evaluation.

2.2 How Do I Migrate OBS Bucket Data from OneRegion to Another?

If you need to migrate OBS bucket data from one region to another, for example,from CN South-Guangzhou to CN North-Beijing1, you can create a migrationtask.

CA UTION

The destination bucket and OMS must be in the same region.

2.3 How Do I Migrate the OBS Bucket Data from OneAccount to Another?

If you need to migrate the OBS bucket data from account A to account B, performthe following operations:

Object Storage Migration ServiceFAQs 2 Product Consulting


https://support.huaweicloud.com/intl/en-us/usermanual-oms/oms_01_0013.html



https://support.huaweicloud.com/intl/en-us/qs-oms/topic_0045916962.html


1. Log in to the management console using account B.2. Click Service List. Under Migration, click Object Storage Migration Service.

The Object Storage Migration Service page is displayed.3. In the upper right corner, click Create Migration Task.

Read the privacy statement, select Agree to the above privacy agreements,and click OK.

4. Enter the AK and SK of account A in the Select Source area, and enter the AKand SK of account B in the Select Destination area.For instructions about how to configure the other parameters, see Creating aMigration Task.

NO TE

For details about how to migrate OBS bucket data from one region to another, seeHow Do I Migrate OBS Bucket Data from One Region to Another?

2.4 How Do I Migrate Data from Multiple SourceBuckets to a Single OBS Bucket on HUAWEI CLOUD?

If you want to migrate data in source buckets A and B to HUAWEI CLOUD OBSbucket H, you can perform the following operations:

1. Create one migration task to migrate data from source bucket A to HUAWEICLOUD OBS bucket H.

2. Create another migration task to migrate data from source bucket B to HUAWEICLOUD OBS bucket H.

If source buckets A and B have objects with the same name, the object migratedfrom source bucket A will be overwritten. For objects with the same name, objectsmigrated earlier will be overwritten by objects migrated later.

2.5 How Do I Migrate Data in the Root Directory?There are two migration methods you can use to migrate data in the rootdirectory: File/Folder and Object Name Prefix

● File/Folder

a. When creating a task, select File/Folder for Select Migration Method.b. Click Select. In the displayed dialog box, select the data to be migrated in

the root directory (displayed by default) and click OK.c. Set advanced parameters and create a task to migrate the selected root

directory data.● Object Name Prefix

a. When creating a task, select Object Name Prefix for Select MigrationMethod.

b. Enter the prefix of the target objects to be migrated, including those inthe root directory, and click Add.You can add multiple prefixes to migrate objects with different prefixes.





c. Set advanced parameters and create a task to migrate the selected rootdirectory data.

2.6 How Does Migration Affect My Source Data? WhatAre the Impacts on Migration If the Source DataChanges?

No Impact on Source Data● Data in the source bucket is not modified during the migration.● Data in the source bucket is not locked during the migration.● Data in the source bucket is not deleted after the migration.

Impacts on Services on Source Platforms● A fast migration may generate additional fees, for instance, CDN fees, or

affect other services on the source platform. You are advised to limit themigration speed based on your service requirements. For details, see "TrafficLimit" described in Creating a Migration Task.

● You may be unaware of the migration fees generated on the source platform,and this may cause your account to fall into arrears, which can affect yourservices. For details about the billing mode of OMS, see Pricing Details.

Impacts Caused by Source Data Changes● Data addition

By default, objects are migrated based on the selected prefixes, inalphabetical order. During the migration, if any data is added to the sourcebucket before OMS finishes listing source objects, the new data will bemigrated, but data added after that point will not.

● Data modificationTo ensure reliability, OMS migrates large objects in parts and, after themigration, compares the sizes and modification times of the source objectwith the corresponding destination objects. If a part is modified during themigration and both of the following conditions are met, data inconsistencyoccurs:– The part size is the same before and after the modification.– The last modification time of the source object is earlier than that of the

destination object.● Data deletion

After migration, if some data in the source bucket is deleted, thecorresponding data in the destination bucket will not be affected.

2.7 How Do I Change the Storage Path of a MigratedObject?

Choose a method based on how many objects there are to migrate:




https://support.huaweicloud.com/intl/en-us/productdesc-oms/oms_01_0028.html

● Few objectsUse the OBS Browser+ tool provided by HUAWEI CLOUD OBS to move thedata.

a. Download OBS Browser+. For details, see Downloading OBS Browser+.b. Log in to OBS Browser+, click the bucket from which you want to move

the target objects (directories or files), and right-click the file or folderand choose Copy from the shortcut menu.

c. Select a path to save the target objects. Right-click the path and choosePaste from the shortcut menu.

● Many objectsUse the obsutil tool to change the object path.

a. Download obsutil and initialize it. For details, see Download andInstallation and Performing Initial Configuration.

b. Use commands to move the objects.The following uses Windows as an example:

– Run obsutil mv obs://src_bucket/key obs://dsc_bucket/dsc_path tomove a single object.

– Run obsutil mv obs://src_bucket/prefix obs://dsc_bucket/dsc_path -f -rto move objects in batches.For batch operations, prefix indicates the name prefix of the objects. Thetool moves all the migrated objects matching that name prefix.For more information about the obsutil commands, see Moving anObject.

2.8 How Does OMS Ensure Data Consistency If Data IsContinuously Written to OBS?

This problem can be described in the following two cases:

1. How to migrate incremental data?For details, see Incremental Data Migration.

2. How to synchronize updated data from the source server?Perform full migration again. OMS automatically identifies the updatedobjects and overwrites them. For details about the overwriting conditions, seeHow Does Migration Affect Data in the Destination Bucket?



https://support.huaweicloud.com/intl/en-us/browsertg-obs/obs_03_1003.html

https://support.huaweicloud.com/intl/en-us/utiltg-obs/obs_11_0003.html





https://support.huaweicloud.com/intl/en-us/bestpractice-oms/oms_01_0015.html

2.9 How Do I Obtain Required Permissions for theSource and Destination Platform Accounts?

Source PermissionsNO TE

The following uses an OBS bucket on HUAWEI CLOUD as the source bucket. If your sourcebucket is on the other cloud service platform, contact engineers of the cloud serviceplatform.

The source platform account needs the following permissions:● Listing all buckets● Obtaining bucket location● Listing objects● Obtaining object metadata● Obtaining object content

You can use either of the following methods to obtain required permissions for thesource platform account:

1. Using system-defined policy OBS AdministratorFor details, see Creating a User Group and Assigning Permissions.

2. Creating a custom policy

a. Log in to HUAWEI CLOUD and click Console in the upper right corner.b. On the management console, hover the mouse pointer over the

username in the upper right corner, and choose Identity and AccessManagement from the drop-down list.

c. In the navigation pane on the left, choose Permissions > Policies/Roles.d. Click Create Custom Policy in the upper right corner.

e. Specify a policy name, and set Scope to Global services and Policy Viewto JSON.

f. Copy the following content to the box next to Policy Content and clickOK.



https://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_03_0001.html

{ "Version": "1.1", "Statement": [ { "Action": [ "obs:bucket:ListAllMyBuckets", "obs:bucket:GetBucketLocation", "obs:bucket:ListBucket", "obs:object:GetObject" ], "Resource": [ "obs:*:*:bucket:*", "obs:*:*:object:*" ], "Effect": "Allow" } ]}

After the custom policy is created, perform the following operations to obtain thepermissions defined by the custom policy:

● If you have joined a user group, assign the custom policy to the user group,and you will obtain the permissions defined by the custom policy. For details,see Assigning Permissions to a User Group.

● If you have not joined a user group, perform the following steps:

a. Create a user group and assign the custom policy to it. For details, seeCreating a User Group and Assigning Permissions.



https://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_03_0001.html#iam_03_0001__section12561378357


b. Add yourself to the user group. You will have the permissions of the usergroup. For details, see Adding Users to a User Group.

CA UTION

It takes 15 to 30 minutes for the authorization to take effects.

Destination PermissionsThe destination platform account needs the following permissions:● Listing all buckets● Obtaining bucket location● Listing objects● Obtaining object metadata● Obtaining object content● Listing uploaded parts● Restoring archive objects

You can use either of the following methods to obtain required permissions for thedestination platform account:

1. Use system-defined policy OBS Administrator.For details, see Creating a User Group and Assigning Permissions.

2. Creating a custom policy.

a. Log in to HUAWEI CLOUD and click Console in the upper right corner.b. On the management console, hover the mouse pointer over the

username in the upper right corner, and choose Identity and AccessManagement from the drop-down list.

c. In the navigation pane on the left, choose Permissions > Policies/Roles.d. Click Create Custom Policy in the upper right corner.

e. Specify a policy name, and set Scope to Global services and Policy Viewto JSON.

f. Copy the following content to the box next to Policy Content and clickOK.





{ "Version": "1.1", "Statement": [ { "Action": [ "obs:bucket:ListBucket", "obs:object:PutObject", "obs:object:RestoreObject", "obs:object:GetObject", "obs:bucket:GetBucketLocation", "obs:object:ListMultipartUploadParts" ], "Resource": [ "obs:*:*:bucket:*", "obs:*:*:object:*" ], "Effect": "Allow" }, { "Action": [ "obs:bucket:ListBucketMultipartUploads" ], "Resource": [ "obs:*:*:bucket:*" ], "Effect": "Allow" }, { "Action": [ "obs:bucket:ListAllMyBuckets" ], "Effect": "Allow" }



]}

After the custom policy is created, perform the following operations to obtain thepermissions defined by the custom policy:

● If you have joined a user group, assign the custom policy to the user group,and you will obtain the permissions defined by the custom policy. For details,see Assigning Permissions to a User Group.

● If you have not joined a user group, perform the following steps:

a. Create a user group and assign the custom policy to it. For details, seeCreating a User Group and Assigning Permissions.

b. Add yourself to the user group. You will have the permissions of the usergroup. For details, see Adding Users to a User Group.

CA UTION

It takes 15 to 30 minutes for the authorization to take effects.

2.10 How Do I Review the Objects Migrated in aMigration Task?

You can perform the following steps to view details about a migration task andobtain the list of objects migrated in the task.

Step 1 Log in to the management console.

Step 2 In the service list on the left, choose Object Storage Migration Service.

Step 3 In the displayed Migration Task page, locate the low where the task you want toview resides, and click View Details in the lower right corner.

Figure 2-1 Migration task details

NO TE

On the task details page, you can view the numbers of source objects, migrated objects,ignored objects, and failed objects.

Step 4 View the object lists.



https://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_03_0001.html#iam_03_0001__section12561378357



If... Then...

The number of migratedobjects is greater than 0

On the task details page, click the path link nextto Migrated Object Storage Path to downloadthe list of migrated objects in the .txt format.NOTE

The path for storing a migrated object list is in the oms/task/Task name/success_object_lists/Timestamp.success_list.txt.

The number of ignoredobjects is greater than 0

On the task details page, click the path link nextto Ignored Object Storage Path to download thelist of ignored objects in the .txt format.NOTE

The path for storing an ignored object list is in the oms/task/Task name/skip_object_lists/Timestamp.skip_list.txt.

The number of failedobjects is greater than 0

On the task details page, click the path link nextto Failed Object Storage Path to download thelist of failed objects in the .txt format.NOTE

The path for storing a failed object list is in the oms/task/Task name/failed_object_lists/Timestamp.failed_list.txt.

For details about the error codes in a failed objectlist, see Viewing the List of Failed Objects.

----End

2.11 How Do I Review the Objects Migrated in aMigration Task Group?

You can perform the following steps to view details about a migration task groupand obtain the list of objects migrated in the task group.



Step 3 In the navigation tree, click Migration Task Group.

The Migration Task Group page is displayed.

Step 4 Locate the migration task group you want to view, and click on the left of thegroup ID.



https://support.huaweicloud.com/intl/en-us/oms_faq/oms_faq_0004.html

Figure 2-2 Migration task group details

NO TE

On the task group details page, you can view the numbers of tasks in the Migrating,Failed, Waiting, Suspended, and Successful state, as well as the numbers of ignored andfailed objects.

Step 5 View the object lists.

If... Then...

The number of migratedobjects is greater than 0

On the task group details page, click the path linknext to Migrated Object Storage Path todownload the list of migrated objects in the .txtformat.NOTE

The path for storing a migrated object list is in the oms/task_group/Task group ID/success_object_lists/Taskname/Timestamp.success_list.txt format.

The number of ignoredobjects is greater than 0

On the task group details page, click the path linknext to Ignored Object Storage Path todownload the list of ignored objects in the .txtformat.NOTE

The path for storing an ignored object list is in the oms/task_group/Task group ID/skip_object_lists/Taskname/Timestamp.skip_list.txt.

The number of failedobjects is greater than 0

On the task group details page, click the path linknext to Failed Object Storage Path to downloadthe list of failed objects in the .txt format.NOTE

The path for storing a failed object list is in the oms/task_group/Task group ID/failed_object_lists/Taskname/Timestamp.failed_list.txt.

For details about the error codes in a failed objectlist, see Viewing the List of Failed Objects.

Step 6 View the list of all source objects to be migrated.

Log in to OBS Console and access the /oms/taskgroup/Task group ID/ path in thedestination bucket.




NO TE

You can view the task group ID on the Migration Task Group page.

----End

2.12 How Do I Review Objects Migrated in aSynchronization Task?

You can perform the following steps to view details about a synchronization taskand obtain the list of objects migrated in the task.

NO TE

Currently, the synchronization task function is only available in CN North-Beijing4.



Step 3 In the navigation tree, click Synchronization Task. The Synchronization Taskpage is displayed.

Step 4 Locate the synchronization task you want to view, and click on the left of thetask ID.

Figure 2-3 Synchronization task details

NO TE

On the synchronization task details page, you can view the numbers of requested,synchronized, failed, and ignored objects as well as the total size of synchronized objects inthis month.

Step 5 Click the link next to Object Lists to obtain the object lists. You will be directed tothe OBS console.● If the number of synchronized objects is greater than 0, you can download the

corresponding .txt file in the success_object_lists folder to view the list ofthese synchronized objects.



NO TE

The path for storing a synchronized object list is in the oms/sync_task/Synchronization task ID/success_object_lists/Sub-task name/Timestamp.success_list.txt.

● If the number of ignored objects is greater than 0, you can download thecorresponding .txt file in the skip_object_lists folder to view the list of theseignored objects.

NO TE

The path for storing an ignored object list is in the oms/sync_task/Synchronizationtask ID/skip_object_lists/Sub-task name/Timestamp.skip_list.txt.

● If the number of failed objects is greater than 0, you can download thecorresponding .txt file in the failed_object_lists folder to view the list of thesefailed objects.

NO TE

The path for storing a failed object list is in the oms/sync_task/Synchronization taskID/failed_object_lists/Sub-task name/Timestamp.failed_list.txt.

For details about the error codes in a failed object list, see Viewing the Listof Failed Objects.

----End

2.13 What Is the Storage Class of Archive DataMigrated to the Archive Destination Bucket?

Migrated data has the same storage class as the destination bucket. For example,if the storage class of the destination bucket is Archive, the storage class of thedata migrated to the destination bucket is Archive.

You can change the storage class of the destination bucket as needed.

For details about how to migrate archive data, see Migrating Archive Data.

2.14 What Are the Application Scenarios of MigrationTasks and Migration Task Groups?

Migration TaskIf there is less than 3 TB of data or less than 5 million objects in a source bucket,you can create a migration task to quickly migrate the source data. For details, seeCreating a Migration Task.

Migration Task GroupIf there is more than 3 TB of data or more than 5 million objects to migrate in asource bucket, you can create a migration task group to migrate the source dataquickly. The system will intelligently group the source objects into multiple sub-tasks for concurrent migration. These sub-tasks are managed in groups, including






https://support.huaweicloud.com/intl/en-us/usermanual-oms/topic_0000001089828284.html

progress query, resumption, and restart. For details, see Creating a MigrationTask Group

NO TE

Compared with a migration task, a migration task group provides a faster and more reliablemigration, to help you easily identify failed objects and quickly migrate them again.

2.15 If I Delete an Object from the Source Object, Willthe Corresponding Object in the Destination Bucket BeDeleted?

No.

You can use a script to list objects in the destination bucket to check whether thedeleted object exists in the destination bucket. If it does, delete it manually.

2.16 How Does OMS Ensure Data Consistency Betweenthe Source and Destination Buckets?

During the migration, whenever an object is migrated, OMS performs aconsistency verification on the object. If a source object is inconsistent with itscorresponding destination one, the object is recorded as failed. OMS records allobjects that fail to be migrated in the task. You can view the statistics in the taskdetails.

If a source object does not meet any of the following criteria, the object fails theverification and is recorded as failed:

● The source object has the same encryption status as the destination one.● The source object has the same size as the destination one.● The last modification time of the source object is earlier than or equal to that

of the destination one.

2.17 How Does Migration Affect Data in theDestination Bucket?

The impacts of migration on existing data in destination buckets depend onwhether objects with the same name exist in the source and destination buckets.





● If no object with the same name exists, the migration has no impacts on theexisting data in the destination bucket.Data in the destination bucket is not modified during the migration.Data in the destination bucket is not deleted after the migration.

● If an object with the same name exists, the existing data in the destinationbucket will be overwritten in the following scenarios:– The encryption statuses of source and destination objects are different.– The sizes of the source and destination objects are different.– The last modification time of the destination object is earlier than that of

the source object.



3 Migration Constraints

3.1 What Are the Constraints on Migrating ObjectStorage Data to a Parallel File System with OMS?

The constraints are as follows:

1. If the object name ends with a slash (/), the object size must be zero.2. The object name cannot contain consecutive slashes (/), for example, test//

test.

3.2 How Do I Fix the Error "Destination bucket must bein the same region as OMS. Select a bucket that islocated in region xxx, or change the OMS region toxxx" When I Create a Migration Task?

To maximize the migration speed, create a migration task in the region where thedestination OBS bucket is located.

As shown in the following figure, ensure the regions selected areas 1 and 2 are thesame.

Object Storage Migration ServiceFAQs 3 Migration Constraints


3.3 How Do I Migrate Data from a Cloud ServiceProvider Not Supported by OMS?

PrerequisitesThe source objects can be accessed through HTTP or HTTPS, and you can list thenames and download links of all the objects to be migrated.

SolutionWhen you create a migration task or task group, set Data Source to HTTP/HTTPSdata sources.

Figure 3-1 Selecting HTTP/HTTPS data sources

NO TE

For differences between migration tasks and migration task groups, see What Are theApplication Scenarios of Migration Tasks and Migration Task Groups?

For HTTP/HTTPS data migration, URL List is set as the migration method bydefault.





● Creating a Migration Task with a Specified URL List● Creating a Migration Task Group with a Specified URL List



https://support.huaweicloud.com/intl/en-us/usermanual-oms/topic_0000001089828284.html#EN-US_TOPIC_0000001089828284__p039312762816

https://support.huaweicloud.com/intl/en-us/usermanual-oms/oms_01_0017.html#oms_01_0017__p039312762816

4 Migration Duration

4.1 How Can I Learn the Progress of a Migration Task?Generally, OMS can migrate 10 TB to 20 TB data per day. However, the speeddepends on the number and size of source objects and the network transmissiondistance between the source and destination buckets. You are advised to create amigration task to test the migration speed. The maximum migration speed is fivetimes the average speed of a single task because up to five tasks can be executedconcurrently in a region.

● You can view the real-time migration speed of a single migration task on theMigration Task page.

● You can view the real-time migration speed of a task group on the MigrationTask Group page.

● If SMN is enabled when a migration task is created, the migration task resultswill be sent to you by email, SMS message, or customized URL.

NO TE

The migration speeds displayed on the OMS console are estimates only.

4.2 Why Is the Migration Speed Slow Sometimes?The migration speed is influenced by many factors, such as the sizes of the sourceobjects, number of objects being migrated, and network transmission speed. If themigration network is normal, the cases may be as follows:

● There are a large number of small objects.

A fixed maximum number of objects can be concurrently migrated in amigration task. Before migration, OMS lists the objects in the source bucket.After an object is migrated, OMS checks whether the object in the sourcebucket is consistent with that in the destination bucket. Compared withmigration of large objects, it takes more time to list, compare, and migrate alarge number of small objects with the same amount of data as a smallernumber of large objects.

Object Storage Migration ServiceFAQs 4 Migration Duration


In this case, you can create a migration task group to intelligently split thesource objects into multiple subtasks for concurrent migration. This canimprove the migration speed.

● There are a small number of large objects.One thread is used to migrate an object. If there are just a few very largeobjects, for example, less than 50, the migration thread concurrency is low,which slows down the migration.In this case, wait patiently.

NO TE

● Large objects are uploaded in parts. Only the data volume of uploaded parts iscounted into the migrated data volume. Therefore, the migration speed mayfluctuate.

● The migration speed is recalculated every 5s. If it takes more than 5s to upload onepart, the instantaneous speed is zero.

● Archive data restoration is used when archive data is migrated.A fixed maximum number of migration tasks can be executed concurrently. Ifyou do not restore the archive data in advance, and rely on the archive datarestoration function of OMS fully, the migration speed will be slow.In this case, you can restore the archive data in batches on the sourceplatform before the migration. For details, see Migrating Archive Data.

Object Storage Migration ServiceFAQs 4 Migration Duration



5 Exception Recovery

5.1 What Can I Do If Migration Tasks Fail To Be CreatedAfter Evaluation?

The number of Creation failures is displayed in the upper left corner of themigration task list.

You can perform the following steps to recreate migration tasks:

1. Click Retry in the upper left corner of the task list.The Re-create Migration Task dialog box is displayed.

2. Select the task to be recreated, enter the AKs and SKs of the source anddestination platform accounts, and click OK.

5.2 What Can I Do If the Migration Fails Because theFrequency of Accessing the Source Object StorageExceeded the Upper Limit?

To ensure a fast migration, OMS invokes the source object storage API at a highfrequency during the migration task execution. Some source cloud vendors haveset an upper limit on the frequency of accessing their object storage. If the accessfrequency exceeded the upper limit, the migration task fails.

Solution:● Contact the source cloud vendor to change the frequency upper limit to meet

migration requirements.● Create a task using the API and specify a smaller value for thread_num

(number of threads used by the migration task) to reduce the accessfrequency of the migration threads.

5.3 What Can I Do If a Migration Task Fails?The following methods can help you handle the exceptions.

Object Storage Migration ServiceFAQs 5 Exception Recovery


Method 1: Viewing Error Messages on the OMS Console

Locate the row that contains the failed task. Hover the cursor over on the rightof the failed task to view the failure cause. Then perform operations as prompted.

NO TE

If the migration task details contain the list of failed objects, rectify the fault by referring toMethod 2.

Method 2: Viewing the List of Failed Objects

NO TICE

You can only obtain the list of failed objects if you enabled the Record FailedObjects function when creating a task.

Step 1 Check the names of the source objects that fail to be migrated.

Locate the row that contains the failed migration task and click View Details. Onthe displayed page, click bucket name under Failed Object List Storage Path toaccess the OBS console and download the list.

Step 2 View the failure cause of the task.

The failed object list contains information about the failed migration task,including the error codes, occurrence time (local UTC), URL-encoded names of thefailed objects, non-URL-encoded names of the failed objects, and error messagesreturned by source SDKs. For details about how to handle a migration failure, seeTable 1.

Figure 5-1 Failed object list

Table 5-1 Error codes returned for failed objects

Error Code Description Solution

GET_SRC_OBJ_META_FAILURE

Failed to obtain themetadata of the sourceobject.

Check whether thesource object exists andwhether you have theread permission on theobject.For details about how toobtain the requiredpermission, seePermissionsManagement.






GET_DST_OBJ_META_FAILURE

Failed to obtain themetadata of thedestination object.

Check whether you havethe read permission onthe destination object.For details about how toobtain the requiredpermission, seePermissionsManagement.

GET_SRC_OBJ_FAILURE Failed to obtain thesource object.

Check whether thesource object exists andwhether you have thedownload permission onthe object.For details about how toobtain the requiredpermission, seePermissionsManagement.

COMPARE_SRC_AND_CDN_FAILURE

The source objects areinconsistent with theobjects mapped to thecustom domain name.

Refresh the CDN cacheso that the cached objecton CDN is the latest.

UPLOAD_DST_OBJ_FAILURE

Failed to upload theobject to the destinationbucket.

Check whether you havethe write permission onthe destination bucket.

COMPARE_SRC_AND_DST_KMS_FAILURE

The encryption statusesof source and destinationobjects are different.

Check whether theencryption status of thedestination object isconsistent with thesetting of KMSencryption of themigration task.

COMPARE_SRC_AND_DST_SIZE_FAILURE

The sizes of the sourceand destination objectsare consistent.

Migrate the object again.

COMPARE_SRC_AND_DST_LAST_MODIFY_DATE_FAILURE

The last modificationtimes of the source anddestination objects areinconsistent.









COMPARE_AND_GET_SRC_OBJ_META_FAILURE

Failed to obtain themetadata of the sourceobject when comparingthe source anddestination objects afterthe migration.

Check whether thesource object exists andwhether you have theread permission on theobject.For details about how toobtain the requiredpermission, seePermissionsManagement.

COMPARE_AND_GET_DST_OBJ_META_FAILURE

Failed to obtain themetadata of thedestination object whencomparing the sourceand destination objectsafter the migration.

Check whether thedestination object existsand whether you havethe read permission onthe object.For details about how toobtain the requiredpermission, seePermissionsManagement.

INIT_DST_OBJ_MULTIPART_FAILURE

Failed to initialize themultipart upload.

Check whether theobject supports multipartupload.NOTE

Objects whoseContentType is multipart/form-data do not supportmultipart upload. You needto use obsutil to uploadsuch objects separately.

COMPLETE_DST_OBJ_MULTIPART_FAILURE

Failed to combine theparts in the destinationbucket.


RESTORE_SRC_ARCHIVE_OBJ_FAILURE

Failed to restore archivedata.

Restore the archive datamanually and migratethe data. Archive thedata after the migration.NOTE

For details about how torestore archive data, seeWhat Should I Do If MyMigration Task Fails Dueto an Archive Data ReadFailure?












MIGRATE_SRC_ARCHIVE_OBJ_FAILURE

Failed to migrate thedata because there isarchive data that has notbeen restored.

Create a migration taskagain and enable theArchive DataRestoration function. Ifa large amount ofarchive data exists, youcan manually restore thedata and then migrate it.NOTE

For details about how torestore archive data, seeWhat Should I Do If MyMigration Task Fails Dueto an Archive Data ReadFailure?

URL_LIST_FORMAT_ERROR

The format of the URLlist is incorrect.

Check for errors basedon the URL list template,rectify the errors, and tryagain.For details, see URL ListFormat Description.

URL_LIST_DOMAIN_FORBIDDEN

The URL list containsforbidden domainnames.

Change the domainnames and try again. Ifyou have any questions,contact technicalsupport.

URL_LIST_IP_FORBIDDEN The URL list containsforbidden IP addresses.

Change the IP addressesand try again. If youhave any questions,contact technicalsupport.

URL_LIST_OBJECT_KEY_TOO_LONG

URLs in the URL list aretoo long.

Modify the URLs (a URLcan contains a maximumof 65,536 characters)and try again.

URL_LIST_INTERNAL_IP The host IP addresses inthe URL list are internalIP addresses.

Change the IP addressesand try again. If youhave any questions,contact technicalsupport.

URL_LIST_INCORRECT_DOMAIN

The domain names inthe URL list are incorrect,irresolvable, or do notexist.

Correct the domainnames and try again.







https://support.huaweicloud.com/intl/en-us/usermanual-oms/topic_0000001089828284.html#EN-US_TOPIC_0000001089828284__li115661073292

https://support.huaweicloud.com/intl/en-us/usermanual-oms/topic_0000001089828284.html#EN-US_TOPIC_0000001089828284__li115661073292

NO TE

If you have any doubts during the processing, submit a service ticket to obtain technicalsupport.

----End

5.4 What Can I Do If a Migration Task Group Fails to BeExecuted?

If a migration task group fails to be executed, perform the following steps:

1. On the Migration Task Group page, locate the target group and click thenumber listed in the Tasks column. On the displayed Migration Task page,sub-tasks contained in the current migration task group are displayed.Generally, a migration task group contains multiple subtasks. The failurecause of each sub-task may be different. You can view the failure cause ofeach sub-task one by one. Then, identify the failure cause by referring toWhat Can I Do If a Migration Task Fails?

Figure 5-2 Migration failed

2. If the Storage Class of the destination bucket for the migration task group isnot Standard, the migration will fail. In this case, change the Storage Classof the destination bucket to Standard and then create a migration task groupagain.

NO TE

For a migration task groups, the system temporarily stores the list file in the oms/directory of the destination bucket. If the destination bucket storage class is notStandard, the migration node cannot read the list file in the directory. As a result, themigration will fail.

5.5 What Can I Do If SMN Is Not Working Properly?If SMN is not available during the creation of a migration task:● After you click Add to select an SMN message topic, if the topic list displayed

in the Select SMN Topic dialog box is empty, log in to the SMN console tocheck whether there are any topics. If there are no topics, create one.

● After you click Add to select an SMN message topic, if the topic list displayedin the Select SMN Topic dialog box contains topics, but you cannot use aselected topic to create a migration task, log in to the SMN console to checkwhether the selected topic exists. If the topic does not exist, select anothertopic or create a topic.

If you have selected an SMN topic during the creation of a migration task and themigration task was successfully created, but the subscribers of the topic are not



https://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex

receiving notifications, ensure that the selected topic contains the requiredsubscription and that the topic has been confirmed. Otherwise, notificationmessages sent using the SMN service will not be sent to designated subscribers. Asubscriber is a role that is subscribed to a topic. When you add a subscription, youneed to specify where the notification is sent.

The following subscribers are supported:● If it is an email subscription, the subscriber is an email address.● If it is an SMS subscription, the subscriber is a phone number.● If it is an HTTP or HTTPS subscription, the subscriber is a URL.

For details about SMN, see Simple Message Notification.

5.6 What Can I Do If a Migration Task Fails Due toInconsistent Encryption Status?

After an object is uploaded to the OBS bucket, OMS checks the status of theobject. If the object is encrypted, while the KMS encryption function was notenabled for the migration task, the system determines that the migration fails. Inthis case, perform the following steps:

Check that Default Encryption is enabled in the basic configurations of thedestination OBS bucket.

● If Default Encryption is enabled and you also want to encrypt the migratedobjects, create a task with the same source and destination as the failedmigration task, and for step Set Task Parameters, select KMS for DataEncryption in the Advanced Settings area.

● If Default Encryption is enabled but you do not want to encrypt the migratedobjects, disable this function and restart the failed task.

● If Default Encryption is not enabled, submit a service ticket to seekHUAWEI CLOUD technical support.

5.7 What Can I Do If the Migration Task Failed Due toan Archive Data Read Failure?

If there is archive data in the source bucket, you need to restore the data beforemigration. To do so, you can perform the following steps:

Step 1 Use a tool or the source platform management console to restore the archive databefore migration. It is recommended that you configure a long restorationduration.● If the source bucket is not on HUAWEI CLOUD, follow the restoration

methods provided by the source cloud service vendors.● If the source bucket is on HUAWEI CLOUD, use obsutil to restore the archive

data. For details, see Restoring Objects from OBS Archive.

Step 2 On the OMS console, create a migration task for the source bucket again andenable the Archive Data Restoration function. OMS will automatically restoreand migrate archive objects that are not restored in advance.



https://support.huaweicloud.com/intl/en-us/smn/index.html

https://console-intl.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex


CA UTION

● If you do not restore archive objects in advance, and if you rely entirely on theautomatic restoration function of OMS, it takes 1 to 5 minutes to restore eachobject, depending on the source object storage services. This can significantlyslow down the migration.

● For OBS buckets with the direct reading function enabled, the archive data inthese OBS buckets can be directly downloaded without being restored inadvance. But OMS cannot migrate such data.

● Direct reading may be a billable function of your source object storage service.Nevertheless, you will be billed for enabling OMS to call or directly using thedirect reading function of HUAWEI CLOUD OBS. For details, see PricingDetails.

● Before you migrate archive data, you are advised to set the storage class of thedestination bucket to Standard. After the migration is complete, confirm thatthe migrated data is correct, and change the storage class of the destinationbucket to Archive.

----End

5.8 What Are the Common Questions and TheirSolutions in the CDN Scenario?

Question 1

The CDN domain name configuration fails the check before the migration, and theerror code COMPARE_SRC_AND_CDN_FAILURE is returned.

Possible Causes

The object metadata obtained by OMS from the source bucket does not match theHTTP response header from the CDN. To be specific, the object sizes are differentor the last modification time of the source object is later than that of the cachedobject on the CDN.

Solution

1. Check whether the URL (CDN domain name followed by the object name)can be used to download the object from the CDN.

a. If the URL cannot be accessed, check whether the CDN anti-leeching isconfigured.

b. If the URL can be accessed, go to 2.2. Check whether the values of Content-Length and Last-Modified in the HTTP

response header from the CDN are the same as those in the metadata of thesource object.

a. If the values are inconsistent, check whether image compression isconfigured on the CDN. If yes, delete the configuration and perform themigration again.



https://support.huaweicloud.com/intl/en-us/usermanual-obs/obs_03_0143.html

https://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/obs

https://www.huaweicloud.com/intl/en-us/pricing/index.html?tab=detail#/obs

Figure 5-3 Metadata

Figure 5-4 Size after image compression

b. If the last modification time of the source object is later than the Last-Modified value obtained from CDN, refresh the CDN cache and performthe migration again.

Question 2

The migration fails and the error code COMPARE_SRC_AND_CDN_FAILURE isdisplayed in the failed object list.



To obtain the list of failed objects, click View Details in the row where the failedmigration task is located, and click the link next to Failed Object List StoragePath.

Figure 5-5 Failed Object List Storage Path

Possible Causes

Before the migration, OMS detects that the size and last modification time of theobject in the source bucket are inconsistent with those of the cached object on theCDN.

This is because that the source object is updated, but the CDN cache is notrefreshed.

Solution

Refresh the CDN cache so that the cached object on the CDN is the latest.

Question 3The migration fails and the error code 403 is displayed in the failed object list.

To obtain the list of failed objects, click View Details in the row where the failedmigration task is located, and click the link next to Failed Object List StoragePath.

Figure 5-6 Failed Object List Storage Path



Possible Causes

403 indicates that OMS does not have the permissions to access the CDN domainname. The causes are as follows:

● For publicly accessible CDNs, the anti-leeching is configured.● For non-publicly accessible CDNs, the authentication information is not

correctly configured on the OMS console.

Solution

● Disable the anti-leeching function. If anti-leeching cannot be disabled forsecurity purposes, contact HUAWEI CLOUD technical support.

● Configure the authentification correctly. For details, see Migration from CDN.



https://support.huaweicloud.com/intl/en-us/usermanual-oms/topic_0000001089828284.html#topic_0000001089828284__p12781437144213

6 Change History

Released On Description

2021-11-12 This issue is the nineteenth official release.Updated How Do I Obtain Required Permissions for theSource and Destination Platform Accounts?

2021-10-27 This issue is the eighteenth official release.Updated What Are the Common Questions and TheirSolutions in the CDN Scenario?

2021-09-26 This issue is the seventeenth official release.Added How Does Migration Affect Data in the DestinationBucket?

2021-06-03 This issue is the fifteenth official release.● Updated What Are the Common Questions and Their

Solutions in the CDN Scenario?● Optimized What Can I Do If a Migration Task Fails?

2021-05-21 This issue is the fourteenth official release.● Added If I Delete an Object from the Source Object, Will

the Corresponding Object in the Destination Bucket BeDeleted?

● Added How Do I Migrate Data from a Cloud ServiceProvider Not Supported by OMS?

● Added How Does OMS Ensure Data Consistency Betweenthe Source and Destination Buckets?

2021-03-31 This issue is the thirteenth official release.● Added What Are the Application Scenarios of Migration

Tasks and Migration Task Groups?● Added What Is the Storage Class of Archive Data

Migrated to the Archive Destination Bucket?● Optimized What Can I Do If a Migration Task Fails?

Object Storage Migration ServiceFAQs 6 Change History



2021-03-12 This issue is the twelfth official release.● Added How Do I Review the Objects Migrated in a

Migration Task?● Added How Do I Review the Objects Migrated in a

Migration Task Group?● Added How Do I Review Objects Migrated in a

Synchronization Task?

2021-03-10 This issue is the eleventh official release.Added How Do I Obtain Required Permissions for the Sourceand Destination Platform Accounts?

2021-02-05 This issue is the tenth official release.Optimized What Can I Do If a Migration Task Fails?

2021-01-11 This issue is the ninth official release.Optimized Question Summary.

2020-12-24 This issue is the eighth official release.● Added How Can I Learn the Progress of a Migration Task?● Added Why Is the Migration Speed Slow Sometimes?● Added How Do I Migrate Data from Multiple Source

Buckets to a Single OBS Bucket on HUAWEI CLOUD?● Added How Does Migration Affect My Source Data? What

Are the Impacts on Migration If the Source DataChanges?

● Added How Do I Fix the Error "Destination bucket mustbe in the same region as OMS. Select a bucket that islocated in region xxx, or change the OMS region to xxx"When I Create a Migration Task?

2020-11-28 This issue is the seventh official release.● Added How Do I Migrate the OBS Bucket Data from One

Account to Another?● Added What Can I Do If the Migration Task Failed Due to

an Archive Data Read Failure?● Added What Are the Constraints on Migrating Object

Storage Data to a Parallel File System with OMS?

2020-08-20 This issue is the sixth official release.Added What Can I Do If a Migration Task Group Fails to BeExecuted?

2020-08-13 This issue is the fifth official release.Added What Are the Common Questions and TheirSolutions in the CDN Scenario?




2020-08-06 This issue is the fourth official release.Added How Do I Migrate OBS Bucket Data from One Regionto Another?

2020-06-19 This issue is the third official release.● Added How Does OMS Ensure Data Consistency If Data Is

Continuously Written to OBS?● Added What Can I Do If a Migration Task Fails Due to

Inconsistent Encryption Status?

2019-08-31 This issue is the second official release.● Updated What Can I Do If a Migration Task Fails?● Added What Can I Do If Migration Tasks Fail To Be

Created After Evaluation?, How Do I Migrate Data in theRoot Directory? and How Do I Change the Storage Pathof a Migrated Object?

2019-07-25 This issue is the first official release.



object storage migration service - huawei cloud

Documents