Using AWS Snowball Edge for large backup restore
Similar to a Snowball backup use case to speed up the backup data movement from datacenter to AWS cloud with limited network bandwidth from customer datacenter to AWS cloud. AWS Snowball Edge can also be used to speed up the restore process by storing AWS backups to AWS Snowball Edge device, shipping it customer datacenter and restore.
Depending on the amount of data that you want to restore from a bucket, it might be worth the time it takes to duplicate the desired images into another bucket. Then export from AWS Snowball Edge from there. This export can be achieved through the use of an image sharing server in the cloud.
The following are two ways to restore AWS backups in a bucket to snowball:
Restoring the entire bucket that is used for the backup to Snowball.
Duplicate the needed backups to a new backup and restore the new bucket to Snowball.
The first option needs to restore the whole backup, which may contain many backups and the amount of data can be huge. The second option allows one to move only the backups that are needed.
During an AWS Snowball Edge export job, data in the bucket being exported is read-only. This limitation is an AWS limitation to prevent race conditions with the data. During data transit, no backups can be made to the bucket.
Depending on network speed, duplication of 1 TB of data from one S3 bucket to another using an image sharing server in an EC2 instance can take time. So for a typical Snowball workload of many TB of data, a duplication can take many hours or up to a few days. The alternative to image sharing is to export directly from the source bucket and not being able to access its data during device transit. Device transit generally takes a few days. The benefits and drawbacks of these two solutions should be weighed for your particular export needs.
To use image sharing to export data by AWS Snowball Edge
- Create an EC2 instance within the same region as both the source and the destination buckets reside. Network performance is important for this workflow. Ensure an S3 endpoint is configured in the VPC the VM is in, as this speeds up network speed between EC2 and S3.
- Install NetBackup on the EC2 instance.
- Configure an MSDP storage server for image sharing.
- Use the web UI to configure a disk pool, disk volume, and storage unit that points to the source bucket.
The volume should have the same name as the original volume the data was created in.
- Import the images.
- Use the CLI to configure a disk pool, disk volume, and storage unit that points to the (empty) destination bucket.
Create a cloud instance alias, run:
/usr/openv/netbackup/bin/admincmd/csconfig cldinstance -as -in amazon.com -sts <storage server> -lsu_name <lsu name>
Create a configuration file, then run the nbdevconfig command.
Configuration file content for adding a new cloud LSU:
V7.5 "operation" "add-lsu-cloud" string V7.5 "lsuName" "<lsu name used in last step> string V7.5 "cmsCredName" "<cms_cred_name>" string V7.5 "lsuCloudBucketName" <destination_bucket_name>" string V7.5 "lsuCloudBucketSubName" "<volume_name_in_bucket>" string
/usr/openv/netbackup/bin/admincmd/nbdevconfig -setconfig -storage_server <storage server> -stype PureDisk -configlist <configuration file path>
Create disk pool by using the nbdevconfig command, run:
/usr/openv/netbackup/bin/admincmd/nbdevconfig -previewdv -storage_servers <storage server name> -stype PureDisk | grep <LSU name> > /tmp/dvlist
#/usr/openv/netbackup/bin/admincmd/nbdevconfig -createdp -dp <disk pool name> -stype PureDisk -dvlist /tmp/dvlist -storage_server <storage server name>
Create storage unit by using bpstuadd command, run:
/usr/openv/netbackup/bin/admincmd/bpstuadd -label <storage unit name> -odo 0 -dt 6 -dp <disk pool name> -nodevhost
Currently, the web UI prevents you from creating additional disk pools on image sharing servers.
- Duplicate the desired images into the destination storage.
- Initiate an export job from the destination bucket with Snowball.
- When the device arrives on-premises, create an image sharing server that points to the AWS Snowball Edge device to perform the desired restores.