PutObject(snowball-auto-extract for small files)
Each copy operation has some overhead; therefore, performing many transfers on individual small files has slower overall performance than transferring the same data in larger files. To significantly improve your transfer speed for small files (files less than 1 MB), batch the small files together. Batching files is a manual process. If the batched files are put to S3 server with the x-amz-meta-snowball-auto-extract header, the batches are automatically extracted when data is imported MSDP S3 Server.
Note:
The x-amz-meta-snowball-auto-extract header is not accepted for un-versioned bucket, and all the batched small files share the same version in S3 server.
Run the tar or gzip command to manually batch small files, and then transfer them to S3 interface for MSDP.
For example: tar -czf <archive-file> <small files or directory of small files>
aws --endpoint https://<hostname>:8443 --profile <profile name> s3api [--ca-bundle <CA_BUNDLE_FILE>] put-object --bucket <bucket name> --key <key path> --body <xxx.tgz> --metadata snowball-auto-extract=true
Keep in mind the following when batching small files:
Maximum batch size of 5 GB.
Recommended maximum of 10,000 files per batch.
Supported archive formats are TGZ.
Request Syntax
PUT /bucket/Key HTTP/1.1 Host: msdps3.server:8443 Content-Length: ContentLength Content-MD5: ContentMD5 x-amz-meta-snowball-auto-extract:true Body
Request Parameters
Bucket
Name of the bucket.
Required: Yes
Type: String
Key
Name of the object.
Required: Yes
Type: String
Request Headers
Enable snowball-auto-extract
Required: Yes
Value: true
Response Syntax
HTTP/1.1 200 ETag: ETag x-amz-version-id: VersionId
Request Headers
x-amz-version-id
The version-id of the object PUT in the bucket.
Possible Error Response
Success
HTTP status code 200.
EntityTooLarge
The object size exceeded maximum allowed size.
HTTP status code 400.
AccessDenied
Request was rejected because user authentication failed.
HTTP status code 403.
NoSuchBucket
The specified bucket does not exist.
HTTP status code 404.
InternalError
Request failed because of an internal server error.
HTTP status code 500.