Working with S3 API

Creating a User and Configuring Access

To work with S3 in the Cloud storage in the Control panel:

  1. Create a separate user.
  2. Assign the created user rights to work with the necessary containers according to the instruction. To provide the user with full access to the required container, select the Read and Write option.
  3. Save the password by checking the Use these credentials to access via S3 protocol box.

Please note that if the user has not saved their password in the Control panel, the authentication will not work because the password is needed to verify the request signature.

Use the login and password of the created user to work in the configuration file:

  • the <Account number>_<Username> format is used as the Access Key;
  • the user’s password is used as the Secret Key;
  • ru-1a is used as the Region.

Please note that the user that is the main user for the account (if the username coincides with the account number in the Control panel) gets read and write access to any bucket.

A bucket is a logical entity that helps store objects. In Swift terminology, these are containers.

POST Method Upload

You can upload files directly to the Cloud Storage from the browser, bypassing proxy servers. To upload, the user needs to assemble a valid HTML form in the multipart/form-data format. This can be done either manually or using the boto3 python library (or others that allow generating form signatures).

There are the following limits:

  • anonymous download is not supported, i.e. all requests must be signed;
  • the maximum expiration value in the Policy is limited to one year.

Multipart Upload

There is a special feature when uploading the objects.

Multipart upload allows you to upload a single object as a set of parts.

Please note that when working with S3 with Multipart uploading of the large objects, all uploaded parts are not collected into a single object in the end, but are stored in the nearest container with the same name and _s3multipartuploads suffix.

This API is used to upload large (the size is variable) objects as a set of parts.

Multipart upload is a three-step process.

Initiation

  1. If the object already exists, it is deleted.
  2. A container with the name of the current one is created, suffix _s3multipartuploads is added to it. Further, the parts of the uploaded object will be stored there.
  3. The uploadId is generated.
  4. A meta object /uploads/<bucketName>_s3multipartuploads/<objectName>_<uploadId>/ is created in the container with parts. It will store the Content-Type and all user metadata before completing.

Parts Upload

Parts are uploading.

Completing

  1. A list of all objects is made using the <bucketName>_s3multipartuploads/<objectName>/<uploadId>/ prefix. It is compared with the list that came from the client.
  2. The total size of the object is calculated and hashes are summed up.
  3. A manifest that will be located in the path of the uploaded multipart object is created.
  4. The Content-Type and user’s metadata are read from the previously created meta object and written to the manifest. The meta object itself is then deleted.

This method is implemented with a slight restriction — only / can be used as a separator.

Please note that buckets show containers with parts.

Supported Methods

Cloud storage supports basic methods for working with the Amazon S3 HTTP API.

Method Command Action Access
GET Service GET / Getting a list of
buckets
The user can get a list of buckets that they have read and write access to
DELETE Bucket DELETE /<bucketName> Deleting a bucket The user can delete a bucket if they have read and write access
GET Bucket (List Objects) Version 1 GET /<bucketName> Getting a list of
objects in the bucket
The user can get a list of objects in the bucket that they have read and write access to
HEAD Bucket HEAD /<bucketName> Getting bucket
status
The user can get the bucket status if they have read and write access
List Multipart Uploads GET /<bucketName>?uploads Getting a list of
uploaded multipart objects
The user can get a list of uploaded multipart objects in the bucket that they have read and write access to
PUT Bucket PUT /<bucketName> Creating a bucket The user can create a bucket if they have read and write access
Delete Multiple Objects POST /<bucketName>?delete Deleting multiple
objects in the bucket
The user can delete multiple objects in the bucket that they have read and write access to
DELETE Object DELETE /<bucketName>/<objectName> Deleting an object The user can delete an object if they have read and write access to the bucket
GET Object GET /<bucketName>/<objectName> Retrieving an object The user can retrieve an object if they have read and write access
HEAD Object HEAD /<bucketName>/<objectName> Retrieving object
metadata
The user can retrieve the object metadata if they have read and write access
PUT Object PUT /<bucketName>/<objectName> Creating an object The user can create an object if they have read and write access
PUT Object - Copy PUT /<bucketName>/<objectName> (x-amz-copy-source) Copying an object * The user can copy an object if they have read and write access
Initiate Multipart Upload POST /<bucketName>/<objectName>?uploads Initiating the
multipart upload
The user can initiate the multipart upload if they have read and write access
Complete Multipart Upload POST /<bucketName>/<objectName>?uploadId=<uploadId> Completing the
multipart upload
The user can complete the multipart upload if they have read and write access
Abort Multipart Upload DELETE /<bucketName>/<objectName>?uploadId=<uploadId> Aborting the
multipart upload **
The user can abort the multipart upload if they have read and write access
List Parts GET /<bucketName>/<objectName>?uploadId=<uploadId> Getting a list of
parts of a multipart object
The user can get a list of parts if they have read and write access
Upload Part PUT /<bucketName>/<objectName>?partNumber=<partNumber>&uploadId=<uploadId> Uploading a part
in the multipart upload ***
The user can create a part of the multipart object if they have read and write access
Upload Part - Copy PUT /<bucketName>/<objectName>?partNumber=<part_number>&uploadId=<uploadId> (x-amz-copy-source) Copying a part
of the multipart object
The user can copy a part of the multipart object if they have read and write access to the source bucket

* When an object is copied to itself, its metadata is modified.

** Each uploaded part is deleted. Then the meta object is deleted too.

*** In this method, the _s3multipartuploads suffix is added to the bucket name, and the object path is replaced with <objectName>/<uploadId>/<partNumber>.