Working with S3 API
Creating a User and Configuring Access
To work with S3 in the Cloud storage in the Control panel:
- Create a separate user.
- Assign the created user rights to work with the necessary containers according to the instruction. To provide the user with full access to the required container, select the Read and Write option.
- Save the password by checking the Use these credentials to access via S3 protocol box.
Please note that if the user has not saved their password in the Control panel, the authentication will not work because the password is needed to verify the request signature.
Use the login and password of the created user to work in the configuration file:
- the
<Account number>_<Username>format is used as the Access Key; - the user’s password is used as the Secret Key;
- ru-1a is used as the Region.
Please note that the user that is the main user for the account (if the username coincides with the account number in the Control panel) gets read and write access to any bucket.
A bucket is a logical entity that helps store objects. In Swift terminology, these are containers.
POST Method Upload
You can upload files directly to the Cloud Storage from the browser, bypassing proxy servers. To upload, the user needs to assemble a valid HTML form in the multipart/form-data format. This can be done either manually or using the boto3 python library (or others that allow generating form signatures).
There are the following limits:
- anonymous download is not supported, i.e. all requests must be signed;
- the maximum expiration value in the Policy is limited to one year.
Multipart Upload
There is a special feature when uploading the objects.
Multipart upload allows you to upload a single object as a set of parts.
Please note that when working with S3 with Multipart uploading of the large objects, all uploaded parts are not collected into a single object in the end, but are stored in the nearest container with the same name and _s3multipartuploads suffix.
This API is used to upload large (the size is variable) objects as a set of parts.
Multipart upload is a three-step process.
Initiation
- If the object already exists, it is deleted.
- A container with the name of the current one is created, suffix
_s3multipartuploadsis added to it. Further, the parts of the uploaded object will be stored there. - The uploadId is generated.
- A meta object
/uploads/<bucketName>_s3multipartuploads/<objectName>_<uploadId>/is created in the container with parts. It will store the Content-Type and all user metadata before completing.
Parts Upload
Parts are uploading.
Completing
- A list of all objects is made using the
<bucketName>_s3multipartuploads/<objectName>/<uploadId>/prefix. It is compared with the list that came from the client. - The total size of the object is calculated and hashes are summed up.
- A manifest that will be located in the path of the uploaded multipart object is created.
- The Content-Type and user’s metadata are read from the previously created meta object and written to the manifest. The meta object itself is then deleted.
This method is implemented with a slight restriction — only / can be used as a separator.
Please note that buckets show containers with parts.
Supported Methods
Cloud storage supports basic methods for working with the Amazon S3 HTTP API.
| Method | Command | Action | Access |
|---|---|---|---|
| GET Service | GET / | Getting a list of buckets |
The user can get a list of buckets that they have read and write access to |
| DELETE Bucket | DELETE /<bucketName> |
Deleting a bucket | The user can delete a bucket if they have read and write access |
| GET Bucket (List Objects) Version 1 | GET /<bucketName> |
Getting a list of objects in the bucket |
The user can get a list of objects in the bucket that they have read and write access to |
| HEAD Bucket | HEAD /<bucketName> |
Getting bucket status |
The user can get the bucket status if they have read and write access |
| List Multipart Uploads | GET /<bucketName>?uploads |
Getting a list of uploaded multipart objects |
The user can get a list of uploaded multipart objects in the bucket that they have read and write access to |
| PUT Bucket | PUT /<bucketName> |
Creating a bucket | The user can create a bucket if they have read and write access |
| Delete Multiple Objects | POST /<bucketName>?delete |
Deleting multiple objects in the bucket |
The user can delete multiple objects in the bucket that they have read and write access to |
| DELETE Object | DELETE /<bucketName>/<objectName> |
Deleting an object | The user can delete an object if they have read and write access to the bucket |
| GET Object | GET /<bucketName>/<objectName> |
Retrieving an object | The user can retrieve an object if they have read and write access |
| HEAD Object | HEAD /<bucketName>/<objectName> |
Retrieving object metadata |
The user can retrieve the object metadata if they have read and write access |
| PUT Object | PUT /<bucketName>/<objectName> |
Creating an object | The user can create an object if they have read and write access |
| PUT Object - Copy | PUT /<bucketName>/<objectName> (x-amz-copy-source) |
Copying an object * |
The user can copy an object if they have read and write access |
| Initiate Multipart Upload | POST /<bucketName>/<objectName>?uploads |
Initiating the multipart upload |
The user can initiate the multipart upload if they have read and write access |
| Complete Multipart Upload | POST /<bucketName>/<objectName>?uploadId=<uploadId> |
Completing the multipart upload |
The user can complete the multipart upload if they have read and write access |
| Abort Multipart Upload | DELETE /<bucketName>/<objectName>?uploadId=<uploadId> |
Aborting the multipart upload ** |
The user can abort the multipart upload if they have read and write access |
| List Parts | GET /<bucketName>/<objectName>?uploadId=<uploadId> |
Getting a list of parts of a multipart object |
The user can get a list of parts if they have read and write access |
| Upload Part | PUT /<bucketName>/<objectName>?partNumber=<partNumber>&uploadId=<uploadId> |
Uploading a part in the multipart upload *** |
The user can create a part of the multipart object if they have read and write access |
| Upload Part - Copy | PUT /<bucketName>/<objectName>?partNumber=<part_number>&uploadId=<uploadId> (x-amz-copy-source) |
Copying a part of the multipart object |
The user can copy a part of the multipart object if they have read and write access to the source bucket |
* When an object is copied to itself, its metadata is modified.
** Each uploaded part is deleted. Then the meta object is deleted too.
*** In this method, the _s3multipartuploads suffix is added to the bucket name, and the object path is replaced with <objectName>/<uploadId>/<partNumber>.