Object Storage#

Object strorage is an integral part of any IT infrastructure. It is designed for storing large amounts of arbitrary data (documents, backups, etc.).

All objects are distributed among containers – buckets.

File Service Protocol (FSP)#

NGN Cloud’s object service has an Amazon S3-compatible API. You can use most of the standard applications that are designed to work with Amazon S3, for example, S3cmd and AWS CLI.

Users are authenticated with a signature specially created for each request.

If object storage is not accessed via the web interface, you must get API access settings. You can do this in the cloud management console. Click on the user login in the upper right corner and select Profile Get API access settings.

A downloaded object contains service address, account name, and a password for signature creation. The downloaded object can be used as an rc file for Bourne Shell shell. In this case, you must specify, in the C2_PROJECT variable, the ID of the project you are currently working on.

This protocol is built on top of the standard HTTP. For most actions, PUT and GET requests are used. In addition to GET requests, there is a HEAD request. A special POST request is used to upload data to a bucket from HTML forms. The data is returned in its original format, while service responses (including error messages) are in XML format.

Objects are addressed by a URL in the /bucket/object format. If an object is not specified in a URL then target object is a bucket. To access special resources additional parameters in a URL are used. For example, /foo/bar?acl is a request to access an ACL of the bar file in the foo bucket.

Request examples:

Request

Description

GET /

Get the list of buckets owned by the user

GET /foo

Get a list of objects in the “foo” bucket

PUT /foo

Create the “foo” bucket

GET /foo/bar

Get an object named “bar” from “foo” bucket

PUT /foo/bar

Upload an object named “bar” to “foo” bucket

GET /foo?acl

Get an access control list for “foo” bucket

PUT /foo?acl

Modify the access control list for “foo” bucket

GET /foo/?website

Get website configuration

PUT /foo?website

Set website configuration (enable website mode)

DELETE /foo?website

Delete website configuration (disable website mode)

If an object with the same name as the uploaded object already exists in the bucket, it will be overwritten and the ACL will be reset. An ACL should be updated each time an object is updated.

There is a special type of a PUT request for copying objects within a bucket. If, when an object is copied, a target object already exists, it will be overwritten and the ACL will be reset.

Create/delete a bucket#

Create a bucket#

To create a bucket:

  1. Go to the Buckets section and click Create.

  2. Enter the bucket name. The bucket name must be unique across the S3 namespace. If the name is not available, then a bucket with that name may have been created by another cloud user.

  3. To enable versioning, select the corresponding checkbox.

  4. To enable bucket monitoring, turn on the relevant switch.

  5. Set the necessary access rights.

  6. Click Create again.

All users of the project where the bucket was created, are automatically granted full control to it, just like the user who created it (i.e. the owner).

Bucket name requirements#

A bucket name must meet the following requirements:

  • be unique;

  • contain from 3 to 63 characters;

  • include lowercase Latin letters, numbers, dots or hyphens;

  • start and end with a letter or digit;

  • not look like an IP address.

  • do not contain the following combinations of dots and hyphens: .., .- and -..

Important

The bucket namespace is shared by all NGN Cloud users.

Delete a bucket#

Before deleting a bucket, make sure it does not contain objects and their versions. To delete a bucket:

  1. Go to the Buckets section.

  2. Select the bucket from the list.

  3. Click Delete.

  4. Confirm the action.

Attention

Be careful. A deleted bucket cannot be recovered.

Once a bucket is deleted, its name becomes available for reuse. If the name is not available, another cloud user may have already created a bucket with that name.

Important

Only the bucket owner can permanently delete it.

Upload/delete an object#

A user who has uploaded an object to a bucket becomes its owner automatically. If an object was uploaded by a project user, then all other users of that project get full rights to it. For object rights for other user categories, see the access rights section.

Upload an object#

To upload an object to a bucket:

  1. Go to the Buckets section.

  2. Locate the bucket where you want to upload the object(s) and go to the bucket page.

  3. Click Download.

  4. In the dialog window, add files for uploading or drag-and-drop them into the uploading area. Object names must be unique across the bucket.

  5. Set access rights for the objects you upload. By default, access is permitted for the owner only.

  6. Click Upload again.

Object name requirements#

When uploading an object, you must specify a name that uniquely identifies the object in the bucket.

Note

An object name must meet the following requirements:

  • contain up to 1,024 characters (if 8-bit encoding is used);

  • contain any UTF-8 characters (certain characters in names may cause problems with some applications and protocols);

  • must not end with the / character. Otherwise, instead of an object, a folder with the specified name will be created.

Delete an object#

To delete one or more objects in the web interface:

  1. Select objects in the bucket.

  2. Click Delete.

  3. Confirm the action.

When deleting objects from the bucket with versioning enabled, specify the object version(s) to be deleted. To select object versions for deletion, enable version display.

Important

Only the object or bucket owner can permanently delete an object version.

Access rights#

The service offers the ability to control user permissions for both buckets and objects. All users of the project where the bucket was created get full rights to work with the bucket and the objects that were uploaded into it.

Access permissions for users of other categories must be explicitly set in a special access control list (Access Control List, ACL) associated with the bucket or object.

Types of access and user categories#

There are four permission types:

  • Read;

  • Write;

  • Read ACL;

  • Write ACL.

With regard to buckets and objects, these permissions are interpreted as follows:

Access

Interpretation for buckets

Interpretation for object

Read

Ability to get a list of objects

Ability to get an object content

Write

Ability to change bucket contents (add or remove objects)

Ability to change an existing object

ACL read

Ability to get an object ACL

Ability to get an object ACL

ACL write

Ability to change a bucket ACL

Ability to change an object ACL

Additional rules:

  • The service API allows you to retrieve a list of only those buckets, which are owned by the user making the request.

  • A bucket may only be deleted by its owner.

  • A bucket owner cannot delete an object he/she does not own.

Important

Please be careful when making changes to access control lists (ACL):

  • The Read ACL permission allows the user to know other users’ access rights;

  • The Write ACL permission allows the user to change his/her own access rules as well as access rules for other users.

In addition to users of the project to which the bucket belongs, permissions can be granted to special groups:

  • Anonymous users – all users;

  • Authenticated users – NGN Cloud users who have successfully passed the authentication procedure;

  • Projects are specific projects in the project@customer format, where project is the project ID in NGN Cloud, customer is the Company ID you specified when registrating in NGN Cloud.

Important

Grant anonymous access to a bucket and objects only when required, for example, in case of static website hosting. You can grant access to all authenticated cloud users or specific projects by specifying them in the Add rule for project field.

Managing access in the web interface#

ACL for a bucket#

To assign access permissions:

  1. Switch to the Buckets section, select the desired bucket, and go to the bucket page.

  2. Open the Access rules tab.

  3. Tick the required permissions for authenticated and/or anonymous users.

  4. To set access rights for a project, click Add rule for project, specify the project name in the project@customer format, and select the required rights.

  5. Once you have set all the required permissions, click Apply.

ACL for an object#

Access rights to an object are assigned like those to a bucket. To grant access rights to a specific object:

  1. Select the object in the Objects tab on the bucket page and go to the object page.

  2. On the object page, open the Access Rules tab.

  3. Tick the required permissions for authenticated and/or anonymous users.

  4. To set access rights for a project, click Add rule for project, specify the project name in the project@customer format, and select the required rights.

  5. Once you have set all the required permissions, click Apply.

In addition, you can grant read-only access to objects for anonymous users with the Allow access to objects button.

Managing access using AWS CLI#

You can use AWS CLI to flexibly control access to objects and buckets via the S3 Project email and S3 User ID service parameters.

S3 Project email can be used when specifying ACL through API. To grant access to an object in the proj1 project for proj2 project users, specify the S3 Project email of the proj2 project in the enabling rule.

aws --profile ngn --endpoint-url https://s3.cloud.ngn.com.tr s3api put-object-acl --bucket bucket1 --key object1 --grant-read emailaddress=proj2@example.com

S3 User ID is used when writing a Policy. You can grant a particular project access to a portion of a bucket. For example, to grant the proj1 project access to a bucket, specify the S3 User ID of the proj2 project in the enabling rule.

aws --profile ngn --endpoint-url https://s3.cloud.ngn.com.tr s3api put-bucket-policy --bucket bucket1 --policy file://policy.json

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": [
       "s3:ListBucket"
     ],
     "Effect": "Allow",
     "Resource": "arn:aws:s3:::bucket1",
     "Principal": {
       "AWS": [
         "arn:aws:iam:::user/09363407-8f1d-4d31-b805-7e2e8adfe915"
       ]
     }
   }
 ]
}

where arn:aws:s3:::bucket1 is a bucket name, 09363407-8f1d-4d31-b805-7e2e8adfe915 is S3 User ID.

Detailed information about projects, S3 Project email, S3 User ID can be found on the profile page. To do this, go to the cloud management console, click on the user login name in the upper right corner and select Profile.

Object Operations#

To rename, download, provide public access to and perform other operations with objects, open the relevant bucket and go to the Objects tab.

Emulate folder structure#

NGN Cloud object storage model has a flat structure. A directory hierarchy is absent but can be emulated since object names contain the / character.

Note

For the ease of navigation, all objects with the prefix dir_1/…/dir_m/object_n are displayed in the web interface as objects with the names object_n within subfolders dir_1/…/dir_m/.

To reproduce the folder structure in a bucket, you can also use a special button in the web interface. To do this:

  1. Go to the bucket page.

  2. In the Objects tab, click Create folder.

  3. In the dialog window, enter the folder name.

  4. Confirm the action.

You can create subfolders if necessary.

Note

When uploading a file <file_name> to a folder /dir1/…/, the full object name will look like /dir1/…/<file_name>

Rename an object#

To rename an object:

  1. Select the object from the list. If necessary, use the bucket search.

  2. Click Rename.

  3. In the dialog window, specify a new name. The new name must be unique across the bucket.

  4. Confirm the action.

Note

10+ GiB objects in a bucket can only be renamed using utilities (for example, AWS CLI or S3cmd).

Download objects#

To download objects from the object storage to your local computer:

  1. Select desired objects in a bucket.

  2. Click Download.

To download a particular object, you can also click on the icon next to its name.

Share a download link#

You can share the link for object download with other users. To do this:

  1. Select the object.

  2. Click Get link.

The object link will be copied to the clipboard. The link is valid for 24 hours and can be used, regardless of the object-related rights.

To get the link for object download, you can also click on the icon next to its name.

Allow access to objects#

You can provide other users, services, and sites with access to objects in a bucket so that they can view/download the objects. To do this:

  1. Select the object or objects which you want to grant access to.

  2. Click Open public access.

  3. In the dialog window, confirm the action.

When you select a folder, public access will be granted to all objects in the folder, including subfolders.

As a result, all users will get read-only access to the selected objects (for details, see about access rights), and the object access status will change to Accessible to everyone. A particular object is accessible at https://s3.cloud.ngn.com.tr/bucket-name/object-name, where bucket-name is the name of the bucket and object-name is the full name of the object, including prefixes out of folder names separated by /.

Creating a snapshot#

You can create volume snapshots from pre-uploaded VMDK, VHD, Qcow2 or RAW files. To create a volume snapshot:

  1. Select an object in a bucket.

  2. Click Create snapshot.

  3. In the dialog window, set the required parameters.

  4. Confirm the action.

For details, see the import procedure for a volume snapshot.

Create an instance image#

You can create instance images from pre-uploaded VMDK, VHD, Qcow2 or RAW files. To create an image:

  1. Select desired object(s).

  2. Click Create image.

  3. In the dialog window, set the required parameters.

  4. Confirm the action.

For details, see the import procedure for an instance image.

Lifecycle management#

Object lifecycle management allows you to automatically delete objects and their versions from a bucket after their retention period has expired.

Add a lifecycle rule#

To set a lifecycle rule for objects in a bucket:

  1. Go to the Object Storage section Buckets.

  2. Find the desired bucket in the resource table and click on its name.

  3. Open the Lifecycle tab and click Add.

  4. In the dialog window, set the rule name and select its scope and actions:

    • Set the rule name in the Name field.

    • If the rule should be applied to some objects only, select the Filtered objects option and specify the common prefix for these objects in the Prefix field.

    • If the rule should be applied to all objects in the bucket, select the Entire bucket option.

    • To set the lifetime for the current version of an object, select the corresponding option and specify the number of days. The count starts from the date of object creation.

      Note

      If versioning is disabled, the option sets the object retention period. Otherwise, it sets the lifetime of the current version the object. As soon as the lifetime expires, the deletion marker is added and the current version becomes non-current.

    • If versioning is enabled in a bucket, you can set the lifetime for non-current versions. To do this, select the Delete non-current versions after option and set the number of days. The count starts from the moment the version ceases to be the latest one.

  5. Click Add to create and enable the rule.

Modify a lifecycle rule#

Attention

If a lifecycle rule was created using API, it may contain additional parameters. Such a rule can only be modified via API.

To modify a lifecycle rule for objects in a bucket:

  1. Go to the Object Storage section Buckets.

  2. In the resource table, find the bucket where you want to modify the rule for objects and click on its name.

  3. Open the Lifecycle tab, select the rule in the table and click Modify.

  4. In the dialog window, change the rule scope and actions, if necessary (for details, see rule creation dialog).

  5. Click Modify to apply the changes.

Enable/disable the rule#

If you need to suspend a lifecycle rule, you can disable it and then enable again.

  1. Go to the Object Storage section Buckets.

  2. In the resource table, find the bucket where the rule should be enabled/disabled and click on its name.

  3. Open the Lifecycle tab, select the rule in the table and click Disable to disable the enabled rule (or Enable to enable the disabled rule).

Delete a rule#

To delete the lifecycle rule:

  1. Go to the Object Storage section Buckets.

  2. In the resource table, find the bucket for which the lifecycle rule you want to delete is set and click on its name.

  3. Open the Lifecycle tab, select the rule in the table and click Delete.

  4. In the dialog window, confirm the deletion.

Metadata#

Each object can have additional metadata attributes. These attributes represent a subset of standard HTTP protocol headers and allow you to fine-tune object processing by HTTP clients. You can create user attributes besides standard ones. User attributes are available as HTTP headers with x-amz-meta- prefix.

Note

Attribute names are not case sensitive. Cache-Control and cache-control correspond to the same attribute.

NGN Cloud uses the following standard attributes:

Attribute

Description

Example

Content-Type

Document MIME type. This parameter is required to correctly display document in a browser. If you specify text/plain for an HTML page, the page will be displayed as a plain unformatted text with all tags. It is recommended to specify character encoding for documents (UTF-8, CP1251, KOI8-R, etc.)

text/html; charset=utf-8

Content-Encoding

Additional document encoding. The most frequent use is document compression by some algorithm to save disk space.

gzip

Content-Disposition

Allows you to call the Save file as dialog and specify desired object name.

attachment; filename=foo.bar

Cache-Control

This attribute transfers directives for all caching mechanisms included in request-reply chain. For example, “no-cache” means that caching is not used at all.

no-cache

Expires

Another header related to caching. It allows you to specify the time, after which a saved document is considered obsolete

Thu, 01 Dec 1994 16:00:00 GMT

Content-MD5

The document checksum in Base64 format

tdQ2hlY2sgSW50ZWdyaXR5IQ==

Managing metadata#

To set, change, or delete user metadata of an object via the web interface, open the Metadata tab on the object page.

To add metadata:

  1. Click Add.

  2. In the dialog window, set the key (attribute) and value. Specifying the prefix x-amz-meta- is not necessary, it will be added automatically.

  3. Confirm the action.

To change metadata:

  1. Select metadata from the list.

  2. Click Change.

  3. In the dialog window, set a new key (attribute) and value. Specifying the prefix x-amz-meta- is not necessary. It will be added automatically

  4. Confirm the action.

To delete metadata:

  1. Select metadata from the list.

  2. Click Delete.

  3. Confirm the action in the dialog window.

You can delete multiple sets of metadata (key-value pairs) at once.

Versioning#

Versioning allows for the multiple versions of an object to be stored in a single bucket, thus protecting users against unintentional object overwriting and deletion.

Enable versioning#

By default, versioning is always disabled. You can enable bucket versioning in the Versioning tab. To do this:

  1. Go to the Buckets section.

  2. Select the bucket where you want to enable versioning and go to the bucket page.

  3. Open the Versioning tab.

  4. Click Enable.

Attention

Versioning cannot be disabled. You can only pause it, but the existing file versions will still be available.

When versioning is enabled, NGN Cloud assigns randomly generated unique version IDs to objects, while the current version is not overwritten. For example, when creating a new version of object.png in a bucket that already contains an object with this name, the original object object.png ID 12121 and its contents will remain unchanged, and NGN Cloud will create a new version ID ID 11111 and add this version to the bucket.

../../_images/fs7.png

Show versions#

You can show or hide existing versions of all objects in a bucket using the Show Versions toggle switch. To do this:

  1. Go to the relevant bucket.

  2. In the Objects tab, set the toggle switch as necessary.

Delete versions#

After deleting an object, all versions remain in the bucket and a Delete Marker is added to the deleted object. You can delete an object version forever, using an object version ID.

Important

Only the bucket owner can permanently delete an object version.

../../_images/fs8.png

Bucket metrics monitoring#

Monitoring allows you to track the history of changes in the data size and object count in a bucket. At any time, you can view the current data size and object count in Information tab on the bucket page.

Note

To enjoy monitoring capabilities, a user should have privileges to use this service.

If monitoring is enabled, you can view the graphs of monitored metrics in the Metrics tab on the bucket page. You can select a preset period for which a metric is shown, or specify your own.

In addition, you can configure alarms for the bucket (for them to work, you need to enable bucket metrics monitoring first). List of alarms configured for this bucket is displayed in the Alarms tab.

Enable monitoring#

To enable bucket metrics monitoring:

  1. Go to the Object Storage section Buckets.

  2. Select the desired bucket in the resource table and click on the bucket name to go to its page.

  3. Open the Information tab.

  4. Turn on Monitoring.

Create alarm#

Alarms help you to monitor critical changes in bucket metrics.

Note

Alarm can also be created in the Alarms section.

  1. Go to the Object Storage section Buckets.

  2. Select the desired bucket in the resource table and click on the bucket name to go to its page.

  3. Open the Alarms tab and click Create.

  4. In the dialog window, select the metrics to be monitored:

    • BucketSizeBytes — Total data size in the bucket.

    • NumberOfObjects — Total number of objects in the bucket.

    Click Next.

  5. Set alarm metrics (for details, see Alarms section):

    • The alarm name and, optionally, its description

    • Statistics

    • The condition of triggering an alarm for the selected metric. It includes a comparison operator and a threshold value. If necessary, you can also change the metric selected in the previous step.

    • The number and duration of time periods, over which metric values are collected.

    If you want to set actions for the alarm later, complete its creation by clicking Create.

  6. Optionally, you can also configure the actions to be executed when the alarm state changes. To do this, go to the next step by clicking Configure actions. After the actions are configured (for details, see Alarms section), click Create.

Change the alarm#

To change the alarm:

  1. Go to the Object Storage section Buckets.

  2. In the table, select the bucket for which the alarm is set and click on the bucket name to go to its page.

  3. Open the Alarms tab, select the desired alarm in the table and click Modify.

  4. The dialog window will open at the Parameters step. Modify the required alarm parameters:

    • Statistics

    • The condition of triggering an alarm for the selected metric. It includes a comparison operator and a threshold value.

    • Monitored metrics.

    • The number and duration of time periods, over which metric values are collected.

    If you do not need to modify actions as per the alarm, save changes by clicking Modify. Otherwise, click Configure actions.

  5. At the Actions step, you can change or cancel already assigned actions when alarm is triggered or add more actions. Click Modify to save the changes.

Delete the alarm#

  1. Go to the Object Storage section Buckets.

  2. In the table, select the bucket wherein the alarm(s) should be deleted and click on the bucket name to go to its page.

  3. Open the Alarms tab and select the alarm to be deleted in the alarm table. You can select multiple alarms at once.

  4. Click Delete and confirm the action in the dialog window.

How to create a static website#

An object storage can readily be used as a website with a static content. To do this, turn the bucket into the website mode:

  1. Go to the Buckets section.

  2. Create an or select a bucket to store the website content, and go to the bucket page.

  3. Open the Web Site tab.

  4. Select the Turn on website mode checkbox.

  5. Specify an index page.

  6. Optionally, specify an error page to be displayed when the requested document is missing and in case of other errors.

  7. Click Apply to enable the website mode.

You can also enable the website mode and set these pages through an external API.

Note

An anonymous access to the bucket and all objects in it must be allowed.

Save changes and test website performance.

When accessing http://bucket-name.website.cloud.ngn.com.tr, an index page http://bucket-name.website.cloud.ngn.com.tr/index.html is displayed, where bucket-name is a bucket with the website mode enabled. If you try to access a non-existing object in the bucket, an error message will appear. If an error page was not set or if the value set in the configuration refers to a non-existing object, then a standard system message appears, which is the same for all users.

If an object name is not specified when accessing a folder, an index document stored in this folder will be displayed. For example, when accessing http://bucket-name.website.cloud.ngn.com.tr/folder a document, which is also accessible by the name http://bucket-name.s3.cloud.ngn.com.tr/folder/index.html, will appear.

See instrutions how to setup a custom domain for a website и HTTPS for a website. A bucket in the website mode can be setup to redirect all or some incoming requests to other buckets or external resources.

Cross-Origin Resource Sharing (CORS)#

Object storage supports Cross-Origin Resource Sharing (CORS) requests to objects in a bucket, so the bucket can be configured in such a way as to give the browser access to the objects.

To set up CORS rules for a bucket:

  1. Go to the bucket page.

  2. In the CORS tab, click Add and set the required parameters:

    • Origin — a list of sources (domains), which are allowed to make cross-domain requests to the bucket. The domain must start with a protocol ID (for example, http) and end with a hostname, or a hostname and a port. Origin of the https://*.example.com format allows requests from any sources in the domain. To allow access from any domain, specify *.

    • Allowed Methods — HTTP methods, which have access to objects. You can allow or prohibit access to objects in a bucket for the GET, PUT, DELETE, POST, HEAD methods.

    • Allowed Headers — a list of allowed request headers. To allow any header, specify *.

    • Expose Headers — headers that can be accessed by a JavaScript application in a browser.

    • Access Control Max Age — an allowable response caching period in seconds. This parameter improves performance since requests coming from the same source are not checked unless necessary.

  3. Click Add.

Note

The Allowed Methods, Allowed Headers, Expose Headers, and Access Control Max Age settings are applied only to requests from the specified sources.

You can change or delete CORS configuration at any time.

To manage CORS rules, you can also use API methods.