Nutanix Objects Tiering to Public Cloud

Anirudha | Tue, 02/02/2021 - 12:33

Nutanix Objects has added tons of features in the last few months. One of the frequently asked features was around moving data to the public cloud without losing the local access to data. While Objects is highly scalable and performant S3 offering, but often you want to move the infrequently accessed data to even colder tier or move it to public cloud such as Amazon AWS S3, to save the cost or getting best out of both private and public cloud.

In this blog we will take a look at :

  1. Objects Tiering
  2. Register Tiering Endpoint
    1. Steps to add and Pre-requisites
  3. Configuring Tiering Policy on Bucket
  4. Deciding tiering policy
  5. Understanding I/O request when tiering enabled.
  6. How does tiering works internally

Objects Tiering :

With Objects 3.1 release, you can now move infrequently accessed data to Tiering Endpoint which can be any S3 compatible public cloud or on-prem solution. This enables customers to realize their hybrid cloud infrastructure getting the best of performance and security without any compromises. It also helps to lower the cost of long-term storage and archival by taking advantage of public cloud infrastructure. This new feature works without requiring any change at application level, your application continues to access the data from Objects endpoint and based on the policies configured on the bucket, Objects will either serve the I/O locally or  serve it from the tiering endpoint.

 

         

Multiple clients connected to Objects cluster and various workload running on it. And Objects is configured to tier to 3 different S3 compatible storage endpoints. Objects not only supports multiple endpoint from different vendors but you can also configure various policies on different buckets based on your requirement.

 

Register Tiering Endpoint :
  • Navigate to Objects UI in Prism Central Server. And Open the Objects Instance Page. On the left panel, click on Tiering Endpoint .

         

  • Tiering Endpoint UI will give you an option to add endpoints. If you already have any endpoint registered, you can see them here.

         

  • Input destination S3 endpoint , bucket name and credentials to access the endpoint. In my case I have created an “objects-tiering-demo” bucket on Amazon AWS S3.

         

Things to Consider :
  • Make sure the user that's being configured in the above window, has all required permissions on the bucket.

    User should have below access on bucket :

    • Head-bucket
    • Head-object
    • Put-object
    • Get-object
    • List-object
    • Delete Object.
  • Make sure you use https endpoint.
  • S3 Endpoint is configured with valid CA certs.
  • All the applicable objects will be tiered to S3 endpoint after policy is configured on the bucket. i.e if you have any existing object then it will not be tiered.
  • Do not configure object lock or lifecycle policy or enable versioning on the remote bucket, this will interfere tiering workflow.
  • Make sure the endpoint is reachable from Objects network.

Please refer Objects user guide to make sure you configure all the required access on the bucket , and take care of all pre-requisite.

 

Configure tiering policy on bucket :

  • Choose the bucket in Objects UI and click on the bucket. On the left panel, click on Lifecycle

          

  • Click on “Create Rule” -> Enter the unique Name and select the scope. By default policy will be applicable on all the objects. 

          

  • You can configure policy on objects with specific prefix or/and tag.In my case I have selected both Tags/Prefix . These both fields are optional and you can choose both or either of them or none based on your requirement.

         

  • Select tiering Endpoint from the Endpoint drop down menu. And enter the time (in days/months/years).

         

  • Confirm the details and click on Done.

         

  • You can view all the details in LifeCycle window :

         

  • Based on the different prefix and object tags, you can configure different tiering policies on the bucket.
  • Along with the tiering you can also configure expiration policies on the bucket.

        

In the above example, I have configured all the objects to tier from demo-bucket after 30 days. And all the objects will be auto deleted after 60 days. If you have any multipart upload which is not finalized for more than 60days then it will get automatically aborted.

You can configure this policy on all objects or on specific object prefixes.

  • You can see the tiering stats in the Objects UI by entering into individual bucket :

          

  • You can also see the cumulative stats in the Tiering Endpoint page :

         

That's it, as simple as it gets.


        Next Read - Understanding Tiering internals.