Part 3b : Configuring expiration policies from S3 APIs

Anirudha | Mon, 03/16/2020 - 07:51

import boto3
session = boto3.session.Session()

#Endpoint information
endpoint_url = "http://statistics.scalcia.com"
access_key = "bVSjIccNo738F5VmpaYjA4BTV_J7h_YH"
secret_key = "_rCf_LXRPea1j9xmuCWKlFsWqu7Qa_1n"

#Bucket, Objectname
bucket = "bucket-from-api"

#Connect to your Objects endpoint.
s3c = session.client(aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
service_name="s3")

#Check if bucket exists and create it.
print "Creating bucket : %s"%(bucket)
try:
  s3c.head_bucket(Bucket=bucket)
except Exception as err:
  s3c.create_bucket(Bucket=bucket)

#Setting LifeCycle policy on bucket.
life_cycle_config = {
    "Rules": [
    {
        "Expiration": {"Days": 10},
        "Filter": {"Prefix": "demo"},
        "Status": "Enabled",
        "ID": "ExpiryPolicy-10Days-ObjectPrefix-demo",
        },
    {
        "Expiration": {"Days": 30},
        "Filter": {"Prefix": "object"},
        "Status": "Enabled",
        "ID": "ExpiryPolicy-1Month-ObjectPrefix-object",
        }
    ]
}

print "Configuring LifeCyclePolicy %s on bucket : %s"%(life_cycle_config, bucket)
s3c.put_bucket_lifecycle_configuration(Bucket=bucket,
LifecycleConfiguration=life_cycle_config)

#Get LifeCyclePolicies configured on Bucket
print "Policies on bucket : %s"%(bucket)
print s3c.get_bucket_lifecycle_configuration(Bucket=bucket)

keys = ["demo_object.txt", "object_demo.txt", "employee_tata.txt"]
#File to upload to Objects endpoint
filename="/tmp/demo_object.txt"

for key in keys:
print "Uploading file : %s as objectname : %s"%(filename, key)
with open(filename, "r") as fh:
  s3c.put_object(Bucket=bucket, Key=key, Body=fh)
print "object : %s, info : %s"%(key, s3c.head_object(Bucket=bucket, Key=key))

Find above code on github

In the code above, I am setting multiple expiration rules in one policy on bucket. You may also notice the “ID” field in life_cycle_config , this is the name we are giving to this policy. 

  1. All objects starting with “demo” will be auto-deleted after after 10 days. Name to this policy is set to : ExpiryPolicy-10Days-ObjectPrefix-demo
  2. All the objects starting with “object” will be auto-deleted after one month. This policy name is set to : ExpiryPolicy-1Month-ObjectPrefix-object

You can configure upto 1000 rules in policy.

Output :

Creating bucket : bucket-from-api

Configuring LifeCyclePolicy {'Rules': [{'Filter': {'Prefix': 'demo'}, 'Status': 'Enabled', 'Expiration': {'Days': 10}, 'ID': 'ExpiryPolicy-10Days-ObjectPrefix-demo'}, {'Filter': {'Prefix': 'object'}, 'Status': 'Enabled', 'Expiration': {'Days': 30}, 'ID': 'ExpiryPolicy-1Month-ObjectPrefix-object'}]} on bucket : bucket-from-api

Policies on bucket : bucket-from-api

{u'Rules': [{u'Filter': {u'Prefix': 'demo'}, u'Status': 'Enabled', u'Expiration': {u'Days': 10}, u'ID': 'ExpiryPolicy-10Days-ObjectPrefix-demo'}, {u'Filter': {u'Prefix': 'object'}, u'Status': 'Enabled', u'Expiration': {u'Days': 30}, u'ID': 'ExpiryPolicy-1Month-ObjectPrefix-object'}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Mon, 16 Mar 2020 00:44:09 GMT', 'vary': 'Origin', 'content-length': '423', 'content-type': 'text/plain; charset=utf-8'}}}

Uploading file : /tmp/demo_object.txt as objectname : demo_object.txt

object : demo_object.txt, info : {u'AcceptRanges': 'bytes', u'ContentType': 'binary/octet-stream', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': '15FCA22526B98907', 'HTTPHeaders': {'content-length': '2066', 'accept-ranges': 'bytes', 'md5sum': 'b13ef685cb8a402269378b57933ee1eb', 'vary': 'Origin', 'server': 'NutanixS3', 'last-modified': 'Mon, 16 Mar 2020 00:44:09 GMT', 'etag': '"b13ef685cb8a402269378b57933ee1eb"', 'x-amz-request-id': '15FCA22526B98907', 'date': 'Mon, 16 Mar 2020 00:44:09 GMT', 'x-amz-expiration': 'expiry-date="Fri, 27 Mar 2020 00:00:00 GMT", rule-id="ExpiryPolicy-10Days-ObjectPrefix-demo"', 'content-type': 'binary/octet-stream'}}, u'LastModified': datetime.datetime(2020, 3, 16, 0, 44, 9, tzinfo=tzutc()), u'ContentLength': 2066, u'ETag': '"b13ef685cb8a402269378b57933ee1eb"', u'Expiration': 'expiry-date="Fri, 27 Mar 2020 00:00:00 GMT", rule-id="ExpiryPolicy-10Days-ObjectPrefix-demo"', u'Metadata': {}}

Uploading file : /tmp/demo_object.txt as objectname : object_demo.txt

object : object_demo.txt, info : {u'AcceptRanges': 'bytes', u'ContentType': 'binary/octet-stream', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': '15FCA22536EC46FF', 'HTTPHeaders': {'content-length': '2066', 'accept-ranges': 'bytes', 'md5sum': 'b13ef685cb8a402269378b57933ee1eb', 'vary': 'Origin', 'server': 'NutanixS3', 'last-modified': 'Mon, 16 Mar 2020 00:44:09 GMT', 'etag': '"b13ef685cb8a402269378b57933ee1eb"', 'x-amz-request-id': '15FCA22536EC46FF', 'date': 'Mon, 16 Mar 2020 00:44:09 GMT', 'x-amz-expiration': 'expiry-date="Thu, 16 Apr 2020 00:00:00 GMT", rule-id="ExpiryPolicy-1Month-ObjectPrefix-object"', 'content-type': 'binary/octet-stream'}}, u'LastModified': datetime.datetime(2020, 3, 16, 0, 44, 9, tzinfo=tzutc()), u'ContentLength': 2066, u'ETag': '"b13ef685cb8a402269378b57933ee1eb"', u'Expiration': 'expiry-date="Thu, 16 Apr 2020 00:00:00 GMT", rule-id="ExpiryPolicy-1Month-ObjectPrefix-object"', u'Metadata': {}}

Uploading file : /tmp/demo_object.txt as objectname : employee_tata.txt

object : employee_tata.txt, info : {u'AcceptRanges': 'bytes', u'ContentType': 'binary/octet-stream', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': '15FCA22546C8532B', 'HTTPHeaders': {'content-length': '2066', 'accept-ranges': 'bytes', 'md5sum': 'b13ef685cb8a402269378b57933ee1eb', 'vary': 'Origin', 'server': 'NutanixS3', 'last-modified': 'Mon, 16 Mar 2020 00:44:09 GMT', 'etag': '"b13ef685cb8a402269378b57933ee1eb"', 'x-amz-request-id': '15FCA22546C8532B', 'date': 'Mon, 16 Mar 2020 00:44:09 GMT', 'content-type': 'binary/octet-stream'}}, u'LastModified': datetime.datetime(2020, 3, 16, 0, 44, 9, tzinfo=tzutc()), u'ContentLength': 2066, u'ETag': '"b13ef685cb8a402269378b57933ee1eb"', u'Metadata': {}}

 

Here you can see: 

  • demo_object.txt” has returned expiration date as : 'Expiration': 'expiry-date="Fri, 27 Mar 2020 00:00:00 GMT", rule-id="ExpiryPolicy-10Days-ObjectPrefix-demo"' . This object will be deleted on 27th March 00:00:00 GMT time.
  • object_demo.txt” has returned expiration details : 'Expiration': 'expiry-date="Thu, 16 Apr 2020 00:00:00 GMT", rule-id="ExpiryPolicy-1Month-ObjectPrefix-object"'. This object will be deleted on 16th April 00:00:00 GMT time.
  • “employee_tata.txt” does not have any expiration set.

 

How to calculate exact expiration date on object :

Expiration policy time starts from midnight on the next day.

E.g if policy is set for 10 days. And you apply policy today at 6pm then convert the time in GMT (for the sake of example : consider GMT time is 1A.M). Now here, GMT time is 1 A.M which means the expiration clock will start after 23 hrs and 10days after that is time will be stamped on object.

In above code, when I uploaded demo_object.txt , my local time was 5.45pm , Mar-15th PST , in GMT it is : 00.52AM on March-16th. So expiration clock starts from March-17th 00:00. Since this object is set to get auto-deleted after 10 days, so this gets deleted on March-26th 11:59:59PM i.e March-27th 00:00 object will no longer be accessible.

You can also setup policies based on the object versions. We will cover this topic separately in other blog.

Refer Nutanix Objects Documentation for complete support for setting up different type of policies.