How to setup your AWS S3 lifecycle management

Published on August 21, 2018
Share this on

Amazon’s S3 bucket is undoubtedly one of the best value offerings from AWS for your object storage. Whether you are using it as a file store or for your website, there are different types of S3 stores with varying pricing, features. Versioning capabilities on an S3 when enabled allows for multiple versions of a file, with prior states that over time can build up without a lifecycle management plan in place.

First, let’s look at the types of S3 storage classes. Due to its relatively low cost, many have replaced their file shares to an S3 bucket. Unlike a file share, S3 buckets are replicated across many availability zones (AZs), to withstand failures of an AZ.Price comparison of S3 storage classes

For these cost savings, one needs to factor that something put into S3 infrequent or single zone bucket will stay there for a minimum of 30 days, or incur a 30-day minimum storage charge. If versioning is enabled on an S3 bucket, and old versions can be expired automatically or infrequently used objects can be moved to non-standard S3 buckets or to Glacier automatically with lifecycle management policies. The lifecycle management policies can only be applied to objects that are in S3 for over 30 days and if the size of the file isn’t tiny at under 128 Kb.

The first step is to check if versioning is enabled on an S3 bucket. To do any of these actions first login to AWS console:search for s3 in aws console

And you should be able to see a listing of your specific S3 buckets:

Get listing of S3 buckets

Click on the specific S3 bucket should take you to the Overview tab:S3 overview tab

Then click on the Properties tab to check if versioning is enabled for that bucket:

s3 properties tab

If multiple versions of each file aren’t needed you can change this attribute and reduce your future storage size from here.

To set up a lifecycle policy go to the Management tab, then click on ‘Add lifecycle rule’:

add s3 lifecycle rule

Click on Lifecycle rule should bring up a dialogue box like below, give a friendly name for this rule. Skip the filter to prefix/tags unless your S3 bucket uses what is called object tags.

s3 lifecycle name and scope

Click on transitions and from here check the current version and previous versions depending on your use of versioning.

In the below example, I have set a transition to move my current version object files to Standard-IA after 90 days and after a year of creation move it to Glacier. For the prior version if retention of all versions is necessary, in the below example they are retained in One-Zone IA after 30 days and moved to Glacier after 90 days.

s3 rule transition setup

Many time prior versions can be deleted permanently. You can do so automatically from a lifecycle management setup at 91 days. Another important setting to have in all buckets to be cleanup incomplete multipart uploads after few days.

how to expire s3 objects

A failed multipart upload is common for a myriad of reasons in S3. If you do not have an S3 policy to remove incomplete multipart uploads these failed uploads start adding to your costs.

multi part objects failed s3 cleanup

A failed multipart upload is common for a myriad of reasons in S3. If you do not have an S3 policy to remove incomplete multipart uploads these failed uploads start adding to your costs.

You can read more on S3 lifecycle management here. Watch a video on the steps here:

If you encounter issues in setup of lifecycle management for an S3 bucket or have a question about any of our AWS Tutorials, please reach out to us with a comment and will add this to the FAQs. If you would like to partner with us and develop AWS open source scripts for these steps, reach out to us.

Download the whitepaper that explains why cloud capacity planning needs a different thought process from traditional on-premise infrastructure.

We wont spam you or sell your email address

Learn more