Nutanix Objects stack is Architected and build by the team who built AOS stack. So it is given that Objects data IO path is highly optimized to take advantage of Nutanix AOS cluster.
- Nutanix Objects is built to support billions of objects. Which also means that, while its data footprint can be in petabytes, its metadata footprint also can be huge.
- If we start writing data and metadata to underlying AOS cluster (Hybrid cluster with two tiers HDD & SSD). Then AOS doesn't differentiate between your Objects data and metadata I/Os, for AOS it's just regular data.
- In this case we will end up in down migrating lot of metadata to the HDD tier as ILM kicks in, even though in most of the cases Objects metadata will be a small part of Objects data, but it still lands as regular IO on AOS .
- The moment Objects metadata is down migrated to HDD tier, you will start seeing the higher latencies in all metadata related operations, such as head on the bucket or object will take longer time Or any list operation will see higher latency, and there will be chances of client hitting timeout before we serve all the data.
To solve this :
Objects stack is highly optimized for its metadata and data I/O path. It doesn’t demand any specific configuration, (eg - keep SSD tier higher than regular or use all flash platform) since that may increase the overall cost per GB.
Now as we understand metadata is very important but we also wants to reduce the cost of overall ownership.
- Objects separates it metadata I/O operation from its data I/O operation. It uses SSD tier intelligently for metadata I/Os and all the regular data I/O goes to HDD tier directly.
- Given the SSD tier will always be within 10% or less, of total storage capacity of AOS (for the deep storage node platform, its around 7%) Objects makes sure it stores all of its metadata in SSD tier and runs the smart garbage collection to avoid over flooding SSD tier usage and avoid risking metadata to get ILM to HDD tier. So your object head or list (pretty much all the metadata operations) are very fast.
S3 storage is not meant to be a filesystem which gives you in sub millisecond latencies, which also mean that you should expect bit higher latency while reading/write data .
- But even in this case, Objects has built magic in its data path layer. While it writes the data directly to HDD, it does quite a lot smart operation. It knows how many vdisks to create on your AOS cluster for Objects data writes, how to spread them intelligently across all your AOS nodes, how to utilize all the vdisks evenly so it doesn't increase the load on any particular AOS node and it knows which data is written where and which AOS node to talk to for accessing the data.
- It does one more magic, it knows how to avoid regular AOS IO path, and it directly talks to stargate for all its read/writes, which further improves latency.
- Objects treats every data as a new data, it further optimizes when to treat given IO as sequential vs random.So when I/O lands on underlying AOS, AOS doesnt need to take any decision to categorize IO as sequential vs random, which further saves SSD resources.
There are advanced algorithms built in the Objects stack which makes your I/Os faster, even though we have limited SSD or HDD in the underlying NOS cluster.
How does Objects helps you to reduce your cost of ownership, continued...