vStorage APIs for Array Integration (VAAI) is a set of features and primitives designed to enable vSphere to offload storage functionality to underlying intelligent storage arrays, reducing network bandwidth, and increasing the overall performance of the process by up to 10x.
Traditionally, this was exclusive to block storage, but with the release of vSphere 5, VMware added the ability to leverage VAAI for NFS for Copy-Offload.
In addition to that, VMware also released vCloud Director 1.5, which includes new Linked Clone functionality, similar to that of VMware View. Unfortunately, the problem still exists where Linked Clones are inherently misaligned.
With VMware View, NetApp developed the Rapid Cloning Utility that is now a native provisioning function in the Virtual Storage Console. I can tell you that there is priority work being done to accomplish this same functionality with regards to vApp cloning within vCloud Director. Very cool stuff, and you’ll see it soon.
Part of my job at NetApp is to build-out use-cases for these sorts of things. My group at NetApp doesn’t have a “product.” We don’t focus on one particular thing. Well, we do, technically, but it isn’t a line item on a PO anywhere. We are in the business of being a think-tank. Ever seen the movie BIG, with Tom Hanks? That’s a bit like what I equate my job to, to an extent. I get to play with cool new technology, and tell people what I think about it, how it could be changed for the better, and tell the end-users how best to make use of our products in conjunction with other partner products. It’s the coolest job I’ve ever had.
So, for today’s post, we’re going to talk about how to put vCloud Director (“vCD”) on top of NetApp, and what cool things come into play when you do so.
Specifically, I want to talk about one key, often-overlooked, new feature in vCD 1.5:
Fast Provisioning
“What is Fast Provisioning, Nick?”
Fast Provisioning is the functional equivalent of Linked Clones in vSphere. As a matter of fact, it’s technically the same, you’re just provisioning vApp’s instead of individual virtual machines. These vApp’s can include one or many VM’s. Most of us are aware of some of the technical fallacies of Linked Clones that led to the development of our Rapid Cloning Utility for vSphere. They’re misaligned.
Let me be clear here: You’ve taken good care to insure the offset of your golden template/image is aligned. Good for you. Unfortunately, this is not the problem. The underlying functionality leverages VMware native snapshots, that are essentially delta disks of the golden image. These are not always going to be written to the same block size. Even though you’ve taken good care to align your golden image’s VMDK, there’s no way you can align each delta snapshot. This is a fallacy of the VMware snapshot engine. We know this. They know this. I’m sure it’s tops on their list to fix at some point.
Since vCloud Director heavily leverages this, how do we combat it? We’re gonna disable Fast Provisioning in vCloud Director.
“HUH?!”
Yup, you heard me right. Turn it off.
“Why in the hell would you disable that kickass technology, Nick?!”
VAAI.
What happens when you disable Fast Provisioning in vCD? vCD will fall back on however the underlying vCenter is configured. And you’ve got VAAI configured and enabled on your vSphere 5 ESXi hosts, don’t you?! I thought so! If you don’t, check out TR-3886 for a complete walkthrough on how to do it on NetApp storage!
This is always a “results may vary” topic based on what kinds of servers you’re using for ESXi5 hosts, and what size storage controllers you’re using, but the general idea is to remove the heavy-lifting of a cloning operation from your servers and offload them to your storage controllers that are better-suited for storage processing anyway. This can improve all sorts of things, but most notably we see a reduction in the total time to clone. This is because instead of sending I/O SCSI commands between hosts and storage controllers, the host is simply saying, “Hey you! Yea you, storage controller! Copy this file!”
All of this works the same on every storage controller out there that I know of, give or take an intricacy here and there.
“But Nick, you’re a NetApp guy… is there anything else that makes this awesome on NetApp?”
Ah! YES! I’m glad you asked!
With VAAI enabled, and certains conditions met (listed in the TR-3886 I noted above), we are able to leverage our snapshotting technology, create many clones near-instantly, and register those into vCenter. This is the fundamental of what our Rapid Cloning Utility does.
UPDATE: To be clear here, we do not have the equivalent of a NetApp “Rapid Cloning Utility” for vApp’s in vCloud Director. Yet. It is something we are diligently working towards. What I am describing is how the offload process would work under the covers by simply turning off Fast Provisioning. As confirmed by Chris Colotti in the comments below, this has been around since before Fast Provisioning existed. What we DO have is the proprietary snapshot technology that enables us to do near-instant snapshot clones instead of the full-copy clones.
So let’s put this all together…
You have a foundation of NetApp storage arrays hosting a vSphere 5 environment with VAAI configured and enabled, which is hosting a vCloud Director 1.5 environment with Fast Provisioning disabled.
Congratulations! You’ve now averted the “linked clone” disaster of potentially overloading your storage controllers and making you look bad! I’m here to help you look like a champion by using NetApp storage, and if you’re already using it, you’re already ahead of the game, regardless of what everyone else will tell you.
“OH HEY! Nick, we’re an all-NFS shop. Does all of this apply for us as well?”
To an extent, yes. VMware’s latest version of vSphere has Copy Offload support for NFS. Configuring VAAI for NFS requires a plugin installation into each of your ESXi5 hosts. We’ve streamlined this into a quick deployment wizard in the latest version of our Virtual Storage Console plugin. You can certainly perform everything I’ve listed above in vCD on vSphere 5 on NFS.
You can participate in the active public BETA of the VSC 4.0 here:
http://communities.netapp.com/groups/vsc-40-external-beta
WE WANT YOUR FEEDBACK!
The deployment wizard for the VAAI NFS plugin is only one of the many, MANY, enhancements made to VSC 4.0. You can read all of the documentation, Release Notes, and feature-adds in the community link above. Look for future posts from me on the other features in the coming weeks, including details about the Online Alignment Tool, and support for our latest version of our unified operating system, DataONTAP 8.1, including “Cluster-Mode.” More on that soon as well! (I know I’ve been absent and busy, but you’ll see why soon!)
I’d love to hear any feedback you have, or anything you’d like to see in the comments below!
Do you currently use the VSC for Host Config of Best Practices? Datastore Provisioning? Backup & Recovery?
How do you currently use the VSC?
If you don’t use it, I’d also love to hear why not?
NetApp SE’s and Partners, I’d like to challenge you to ask your customers these same questions! If you’re having problems and need help, that’s what we’re here for!
As always, thanks for reading!
-Nick
I am not sure I am following your drift here, unless I am missing something or misread your post. Instead of using Linked Clones for vCloud Director (aka fast provisioning) you are suggesting to use NetApp snapshot technology to clone these and register in vCenter Server?
What about vCloud Director? Does that mean I would need to import them also in to vCloud Director? Which also means I need to give my tenants access to every part of my management layer?Again, maybe I am missing something here…
Actually, what we’re suggesting is to use VAAI. This should be invisible to vCD, View or even Lab Manager. Each of those has a way to turn off “fast clones” and simply call the ESX primitive to “full copy”. ESX determines if VAAI is available for the source/dest datastore(s) in question. If VAAI isn’t there, it’s a slow copy. If it is, ESX tells storage “Copy this for me”.
While VSC/RCU has a way to register VMs in View, there is no such VSC functionality for vCD or LM.
As for the support question, I’d really like to know what the issue is, other than QA just hasn’t tried it yet. I can’t see how vCD (or View or LM) would know or care.
I have to agree with Duncan, I do not see how the vCD use case is supported here. Another point besides the operational model being broken in the example given, is vCD 1.5 does not support the NFS VAAI primitive.
Wade I have to check. VAAI is a vSphere function not a vCD function. vCD does not really know if VAAI is enabled or not it is relying on vCenter for that. So if vCenter supports it vCD may not need to, but there is that fine line. I have an EMail into someone to see where the line blurs. I am almost positive in past emails I have seen that vCD does not care one way or the other since VAAI is not a vCD deal.
VAAI definitely isn’t supported on NAS with vClould Director.
The reason for this is that at the vSphere level NAS VAAI uses a different VIM API to VMFS which vCloud Director does not make use of at the moment.
Yes I was finding some internal posts about that in fact. 🙂
The confusing part is some references speak to VAAI/NFS WITH fast provisioning and others without, so frankly I am completely confused. It is too late on a Sunday to think about this any more.
Could you clarify as to whether this is with or without Fast Provisioning? The whole point of this post was to show that vCD doesn’t make use of VAAI, whether it is VMFS or NAS. The underlying vSphere does. And if you turn off the linked clone functionality of vCD, it will fall back onto however vSphere is configured. If that vSphere is vSphere 5, and you have support for NFS VAAI in vSphere 5, what am I missing? If vSphere 5 has support for NFS VAAI, and that is what vCD uses for cloning operations, why would it not be supported?
If you want a reference detailing the support position, please consult the vCloud Architecture Toolkit 2.0.
http://www.vmware.com/files/pdf/vcat/Architecting-VMware-vCloud.pdf
Page 23, Section 4.3.3 – VAAI:
vCloud Director 1.5 supports block-based primitives (FC, iSCSI), but not the new NAS primitives introduced in vSphere 5.
If you turn off Fast Provisioning in vCD and try to Deploy a vApp from a template, it falls back to doing full clones via however vCenter is configured. (this is universal)
If you’ve got VAAI configured (yes it works with NFS VAAI as well so long as you have plugins installed), it will offload that full clone to the storage controller (this is universal).
The part that gets interesting is if you’ve got your gold template in the same volume/datastore on a NetApp filer, it will snapshot the clone, which is a near instant operation.
There is no RCU (yet, due to lack of vCD extensibility or plugin framework), but this is one way to almost get there.
Nick I think what is unclear is this statement:
“we are able to leverage our snapshotting technology, create many clones near-instantly, and register those into vCenter. This is the fundamental of what our Rapid Cloning Utility does.”
Since there is not RCU with vCD stating this may add confusion. I think everyone agrees that without Fast Provisioning Enabled you will revert to full clones and VAAI can help with that. I even talked about this in the “Clone Wares” Series. I think that statement of registering VM’s is simply misplaced since vCD and vCenter create the Virtual Machine objects in the vSphere Database. Maybe the RCU aspect is just adding fog to the post since it is not vCD integrated. Just a thought, but just leave it as VAAI is in place and increases the full clone.
This has been the case since vCD was released that if VAAI was enabled (before fast provisioning)) it would always get used. Nothing has really changed in that so I see why you are suggesting to turn it off.
Chris,
Awesome! Thanks for clarifying. You’re right, and I am working on a way to re-word that better, as I can see how it can be misleading. I more intended it to be a high-level overview of what was happening under the covers, and equating it to something folks might already be familiar with.
But to be clear, there is no plugin framework available for vCD, e.g. there is no “Rapid Cloning Utility” for vCD. We are exploring other ways to do this.
I would love an update from you/Duncan as to whether using NFS Datastores that are properly configured for VAAI is or is not a supported configuration for a PVDC to use.
Thanks for the banter, gentlemen!
See my comment to Wade’s post. I will have an answer soon, but I recall other email exchanges on VAAI with vCD and I cannot be 100% sure but I think vCD just does not know about it either way (enabled or not). I will confirm and let you know.
Great! Thanks Chris! I will also forward up to our PM’s!
Like I said before, not supported.
So even with the new Sub-block sizing and unified block size we still get mis-aligned Delta files from the snapshot? Wow, I am going to have to do some homework on this. We are currently re-evaluating our current standards. I would really not want to implement a feature that could end up blowing up in your face…
@twitter-51132403:disqus ,
Yes, your assessment is correct, unfortunately. Linked Clones, whether in View or vCD are still inherently misaligned because of the use of delta disks off of the gold master. It can bring a storage system from any vendor to it’s knees if the partial-write I/O gets out of hand.
Nick, First-off big thanks for sharing this information. Is there any way to really avoid this problem? I don’t think we plan on using fast provisioning because we do utilize VAAI. I have seen some information on it but really not much.
I never heard back but no big deal. I respect you guys because you are probably busy. According to the article I don’t see a way. Do you know of any documentation that would provide details on the implications of this to ones storage array? Obviously there are some concerns I have from a scaling standpoint when it comes to storage.. Adding thin provisioning and linked clones represent a HIGH VM density on your datastores..
@twitter-51132403:disqus ,
As of this past Friday, VMware as added an errata doc to the vCAT stating that NFS VAAI is now supported in vCloud Director environments. I have an update post coming. Stay tuned!