I have blogged a decent amount recently about VVols and in many of those posts I mention config VVols. When using vSphere Virtual Volumes, VMs have one, some, or all of the following VVols types:

• Data VVols–every virtual disk you add creates a data VVol on your array
• Swap VVol–when you power on a VVol-based VM, a swap VVol is created. When you power it off, this is deleted.
• Memory VVol–When you create a snapshot and store the memory state or when you suspend a VM, this is created.
• Config VVol–represents a folder on a VVol datastore.

This statement about config VVols deserves a bit more attention I think. What does that really mean? Understanding config VVols is important  when it comes to recovery etc. So let’s dig into this.

When you create a VM on VMFS (or NFS) what happens? Well a folder is created with various files.

So if I have datastore “FlashArrayVMFS” that is empty:

Then I create a VM called “VM-1” with one 40 GB virtual disk, I see the following:

In that folder I see the various files that make up a VM; .vmx, .vmdks, etc.

What we are looking at is an actual directory in a file system, in this case VMFS. If I power-on the VM the vswp file is created and some various other ones.

Great, so this makes sense. A VM is a bunch of files. These files sit in directories on a file system (VMFS or NFS).

But what about with Virtual Volumes. There is no more file system. Does what a VM is fundamentally change?

No. Doing so would break so many things–VMware could not change what a VM is. A VM has a VMX file. A VM has VMDKs. A VM has a swap file. Etc. If they changed what a VM was, pretty everything that integrates with VMware would no longer work with VVol-based VMs.

This is a KEY design feature of VVols–to not change the look and feel and user interaction with itself just because they are using VVols. Furthermore, how APIs (VADP for instance) work should not really change. VMware admins should not really have to change their PowerCLI scripts or re-learn wizards just to create a VM.

So the files cannot go away. So where do these files get stored in the absence of a file system?

VVol Datastore

So first off, the VVol datastore is not a file system. It is not a “LUN” or a “device” or a “volume”. It is a capacity limit that represents a given array (an array can have more than one VVol datastore each with its own designated size limit).

So when you choose a VVol datastore, where does the VMX file or VMDK files go?

The config VVol. Any time a VM is created on a VVol datastore, or a new virtual disk is added to a VM on a new VVol datastore, a config VVol is created. A config VVol represents a folder on a VVol datastore.

So let’s start with something simple. Creating a new VM on a VVol datastore.

So my VVol datastore is currently “empty”:

I create a new VM called VM-2 on my VVol datastore:

Then I see a folder:

And in that folder I see a bunch of files:

So where are these files stored!?

Well not on the VVol datastore. Because, like I said before, it isn’t a file system of any kind.

Instead they are stored on the config VVol. When a new “folder” is “created” on a VVol datastore (for instance like creating a new VM), a config VVol is created. So in this case, the following config VVol was created:

A config VVol is actually internally formatted with VMFS, and all of these files are stored on it. The original folder though is not a directory on a file system, it is actually just a mount point to the file system on the config VVol.

So a VVol datastore is really just a collection of mount points (config VVols).

Inside of the config VVol is where the files are actually stored. So the VMX file is there. The VMDKs files are there. The vSwap file is there. But if every virtual disk is a actual volume on the array, why is there a VMDK file? Well, because that how VMware works. RDMs have VMDK files. So do VVols. But a VMDK file in those situations are just pointer files. The config VVol holds a bunch of pointer files. VMDKs. vSwap. Snapshots.

Look at a VMDK pointer file for an RDM:

VVol VMDK:

The RDM VMDK is a pointer to the pRDM and the VVol VMDK is a pointer to the VVol UUID (each VVol has its own VVol ID that is unique, similar to a volume serial number).

So if I create a folder in a VVol datastore, what happens?

 

The folder appears in the VVol datastore

And if we look at the array a config VVol for “codyfolder” is created too.

So really all a config VVol is, is a small file system that is presented through a VVol datastore as a folder object. But all of that difference is totally abstracted from the end user. You can still cd into /vmfs/volumes and navigate with ls and cd etc. You can still use the browse datastore function in the vSphere Web Client. All of the stuff superficially looks the same, but in reality what happens is very different.

Recovering a Config VVol

So what happens if I delete a config VVol? Well let’s take the situation with my folder I created earlier “codyfolder”.

If I delete it on my FlashArray:

The folder goes away:

Now on the FlashArray, when I delete a volume, it goes into a recycling bin for 24 hours, so I can recover it after deletion.

So if I recover it, it immediately appears back in my VVol datastore.

And the folder is back:

So a VVol datastore is a very flexible thing–since it is simply an abstraction and continuous assembly of the VASA provider on your array. So recovering a VM is a very similar process.

Note that not all arrays have a “recycling bin” concept, but for the ones that do, there is a nice immediate benefit you can get with VVols.

So what if I delete a VM? I already posted about that in this blog post:

Recovering a Deleted Virtual Machine with VVols

But the important thing to note is that when ESXi deletes a VM, it first deletes all of the files on the config VVol. Just like with VMFS or NFS, when you delete a VM, it deletes the files the describe the VM, then lastly it deletes the folder.

If I delete my folder from above:

The process completes:

Then the config VVol is deleted:

Similar if I delete a VM:

There goes the folder and all of the files in it:

So if I restore the config VVol:

The VM folder appears back, just like we would expect:

But if we look in the folder…it is empty! Well almost empty.

All that is there is the sdd.sf folder, which is probably familiar to you. If you look at any VMFS volume you will see that too:

This holds various VMFS information. But everything else is gone, this is of course because VMware deleted it. Just like it would with VMFS or NFS. So to restore it you need a backup of the config VVol. Like an array-based snapshot.

I have a snapshot of my config VVol that was created earlier.

If I restore from that snapshot:

My files are restored in my VM folder instantly!

Super simple. To get the VM back, just choose the VMX and register it back. The VMDKs are back so the pointers are restored so there is no need to individually re-import the data VVols, just make sure they exist on the array still of course.

If you only delete a single data VVol (choose a VMDK and then “delete from disk”) follow the process here.

One of the nice things about how we did VVols on the FlashArray is that the configuration is distributed to the volumes. Meaning there isn’t a VASA database–the VASA provider just looks at the volume metadata and presents it as such. If you recover a config VVol, it has meta data saying “hey I belong to this VVol datastore”. So the VASA process represents it.

The process above will vary somewhat between vendors (and may or may not be possible) but VMware built VVols in a very flexible way, which gives a lot of room for vendors to build some cool stuff and value add. Recovery like above into vCenter plugins etc. Or do this in vRO. And so on.

Final Thoughts

So a few things to get from this. Understand how your array deals with VVol deletion and management. If you have a recycling bin there are some nice options to easily restoring accidentally deleted VMs in a simple way.

I think a good best practice is to have snapshots created of your config VVols created every so often. A good way to make sure of this is if you VVol vendor’s VASA provider has a snapshot schedule capability.

So a policy like so which I called “configpolicy”

Then go to your VM and go to Configure then Policies then click Edit Storage Policies.

Then for the “home” which is the policy for the config VVol add that snapshot policy.

Verify the policy is configured.

Then apply it.

The array will now snapshot the config VVol once a month and keep each one for a month. If someone messes with the array-side configured VMware will tell you:

In the case above, I went to my array and changed the configuration from 1 month to 2 months. Violating the VMware policy. So VMware lets me know.