VAAI XCOPY not being used with Powered-On Windows VM

This is an issue I discovered along with my good friend and former colleague Drew Tonnesen a few years back which has cropped up a few times in recent days. I noticed there wasn’t really any information about it online, so made sense to put a quick post together.

In short, Windows 2012 R2 virtual machine clone or Storage vMotion operations complete much slower when powered-on as compared to when power-off. The common explanation is that VAAI XCOPY does not work when the VM is powered-on. This is not exactly true. Let me explain.

For the official list of reasons VAAI XCOPY isn’t engaged for a particular copy session, please refer to this KB article from VMware:

https://kb.vmware.com/kb/1021976

As referenced in the above link, there are multiple reasons why XCOPY might not be used. Moving a VM from one array to another is the most common. Eagerzeroedthick to thin is another. And more. But there are other reasons that XCOPY does not get engaged that are not listed there, and instead traditional software copy (Network File Copy/NFC) is used (standard reads from the source and standard writes to the target). For this post, I am specifically referring to an issue in Windows.

By default, when VMware tools is installed (in vSphere 4.1 or later) an option called disk.EnableUUID is added to the virtual machine advanced options and is set to true. This enables VMware tools to quiesce the virtual disks. Unfortunately, this also prevents VAAI XCOPY from being used for any clone or storage vMotion sessions involving that VM when it is powered-on.

Besides just seeing that the VM takes longer to move/clone you can prove this in esxtop, by looking at the XCOPY counters, called CLONE_RD and CLONE_WR. Every XCOPY operation as a bunch of XCOPY reads to a source and then a series of related XCOPY writes to the target. During an XCOPY session these both will increase identically.

If you do not see XCOPY going up here, this is possibly an issue if the VM is online. Check the advanced options for the VM:

If this option is missing or is set to false it will work. If it is set to true, XCOPY will not be used.

And that might be totally fine, if this setting is disabled/set to false certain quiescing operations by most notably backup software will fail. So if that is something you are using, I wouldn’t disable this setting. If you really want it to use XCOPY go ahead and disable it. But note that this is an offline change generally, though there are ways to reload the VMX file once it has been changed online via CLI or of course a regular old vMotion to a new host.

There are a lot of links that talk about this setting:

https://kb.vmware.com/kb/2035736

http://blogs.vmware.com/kb/2013/03/setting-disk-enableuuidtrue-in-vmware-data-protection.html

https://kb.vmware.com/kb/1028881

etc.

Generally, I think it is best to just leave it be and live without XCOPY for that VM. Unless it is a huge VM that you need to clone or for whatever reasons gets storage vMotion’ed around a lot I guess.

This is not a problem with Linux as far as I can tell, which makes sense as this generally is a Windows VSS issue.

The main point here is an FYI. Just having a VM online does not stop XCOPY from being used, but situations like this with Windows can.