I posted shortly after ESXi 6.0 came out a while back explaining how to do in-guest UNMAP with Windows. See the original post here:
Direct Guest OS UNMAP in vSphere 6.0
The high-level workflow if you don’t want to read the post is:
- You delete a file in Windows
- Run Disk Optimizer to reclaim the space
- Windows issues UNMAP to the filesystem
- ESXi shrinks the virtual disk
- If EnableBlockDelete is enabled on the ESXi hosts, ESXi will issue UNMAP to reclaim the space on the array
This had a few requirements:
- ESXi 6.0+
- VM hardware version 11+
- Thin virtual disk
- CBT cannot be enabled (though this restriction is removed in ESXi 6.5 see this post)
One thing really bothered me about this, which is why doesn’t automatic UNMAP work? In Windows 2012 R2 and later, by default, when you delete a file UNMAP is supposed to be issued automatically to reclaim the space. This never seemed to work.
Well I finally found my problem. Long story short, make sure when formatting your NTFS that the allocation unit is 64K. Anything else will prevent automatic in-guest UNMAP with virtual disks from Windows.
So make sure your NTFS volumes are using a 64K allocation unit.
UPDATE: This is fixed in 6.5 Patch 1! You no longer need the right allocation unit size! See this post: In-Guest UNMAP Fix in ESXi 6.5 Part I: Windows
NOTE: This behavior is common to ESXi 6.0, 6.0 U2 and 6.5. The reason I am blogging about it just now is that I just figured it out.
This is a common SQL best practice so many of you probably are have NTFS configured as such. To check, here is a short PowerShell script from this blog post that will do it easily:
$wql = "SELECT Label, Blocksize, Name FROM Win32_Volume WHERE FileSystem='NTFS'" 2 Get-WmiObject -Query $wql -ComputerName '.' | Select-Object Label, Blocksize, Name
Paste that into a PowerShell window and you will get your answer.
Note all of this was on Windows 2012 R2. I need to still test Windows 10 and 2016.
For details on some testing: read on…
I was testing some other UNMAP stuff and I couldn’t get UNMAP to work fully.
I had copied an ISO to a thin virtual disk which made it 3.1 GB in size:
The virtual disk:
I would permanently delete the file and then run the disk optimizer tool, not everything would be reclaimed though. I got back about half of what I expected.
About 1.5 GB was returned. Not perfect but something.
So during testing I decided to try deleting the virtual disk instead of removing and adding an entirely new virtual disk.
I noticed that the remaining 1.5 GB virtual disk, was now ~80 MB. It had fully reclaimed! But how/why?
Well I first thought that maybe I didn’t wait long enough for UNMAP to finish. But I quickly proved that wrong. Then after a quick search I found this:
I had completely forgot about that behavior. NTFS format triggers a fully UNMAP first, then it allocates its metadata, in this case about 87 MB. You can see it goes basically down to zero then up to 87, as I would run the ls -s command during the format and it goes from 1.5 GB down to 30 MB then finishes at 87 MB.
Since it was essentially issuing UNMAP to a raw device, it got be thinking about alignment issues and how with the new Linux in-guest it has to be aligned at 1 MB for UNMAP to work. So I started trying different NTFS allocation units. Starting with 64k-which is SQL server recommendation. Sure enough, when I set it to that automatic UNMAP worked, and it worked well. As soon as I deleted my file(s) I got 100% of my space back–my virtual disk shrunk down to the original size.
So my recommendation is to make sure you format NTFS using a 64k allocation unit. Most applications recommend it anyways, so you might as well. For UNMAP, it is far more efficient (you get all of your space back) and it is automatic (as soon as you delete a file it will reclaim the space and shrink the virtual disk.
Stay tuned on more details on why this is required.
Good find Cody! Did you try if this works for a system drive also formatted @ 64K?
It should, but I have not created a system drive with 64K. I would think it would work fine. I need to test to be sure
Following… We are setting up a Windows 2012 template right now and the default format appears to be 4k for the system drive.
What’s about UNMAP for Windows 2008 / 2008 R2?
Windows 2008 unfortunately does not support UNMAP, so this is only for 2012 and later
You can run unmap from the esxi host CLI. Does this accomplish the same thing fo win 2008 R2 guests? We are doing it to restore large swaths of storage after a vsphere migration to a different VVol. It is only an occasional process.
The esxcli UNMAP though is only for VMs (or VMDKs) that have been deleted–it will reclaim the space that they took up on the VMFS. In-guest is for files that have been deleted on the guest filesystem (in this case NTFS) on the virtual disk. To reclaim this space you need the guest OS in the VM to issue UNMAP. 2008 does not support this, so the only way to really reclaim in a Windows guest is through zeroing.
Ok, and zeroing the FS is a step we have taken after a system cleanup, but before esxcli Unmap. Like this:
1. VM guest cleanup, for Win 2008 R2, hack cleanmgr if desktop experience is not installed
2. Run sdelete.exe to zero FS
3. Run esxcli unmap
Those steps work well for us, but we allow our SAN to handle the thin provisioning in most cases. Are you using the unmap in the OS with thin provisioned virtual machine guests? I think the scope is different in that case, but the effects might be similar.
ESXCLI UNMAP isn’t going to help you after a zeroing operation because ESXCLI UNMAP only issues unmap to unused space on the VMFS. Zeroing is issued to the VMDK and is still seen as written/used data on the VMFS. Once the zeroes are written the zeroes will need to be removed on the array itself. For a data-reduction array like the FlashArray it will happen immediately because contiguous zeroes are removed inline and never stored. On many hybrid arrays you would need to run a zero space reclamation to get it back. The only way esxcli unmap will return you space is if you delete the VM or the VMDK and then run it. In my case (well the target of this blog post) I am running UNMAP in the guest to a thin VMDK. The thin VMDK is then shrunk by the UNMAP and then UNMAP is issued to the VMFS to reclaim the space on the array. If you get space back after zeroing then running UNMAP it is either because
1) your array supports inline zero removal (which would not require you to run esxcli unmap as it would not do anything)
2)there is other dead space on the VMFS from some previously deleted VMDK that is incidentally reclaimed from esxcli unmap.
Does that make sense?
Great find! I did a bit of testing with Windows Server 2012 and found that guest automatic UNMAP seems to work with 32K or 64K allocation unit sizes. I couldn’t get 100% of the space reclaimed using 64K size…close, but not 100%. Using 32K size showed that 100% of the space was reclaimed in my tests. Test storage was a zVol with 16K block size on FreeNAS 9.10.2.
Interesting! I thought in my tests I couldn’t get 32k to work, but let me try again and get back to you. Still working with VMware to explain this behavior–I know it has to do with UNMAP granularity/alignment, but I am looking for a more detailed explanation.
Yeah, it makes no sense to me. If the optimize-volume cmdlet with the -retrim parameter works with any cluster size, you’d think automatic unmap would work too. I tested them all and 32k and 64k are the only ones. Hopefully we can get an answer as to why it doesn’t work and if they can/will fix it.
Hi, can this same command work if the VMDK files are set to Thick Provisioned Eager Zeroed? We were told by HPE for our 3Par that our vSphere VMDK disks, other than OS should be configured as Thick Provisioned Eager Zeroed, so that the 3Par handles the Thin provisioning of the VMDK disks directly.
I am not sure if unmap from vSphere would be required if that is the case. The 3Par may just need to reclaim space for the OS on the 3Par itself – but I’m not certain.
Hi–no this will only work if the virtual disks are thin provisioned. This is one of the reasons I really recommend using thin unless there is a specific intense-performance use case. Then EZT will be a little better. You lose all of this capacity reclamation if you use anything but thin virtual disks
Any new news on this being a bug or not? Not being able to handle OS disks at default 4K is an annoyance if nothing else. That, and not being 4K disables NTFS compression (useful to reduce flying bytes from/to your array, though hobbles array based compression)
I actually just sent a request back to VMware yesterday to inquire on the status of the ticket on this. I also have a request into Microsoft. Will post as soon as I hear back!
Looks like VMware released a patch today to get automated UNMAP working on NTFS volumes formatted with 4K cluster sizes.
http://cormachogan.com/2017/03/15/vsphere-6-5-p01-important-patch-users-automated-unmap/
Just posted a blog about it, it works great! https://www.codyhosterman.com/2017/03/in-guest-unmap-fix-in-esxi-6-5-patch-1/