I’ve noticed I am beginning to have some blog post sprawl as I update my UNMAP script over and over so I will be using this post from now on to record future updates. Please use this post as the final word on what is new with my UNMAP Script.
This script runs UNMAP on all FlashArray datastores in your vCenter environment to reclaim dead space. All operations are logged to a file in a directory of your choosing. This script requires that a datastore be accessible by at least one ESXi version 5.5 host or later to run UNMAP. Please refer below for detailed changes and requirements for specific versions. The newest version description will be directly below. The script can be found here:
https://github.com/codyhosterman/powercli/blob/master/unmapsdk.ps1
Note there is an unattended version of this script too, if you want to run it as a scheduled task. Find it here:
https://www.codyhosterman.com/2017/04/unattended-flasharray-vmfs-unmap-script/
For required vCenter permissions please see this post:
Script Version 5.0 January 27th, 2017
This latest version has a few major changes:
- Support for VMFS-6. VMFS-6 by default enables automatic background UNMAP and therefore does not require manual UNMAP to be run. My script now identifies VMFS-6 volumes and checks to see if automatic UNMAP is turned on. Any VMFS-6 that has it turned on it will skip
- No longer ever need to go into the script and hard-code variables. They are now asked interactively when you run the script. This includes:
- Log file directory
- FlashArray connection information
- vCenter connection information
- Whether to send results to Log Insight
- Improved logging in general.
- Changed how space reclamation results are reported. It now uses the method described in the post here. Far more accurate and useful than before. Also it reports at the end, not for each datastore logging section to make it easier to read. Furthermore, it gives the array more time to make sure the space reporting is accurate. Usually takes a few seconds and doing it immediately after the UNMAP is not always accurate. It is reported on the array level and the datastore level in a table at the end of the log and also on-screen.
- Returns more information to the screen. Which datastore it is running UNMAP on and the final total space reclaimed (both virtual and physical) for all entered arrays in GB.
- Checks and imports both the PowerCLI modules and the FlashArray PowerShell SDK. They must both be installed.
- Improved error handling
Another note on space reporting. I report three numbers for a given volume:
- Expected minimum virtual space to be reclaimed. This is by taking the difference between the VMFS allocation and what the array sees as written overall (virtual space)
- Actual reclaimed virtual space. This is the difference between virtual space for a volume before UNMAP as compared to after. If the actual is much higher than the expected, this is due to the presence of thick provisioned virtual disks. This makes the expected lower than it should be.
- Actual physical space reclaimed. This is how much unique space actually written down to SSD was reclaimed.
Note, if you want a report about how much will be reclaimed without actually running it, you can run this script:
Detecting what FlashArray VMFS Volumes Have Dead Space
Integration with Log Insight
See a more in-depth post about how I send to Log Insight here:
Using PowerShell with the VMware Log Insight REST API
The UNMAP script also optionally supports sending per-datastore UNMAP results to Log Insight.
If you do not use Log Insight, or don’t want to send results to it, then enter n and move on.
If you do, enter y, the Log Insight server and a UUID to use to identify the messages.
If this is enabled, the script will send a REST call with the PowerShell InvokeRest-Method command to Log Insight. You enable this feature by simply changing the $loginsightserver from $null to your server. The REST call sends the following information:
- VMFS volume that UNMAP ran on
- Network Address Authority (NAA) of the underlying device
- The FlashArray volume name
- The FlashArray
- The ESXi server that ran the UNMAP process
- The virtual space reclaimed in GB
The Log Insight message will look like below (with the corresponding fields as I named them in the REST call):
Hopefully will be a decent way to keep track over time how much space has been reclaimed by UNMAP procedures. Now I can run queries and show how much has been reclaimed overall:
Hi Cody,
I just tested your script on my infrastructure (2 clusters vSphere 6.0 U2 with FA-405 and //m20) but I got the followiong error:
Cannot convert value “397.347,871” to type “System.Int32”. Error: “Input string was not in a correct format.”
At D:\Pure Storage\unmapsdk.ps1:332 char:21
+ $unmapsavings = [math]::Round(($volphysicalcapacity – $volph …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvalidCastFromStringToInteger
I see that you format the vol* variables in a string value using the .NET Framework formatting method {0:N3} when you set them. In Italy the decimal separator is “,” and the thousands separator is “.”, so the Round function doesn’t work because the string cannot be implicitly converted to an integer.
I modified the script formatting the variable values on the log writes:
296 $volreduction = $volinfo.data_reduction
297 $volphysicalcapacity = $volinfo.volumes/1024/1024
…
299 add-content $logfile (‘The current data reduction for this volume before UNMAP is {0:N3} to 1’ -f $volreduction)
300 add-content $logfile (‘The physical space consumption in MB of this device before UNMAP is {0:N3}’ -f $volphysicalcapacity)
…
330 $volreduction = $volinfo.data_reduction
331 $volphysicalcapacitynew = $volinfo.volumes/1024/1024
…
335 add-content $logfile (‘The new data reduction for this volume after UNMAP is {0:N3} to 1’ -f $volreduction)
336 add-content $logfile (‘The new physical space consumption in MB of this device after UNMAP is {0:N3}’ -f $volphysicalcapacitynew)
337 add-content $logfile (‘{0:N2} in MB has been reclaimed from the FlashArray from this volume’ -f $unmapsavings)
…
360 add-content $logfile (‘Space reclaim operation for all volumes is complete. Total immediate space reclaimed is {0:N2} MB’ -f $totalspacereclaimed)
Now the script works without any issue.
Thanks,
—
Lorenzo
Ah thanks for fixing this! I will work this into my main code branch in the next release. Feel free to fork off on GitHub though!
LOL! “Fork off on GitHub”. Heh heh heh!
Cody – Thank you so much for this comprehensive unmap script! I ran it against my environment and I’m seeing that for some of my volumes the reclaimed space value is a positive integer and for other volumes the reclaimed space value is a negative integer. What could account for the negative values, or put another way how can an unmap operation result in *increased* physical space consumption for a given device?
****results with positive value*******
The current data reduction for this volume before UNMAP is 3.571 to 1
The physical space consumption in MB of this device before UNMAP is 61,064.189
The maximum allowed block count for this datastore is 20076
The new data reduction for this volume after UNMAP is 3.073 to 1
The new physical space consumption in MB of this device after UNMAP is 60,180.549
883 in MB has been reclaimed from the FlashArray from this volume
****results with negative value*******
The current data reduction for this volume before UNMAP is 2.856 to 1
The physical space consumption in MB of this device before UNMAP is 75,507.639
The maximum allowed block count for this datastore is 31207
The new data reduction for this volume after UNMAP is 2.837 to 1
The new physical space consumption in MB of this device after UNMAP is 75,606.130
-98 in MB has been reclaimed from the FlashArray from this volume
You’re welcome! It’s not surprising. What the script is doing is finding all of FlashArray volumes, then going through them one by one. For each volume, it finds out what the FlashArray is reporting for space usage, then it runs UNMAP and then finds the space usage again afher UNMAP is complete. Often volumes will not really have any dead space, so nothing gets reclaimed. The expectation there would then be that the value is zero. But… In the case where nothing (or very little) is reclaimed, additional capacity could be written to the volume by VMs on them (either the virtual disks grow in the case of thin or new writes inside of a thick virtual disk). This would cause more space to be used after UNMAP completes. Therefore would give a negative value. The numbers in the log are really meant to just give an idea what is happening. Does that make sense?
Makes perfect sense – the vms continue to generate I/O while the UNMAP operation is in progress. Thanks!
Cody, what you’re saying here makes sense, but I have a couple of volumes that are in the Terabyte range, which is super weird because our array really isn’t that large.
I’m not sure I understand. Can you clarify your question? Thanks!
Trying to get this script to run via the task scheduler on my vcenter server.
I added the following to the start of the script:
Import-Module -Name PureStoragePowerShellSDK
I also added the following to the end of the script:
#- Send Mail
send-mailmessage -to [email protected] -from [email protected] -subject “Weekly PURE Optimization Results” -smtpserver smtp.xxxx.com -body (get-content $logfile)
My parameters in the task are:
Program
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Arguments
-noe -c “. \”D:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1\” $true .\”D:\LOGS\PURE-unmap.ps1″ /$true”
Am I missing something?
Sorry took so long to respond. What error are you getting? Anything?
Cody,
Thanks for your script. It works on 1 of my volumes, but fails on the other two. Any idea why? The environment was just upgraded to 6.0U2 but the same problem existed in 5.5.
Failed to run UNMAP to this volume. Most common cause is the device is locked by another process.
Message: An error occurred while communicating with the remote host.;
InnerText:
Skipping volume…
Hmm. Out of curiosity have you tried running esxcli storage vmfs unmap on the datastores that fail? I’d be curious to know if it fails that way too. This seems like a communication problem though with the selected host from PowerCLI
Hey Cody,
Ive been using the PowerActions version of your script. Ive gone ahead and updated my PS SDK to 1.6 and PowerCLI to 6.3.0-3737840. Should I/can I use the most recent version of unmapsdk.ps1 from GitHub thru PowerActions?
I know previously you had created a PowerActions version of this script, but I dont see that its been update recently. I know youve gotten so many scripts to keep updated, so Im not trying to add more to your pile.
Id just like to make it easy for my fellow admins to use the VCenter Web Interface, along with this script. What is the best way to do that?
Thanks again!
Ah I haven’t updated that script, so it will not work out of the box with PowerActions. I can do that pretty easily though. Just need to make a few changes. Though, we are releasing a new version of our Web client plugin pretty soon, which has UNMAP management built-in.
Hey Cody,
Excellent script. I’ve used it in the past at a former employer. Trying to run this now at my new position. Everything seems to run through although much faster than I would expect. The troubling thing is that it doesn’t seem to reclaim any blocks. The final output from the log states that, “Space reclaim operation for all volumes is complete. Total immediate space reclaimed is 0 MB”. We add/move/delete VM’s on this array all the time. Do you know of an underlying issue that would cause this?
Hey! Thanks a lot! Glad you’ve found it useful. I am working on revamping this script entirely when it comes to reporting, I will have it done soon. I will post on it shortly, try that one and let me know!
Hi Cody,
Script looks awesome! My concern is will this work on iSCSI / SAN types? May be few edits would make it work. Adding to that, I have made up another script which probably could help on SAN but need to test it.
My ask is some what simple, I am looking out for a script which could help me unmap datastores mapped to single ESXi Host.
This works on FC or iSCSI. It doesnt really matter–though my script is currently configured to only work with FlashArray volumes, but that can be changed pretty easily. You can easily just do on ESXi host. Instead of the credentials/IP of a vCenter just do the creds/IP for the ESXi host. Or you can hard code the ESXi name into the script.
Hi Cody,
I’m getting below error when I run unmap script in my vSphere 6.0 U3 environment.
Pure Storage FlashArray VMware ESXi UNMAP Script v5.0
—————————————————————————————————-
How many FlashArrays do you need to enter (enter a number): 1
Please enter a FlashArray IP or FQDN: 10.10.5.20
Please enter a vCenter IP or FQDN: mts-vcspwv01.mts.local
Re-use the FlashArray credentials for vCenter? (y/n): n
Please be patient–this process can take a long time
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:343 char:21
+ $arrayname = Get-PfaArrayAttributes -array $EndPoint[$arrayc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:349 char:21
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$array …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Running UNMAP on VMFS named MTS-A001_01PUR_VMFS_P_NR_0000_10…
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:343 char:21
+ $arrayname = Get-PfaArrayAttributes -array $EndPoint[$arrayc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:349 char:21
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$array …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Running UNMAP on VMFS named MTS-A001_01PUR_VMFS_P_NR_1011_11…
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:343 char:21
+ $arrayname = Get-PfaArrayAttributes -array $EndPoint[$arrayc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:349 char:21
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$array …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Running UNMAP on VMFS named MTS-A001_01PUR_VMFS_P_NR_1014_12…
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:343 char:21
+ $arrayname = Get-PfaArrayAttributes -array $EndPoint[$arrayc …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:349 char:21
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$array …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Running UNMAP on VMFS named MTS-A001_01PUR_VMFS_P_NR_66C3_254…
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:436 char:5
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$arraychoice] -VolumeN …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:436 char:5
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$arraychoice] -VolumeN …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:436 char:5
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$arraychoice] -VolumeN …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Source\unmapsdk.ps1:436 char:5
+ $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$arraychoice] -VolumeN …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
FlashArray-level Reclamation Statistics:
FlashArray VirtualSpaceGBReclaimed PhysicalSpaceGBReclaimed
———- ———————– ————————
MTS-A000101PUR 0 8
By any chance does your array serial number start with a zero?
Hi Cody,
Can you please let me know how to retrieve Array serial number from GUI or Powershell? FYI, All my LUN serial numbers started with 9BEAB6FC52570F6D0001XXX
Simplest way is with get-pfaarrayid
PS C:\Users\cody> $creds = Get-Credential -Credentials $creds -IgnoreCertificateError
PS C:\Users\cody> $fa = new-pfaarray -EndPoint
PS C:\Users\cody> Get-PfaArrayId -Array $fa
version revision array_name id
——- ——– ———- —
99.9.9 201802221714+ca5867a flasharray-m20-1 4bfab20d-4b69-4df7-8540-866deae684d7
Thank you! Array details below
version : 4.10.8
revision : xxxxxxxxxxxxx
array_name : xxxxxxxxxxxxxxx
id : 09bxxxxxxxxxxxxxxxxxxxxxxxxxxx
Yep. There’s the zero. That’s the issue. I will update the script to take this into account. Stay tuned.
Cody, as you appear to still respond to comments on this thread, I have run this powershell script successfully on other arrays but I am encountering a similar error that Mohamed had:
Running UNMAP on VMFS named puredatastore10…
Index operation failed; the array index evaluated to null.
At C:\Work Area\pure_unmap\unmapsdk.ps1:353 char:21
+ … $arrayname = Get-PfaArrayAttributes -array $EndPoint[$arr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
Index operation failed; the array index evaluated to null.
At C:\Work Area\pure_unmap\unmapsdk.ps1:359 char:21
+ … $volinfo = Get-PfaVolumeSpaceMetrics -Array $EndPoint[$ar …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArrayIndex
In the above puredatastore10 is a 10TB VMFS5.5 datastore on my cluster running 6.0 Update 3. Purity version is 4.10.10. This error generates on every datastore/volume and there is about 35 total. The script states “serial issue fixed” at the top of the page and my array does begin with “004”. Is there any other potential issue or should this be working correctly?
I am not sure if the script is running correctly or not as it goes through each datastore, takes a long time, and once it completes it shows that 0GB reclaimed on the datastores. Either there is nothing to reclaim or this error is preventing it from perform reclamation.
Hmm I will take a look again at it for the zero issue. I thought I fixed it–I will verify. With that being said, it will still “work”. It still runs UNMAP, it just cant reach out to the array to report the before and after stats
It looks like the v5.0 script has a check for esxi version that doesn’t include version 7:
where-object {($_.version -like ‘5.5.*’) -or ($_.version -like ‘6.*’)}
I changed this to include 7.* and it’s working again (probably should have just excluded the version check because we don’t have anything below a level that wouldn’t work).
Anyway, putting this here in case it helps someone else.