***********UPDATE PLEASE REFER TO THE POST AT THIS LINK FOR UPDATED INFORMATION ON THIS SCRIPT***********************
The most common request I get for scripts here at Pure Storage is an UNMAP script using PowerCLI. I have a basic one here that does the trick–UNMAPs Pure Storage volumes in a vCenter. That being said it is pretty dumb–doesn’t tell you much about what happened other than what volumes it is reclaiming (or not reclaiming) and moves on. A few requests have come in recently for something a little more in-depth. Most notably the ability to see how much space has been reclaimed. This information cannot be gathered from the VMware side of things–it has to come from the FlashArray.
There are two options here–either use our REST APIs or use our PowerShell toolkit to get this information (which just wraps the REST calls). For this script I chose to use the REST API directly from within PowerShell. What this script does is:
- Connects to the vCenter and FlashArray
- Finds all of the datastores and counts how many are actually Pure Storage volumes (NAA comparison)
- Iterates through all of the datastores
- Skips it if it is not Pure
- If it is, the current data reduction ratio is reported and so the is current physical written capacity on the FlashArray.
- Runs UNMAP on the datastore
- Reports the new data reduction and physical space after UNMAP completes and how much was reclaimed.
- Repeats for the rest of the volumes.
The script reports all of this to the console window, but it always throws it in a log file through add-content. If you don’t want it to return the info to the console, simply delete the write-host lines. If you don’t want it to log, delete the add-content lines.
There are a few required parameters–vCenter information (IP, username, password), FlashArray info (IP, username, password), UNMAP block count and a log file location. These are hard-coded parameters, but that can easily be changed by altering it to a read-host.
You may also note that after each UNMAP the script sleeps for 60 seconds–I do this so I make sure the FlashArray has time to update its information right after the UNMAP. 60 seconds is VERY conservative–probably 10 or so is fine, so feel free to mess with that number if you don’t like waiting. I also have another sleep at the end of each datastore operation to give a quick chance to review the latest results before it starts spewing the next datastore information on the screen (note this update didn’t make it into the video demo below–it doesn’t wait after each datastore).
See the script in action below. Essentially I am deleting a bunch of VMs across 4 datastores and then running the UNMAP. You can see the space get reclaimed on the FlashArray.
Note: You need particular access (see a blog post about that here) to vCenter to run UNMAP. For the FlashArray only Read Only is needed (higher of course is fine too).
Get the script here:
https://github.com/codyhosterman/powercli/blob/master/unmapsdk.ps1
Cody,
We’ve tried to utilize your script, put in our site variables, went to run it, and we get the following. We are we missing? Is there a prior step to enable the REST APIs or anything?
Invoke-RestMethod : {“msg”: “invalid credentials”}
At line:46 char:13
+ $ApiToken = Invoke-RestMethod -Method Post -Uri “https://${purevip}/api/1.2/auth …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
Invoke-RestMethod : {“msg”: “Invalid api_token”}
At line:50 char:1
+ Invoke-RestMethod -Method Post -Uri “https://${purevip}/api/1.2/auth/session” -B …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
Seems like the FlashArray doesn’t like the user credentials supplied in the token–are you using pureuser or external (AD) credentials?
I was using AD credentials. Is there specific credentials to Pure this user would need? I assumed to Pure it only needed RO, and to VCenter a bit more rights, so in that case I was using my own VCenter login.
This script doesn’t make any change calls so read only should be fine. I have’t tested it out with read only though. Let me run a test and see if I have the same issue.
I just tested it as read only and it is working for me. Something must be up with the credentials maybe?
I’m almost positive ours is the same issue as the poster above, dealing with self signed certs. I’m going to try with your new line of the script tomorrow.
Would still using Pure self signed certs have anything to do with it? I have the same issue with the REST Snap Script. and it doesn’t matter whether I use AD credentials or pureuser. I am trying to run it against Purity 4.1.2
This line added to a script should solve that issue:
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
Cody can this line go anywhere?
and am I mistaken Cody, isn’t this line already in your script? Around line 42.
Sorry yeah it is in this script–that note was meant for anyone who didn’t already have it. but put it near the start for those writing their own
Ok Cody I think we got it. We were making a mistake with credentials all along. But of course we hit something else. Should we be worried this message came up/and is taking a while to go away/run:
Connection to FlashArray successful
Connection to vCenter successful
Initiating VMFS UNMAP for all Pure Storage volumes in the vCenter
Searching for VMFS volumes to reclaim (UNMAP)
Found 174 VMFS volume(s).
get-scsilun : 3/17/2015 9:02:16 AM Get-ScsiLun ScsiLun info is available only for Vmfs datstores.
At line:49 char:26
+ $purevols = $datastores |get-scsilun |where-object {$_.CanonicalName -like “naa. …
+ ~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Get-ScsiLun], VimException
+ FullyQualifiedErrorId : Client20_StorageServiceImpl_GetScsiLun_InvalidDatastoreType,VMware.VimAutomation.ViCore.Cmdlets.Commands.Host.GetScsiLun
Nah this probably means you have NFS in your environment. I have updated the script so that this error doesn’t appear anymore (it now can detect and skip NFS) in the latest version.
We have 174 VMFS volumes, 18 total Pure datastores. Would you recommend any other tweaks to the script based on my site’s details?
Looks like it runs solidly, Cody. I mentioned our datastore count just because, as I am sure you’re aware, the checks on whether they are Pure or not, just take time. If there’s a way to skip them quicker, great, if not, that’s fine. Before your script we had nothing but a manual process anyway, and this definitely beats that. Thanks again.
and I spoke to soon. We ran the script against a test cluster of ours, it found the Pure datastore as well as ID’d the local datastore as not Pure, but it then spit this error at us:
This volume has a data reduction ratio of 9.872 to 1 prior to reclamation.
This volume has 12.066 GB of data physically written to the SSDs on the FlashArray prior to reclamation.
Initiating reclaim…Operation time will vary depending on block count, size of volume and other factors.
Permission to perform this operation was denied. Required privilege ‘Host.Config.Storage’ on managed object with id ‘ReflectManagedMethodExecuter-ManagedMethodExecuter-34’.
At line:132 char:13
+ $esxcli.storage.vmfs.unmap($UNMAPBlockCount, $datastore.Name, $null) …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], NoPermission
+ FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.NoPermission
The latest version of the script removes that first sort which takes forever (starts with $purevols =) and it isnt really necessary. That will save a lot of time. Remove the subsequent line too that has $purevols in it. Or just download my latest version.
For the error, how long did it take to get to that step? Maybe the vCenter connection timed out?
Actually it gives me it immediately. It’s a very small VCenter with only 1 ESX host, 1 non pure datastore, and 1 Pure DS. And actually I thought I downloaded your latest version. From this link, correct, from 3 days ago? https://github.com/codyhosterman/powercli/blob/master/unmap.ps1
Just to clarify, the vcuser and pureuser I’m using has RO access to VCenter and the Pure array. That is enough, correct?
Ah that will do it. You only need RO on the FlashArray but the UNMAP process in ESXi is a change process so you need admin rights to that object. I will add that in the post.
Cody,
Can you speak to a permission we could create that could limit the rights to just this?
Let me clarify: a role in VCenter for specifically this script.
Let me look into it and I will get back to you.
Hey Cody, any luck on that role? Sorry to be a pest. Thanks a lot in advance.
Not a pest at all! Sorry my lab was being moved and I just got it up but then went on vacation for a week and a half and just got back. I plan on looking into this in the next few days. Apologize for the delay.
Vacation? What? Who ever heard of vacations…..
Thanks again Cody. Looking forward to automating this process.
Haha yeah seriously… So I finally got around to this, apologies for the delay.
https://www.codyhosterman.com/2015/04/20/required-esxi-permissions-for-unmap-through-powercli/
Hi Cody, great script, thank you. However, we have a vcenter infrastructure spread across multiple sites, with a PURE array at each site. Any way to allow for it working through more than one pure array or should I just have a script for each site and just connect to a single VM host, rather than vcenter server, and specify the PURE array for that site?
Kind regards,
Neil
Yeah you can create connections for multiple arrays. You will just need to create multiple session /connection variables, one for each array. You will just need to run the connect/session create commands multiple times, storing them in differently named variables.
Great, thanks for getting back to me. 🙂
We have deleted vm in vcenter, which was using pure datastore, now in vcenter, we are getting proper free space, but in pure storage still volume is showing used space 157 GB, so how can we reclaim free space on pure storage, our purity version is 6.1.16 & we are using vcenter7.0.3.