Changing the default IOPS value for VMware Round Robin and Symmetrix Devices

A common recommendation from storage vendors is to change the default IOPS setting for VMwares’ Native Multi-Pathing (NMP) Path Selection Policy (PSP) Round Robin. The IOPS setting controls how many I/Os are sent down a single logical path before switching to the next path. By default this number is 1,000 I/Os. The VMAX recommendation is to set this to 1. The purpose of this blog post is not to debate the setting, but to help those who want to use it. Regardless, I have seen many customers benefit from this recommendation. Once they see a benefit they want to know–can I make this setting a default?

It is somewhat tedious to change, as it needs to be set on each device on each host. Making adding new devices a scripting nightmare. But the answer is yes! You can change it so new devices inherit a setting of IOPS=1 instead of 1,000. The storage guru at VMware, Cormac Hogan, blogged about this recently here:

http://cormachogan.com/2013/07/08/automating-the-iops-setting-in-the-round-robin-psp/

Since he posted about this I have had more than a few customers who use IOPS=1 ask me how to do this for Symmetrix. His instructions are generic to all arrays and some people were having a hard time getting this change to work for Symmetrix. So I thought I would give his process a try and see what I could do and could I get it to work.

Here it is (technically you can skip step 1 and 2 but for information purposes…):

Note that I am doing this locally via SSH on the ESXi service console, but this can be done just as easily with remote CLI tools or PowerCLI.

1. Identify the names for the Symmetrix model and vendor. This will be the same for all Symmetrix arrays and Cormac provided an easy process for finding this. Run a rescan and grep the vmkernel log for the information. After running a rescan I ran:

grep -i scsiscan /var/log/vmkernel.log

And got:

2013-08-19T21:35:50.429Z cpu7:8315)ScsiScan: 888: Path 'vmhba1:C0:T2:L7': Vendor: 'EMC     '  Model: 'SYMMETRIX       '  Rev: '5874'

The vendor being EMC and the model is SYMMETRIX. Note that the vendor and model have appended spaces, in his post Cormac stated these might need to be included in the command to change the default. This is NOT the case for the Symmetrix, you must remove the spaces.
T
2. Now identify the name of the Symmetrix Storage Array Type Plugin (SATP). This can be found by running:

esxcli storage core plugin registration list -N SATP

You will see what will probably be the first in the list:

Module Name Plugin Name Plugin Class Dependencies Full Path
------------------- ------------------- ------------ ----------------------------- ---------
vmw_satp_symm VMW_SATP_SYMM SATP

The SATP name is VMW_SATP_SYMM. Note this is all caps and is CASE SENSITIVE I found out.

3. Check the configuration of an existing device using esxcli. You can get the NAA a million different ways http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014953

esxcli storage nmp psp roundrobin deviceconfig get --device naa.60000970000195701248533030363332

This returns:

Byte Limit: 10485760
 Device: naa.60000970000195701248533030363332
 IOOperation Limit: 1000
 Limit Type: Default
 Use Active Unoptimized Paths: false

4. So the command to alter the SATP default to 1 would be:

esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1"

The command is universal to all Symmetrix arrays and is exactly what you will run. Feel free to copy/paste 🙂

5. Now the rule will not take into effect immediately for existing claimed devices.

For devices that have not been presented to the ESXi host yet they will get this configuration as soon as they are discovered by a rescan and will be claimed with the new IOPS parameter. For existing claimed devices they need to be unclaimed and reclaimed by the SATP. Two ways to do this. Simplest option is to just reboot the host–of course this means downtime etc, so plan accordingly. Rebooting will change the IOPS setting for all existing devices under that SATP upon power up. The other option is to manually unclaim devices one by one and then reclaim them. Unclaim a device like so (note that this can only be done to paths with no outstanding I/O):

esxcli storage core claiming unclaim -t device -d naa.60000970000195701248533030363332

Then load the claimrule:

esxcli storage core claimrule load

Then rescan the host.

6. Verify the existing device is configured properly now:

esxcli storage nmp psp roundrobin deviceconfig get --device naa.60000970000195701248533030363332
 Byte Limit: 10485760
 Device: naa.60000970000195701248533030363332
 IOOperation Limit: 1
 Limit Type: Default
 Use Active Unoptimized Paths: false

Yay!

7. All set on that host!! Check the below GIF for the process of a new device was just masked to the host and formatted and it automatically has IOPS set to 1.

8. To change or remove the rule you cannot simply just run the command again to change the rule back to 1,000 or whatever number. You must first remove the rule and then you can create a new one with a different number, or leave it without a rule to use 1,000 again.

 esxcli storage nmp satp rule remove -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1"

If you don’t remember what you set or want to take a look at the existing rules, run:

esxcli storage nmp satp rule list

Enjoy!