I jumped on a call the other day to talk about iSCSI setup for a new FlashArray and the main reason for the discussion had to do with co-existence of a pre-existing array from another vendor. They were following my blog post on iSCSI setup and things didn’t quite match up.
To setup multi-pathing (the recommended way) for Software iSCSI is to configure more than one vmkernel port that each have exactly one active host adapter (physical NIC). You then add those vmkernel ports to the iSCSI software adapter and the iSCSI adapter will then use those specific NICs for I/O transmission and load-balance across those ports.
The issue appeared at the step of configuring the vmkernel ports. As I mentioned above, each vmkernel port must have only one active NIC and the rest must be designated as in-active (not even standby). If they are not configured as such, they will be filtered out of the available vmkernel ports that can be added to the iSCSI adapter:
Or if it is added because it is configured properly and then it is changed later (let’s say a new active adapter was added) it will become non-compliant and marked as not usable by the iSCSI adapter until it is fixed.
In this case the customer had one pre-existing vmkernel port with two active adapters AND were still using the Software iSCSI Adapter. There were no network port bindings in the iSCSI adapter though but it still saw storage fine, when they got to the step in my blog post to create the network port bindings it couldn’t be done because the vmkernel port was non-compliant.
The iSCSI adapter was using their previous storage fine, saw the paths and presented the volumes. So this begs the questions–what NIC/vmkernel ports was it using and how was it multipathed? Can you then actually use vmkernel ports with active-active host adapters? Is everything a lie? Santa…?
So the answer to all of this is “kinda, but not really.”
Network port bindings in a Software iSCSI Adapter serves two main purposes:
- Specifically indicate what vmkernel ports and by association, physical NICs (host adapters), iSCSI should use
- Provide for the iSCSI Adapter to manage multi-pathing
So essentially, you do not have to create network port bindings to get the Software iSCSI Adapter to work and access storage. If no port bindings exist, the iSCSI Adapter will simply look for the first vmkernel port it finds (vmk0, then 1, then 2 etc.) that can communicate with the configured targets in the static target list in the iSCSI adapter refer to the vmkernel route table for the specified connection.
So while it is predictable to what vmkernel adapter it will choose, it is not obviously apparent by looking at the vmkernel port or the iSCSI adapter. There is no configured relationship between them really.
***UPDATE: The choice of vmk ports is more sophisticated than I mention above. It has to do with the vmkernel routing table as Andy Banta notes in the comments. Thanks so much Andy!***
Furthermore, the multipathing is not really active/active–even though there are two active adapters for the vmkernel port. Instead a single vmkernel port is chosen and only one of the host adapters/NICs is used at a time. So basically this turns the active-active host adapters into active/standby–so their configuration state is somewhat misleading. There is no load balancing across the adapters–they are just failed over when there is a failure.
So below we have zero network port bindings but we do have a vmkernel port with two active-active physical adapters (as indicated by the two orange lines).
If I were to run I/O to an iSCSI device we will see in esxtop that only one NIC is being used but the I/O is still spread across the array front end. So it is multipathed on the array but not on the host–all of the I/O is going through one physical adapter. The first image shows esxtop and the fact that only one vmkernel adapter (vmk1) is being used and only vmnic2. The second image shows the balance on the FlashArray across both controllers. You can see they are pretty well balanced.
Now what if we configure network port mappings in the Software iSCSI Adapter? In this case I have two vmkernel ports each with their own physical adapter (vmk1 and vmk2, with vmnic2 and vmnic3 respectively).
Let’s restart the I/O. You can see in esxtop that both vmkernel ports and their respective physical adapters are bring used and quite evenly. Also the FlashArray controllers remained balanced. Lastly note the increase in throughput by leveraging two physical adapters.
Besides improvements in multipathing, assigning network port bindings help make sure you do not accidentally remove access for the software iSCSI adapter by deleting the wrong vmkernel port. If you don’t have network port bindings and instead are just having the iSCSI adapter pick one, there is no protection against deletion because the Web Client or CLI or whatever doesn’t know about the relationship.
If you try to remove a vmkernel port that is not bound to the iSCSI adapter the “Analyze impact” check doesn’t warn you:
If it is bound to an iSCSI adapter you do get a warning:
Okay, so I think you get the point. Let’s all use network port binding from now on.
Before I go–a few recommendations:
- Do have Round Robin enabled and the IO Operations Limit set to one.
- DO set the iSCSI “LoginTimeout” to 30 seconds and “DelayedAck” to disabled. Set these on the dynamic target and let it cascade to the static targets. DO NOT set them on the iSCSI adapter itself–this will provide for the possibility of different targets that might have different recommendations down the road.
- DO use network port bindings to more than one physical adapter if possible. Redundancy and performance.
- DO NOT configure the physical adapter teaming and failover only at the virtual switch layer and then let the vmkernel ports inherit them. If you add a new active adapter at some time later the vmkernel ports will inherit them and become non-compliant. Always override the teaming and failover settings on the vmkernel ports themselves and choose just one active physical adapter.
- OPTIONALLY use Jumbo Frames–this is at your discretion.
A few comments here: If you’re using the default path to strorage (not using bound ports), the vmknic that is used is determined by the ESX vmkernel route table. It is not jsut the first vmknic that can see the storage.
Also, the sweet spot on round-robin IO operations is rarely 1. In many cases, it’s closer to 10. This is partially due to the path switch overhead in the vmkernel and partially in how queues are handled by the specific storage
I was hoping you would chime in! Thanks for reading Andy–I am new to a lot of this iSCSI stuff so I was hoping to be fact checked 🙂
I am planning on doing some performance testing with the IO operations with iSCSI shortly. That is good to hear–I will focus my testing in that area.
No problem, Cody. I know you’re not confused, but I just want to make sure anyone else who makes it all the way to the comments knows I’m at SolidFire at this point, not still whipping up iSCSI stuff at VMware.
Cody, still recommending round robin IO operations limit set to 1? Saw that you were going to do some testing after Andy stated the sweet spot was rarely 1 on VMware ESX.
Of note, I’m setup exactly this way, but seeing about 1/3rd of the traffic on one vmk and vmnic and 2/3rds on the other for some reason!
Yes. We have seen no significant difference in changing it to anything higher than 1. There can be a very slight CPU difference but it is rarely even detectable. At this point we see no reason to change that recommendation. Anyways, moving forward we will likely be changing our recommendation to the new latency-based policy, which is certainly superior to any configuration using the “IOPS” based policy. Though we still have some testing to do before we make that recommendation.
If you are seeing that type of imbalance, it could be that either your logical paths are imbalanced across your physical paths for some reason (VLANs?) Or potentially some devices are not set to round robin?
Reblogged this on SutoCom Solutions.
Thanks very informative. So If I have two NICS should I have one vSwitch with two vMkernels mapped to one vmnic and a second switch configured the same way?
Thanks
You’re welcome! Doesn’t matter. You can have two vSwitches, one with a NIC and each one vmkernel, or one vSwitch with both NICs. If you do the latter you just need to make sure the active/unused NIC configs inside the vmkernel ports are correct. One vmkernel port per NIC is fine either way.
In general you should use port binding only if both vmkernels and iSCSI targets are in the same subnet.
Please follow VMware articles:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038869
http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-multipathing-configuration-software-iscsi-port-binding-white-paper.pdf
I have followed the steps tried to set up the port binding with two physical NICs but found that only one vmkernel NIC is working. If I unplug the cable of the working NIC, the storage is disconnected. How can I troubleshoot it?
I would try to use vmkping, and ping the iSCSI target by specifying the exact vmk to ping through. I suspect there might be a routing issue. Or maybe a VLAN problem. You might want to open a ticket with Pure Storage support so that they can help you.
The vmkping with the two vmkX interfaces works. The storage and the hosts are connected with a standard switch. There is no VLAN settings on the switch. Besides open a ticket, is there any way to diagnose the problem on my own?
Sorry for the delay, have been traveling all week. Hmm. If one nic doesnt see the storage but the other one does, but the pings work which is strange. Can you ping all of the iSCSI target IPs from both vmk? Which IP did you ping?
It’s ok. I have successfully set it up with the commands mentioned in http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-multipathing-configuration-software-iscsi-port-binding-white-paper.pdf. It is weird that configuring it via ESXI web client doesn’t work.
Hi Cody,
In case of adding a FlashArray to an existing iSCSI SAN connected to vSphere hosts, is it best practice to just stick it on the iSCSI subnet of the existing array and hit it with the existing vmkernels or should it be placed in a separate VLAN/subnet on the iSCSI switches, with dedicated vmkernels added to the port binding?
Broadcast domain vs simplicity I guess, not sure which one is the best.
Cheers,
Hey! I would probably just keep it simple and re-use them. I am not aware of a benefit from separating them from a technical perspective. But if you need the traffic separated for compliance reasons maybe then
Where do you enable round robin on 6.5?
You can do it in the vSphere Web Client or use the CLI, I recommend using SATP rules to do it: https://www.codyhosterman.com/2014/05/changing-the-default-vmware-round-robin-io-operation-limit-value-for-pure-storage-flasharray-devices/
Hi,
I’m installaling a new infrastructure ( 2 x Esxi6.5 + 1 x Lenovo Storage DS4200 + At side a Synology all under ISCSi).
I would like to use more that only on Network card ( & be limited to 1GB).
The best practice, should to use Port Binding & get 2x 1GB, or 3X, etc..) . Under Vmware no problem, same for Synolgy.
By the way, for Lenovo DS4200, i can’t found the way to make a teaming of 2 Network interace on controller.
Each network card need to get is own IP ( Same range, or different range, i can choose).
In this case, i think that i can’t use Port Binding, right?
What’s the other option ?
Thanks a lot.
S.
Hi, I am not really familiar with Lenovo, so I would recommend asking their support, but that being said, nothing that you say, at this point, sounds like you cannot use port binding. There are a few restrictions in vSphere 6.0 and earlier, but with 6.5 those mostly went away. Regardless it doesn’t sound like you situation anyways (since you are on 6.5/not routing).
Hi,
Thanks for the reply.
I opened a ticket to Lenovo, we are exchanging some email, waiting they feedback now.
Btw, i’m looking for a workarround since the Port Binding is available on most storage system ( HP, EMC, & Low cost with Synology….).
Br
S.
This is really great. I’ve used this a couple of times now.
I have a somewhat unrelated question, but I can’t seem to find anything about it. In our environment, I see a fairly large difference between what the vmnic shows in top, and what the vmk shows. Bandwidth is similar, but number of packets and packet size is not. We have jumbo frames enabled. It’s almost like the vmkernel port is always using jumbo frames, but some other traffic is using the vmnic but not via the vmk. The vmnic is dedicated only to iSCSI, so I’m not really sure what’s going on. I would really appreciate any input. Love the site!
Thanks! Glad it is helpful! Hmm, if the bandwidth is the same I would suspect the vmk is consolidating the packets together somehow. Maybe LRO? Check this out https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.networking.doc/GUID-514BC149-CDE7-4E07-A922-E3DFB663DC13.html
I appreciate the response! I’ll check that out.
Thanks.