What’s New in vSphere 7.0 U2 Storage: Increased iSCSI Path Limit

A continuation of the series I started here.

What’s New in vSphere 7.0 U2 Storage: Multiple SPBM Configurations

This is a simple but important one. iSCSI path limits. Since the dawn of human, ESXi has had a disparity in path limits between iSCSI and Fibre Channel. 32 paths for FC and 8 (8!) paths for iSCSI.

To define a “path” it is a logical path to the target volume on the array.

C = “channel”. T = “target” and L = “LUN ID”.

So the key here is that this can be a logical path, not every path is entirely physically separate from the others.

In general, best practices dictate redundancy. Redundancy also often means (in my view) that one failure should not put you on the edge of death. Meaning that as long as both objects in redundant pair don’t fail a double failure will not take you down. And a single failure will not make HALF of your connections useless. To explain, the bare minimum is:

1. Two host physical adapters
2. Two switches
3. Two controllers
4. Two ports on each controllers

And importantly, each adapter should use both switches, and have connections to both controllers, and use two ports on each controllers.

So:

“as long as both objects in redundant pair don’t fail a double failure will not take you down“

If you lose both adapters on the host, that host is losing connection. If you lose both switches, well all hosts using them will go down. But if you lose one adapter and one controller, the other adapter will keep on chugging to the other controller. If you lose one, adapter, one switch, one controller–that other adapter will work through the surviving switch, to the surviving controller.

And a single failure will not make HALF of your connections useless

If everything is redundantly connected, one failure will not make half of your connection paths lose usefulness. If you are using one port on each controller and a cable goes bad, that entire controller becomes useless and now you are down to one more failure taking you down entirely. If you are using two ports, one cable does not make that controller useless to your hosts.

Cool.

So if we follow this math, what is the bare minimum path recommendation?

2 adapters x 2 controller x 2 ports (switches don’t inherently add paths–they just enable both adapters to use the same port on the array).

This is 8. 8 is the maximum in ESXi. This is not ideal–many, customers want additional paths for performance and/or resiliency. It is EASY to surpass this limit. And because 8 is not a hard limit–I would wager more than half of our iSCSI customers surpass it. While VMware has been good about not rejecting support cases due to “exceeding limits” they would be within in their rights to do so, so it makes customers uncomfortable, and well it makes me nervous too.

This is particularly exacerbated with stretched storage where paths connections instantly double (second array, second set of controllers, second set of ports etc).

This has led some customers down the paths (no pun intended) to use NIC teaming, or LACP with iSCSI which has its own complexities. Which if not dealt with carefully can lead to problems worse than simply exceeding supported limits.

Well this is something we brought up to VMware quite a bit for the benefit of our joint customers and were very pleased to see them take this task on of officially increasing this limit. And as of ESXi 7.0 U2 this limit is now 32 paths:

If you are an iSCSI customer, this is an attractive reason to start your upgrade plans.

With 32, we can now have configurations like:

4 adapters x 2 controllers x 4 ports = 32

2 adapters x 2 controllers x 8 ports = 32

4 adapters x 4 controllers x 2 ports = 32 (stretched)

2 adapters x 4 controllers x 4 ports = 32 (stretched)

Definitely plenty now!