Posted by Cormac Hogan
Technical Marketing Manager (Storage)
If you've got an ALUA array, you've probably wondered what all those obscure configration settings mean in the esxcli device listing. I certainly have.
Let me show you what I mean.
~ # esxcli storage nmp device list -d naa.xxx
naa.xxx
Device Display Name: DGC Fibre Channel Disk (naa.xxx)
Storage Array Type: VMW_SATP_ALUA_CX
Storage Array Type Device Config: {navireg=on, ipfilter=on}{implicit_support=on;explicit_support=on; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=AO}{TPG_id=2,TPG_state=ANO}}
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba2:C0:T2:L100;current=vmhba2:C0:T2:L100}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T2:L100
Is Local SAS Device: false
~ #
Now, there does seem to be a lot of configuration options there, doesn't there? Before describing what they are, we first need to know a little bit about ALUA. ALUA, or Asymmetric Logical Unit Access, occurs when access characteristics of one controller port on a storage array that is presenting a LUN to a host has different access characteristics when compared to another controller port on the same array.
ALUA provides a way of allowing devices report the state of its ports to hosts. This state can then be used by hosts to prioritize paths and make failover/load balancing decisions.
Let's delve a little deeper into possbile ALUA characteristics.
Explicit/Implicit ALUA
ALUA devices can operate in two modes: implicit and/or explicit.
- Explicit ALUA devices allow host to use the "Set Target Port Group" task management command to set the Target Port Group (TPG) state. We will look at target port groups & their various states shortly. This is not configurable by the way, as it is an attirbute of the device.
- In implicit ALUA, a device's TPG state will be managed by the target device itself. This is not configurable either, as again it is an attirbute of the device.
Optomized/Non-Optomized Paths
ALUA is typically associated with Asymmetrical Active-Active (what we will term AAA) arrays. In an AAA array, both controllers can receive I/O commands (active-active), but only one controller can issue I/O to the LUN. This is the asymmetrical part. The opposite of a AAA array is a Symmetric Active-Active (SAA) array. These SAA arrays can issue I/O commands to the LUN via both controllers
The controller in an AAA array who can issue commands is called the managing controller. Paths to the LUN via ports on this controller are called optimized paths. I/O sent to a port of the non-owning controller must be transferred to the owning controller internally. This increases latency and impacts on the performance of the array. Therefore paths to the LUN via the non-managing controller are called non-optimized paths.
Target Port Group Support
TPGS provides a method for determining the access characteristics of a path to a LUN through a target port. It supports soliciting information about different target port capabilities & supports routing I/O to the particular port or ports that can achieve the best performance. TPGS is often used with arrays that handle load balancing and failover within the array controllers. Target Port Groups allows path grouping. Each port in the same TPG has the same port state, which can be one of the following state: Active/Optimized, Active/Non-optimized, Standby, Unavailable, and In-Transition. A TPG is defined as a set of target ports that are in the same target port asymmetric access state at all times.
The ALUA SATP plugin sends RTPG (Relative Target Port Group) commands to the array to get the device server's TPG identifiers and states.
Follow-Over Algorithm & Path Thrashing
To understand the ‘follow-over’ algorithm, let’s revisit a scenario which may result in path-thrashing. If a customer uses a fixed path policy on two hosts sharing a LUN from an active-passive array and sets a preferred path to array controller A on ESX A and a preferred path to array controller B on ESX B, the hosts would always try to activate the preferred path each time path states changes. When host B changes the path state, all I/Os from host A will see an error indicating they are talking to a standby controller. Host A will then try to activate the path again after which host B will see errors and so on). In both cases, if the ESX pulls the LUN back to its preferred controller, then the other ESX will just pull it back again and you are in a path thrashing state.
Using the 'follow-over' algorithm means that if an ESX host sees the array controller of its 'preferred‘ path become inactive on the array, then it is either because:
- Some other host lost access to the array controller of your preferred path.
- Some other host had their preferred path changed to use that array controller.
In ALUA, a host can activate a standby preferred path only if it caused a failover that initially put it into standby. A host just accepts that it cannot use its 'preferred' path in some cases. This is what "follow-over" means. You follow the lead of other hosts in certain cases of failovers or preferred path changes.
‘Follow-Over’ Algorithm Example
This is a bit complicated, so lets try to explain it again using a real life example. Say host A loses access to array controller port B0 (which is currently active & preferred across all hosts). It must then select a new path. It may try to select controller port B1 then, but if this is also dead, it might select port A0 and move all its I/O over to that path. Since we are switching I/O to a different controller, this will cause a trespass where LUNs may have to move ownership from controller B to controller A. If controller B later becomes available, then host A can switch back to its original preferred path since it was the cause of the original failover.
However other hosts, which had to move to controller A because of the actions taken by host A, cannot switch I/O back to controller B. Those other hosts do not know if host A can still access controller B (even if they can access it). But if host A can see the path to port B0 again, then it can try to switch back to the preferred path.
On the other hand, if it was another host which lost access to controller B and they moved their active path to another port on controller A, this also implies that host A loses access to controller B as I/O has to move to the new active path on controller A. In this case host A cannot pull back the LUN to its preferred path on controller B, even if it has access to it. This is because host A did not initiate the failover and make the paths to controller B standby.
Phew! That's quite a bit of information. However, with all of the above in mind, we should now be able to get a good understanding of the settings in the esxcli output. Let's remind ourselves again about them:
Storage Array Type Device Config: {navireg=on, ipfilter=on}{implicit_support=on;explicit_support=on; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=AO}{TPG_id=2,TPG_state=ANO}}
Let's skip navireg & ipfilter for the moment as they are not related to ALUA, and let's look at the other options.
- explicit_support: Shows whether or not the device supports explicit ALUA; this option cannot be set by the user as it is a property of the LUN.
- explicit_allowed: Shows whether or not the user allows the SATP to exercise its explicit ALUA capability if the need arises during path failure. This only matters if the device actually supports explicit ALUA (i.e. explicit_support is 'on'). This option is turned on using 'enable_explicit_alua' and turned off using 'disable_explicit_alua' (esxcli command).
- alua_followover: Shows whether or not the user allows the SATP to exercise the 'follow-over' policy, which prevents path thrashing in multi-host set-ups. This option is turned on using 'enable_alua_followover' and turned off using 'disable_alua_followover'. (esxcli command)
- There are two Target Port Groups. TPG_id=1 has all the Active Optomized paths (AO). TPG_id=2 has all the Active Non-Optomized paths (ANO)
A final note on the navireg and ipfilter entries. These are not related to ALUA, but are related to the specific SATP used. The more observant of you will have figured out that this is an EMC CX series storage array and the Storage Array Type is VMW_SATP_ALUA_CX. The Storage Array Type Device Config entries {navireg ipfilter} are specific to the VMW_SATP_ALUA_CX (as well as some other EMC SATPs). The accepted values are: navireg_on, navireg_off, ipfilter_on, ipfilter_off. To complete the post, I'll include some detail about those options for the more curious amongst you.
- navireg_on starts automatic registration of the device with Navisphere
- navireg_off stops the automatic registration of the device
- ipfilter_on stops the sending of the host name for Navisphere registration; used if host is known as "localhost"
- ipfilter_off enables the sending of the host name during Navisphere registration
That completes the post. Hope those configuration settings make a bit more sense now.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: 
@VMwareStorage
