The Linux multipath implementation

Original author : Christophe Varoqui
Creation : Feb 2004
Last update : Apr 2004

This document is shared under the OpenContent License (http://www.opencontent.org/opl.shtml)

Introduction

The most common multipathed environment today is a Fibre Channel (FC) Storage Area Network (SAN). This beasts can be found in most Datacenters. The lego blocks forming a SAN are :

• FC switches : core switches (multi protocols chassis and FC boards), or stacked switches linked by Inter Switch Links (ISL). This layer, the Fabric layer, can be subdivided in two major fabric types :

  • Simple fabrics : all storage ports can be routed to all hosts ports. Hosts and storage controllers can have a single attachment to the the fabric.

  • Dual independent Fabrics : two sets of switches are completely segregated (no ISL). They form two independent naming domains. The hosts and storage controllers must be attached to the two fabrics to ensure redundancy. This technology is used to provide the maximum availability as one fabric can be shut, for planned or unplanned reasons, without perturbation seen on the other one.

• FC storage controllers : most of them provide multiple ports to attach to the switches layer. The physical storage they drive is arranged in virtual drives we will refer to as Logical Units (LU). Each LU is provided by its host controller a unique identifier as per the SCSI standard. We will refer to this identifier as the World Wide Identifier (WWID) or World Wide Name (WWN).

• Host Bus Controllers (HBA) : the PCI / FC coupling adapters. A server can embed multiple HBA.

The multipath term simply means that a host can access a LU by multiple paths, the path being a route from one host HBA port to one storage controller port.

The Linux kernel choose not to mask the individual paths, that appear as normal SCSI Disks (SD).

Multipath awareness and support for an operating system can be described as :

• Provide a single block device node for a multipathed LU

• Ensure that IO are re-routed to available paths when a loss of path occurs, with no userspace process disruption other than an short pause.

• Ensure that failed paths get revalidated as soon as possible

• Ensure stability of the naming of that node

• Configure the multipaths to maximize performance : spread IO when possible path switching is free, and not spread when it's costly.

• Configure the multipaths automatically early at boot to permit OS install and boot on a multipathed LU

• Reconfigure the multipaths automatically when events occur

• The multipath must be partitionable

• In the Linux way : simple and hardware vendor agnostic

All these goals are met by leveraging a set of userspace tools ans kernel subsystems :

• the kernel device mapper

• the hotplug kernel subsystem

• the udev device naming tool

• the multipath userspace configuration tool

• the multipathd userspace daemon

• the kpartx userspace configuration tool

• the early userspace Linux kernel boot environment

The rest of this document describes these individual tools and subsystems and their interactions.

Device Mapper

Starting with Linux kernel 2.6, a new lightweight block subsystem named Device Mapper enables advanced storage management with style. This component features a pluggable design. At the time of this writing available plugins are :

• segments concatenation

• segment striping

• segment snapshotting

• segment mirroring, with and without persistence

• segment on-the-fly encryption

• segment multipathing

This last policy is the core component of the multipath tool chain. It is not included in the main kernel tree as of linux-2.6.5. It is part of a patchset maintained by Joe Thornber (thornber at redhat dot com) that can be downloaded at http://people.sistina.com/~thornber/dm/

This component fills the following requirements :

• Provide a single block device node for a multipathed LU

• Ensure that IO are re-routed to available paths when a loss of path occurs, with no userspace process disruption

So, let's see how it works.

The Device Mapper is configured one map at a time. A device map, also referred to as a table, is a list of segments in the form of :

The first 2 parameters of each line are the segment starting block in the virtual device and the length of the segment. The next keyword is the target policy (linear). The rest of the line is the target parameters.

The Device Mapper can be fed its tables through the use of a library : libdevmapper. Dmsetup, LVM2, the multipath configuration tool and kpartx all link this lib. A table setup boils down to sprintf'ing the right segment definitions in a char *. Should the DM user-kernel interface change from being ioctl based to a pseudo filesystem, the libdevmapper API should remain stable.

The multipath target parameters are :

• $0 : the “multipath” keyword

• $1 : the number of priority groups for the segment

• $2 : the first priority group parameters :

  • $2.1 : the scheduler to be used to spread IO inside the PG

  • $2.2 : the number of paths in the PG

  • $2.3 : the number of paths parameters (usualy 0)

  • $2.4 : the paths list for this PG

• $3 : next priority group

Upon device map creation, a new block kernel object named dm-[0-9]* is instantiated, and a hotplug call is triggered. Each device map can be assigned a symbolic name when created through libdevmapper, but this name won't be available anywhere but through a libdevmapper request.

hotplug subsystem and udev

Starting with Linux kernel 2.6, the hotplug callbacks are communized through the presence of a new pseudo filesystem : sysfs. This filesystem presents to userspace kernel objects like bus, driver instances or block devices in a hierarchical and homogeneous manner. The hotplug subsystem is leveraged by triggering a /sbin/hotplug call upon file creation and deletion in the sysfs filesystem.

For our needs this facility provides :

• userspace callbacks upon paths additions and suppressions

• userspace callbacks upon device maps additions and suppressions

Since linux-2.6.4, and its integration of the transport class for sysfs, it can also provide callbacks upon FC transport events like a “Port Database Rescan”. These callbacks could now be used to trigger SCSI Bus Rescan to bring a fully dynamic storage layer. (Or am I wrong ?)

Here is how we use this callbacks for the multipath implementation :

• The paths additions and suppressions callbacks are routed to the multipath userspace configuration tool described later. This tool ensure the multipath maps are always up-to-date with the fabric topology, and this ensure optimal performance by adding new paths to the existing maps as soon as they become available.

• The udev userspace tool is triggered upon every block sysfs entry creation and suppression, and assume the responsibility of the associated device node creation and naming. Udev default naming policies can be complemented by add-on scripts or binaries. As it does not currently have a default policy for device maps naming, we plug a little tool named devmap_name that resolve the sysfs dm-[0-9]* names in map names as set at map creation time. Provided the map naming is rightly done, this plugin provides the naming stability and meaningfulness required for a proper multipath implementation.

• The userspace callbacks upon device maps additions and suppressions also trigger the kpartx tool to create the device maps over eventual partitions

Udev is a reimplementation in userspace of the devfs kernel facility. It provides a dynamic /dev space, with an agnostic naming policy. Greg Kroah-Hartman is the main developer and maintainer of this package. It can be found at http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/

To synthesize what implementation details these subsystems fill :

• Ensure naming stability of the multipathed LU

• Reconfigure the multipaths automatically when hotplug events occur

• The multipath are partitionable

multipath userspace config tool

This tool is responsible for the paths coalescing and device maps creation. As seen earlier, it is triggered by the hotplug calls on new paths additions and suppressions. It must deal with hardware specifics and abstract them for the others subsystems.

Here is how it works :

• draw a list of all available devices in the system through a sysfs scan. For each device, get a bunch of information :

  • Host / Bus / Target / Lun tuple

  • SCSI Device Strings : Vendor / Model / Revision

  • SCSI Serial String

• Considering the informations fetched, elect a LU UID fetching method and an IO spreading policy. Ie deal with hardware specifics.

• Get the LU UID with the elected method. This method defaults to the standard 128 bit EUID found in the EVPD 0x83 inquiry page of the device.

• Coalesce the paths to form the multipath structs, ie group paths by UID.

• Create and name the device maps associated with the multipath structs with the selected IO spreading policy

There are currently 4 spreading policy implemented :

• failover : 1 path per priority group. IO thus get routed to one path only.

• multibus : 1 priority group containing all paths to the LU. Brings the maximum spreading, but assumes that all paths are excitable without penalty.

• group_by_serial : 1 priority group per storage controller (serial), paths through one controller are assigned to the associated PG. This policy applies to controllers that impose a latency penalty on LU management hand-over between a pair of redundant controllers.

• group_by_tur : 2 priority groups. 1 for active paths and 1 one for failed paths. The active paths' PG is alway set first, so that IO only get routed to known working paths, as long as the PG doesn't get empty. This policy is dedicated to dual storage controllers that can't handle a LU at the same time on the two controllers. In this case, half the paths are failing. This policy groups them in the secondary PG and set their state as active. In case of a LU handling switch, all path in the primary PG fail at once, and IO get routed to the secondary PG whose paths are now really active.

Policy assignment can be set manually at the command line. This one sets the policy to multibus for the multipath containing the device with major 8 and minor 0 (/dev/sda)

Alternatively, default settings are defined into the multipath tool source code. Here is how it is :

These policies can optionally be stored in a config file (/etc/multipath.conf). If the file is present, its content override the in-code defaults. All multipath hardware you will use must be described in either the config file if you have one, or the code defaults table if not, for the multipath tool to work.

Default settings are applied per LU. The LU is characterized by its vendor_id / product_id tuple ( respectively col1 & col2). The third column sets the default policy and the last set the default function used for fetch the unique identifier necessary for coalescing the paths.

The device maps naming policy is “name by LU WWID”.

The first section shows the list of all paths detected on the host. The second shows the multipath structs produced by the coalescing logic. The third shows the device maps submitted to the Device Mapper.

Of interest is the creation of device maps for single path LU : this enables systems to operate normally when booted in a degraded SAN context. The missing paths will be added to the maps when they become available.

This tool is packaged with udev, in the extras/ section. The devmap_name tool and multipathd daemon are distributed in the same tree. Alternatively the multipath-tools package can be found at http://christophe.varoqui.free.fr/

The implementation requirements filled by this tool are :

• Ensure naming stability of the multipathed LU (in complement of udev)

• Configure the multipaths to maximize performance : spread IO when possible path switching is free, and not spread when it's costly.

• Configure the multipaths automatically at boot

• Reconfigure the multipaths automatically when events occur (in complement of hotplug)

the multipathd daemon

This daemon is in charge of checking the failed paths in case they come up. When this occurs, it will reconfigure the multipath map the path belongs to, so that this map regain its maximum performance and redundancy.

This daemon executes the external multipath config tool when events occur. In turn, the multipath tool signals the multipathd daemon it is done with devmap reconfiguration, so that it can refresh its failed path list.

A logical diagram can be found at http://christophe.varoqui.free.fr/

The implementation requirements filled by this daemon are :

• Ensure that failed paths get revalidated as soon as possible

• Reconfigure the multipaths automatically when events occur

kpartx userspace config tool

This tool, derived from util-linux' partx, reads partition tables on specified device and create device maps over partitions segments detected. It is called from hotplug upon device maps creation and deletion.

kpartx is part of the multipath-tools package distributed on http://christophe.varoqui.free.fr/

Early userspace

Starting with Linux kernel 2.6, an early userspace execution environment is available in the name of initramfs. The grand plan is to package a set of tools in a cpio archive concatenated to the kernel. This archive is expanded in an in-memory filesystem early at boot and the tools are called to assume logics that previously belonged in the kernel : dhcp requests and setups, nfsroot stuffing ...

Being concatenated to the kernel, the size of this archive matters a lot. A slim libc implementation is required and provided in the name of klibc, maintained by Hans Peter Anvin.

The multipath implementation toolbox fits in this early userspace definition. Udev, multipath and kpartx are linked against klibc and can be packaged with the cpio archive to bring up the multipathed device early enough to boot on.

So is met the last multipath implementation requirement.

Quick installation guide

• Download and unarchive a recent 2.6 kernel

• Download, unarchive and apply the associated udm patchset from http://people.sistina.com/~thornber/dm/

• Configure, compile and install the kernel, reconfigure the bootloader if needed

• Download udev, at least 017, from http://ftp.kernel.org/pub/linux/utils/kernel/hotplug/

• Compile udev with klibc linking (edit the Makefile to uncomment KLIBC=true). Make install.

• Download the last multipath-tools package from http://christophe.varoqui.free.fr/, unpack udev*/extras/, make and make install

• Reboot under the new kernel and see the magic operate

Tested environments

Last updated on Mon Feb 16 2004