Mastering Linux DM Multipathing and iSCSI Block Storage on SUSE Linux Enterprise Server
Linux Device Mapper Multipath (DM Multipath) is a feature used in enterprise storage environments to provide redundancy and improved performance by allowing a system to access storage devices over multiple physical paths.
For this post I will use iSCSI block storage provided by a Microsoft iSCSI Target server to show how we can set up DM multipath on a SUSE Linux Enterprise Server.
Finally we connect an iSCSI target (virtuell disk) to a SUSE Linux Enterprise Server by using multiple NICs/IP addresses on the iSCSi Target server to provide at least two physical paths to connect the disk and showing how DM Multipathing works.
About how to set up DM multipath on Red Hat Enterprise Linux you can read my following post.
Set up the iSCSI Target Server
About how to set up a Microsoft iSCSI Target Server step by step you can also read my following post.
Set up the iSCSI Initiator on SUSE Linux Enterprise
The iSCSI initiator on SUSE Linux Enterprise Server we can install by using the following packages:
- open-iscsi
- yast2-iscsi-client
The iSCSI initiator can be used to connect to any iSCSI target. The configuration of iSCSI initiator involves two major steps: the discovery of available iSCSI targets and the setup of an iSCSI session. Both can be done with YaST.
Install open-iscsi.
# zypper install open-iscsi
Install yast2-iscsi-client.
# zypper install yast2-iscsi-client
Open Yast and navigate to Network Services -> iSCSI Initiator.
Switch to the Service tab.
Under After writing configuration, define what to do if there is a configuration change. Bear in mind that available options depend on the service current status.
The Keep current state option keeps the service in the same status.
I will select Start.
In the After reboot menu specify action that will take place after reboot:
- Start on boot – to start the service automatically on boot.
- Start on demand – the associated socket will be running and if needed, the service will be started.
- Do not start – the service is not started automatically.
- Keep current settings – the service configuration is not changed.
I will select Start on boot.
Specify or verify the Initiator Name.
Specify a well-formed iSCSI qualified name (IQN) for the iSCSI initiator on this server. The initiator name must be globally unique on your network. The IQN uses the following general format:
iqn.yyyy-mm.com.mycompany:n1:n2
where n1 and n2 are alphanumeric characters. For example:
iqn.1996-04.de.suse:01:a5dfcea717a
The Initiator Name is automatically completed with the corresponding value from the /etc/iscsi/initiatorname.iscsi file on the server.
Connect to iSCSI Target manually by using Yast
To connect to an iSCSI target, select iSCSI Initiator in YaST and then select the Discovered Targets tab.
Click Discovery to open the iSCSI Initiator Discovery dialog.
Enter the IP address and change the port if needed. The default port is 3260.
If authentication is required, deselect No Discovery Authentication, then specify the credentials for Authentication by Initiator or Authentication by Targets.
Because my iSCSI Target server is using two NICs to provide multipathing, we will see the target (same virtuell disk) for each of these NICs and its IPs.
I will first just connect to the iSCSI Target by using its first IP address 10.0.0.40 selecting and clicking Connect. Afterwards the iSCSI Target is shown up under connected targets.
To connect also the disk by using the iSCSI Target servers second IP address 10.0.0.41, we first need to configure multipath on the client, otherwise we risk data corruption as the disk will shown up finally as two different disks on the client.
Further on the iSCSI Target server we can see our new SUSE Linux Enterprise client (iSCSI initiator) is connected to the iSCSI Target.
The disk is now shown up in Linux, we can therefore use the lsblk (list block devices) command.
# lsblk
Or by using the lsscsi command.
# lsscsi
We can also use the iSCSI Administration Tool (iscsiadm) to list active iSCSI sessions on the system.
# iscsiadm -m session or # iscsiadm -m session -P 0 -P printlevel or with more information # sudo iscsiadm -m session -P 3
Source: https://documentation.suse.com/sles/15-SP6/html/SLES-all/cha-iscsi.html#sec-iscsi-initiator
Enable Multipathing
Below I want to show how to manage failover and path load balancing for multiple paths between the servers and block storage devices by using Multipath I/O (MPIO).
When trying to connect to the same iSCSI target on the iSCSI Target servers second IP address, I will get the following warning message.
So as mentioned earlier, we first need to configure DM multipathing on the client to connect to the same target by using the iSCSI Target servers both IP addresses.
Multipathing is the ability of a server to communicate with the same physical or logical block storage device across multiple physical paths between the host bus adapters in the server and the storage controllers for the device, typically in Fibre Channel (FC) or iSCSI SAN environments.
Linux multipathing provides connection fault tolerance and can provide load balancing across the active connections. When multipathing is configured and running, it automatically isolates and identifies device connection failures, and reroutes I/O to alternate connections.
Multipathing provides fault tolerance against connection failures, but not against failures of the storage device itself. The latter is achieved with complementary techniques like mirroring.
The traditional, generic implementation of multipathing under Linux uses the device mapper framework. For most device types like SCSI devices, device mapper multipathing is the only available implementation. Device mapper multipath is highly configurable and flexible.
Source: https://documentation.suse.com/sles/15-SP6/html/SLES-all/cha-multipath.html
To configure and manage Device mapper multipath we can use the multipath-tools on Linux, on SUSE Linux Enterprise they are installed by default.
Next we need to enable the multipath daemon/service.
# sudo systemctl enable multipathd --now
Checking the status.
# sudo systemctl status multipathd or # sudo systemctl status multipath.service
Create or edit the /etc/multipath.conf file.
A basic configuration will looks like:
defaults {
user_friendly_names yes
find_multipaths yes
}
blacklist {
devnode "^sda" # Avoid multipathing the local boot disk
}# vi /etc/multipath.conf
A better and more secure way to exclude disks from multipathing, is by using its World Wide Name (WWN) or World Wide Identifier (WWID).
To determine the WWID of the disk, we can use the lsscsi command.
# lsscsi -i -s --scsi_id|-i show udev derived /dev/disk/by-id/scsi* entry --size|-s show disk size
As we should avoid multipathing the root/boot device unless specifically required, I will add also the second /dev/sdb disk which contains the root file system to the blacklist. Both by using its WWID instead its device label.
Rebuild initramfs (If Root is on Multipath or for Boot Support)
This setup guarantees that multipath devices are available early in the boot process, which is crucial for maintaining the integrity of the storage setup and preventing potential issues.
# dracut -f # systemctl restart multipathd
Check if both disk are not used by multipathing. Looks good, the multipath -ll command shouldn’t show our /dev/sda and /dev/sdb disks.
# lsblk # multipath -ll -ll show multipath topology (maximum info)
We can now connect the iSCSI target (disk) by using both iSCSI Target servers IP addresses for DM multipathing.
Connecting the targets like previously shown by using yast.
Because multipath is enabled, the disk connected through the iSCSI Target servers both IP addresses (iSCSI Targets), the disk is shown up just once and its device mappers name mpatha (mpathN).
We can check this by using the multipath and lsblk command.
# multipath -ll -ll show multipath topology (maximum info)
In the output of multipath -ll, the status= field reflects the state of the path group, and values like active and enabled have distinct meanings related to how Device Mapper Multipath manages I/O paths.
status=active This path group is currently being used for I/O.
status=enabled This path group is available, but not currently in use. It’s a standby or failover group ready to take over if the active group fails.The policy= in multipath -ll output refers to the path selector policy, which defines how multipath decides which paths to use for I/O.
service-time Distributes I/O based on the response time of each path. Paths that return requests faster get more I/O. Designed to optimize throughput and latency in environments where path performance varies e.g., asymmetric storage, mixed-speed links, or overloaded paths.
Other policies available are: round-robin which sends I/O in a rotating fashion across all active paths. queue-length which chooses path with shortest I/O queue. failover which uses one path; switches only if it fails.
The policy mentioned above we can change within the /etc/multipath.conf file and its defaults section. Below for example I will change it from the default service time to round robin.
The 0 here is a placeholder, some implementations accept parameters, but it’s typically safe to leave as is.
defaults {
user_friendly_names yes
find_multipaths yes
path_selector "round-robin 0"
}Run the following commands after changing the /etc/multipath.conf file.
# dracut -f # systemctl restart multipathd
This means /dev/sdc and /dev/sdd are now managed as a single multipath device (/dev/mapper/mpatha).
# lsblk
# ls -la /dev/mapper
Finally we can now mount the disk and create the file system if not already done. I will first need to create a file system on the disk.
# # mkfs.ext4 /dev/mapper/mpatha
Now I can finally mount the new disk. Therefore I was first creating a new folder on the root file system named data to mount the disk to.
# mount /dev/mapper/mpatha /data/
Looks good.
# df -hT
To make the mount persistent we also need to add an entry in the /etc/fstab file.
The _netdev option is useful if the multipath device is over iSCSI or FC and depends on network/storage being ready.
The mount is deferred until the network is up. Without _netdev, the system might attempt to mount the filesystem before the network (or multipath stack) is ready, leading to boot errors or dropped mounts.
/dev/mapper/mpatha /data/ ext4 defaults,_netdev 0 0
Keeping the initramfs synchronized
If multipathing is enabled in the system, it also needs to be enabled in the initramfs and vice versa.
The multipath configuration must be synchronized between the booted system and the initramfs. Therefore, if you change any of the files: /etc/multipath.conf, /etc/multipath/wwids, /etc/multipath/bindings, or other configuration files, or udev rules related to device identification, rebuild initramfs using the command:
sudo dracut -f
If the initramfs and the system are not synchronized, the system will not boot properly, and the start-up procedure may result in an emergency shell.
Using DM Multipath Disks for LVM
We can also use the DM multipath disk with the Logical Volume Manger (LVM). Therefore we must initialize the /dev/mapper/mpathN disk as a physical volume (PV).
# pvcreate /dev/mapper/mpatha # pvs
Creating the volume group (VG).
# vgcreate data_vg /dev/mapper/mpatha
Creating a logical volume (LV).
# lvcreate -l 100%FREE -n data data_vg
Creating the file system.
# mkfs.ext4 /dev/data_vg/data
Finally we can mount our logical volumes.
# mount /dev/data_vg/data /data
To make the mount persistent we also need to add an entry in the /etc/fstab file.
The lsblk command output now looks like below.
# lsblk
The multipath -ll command will still display the disk labels with sdc and sdd.
More about creating LVM volume groups and logical volumes you will find in my following post.
Links
ISCSI Multipath
https://pve.proxmox.com/wiki/ISCSI_MultipathDevice Mapper Multipathing
https://docs.redhat.com/de/documentation/red_hat_enterprise_linux/6/html/dm_multipath/mpio_overviewManaging multipath I/O for devices
https://documentation.suse.com/de-de/sles/15-SP6/html/SLES-all/cha-multipath.htmlMultipath commands
https://www.ibm.com/docs/en/svfpc/8.6.0?topic=hosts-multipath-commandsAppendix A. The Device Mapper
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/device_mapperUnderstanding Device Mapper and Filter Driver
https://www.msystechnologies.com/blog/understanding-device-mapper-and-filter-driver/Mass storage over IP networks: iSCSI
https://documentation.suse.com/sles/15-SP6/html/SLES-all/cha-iscsi.htmlInstalling the iSCSI LIO target server and iSCSI initiator
https://documentation.suse.com/sles/15-SP6/html/SLES-all/cha-iscsi.html#sec-iscsi-install
