Linux Logical Volume Management
There's a
separate page about handling dead/dying disks, which includes extending a mirrored LVM vg to include a third disk prior to removing the suspect one.
mounting your new lv using UUID via blkid or /dev/disk/by-uuid
Mirroring (and striping) via md devices
My build notes suggest I've been using Linux LVM since at least 2001. Unix-wise, I started out on AIX, which ships with logical volume management, so working with non-logical disk devices on Linux was a pain. After I lost my file server to a dead disk (the system was on a single IBM drive) I required mirroring as well.
My current approach is to construct MD devices, and use them as physical volumes for LVM.
The example below shows the logical volume, the RAID device its built over, the physical devices it contains. There's also a quick way to see the status of all the RAID devices in the system, in /proc.
# lvdisplay -m /dev/rootvg/lv00sys00
--- Logical volume ---
LV Name /dev/rootvg/lv00sys00
VG Name rootvg
..
LV Size 4.00 GB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0
--- Segments ---
Logical extent 0 to 1023:
Type linear
Physical volume /dev/md2
Physical extents 0 to 1023
# mdadm --misc --detail /dev/md2
/dev/md2:
Version : 00.90.01
Creation Time : Mon Sep 4 06:00:22 2006
Raid Level : raid1
Array Size : 10490304 (10.00 GiB 10.74 GB)
Device Size : 10490304 (10.00 GiB 10.74 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Sat Oct 20 22:27:51 2007
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 415839bb:c0795a74:0d5fb4cf:9e362e4d
Events : 0.9374650
Number Major Minor RaidDevice State
0 8 5 0 active sync /dev/sda5
1 8 21 1 active sync /dev/sdb5
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[0]
4200896 blocks [2/2] [UU]
md2 : active raid1 sdb5[1] sda5[0]
10490304 blocks [2/2] [UU]
|
However, shortly after a rebuilding a machine, I tried to extend a logical volume from 130gb to 230gb, and it refused. It wanted around 230gb of free LVM extents, not 100gb.
Turns out the volume had been created, by SuSE YAST, with striping, and this placed constraints on how it would handle growing the logical volume and what physical volumes it would accept. But the documentation also referred to *new* native disk mirroring in LVM.
I'd originally partitioned the physical devices and created a series of RAID devices to help speed up recovery time. Its a pain when a 200GB array becomes unclean, because you have to wait for all of it to be resynchronised. However, if the filesystem were built over four 50GB devices, you'd have made some permenant progress after the first 50gb. It looks likely that LVM will rebuild the mirror far more intelligently.
Creating a volume group
LVM2 disk mirroring
|
Recommendation
Don't use LVM2 disk mirroring.
This statement is based on (Ubuntu) Kernel 2.6.24
|
Mirroring the OS with LVM is particularly flawed.
-
Test what will happen if you lose one of the disks in the mirror. I found the system would not then boot; md devices will happily run off one disk.
-
You cannot grow mirrored LVM2 volumes online. They comprise two separate LVM logical volumes for each disk underneath the third one that you actually created. So it doesn't look that hard, but it doesn't (yet) work. This is a pretty essential feature, I think.
-
If you have a suspect physical disk and want to swap it out .. md devices allow you to add a third disk to a RAID1 set, and then remove the suspect one. LVM2 does not support more than two disks in the mirror, though it does appear possible.
-
You have to provide a third device for LVM logging; if you've only got two disks to play with, you're off creating md devices anyway.
Recommendation is to mirror your disks/slices with mdadm and then consume those devices in LVM.
Creating and growing LVM2 mirrored devices
Creating mirrored logical volumes, apparently, requires you to specify three destination volumes.
# pvscan
PV /dev/md5 VG rootvg lvm2 [10.00 GB / 5.52 GB free]
PV /dev/md1 VG datavg lvm2 [68.52 GB / 0 free]
PV /dev/md2 VG datavg lvm2 [68.50 GB / 24.00 MB free]
PV /dev/md0 lvm2 [1.00 GB]
PV /dev/sda6 lvm2 [185.30 GB]
PV /dev/sdb6 lvm2 [185.30 GB]
# vgcreate mirrorvg /dev/md0 /dev/sda6 /dev/sdb6
Volume group "mirrorvg" successfully created
# lvcreate -L 1G -m1 -nlv02srv00 mirrorvg /dev/sda6 /dev/sdb6 /dev/md0
Logical volume "lv02srv00" created
|
Growing logical volumes is quite simple, except for the occasional glass ceiling.
# lvextend -L+9G /dev/mirrorvg/lv02srv00
Extending 2 mirror images.
Extending logical volume lv02srv00 to 10.00 GB
# lvextend -L+140G /dev/mirrorvg/lv02srv00
Extending 2 mirror images.
Extending logical volume lv02srv00 to 150.00 GB
Logical volume lv02srv00 successfully resized
|
.. ie: every so often, however, you might trip over this error.
# lvextend -L+1G /dev/mirrorvg/lv02srv00
Extending 2 mirror images.
Extending logical volume lv02srv00 to 151.00 GB
Insufficient free space: 38656 extents needed, but only 18329 available
# vgdisplay mirrorvg
--- Volume group ---
VG Name mirrorvg
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 0
Max PV 0
Cur PV 3
Act PV 3
VG Size 371.60 GB
PE Size 4.00 MB
Total PE 95130
Alloc PE / Size 76801 / 300.00 GB
Free PE / Size 18329 / 71.60 GB
|
The allocation policy appears to be key to this issue. By default, the policy is set in the volume group - to normal - and inherited by the logical volumes. I can't find a way to check the volume group policy, but you can display the lv policies change a logical volume as follows. The options are contiguous, cling, normal, anywhere, or inherit. Avoid 'anywhere' if at all possible.
# lvdisplay /dev/mirrorvg/lv02srv00
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00
VG Name mirrorvg
..
Allocation inherit
Read ahead sectors 0
Block device 253:5
# lvchange --alloc normal /dev/mirrorvg/lv02srv00
|
I fixed it in this instance by dropping the 150gb logical volume, dropping the _mlog volume manually, and then recreating the lv as 180GB, which was what I wanted.
Growing logical volumes within a mirror across physical volumes appears to be fairly foolproof. Here's the start state.
# lvdisplay -m /dev/mirrorvg/lv02srv00
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00
VG Name mirrorvg
..
# open 2
LV Size 180.00 GB
Current LE 46080
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:5
--- Segments ---
Logical extent 0 to 46079:
Type mirror
Mirrors 2
Mirror size 46080
Mirror log volume lv02srv00_mlog
Mirror region size 512.00 KB
Mirror original:
Logical volume lv02srv00_mimage_0
Logical extents 0 to 46079
Mirror destinations:
Logical volume lv02srv00_mimage_1
Logical extents 0 to 46079
# lvdisplay -m -a /dev/mirrorvg/lv02srv00_mimage_0
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00_mimage_0
VG Name mirrorvg
..
LV Size 180.00 GB
Current LE 46080
Segments 1
Allocation inherit
Read ahead sectors 0
--- Segments ---
Logical extent 0 to 46079:
Type linear
Physical volume /dev/sda6
Physical extents 0 to 46079
# lvdisplay -m -a /dev/mirrorvg/lv02srv00_mimage_1
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00_mimage_1
VG Name mirrorvg
..
LV Size 180.00 GB
Current LE 46080
Segments 1
Allocation inherit
Read ahead sectors 0
--- Segments ---
Logical extent 0 to 46079:
Type linear
Physical volume /dev/sdb6
Physical extents 0 to 46079
|
Background: the existing two physical volumes aren't fully utilised yet; and there's four further 68gb volumes in the volume group; two on one physical disk, two on another.
I assumed, as I had to specify the physical volumes at creation, that I'd need to now as well. However, this generated a 'insufficient free space' error. I think it thought I wanted to move the logical volume to the specified devices.
# lvextend -t -L +60G -m1 /dev/mirrorvg/lv02srv00 /dev/hda7 /dev/hdc7 /dev/md0
Test mode: Metadata will NOT be updated.
Extending logical volume lv02srv00 to 300.00 GB
Insufficient free space: 76800 extents needed, but only 35329 available
|
Instead, providing the allocation policy is set right (I'm assuming its the default of 'normal') simply growing the lv seems to do the trick. NB - it has to be offline to do this.
(If this actually doesn't work, this is what I *actually* did. I cycled the allocation policy on the logical volume. First I switched it from inherit to normal, then to cling, then to anywhere. Up to this point, with the test switch, extending the lv qualified with PV names didn't work. So, I tried it without specifying the physical volume names, and it did work. So I set it back to normal and then back to inherit, checking it would still apply the change.)
The -t switch allows you to test drive commands, though if the change involves a couple of dependent steps, the test drive feature will fail when the dependencies aren't in place.
I've included -m1, though it is probably redundant, to enforce my expectation that it'll be mirrored.
# lvextend -t -L +60G -m1 /dev/mirrorvg/lv02srv00
Test mode: Metadata will NOT be updated.
Mirrors cannot be resized while active yet.
# umount /srv/data ; lvchange -an /dev/mirrorvg/lv02srv00
# lvextend -L +60G -m1 /dev/mirrorvg/lv02srv00
Extending logical volume lv02srv00 to 240.00 GB
Logical volume lv02srv00 successfully resized
# lvdisplay -m -a /dev/mirrorvg/lv02srv00_mimage_0
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00_mimage_0
VG Name mirrorvg
..
LV Size 240.00 GB
Current LE 61440
Segments 2
Allocation inherit
Read ahead sectors 0
--- Segments ---
Logical extent 0 to 47436:
Type linear
Physical volume /dev/sda6
Physical extents 0 to 47436
Logical extent 47437 to 61439:
Type linear
Physical volume /dev/hda6
Physical extents 0 to 14002
# lvdisplay -m -a /dev/mirrorvg/lv02srv00_mimage_1
--- Logical volume ---
LV Name /dev/mirrorvg/lv02srv00_mimage_1
VG Name mirrorvg
..
LV Size 240.00 GB
Current LE 61440
Segments 2
Allocation inherit
Read ahead sectors 0
--- Segments ---
Logical extent 0 to 47436:
Type linear
Physical volume /dev/sdb6
Physical extents 0 to 47436
Logical extent 47437 to 61439:
Type linear
Physical volume /dev/hdc6
Physical extents 0 to 14002
# lvchange -ay /dev/mirrorvg/lv02srv00
# resize_reiserfs -s+60G /dev/mirrorvg/lv02srv00
resize_reiserfs 3.6.19 (2003 www.namesys.com)
ReiserFS report:
blocksize 4096
block count 62914560 (47185920)
free blocks 34672787 (18944627)
bitmap block count 1920 (1440)
Syncing..done
resize_reiserfs: Resizing finished successfully.
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
lv00hom00 merrimacvg -wi-ao 500.00M
lv00sys00 merrimacvg -wi-ao 4.00G
lv02srv00 mirrorvg mwi-a- 240.00G lv02srv00_mlog 75.17
# mount /srv/data
|
Sources
RHEL-5: Creating logical volumes (opens in a new window)
RHEL-5: Extending a striped volume (opens in a new window)
Striped LVs have >1 segments (opens in a new window)
LVM basics from IBM developerworks (opens in a new window)
Adding mirroring to existing volumes
Ubuntu 8.04 server wouldn't let me build from scratch using LVM mirroring. The text installer was never going to play game, and attempts to construct the logical volumes on the command line were met with a usage statement. The Kernel wasn't presenting all the information required; device mapping or something.
So, the plan is to construct this system across two 250gb disks, with a 1gb RAID device for logging. Post install, it looks like this:
root@farragut:~# pvscan
PV /dev/sda4 VG farragutvg lvm2 [227.99 GB / 58.99 GB free]
PV /dev/md2 lvm2 [956.88 MB]
PV /dev/sdb4 lvm2 [228.87 GB]
Total: 3 [457.80 GB] / in use: 1 [227.99 GB] / in no VG: 2 [229.80 GB]
root@farragut:~# lvscan
ACTIVE '/dev/farragutvg/lv00sys00' [4.00 GB] inherit
ACTIVE '/dev/farragutvg/lv00srv00' [70.00 GB] inherit
ACTIVE '/dev/farragutvg/lv00srv01' [80.00 GB] inherit
ACTIVE '/dev/farragutvg/lv00srv02' [15.00 GB] inherit
root@farragut:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
lv00srv00 farragutvg -wi-ao 70.00G
lv00srv01 farragutvg -wi-ao 80.00G
lv00srv02 farragutvg -wi-ao 15.00G
lv00sys00 farragutvg -wi-ao 4.00G
|
Next step, I add sdb4 and md2 into the volume group.
root@farragut:~# vgextend farragutvg /dev/sdb4 /dev/md2
Volume group "farragutvg" successfully extended
root@farragut:~# pvscan
PV /dev/sda4 VG farragutvg lvm2 [227.99 GB / 58.99 GB free]
PV /dev/sdb4 VG farragutvg lvm2 [228.87 GB / 228.87 GB free]
PV /dev/md2 VG farragutvg lvm2 [956.00 MB / 956.00 MB free]
|
lvdisplay -m -a <lv> shows which extents are assigned to each logical volume. Working up from zero, I start by mirroring lv00sys00, which is mounted on root. Note that 'lvextend' and 'lvchange' don't play a role here, we use 'lvconvert':
root@farragut:~# lvconvert --mirrors 1 /dev/farragutvg/lv00sys00 /dev/sdb4 /dev/md2
Logical volume lv00sys00 converted.
root@farragut:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
lv00srv00 farragutvg -wi-ao 70.00G
lv00srv01 farragutvg -wi-ao 80.00G
lv00srv02 farragutvg -wi-ao 15.00G
lv00sys00 farragutvg mwi-ao 4.00G lv00sys00_mlog 5.96
|
I now want to check its doing what I expect .. and it seems to be.
The volume now lists these mimage devices, which are in turn logical volumes mapped to physical devices. I've got the first 1024 extents on both disks, plus one extent on the mirror log device, all of 4mb. I guess 1GB was too big!
root@farragut:~# lvdisplay -m -a /dev/farragutvg/lv00sys00
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00
VG Name farragutvg
LV Size 4.00 GB
Current LE 1024
Segments 1
Allocation inherit
--- Segments ---
Logical extent 0 to 1023:
Type mirror
Mirrors 2
Mirror size 1024
Mirror log volume lv00sys00_mlog
Mirror region size 512.00 KB
Mirror original:
Logical volume lv00sys00_mimage_0
Logical extents 0 to 1023
Mirror destinations:
Logical volume lv00sys00_mimage_1
Logical extents 0 to 1023
root@farragut:~# lvdisplay -m -a /dev/farragutvg/lv00sys00_mlog
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mlog
VG Name farragutvg
LV Size 4.00 MB
Current LE 1
Allocation inherit
--- Segments ---
Logical extent 0 to 0:
Type linear
Physical volume /dev/md2
Physical extents 0 to 0
root@farragut:~# lvdisplay -m -a /dev/farragutvg/lv00sys00_mimage_0
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mimage_0
VG Name farragutvg
LV Size 4.00 GB
Current LE 1024
Allocation inherit
--- Segments ---
Logical extent 0 to 1023:
Type linear
Physical volume /dev/sda4
Physical extents 0 to 1023
root@farragut:~# lvdisplay -m -a /dev/farragutvg/lv00sys00_mimage_1
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mimage_1
VG Name farragutvg
LV Size 4.00 GB
Current LE 1024
Allocation inherit
--- Segments ---
Logical extent 0 to 1023:
Type linear
Physical volume /dev/sdb4
Physical extents 0 to 1023
|
lv00srv00 has extents 1024 to 18943, lv00srv01 has 18944 to 39423, lv00srv02 has 39424 to 43263. I'll mirror them up in that order:
root@farragut:~# lvconvert --mirrors 1 /dev/farragutvg/lv00srv00 /dev/sdb4 /dev/md2
Logical volume lv00srv00 converted.
root@farragut:~# lvconvert --mirrors 1 /dev/farragutvg/lv00srv01 /dev/sdb4 /dev/md2
Logical volume lv00srv01 converted.
root@farragut:~# lvconvert --mirrors 1 /dev/farragutvg/lv00srv02 /dev/sdb4 /dev/md2
Logical volume lv00srv02 converted.
root@farragut:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
lv00srv00 farragutvg mwi-ao 70.00G lv00srv00_mlog 4.07
lv00srv01 farragutvg mwi-ao 80.00G lv00srv01_mlog 2.20
lv00srv02 farragutvg mwi-ao 15.00G lv00srv02_mlog 0.31
lv00sys00 farragutvg mwi-ao 4.00G lv00sys00_mlog 100.00
root@farragut:~# lvdisplay -m -a | egrep 'Physical v|Physical e|--- Seg|LV Na|LV Si|Mirror l| Mirror o|Mirror d|Logical v'
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00
LV Size 4.00 GB
--- Segments ---
Mirror log volume lv00sys00_mlog
Mirror original:
Logical volume lv00sys00_mimage_0
Mirror destinations:
Logical volume lv00sys00_mimage_1
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv00
LV Size 70.00 GB
--- Segments ---
Mirror log volume lv00srv00_mlog
Mirror original:
Logical volume lv00srv00_mimage_0
Mirror destinations:
Logical volume lv00srv00_mimage_1
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv01
LV Size 80.00 GB
--- Segments ---
Mirror log volume lv00srv01_mlog
Mirror original:
Logical volume lv00srv01_mimage_0
Mirror destinations:
Logical volume lv00srv01_mimage_1
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv02
LV Size 15.00 GB
--- Segments ---
Mirror log volume lv00srv02_mlog
Mirror original:
Logical volume lv00srv02_mimage_0
Mirror destinations:
Logical volume lv00srv02_mimage_1
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mlog
LV Size 4.00 MB
--- Segments ---
Physical volume /dev/md2
Physical extents 0 to 0
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mimage_0
LV Size 4.00 GB
--- Segments ---
Physical volume /dev/sda4
Physical extents 0 to 1023
--- Logical volume ---
LV Name /dev/farragutvg/lv00sys00_mimage_1
LV Size 4.00 GB
--- Segments ---
Physical volume /dev/sdb4
Physical extents 0 to 1023
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv00_mlog
LV Size 4.00 MB
--- Segments ---
Physical volume /dev/md2
Physical extents 1 to 1
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv00_mimage_0
LV Size 70.00 GB
--- Segments ---
Physical volume /dev/sda4
Physical extents 1024 to 18943
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv00_mimage_1
LV Size 70.00 GB
--- Segments ---
Physical volume /dev/sdb4
Physical extents 1024 to 18943
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv01_mlog
LV Size 4.00 MB
--- Segments ---
Physical volume /dev/md2
Physical extents 2 to 2
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv01_mimage_0
LV Size 80.00 GB
--- Segments ---
Physical volume /dev/sda4
Physical extents 18944 to 39423
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv01_mimage_1
LV Size 80.00 GB
--- Segments ---
Physical volume /dev/sdb4
Physical extents 18944 to 39423
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv02_mlog
LV Size 4.00 MB
--- Segments ---
Physical volume /dev/md2
Physical extents 3 to 3
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv02_mimage_0
LV Size 15.00 GB
--- Segments ---
Physical volume /dev/sda4
Physical extents 39424 to 43263
--- Logical volume ---
LV Name /dev/farragutvg/lv00srv02_mimage_1
LV Size 15.00 GB
--- Segments ---
Physical volume /dev/sdb4
Physical extents 39424 to 43263
root@farragut:~#
|
linux-lvm post
First boot off LVM fails
Built a netbook off USB with Ubuntu 9.10 using LVM for the rootdisk.
There's no other obvious way to do this; the alternative CD isn't, I assume, going to support building a bootable USB disk.
System fails to boot, unable to find logical volume manager.
The symptoms are similar to those encountered swapping out a faulty root disk:
Missing device - md
.. and it seems to fix is broadly the same as the one attempted there - boot back off USB, mount the system up, chroot, and add lvm2.
ubuntuforums
/target failed to unmount
probably /proc and /sys that are holding it open, so we'll try and limit the damage
initramfs includes the basic shell environment and tools to rescue the system if the kernel can't get it bootstrapped; one assumes logical volume manager is in there now.
Printed and hosted by Prater Raines Ltd, 98 Sandgate High Street, Folkestone CT20 3BY.
Published and promoted by Ben Prescott, 14, St James's Square, Bournemouth, BH5 2BX. All rights reserved.
The views expressed are solely those of the author, not of the service provider.
|
