progetti:cloud-areapd:ceph:replace_a_ssd_disk
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revision | |||
progetti:cloud-areapd:ceph:replace_a_ssd_disk [2019/05/14 14:10] – [Replace a SSD disk used as db for bluestore] sgaravat@infn.it | progetti:cloud-areapd:ceph:replace_a_ssd_disk [2019/05/14 14:13] (current) – [Replace a SSD disk used as db for bluestore] sgaravat@infn.it | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Replace a SSD disk ====== | ||
+ | |||
+ | ==== Replace a SSD disk used as journal for filestore ==== | ||
+ | |||
+ | Let's suppose that we need to replace /dev/sdb. | ||
+ | This device is used for journal for osd.15 .. osd.19: | ||
+ | |||
+ | <code bash> | ||
+ | [root@ceph-osd-02 ~]# ceph-disk list | grep sdb | ||
+ | /dev/sdb : | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | </ | ||
+ | |||
+ | |||
+ | Let's tell ceph to not rebalance the cluster as we stop these OSDs for maintenance: | ||
+ | |||
+ | <code bash> | ||
+ | ceph osd set noout | ||
+ | </ | ||
+ | |||
+ | Let's stop the affected OSDs: | ||
+ | |||
+ | <code bash> | ||
+ | systemctl stop ceph-osd@15.service | ||
+ | systemctl stop ceph-osd@16.service | ||
+ | systemctl stop ceph-osd@17.service | ||
+ | systemctl stop ceph-osd@18.service | ||
+ | systemctl stop ceph-osd@19.service | ||
+ | </ | ||
+ | |||
+ | Let's flush the journals for these OSDs: | ||
+ | |||
+ | <code bash> | ||
+ | ceph-osd -i 15 --flush-journal | ||
+ | ceph-osd -i 16 --flush-journal | ||
+ | ceph-osd -i 17 --flush-journal | ||
+ | ceph-osd -i 18 --flush-journal | ||
+ | ceph-osd -i 19 --flush-journal | ||
+ | </ | ||
+ | |||
+ | Let's replace the device sdb. | ||
+ | In case, let's zap it: | ||
+ | |||
+ | <code bash> | ||
+ | ceph-disk zap /dev/sdb | ||
+ | </ | ||
+ | |||
+ | Let's partition the new disk, using this script: | ||
+ | |||
+ | <code bash> | ||
+ | #!/bin/bash | ||
+ | |||
+ | osds=" | ||
+ | journal_disk=/ | ||
+ | part_number=0 | ||
+ | for osd_id in $osds; do | ||
+ | part_number=$((part_number+1)) | ||
+ | journal_uuid=$(cat / | ||
+ | echo " | ||
+ | echo " | ||
+ | sgdisk --new=${part_number}: | ||
+ | done | ||
+ | </ | ||
+ | |||
+ | Then: | ||
+ | |||
+ | <code bash> | ||
+ | ceph-osd --mkjournal -i 15 | ||
+ | ceph-osd --mkjournal -i 16 | ||
+ | ceph-osd --mkjournal -i 17 | ||
+ | ceph-osd --mkjournal -i 18 | ||
+ | ceph-osd --mkjournal -i 19 | ||
+ | </ | ||
+ | |||
+ | Let's restart the osds: | ||
+ | |||
+ | <code bash> | ||
+ | systemctl restart ceph-osd@15.service | ||
+ | systemctl restart ceph-osd@16.service | ||
+ | systemctl restart ceph-osd@17.service | ||
+ | systemctl restart ceph-osd@18.service | ||
+ | systemctl restart ceph-osd@19.service | ||
+ | </ | ||
+ | |||
+ | Finally: | ||
+ | |||
+ | <code bash> | ||
+ | ceph osd unset noout | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Replace a SSD disk used as db for bluestore ==== | ||
+ | |||
+ | Let's suppose that we need to replace /dev/vdk. | ||
+ | |||
+ | Using: | ||
+ | |||
+ | <code bash> | ||
+ | vgdisplay -v | ||
+ | </ | ||
+ | |||
+ | we find the volume group used for this physical device. In our case it is: | ||
+ | |||
+ | ceph-db-16-19 | ||
+ | |||
+ | |||
+ | <code bash> | ||
+ | |||
+ | [root@c-osd-5 /]# vgdisplay -v ceph-db-16-19 | ||
+ | --- Volume group --- | ||
+ | VG Name | ||
+ | System ID | ||
+ | Format | ||
+ | Metadata Areas 1 | ||
+ | Metadata Sequence No 57 | ||
+ | VG Access | ||
+ | VG Status | ||
+ | MAX LV 0 | ||
+ | Cur LV 4 | ||
+ | Open LV 4 | ||
+ | Max PV 0 | ||
+ | Cur PV 1 | ||
+ | Act PV 1 | ||
+ | VG Size < | ||
+ | PE Size 4.00 MiB | ||
+ | Total PE 51199 | ||
+ | Alloc PE / Size 50944 / 199.00 GiB | ||
+ | Free PE / Size 255 / 1020.00 MiB | ||
+ | VG UUID | ||
+ | |||
+ | --- Logical volume --- | ||
+ | LV Path / | ||
+ | LV Name db-16 | ||
+ | VG Name ceph-db-16-19 | ||
+ | LV UUID YVoGSB-vBIl-sorZ-wLNR-8bX2-XxIc-sYHbkj | ||
+ | LV Write Access | ||
+ | LV Creation host, time c-osd-5.novalocal, | ||
+ | LV Status | ||
+ | # open 2 | ||
+ | LV Size 50.00 GiB | ||
+ | Current LE 12800 | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 8192 | ||
+ | Block device | ||
+ | |||
+ | --- Logical volume --- | ||
+ | LV Path / | ||
+ | LV Name db-17 | ||
+ | VG Name ceph-db-16-19 | ||
+ | LV UUID 4kdL1r-RMPt-MxXj-Ve7y-Czxc-eJRX-476pzt | ||
+ | LV Write Access | ||
+ | LV Creation host, time c-osd-5.novalocal, | ||
+ | LV Status | ||
+ | # open 2 | ||
+ | LV Size 50.00 GiB | ||
+ | Current LE 12800 | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 8192 | ||
+ | Block device | ||
+ | |||
+ | --- Logical volume --- | ||
+ | LV Path / | ||
+ | LV Name db-18 | ||
+ | VG Name ceph-db-16-19 | ||
+ | LV UUID 3378B5-5jFJ-d3jZ-nuSV-zhB9-fEhc-efNezZ | ||
+ | LV Write Access | ||
+ | LV Creation host, time c-osd-5.novalocal, | ||
+ | LV Status | ||
+ | # open 2 | ||
+ | LV Size 50.00 GiB | ||
+ | Current LE 12800 | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 8192 | ||
+ | Block device | ||
+ | |||
+ | --- Logical volume --- | ||
+ | LV Path / | ||
+ | LV Name db-19 | ||
+ | VG Name ceph-db-16-19 | ||
+ | LV UUID fDAejd-teoj-MkTb-AaOb-O4Gy-tztQ-1oXjIL | ||
+ | LV Write Access | ||
+ | LV Creation host, time c-osd-5.novalocal, | ||
+ | LV Status | ||
+ | # open 2 | ||
+ | LV Size 49.00 GiB | ||
+ | Current LE 12544 | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 8192 | ||
+ | Block device | ||
+ | |||
+ | --- Physical volumes --- | ||
+ | PV Name / | ||
+ | PV UUID | ||
+ | PV Status | ||
+ | Total PE / Free PE 51199 / 255 | ||
+ | |||
+ | [root@c-osd-5 /]# | ||
+ | </ | ||
+ | |||
+ | I.e. it is used as db for OSD 16..19 | ||
+ | |||
+ | Let's ' | ||
+ | |||
+ | <code bash> | ||
+ | [root@c-osd-5 /]# ceph osd crush reweight osd.16 0 | ||
+ | reweighted item id 16 name ' | ||
+ | [root@c-osd-5 /]# ceph osd crush reweight osd.17 0 | ||
+ | reweighted item id 17 name ' | ||
+ | [root@c-osd-5 /]# ceph osd crush reweight osd.18 0 | ||
+ | reweighted item id 18 name ' | ||
+ | [root@c-osd-5 /]# ceph osd crush reweight osd.19 0 | ||
+ | reweighted item id 19 name ' | ||
+ | [root@c-osd-5 /]# | ||
+ | </ | ||
+ | |||
+ | Wait that the status is HEALTH-OK. Then: | ||
+ | |||
+ | TBC | ||
+ | |||
+ | <code bash> | ||
+ | ceph osd out osd.16 | ||
+ | ceph osd out osd.17 | ||
+ | ceph osd out osd.18 | ||
+ | ceph osd out osd.19 | ||
+ | |||
+ | ceph osd crush remove osd.16 | ||
+ | ceph osd crush remove osd.17 | ||
+ | ceph osd crush remove osd.18 | ||
+ | ceph osd crush remove osd.19 | ||
+ | |||
+ | systemctl stop ceph-osd@16.service | ||
+ | systemctl stop ceph-osd@17.service | ||
+ | systemctl stop ceph-osd@18.service | ||
+ | systemctl stop ceph-osd@19.service | ||
+ | |||
+ | ceph auth del osd.16 | ||
+ | ceph auth del osd.17 | ||
+ | ceph auth del osd.18 | ||
+ | ceph auth del osd.19 | ||
+ | |||
+ | ceph osd rm osd.16 | ||
+ | ceph osd rm osd.17 | ||
+ | ceph osd rm osd.18 | ||
+ | ceph osd rm osd.19 | ||
+ | |||
+ | umount / | ||
+ | umount / | ||
+ | umount / | ||
+ | umount / | ||
+ | </ | ||
+ | |||
+ | |||
+ | Destroy the volume group created on this SSD disk (be sure to have saved the output of ' | ||
+ | |||
+ | <code bash> | ||
+ | [root@c-osd-5 /]# vgremove ceph-db-16-19 | ||
+ | Do you really want to remove volume group " | ||
+ | Do you really want to remove active logical volume ceph-db-16-19/ | ||
+ | Logical volume " | ||
+ | Do you really want to remove active logical volume ceph-db-16-19/ | ||
+ | Logical volume " | ||
+ | Do you really want to remove active logical volume ceph-db-16-19/ | ||
+ | Logical volume " | ||
+ | Do you really want to remove active logical volume ceph-db-16-19/ | ||
+ | Logical volume " | ||
+ | Volume group " | ||
+ | [root@c-osd-5 /]# | ||
+ | </ | ||
+ | |||
+ | Sostituiamo il disco SSD. Supponiamo che quello nuovo si chiami sempre vdk. | ||
+ | |||
+ | Ricreo volume group e logical volume (fare riferimento all' | ||
+ | |||
+ | <code bash> | ||
+ | [root@c-osd-5 /]# vgcreate ceph-db-16-19 / | ||
+ | Physical volume "/ | ||
+ | Volume group " | ||
+ | [root@c-osd-5 /]# lvcreate -L 50GB -n db-16 ceph-db-16-19 | ||
+ | Logical volume " | ||
+ | [root@c-osd-5 /]# lvcreate -L 50GB -n db-17 ceph-db-16-19 | ||
+ | Logical volume " | ||
+ | [root@c-osd-5 /]# lvcreate -L 50GB -n db-18 ceph-db-16-19 | ||
+ | Logical volume " | ||
+ | [root@c-osd-5 /]# lvcreate -L 50GB -n db-19 ceph-db-16-19 | ||
+ | Volume group " | ||
+ | [root@c-osd-5 /]# lvcreate -L 49GB -n db-19 ceph-db-16-19 | ||
+ | Logical volume " | ||
+ | [root@c-osd-5 /]# | ||
+ | </ | ||
+ | |||
+ | Facciamo un lvm zap (v. https:// | ||
+ | |||
+ | <code bash> | ||
+ | [root@c-osd-5 /]# ceph-volume lvm zap / | ||
+ | Running command: / | ||
+ | --> Zapping: / | ||
+ | Running command: wipefs --all / | ||
+ | Running command: dd if=/ | ||
+ | | ||
+ | 10+0 records out | ||
+ | 10485760 bytes (10 MB) copied | ||
+ | | ||
+ | --> Zapping successful for: / | ||
+ | </ | ||
+ | |||
+ | Creiamo l'OSD: | ||
+ | |||
+ | <code bash> | ||
+ | [root@c-osd-5 /]# ceph-volume lvm create --bluestore --data ceph-block-16/ | ||
+ | Running command: / | ||
+ | Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring / | ||
+ | Running command: / | ||
+ | Running command: mount -t tmpfs tmpfs / | ||
+ | Running command: restorecon / | ||
+ | Running command: chown -h ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph /dev/dm-13 | ||
+ | Running command: ln -s / | ||
+ | Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring / | ||
+ | | ||
+ | Running command: ceph-authtool / | ||
+ | | ||
+ | added entity osd.16 auth auth(auid = 18446744073709551615 key=AQCiytpcTgRcNhAAOK69zFKcnPvLR0QiVUusGA== with 0 caps) | ||
+ | Running command: chown -R ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph / | ||
+ | Running command: chown -h ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph /dev/dm-1 | ||
+ | Running command: / | ||
+ | --> ceph-volume lvm prepare successful for: ceph-block-16/ | ||
+ | Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev / | ||
+ | Running command: ln -snf / | ||
+ | Running command: chown -h ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph /dev/dm-13 | ||
+ | Running command: chown -R ceph:ceph / | ||
+ | Running command: ln -snf / | ||
+ | Running command: chown -h ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph /dev/dm-1 | ||
+ | Running command: chown -h ceph:ceph / | ||
+ | Running command: chown -R ceph:ceph /dev/dm-1 | ||
+ | Running command: systemctl enable ceph-volume@lvm-16-86f9420c-4d3f-4eee-ade5-622efbe12fec | ||
+ | | ||
+ | Running command: systemctl enable --runtime ceph-osd@16 | ||
+ | Running command: systemctl start ceph-osd@16 | ||
+ | --> ceph-volume lvm activate successful for osd ID: 16 | ||
+ | --> ceph-volume lvm create successful for: ceph-block-16/ | ||
+ | [root@c-osd-5 /]# | ||
+ | </ | ||
+ | |||
+ | |||
+ | NB: Probabilmente il 'ceph volume' | ||
+ | |||
+ | |||
+ | Stesso per gli altri: | ||
+ | |||
+ | <code bash> | ||
+ | ceph-volume lvm create --bluestore --data ceph-block-17/ | ||
+ | ceph-volume lvm zap / | ||
+ | ceph-volume lvm create --bluestore --data ceph-block-17/ | ||
+ | |||
+ | ceph-volume lvm create --bluestore --data ceph-block-18/ | ||
+ | ceph-volume lvm zap / | ||
+ | ceph-volume lvm create --bluestore --data ceph-block-18/ | ||
+ | |||
+ | ceph-volume lvm create --bluestore --data ceph-block-19/ | ||
+ | ceph-volume lvm zap / | ||
+ | ceph-volume lvm create --bluestore --data ceph-block-19/ | ||
+ | |||
+ | </ | ||
+ | |||
progetti/cloud-areapd/ceph/replace_a_ssd_disk.txt · Last modified: 2019/05/14 14:13 by sgaravat@infn.it