Gluster

The brick is the storage filesystem that has been assigned to a volume. To check the bricks from one each servers host you have to ensure there is a device proper mounted in a path with xfs or ext4 filesystem.

<code_bash>

 df -hT

</code> There shoud be one or more lines like (one each brick to put in gluster cluster): <code_bash>

 ...
 /dev/mapper/mpathc                 xfs             2.2T  1G  2.2T   1% /brick-nova
 ...

</code>

NFS

The exports is the directory that will be exported to remote host. Check the is a directory mounted and it is inserted in /etc/exports. <code_bash>

 df -hT

</code> There shoud be one or more lines like (one each brick to put in gluster cluster): <code_bash>

 ...
 /dev/mapper/mpathc                 xfs             2.2T  1G  2.2T   1% /nfs-path
...

</code>

Gluster

Check the glusterd status also with a ps -ef or ps aux command and ensure the gluster is running. <code_bash>

 service glusterd status
 or
 /bin/systemctl status  glusterd.service
 
 ps -ef | grep gluster

</code>

NFS

Check all the NFS cluster status also with a ps -ef or ps aux command and ensure the NFS is running. <code_bash>

 service nfs status
 service pcsd status
 ...
 or
 /bin/systemctl status  nfs.service
 /bin/systemctl status  pcsd.service
 ....

ps -ef | grep nfs

 ps -ef | grep pcsd
 ps -ef | grep iscsi
 ...

</code>

Gluster

Check the peer status of the gluster servers in the cluster. <code_bash>

 gluster peer status

</code>

NFS

Check the cluster NFS status of the NFS servers in the cluster.

Gluster

After the volume creation and after the volume has been started check with gluster command the volume status. <code_bash>

 gluster volume status <volume_name>

</code>

NFS

After the LVM creation and check with LVM command the volume status. <code_bash>

 lvm vgdisplay
 lvm lvdisplay

</code>

Gluster

After the volume creation and after the volume has been started check with gluster command the volume info. <code_bash>

 gluster volume info <volume_name>

</code>

NFS

After the LVM creation and check with LVM command the volume status. <code_bash>

 lvm vgdisplay
 lvm lvdisplay

</code>

Gluster

<code_bash>

 mount -t glusterfs <ip_gluster_server_1>:<volume_name> /mnt

</code>

NFS

<code_bash>

 mount -t nfd <ip_vip_cluster>:<exports> /mnt

</code>

Gluster

<code_bash>

 mount -t glusterfs <gluster_server_1>:<volume_name> /mnt

</code>

NFS

<code_bash>

 mount -t nfs <ip_vip_cluster>:<exports> /mnt

</code>

<code_bash>

 mount -t glusterfs <gluster_server_2>:<volume_name> /mnt

</code>

Not need in NFS using VIP. Restart all glusterd or NFS process and mount the gluster volume to a path like /mnt

Gluster

Write a file in the gluster volume just mounted. (the file created will be of 1GB) <code_bash>

 dd if=/dev/zero of=/mnt/testFile1 bs=1M count=1024 conv=fdatasync | md5sum

</code>

NFS

Write a file in the NFS volume just mounted. (the file created will be of 1GB) <code_bash>

 dd if=/dev/zero of=/mnt/testFile1 bs=1M count=1024 conv=fdatasync | md5sum

</code>

Gluster

Read a file in the gluster volume just mounted. <code_bash>

 head /mnt/testFile1
 md5sum /mnt/testFile1

</code>

NFS

Read a file in the NFS volume just mounted. <code_bash>

 head /mnt/testFile1
 md5sum /mnt/testFile1

</code>

Mount the NFS or gluster volume from 2 clients write two file concurrently in the NFS or gluster volume. Change the X value to 1 or 2 in <client_X> in the command below From each clients exec: (the file created will be of 10GB) <code_bash>

 dd if=/dev/zero of=/mnt/testFile_<client_X>_1 bs=100M count=1024 conv=fdatasync | md5sum

</code>

Mount the NFS or gluster volume from 2 clients read the same file concurrently in the NFS or gluster volume. From each clients exec: <code_bash>

 cat /mnt/testFile1_client_1

</code>

Mount the NFS or gluster volume from 2 clients write two file concurrently in the NFS or gluster volume. And read the file while they are writing data. Change the X value to 1 or 2 in <client_X> in the command below From each clients exec: (the file created will be of 10GB) Please open two shell each clients: <code_bash>

 dd if=/dev/zero of=/mnt/testFile_<client_X>_1 bs=100M count=1024 conv=fdatasync 

</code> On the other shell: <code_bash>

 tail -f /mnt/testFile_<client_X>_1 

</code>

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data stop NFS/pcsd or glusterd process from one server. Check with ps -ef in the server there are no NFS/pcsd or gluster process (if there are glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server gluster or NFS

While the file copying is running

Gluster

<code_bash>

service glusterd stop
ps -ef | grep -i gluster

</code>

NFS

<code_bash>

service nfs stop
service pcsd stop
ps -ef | grep -i nfs
ps -ef | grep -i pcsd

</code>

If there are process NFS/pcsd or gluster running try to:

Same test 2.7 but stopping NFS/pcsd or glusterd process from server_2

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data kill -9 NFS/pcsd or glusterd process from one server.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server gluster or NFS

While the file copying is running

<code_bash>

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done 

</code>

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data poweroff server_1.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server NFS or gluster

While the file copying is running <code_bash> poweroff </code>

Same test 2.11 but poweoff server_2

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data ifdown interface of server_1 where NFS or gluster are using. Check with ps -ef in the server there are no gluster or NFS process (if there are NFS/pcsd or glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server NFS or gluster

While the file copying is running. We suppose glusterd is using eth1 <code_bash> ifdown eth1 ps -ef | grep -i gluster ps -ef | grep -i nfs ps -ef | grep -i pcsd </code>

If there are process NFS/pcsd or gluster running try to:

Same test 2.11 but put down the eth1 from server_2

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the gluster volume. While the file is writing data stop NFS/pcsd or glusterd process from all server. Check with ps -ef in the server there are no NFS/pcsd or gluster process (if there are NFS/pcsd or glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From all servers NFS/gluster

While the file copying is running <code_bash>

service glusterd stop
ps -ef | grep -i gluster
ps -ef | grep -i nfs
ps -ef | grep -i pcsd

</code>

If there are process gluster running try to:

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data put down the network interface from all server. After 3 minutes put up the network interfaces.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash>

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From all server

While the file copying is running <code_bash> ifdown eth1 ps -ef | grep -i gluster ps -ef | grep -i pcsd ps -ef | grep -i nfs

sleep 180; ifup eth1 </code> If there are process NFS/pcsd or gluster running leave the process running

ONLY FOR GLUSTER (if needed) All the test above has been done with simple mount. If some tests fail try to mount the volume in back up mode and run again all the tests above: <code_bash> mount -t glusterfs -obackup-volfile-servers=<ip_gluster_server_2> <ip_gluster_server_1>:<volume_name> /mnt </code>

from compute

Use only a compute node which has the nova /var/lib/nova/instances path mounted in NFS or gluster volume.

Gluster

<code_bash> mount -t glusterfs -obackup-volfile-servers=<ip_gluster_server_2> <ip_gluster_server_1>:<volume_name> /var/lib/nova/instances </code>

NFS

<code_bash> mount -t nfs <vip_nfs_server> :<volume_nfs> /var/lib/nova/instances </code>

form NFS or gluster server

Ensure the volume has the right permissions for the user nova (usually by default uid 162 and gid 162). From the one server:

Gluster

<code_bash> gluster volume set <volume_name> storage.owner-uid 162 gluster volume set <volume_name> storage.owner-gid 162 </code>

NFS

<code_bash> chown 162.162 <lvm_name_path> </code>

Create 5 virtual hosts in the volume mounted from gluster, and check the virtual hosts are running and filesystem is not in read mode every step in the testplan below:

form gluster server 1

<code_bash>

service glusterd stop

</code>

form NFS server 1

<code_bash>

service nfs stop
service pcsd stop

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

form server 1

Gluster

<code_bash>

service glusterd start

</code>

NFS

<code_bash>

service nfs start
service pcsd start

</code>

form server 2

Gluster

<code_bash>

service glusterd stop

</code>

NFS

<code_bash>

service nfs stop
service pcsd stop

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

Start all NFS/pcsd or glusterd on all the servers

form NFS or gluster server 1

<code_bash> poweoff </code>

from dashborad or from nova client

Check the vm are running and they works fine.

form NFS or gluster server 1

power on and check the NFS/pcsd or gluster process are running. Check with the volume status and info the integrity of the volume or with lvm commands.

Gluster

<code_bash> service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> service nfs status service pcsd status lvm lvdisplay </code>

form gluster server 2

<code_bash> poweoff </code>

from dashborad or from nova client

Check the vm are running and they works fine.

form gluster server 2

power on and check the NFS/pcsd or gluster process are running and check with the volume status and info the integrity of the volume.

Gluster

<code_bash> service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> service nfs status service pcsd status lvm lvdisplay </code>

from dashborad or from nova client

Check the vm are running and they works fine.

form NFS or gluster server 1 <code_bash>

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done 

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

form NFS or gluster server 1 <code_bash> service glusterd start service nfs start service pcsd start </code>

form NFS or gluster server 2 <code_bash>

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done 

</code>

form NFS or gluster server 1 (we suppose the eth1 is the interface where NFS or gluster is running

<code_bash> ifdown eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

form NFS or gluster server 1 (we suppose the eth1 is the interface where gluster is running

Gluster

<code_bash> ifup eth1 service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> ifup eth1 service nfs status service pcsd status lvm lvdisplay </code>

form gluster server 2 (we suppose the eth1 is the interface where gluster is running

<code_bash> ifdown eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

form NFS or gluster server 1 (we suppose the eth1 is the interface where NFS or gluster is running)

<code_bash> ifdown eth1 sleep 120; ifup eth1 </code>

form gluster server 2 (we suppose the eth1 is the interface where gluster is running)

<code_bash> ifdown eth1 sleep 120; ifup eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

Find the switch port where the network interface of the fist server, the one NFS or gluster process is using and block the port, simulating a cable cut off.

from dashborad or from nova client

Check the vm are running and they works fine.

Find the switch port where the network interface of the second server, the one NFS or gluster process is using and block the port, simulating a cable cut off from second server.

from dashborad or from nova client

Check the vm are running and they works fine.

from dashborad or from nova client

Check the vm are running and they works fine, before, during and after the port blocking.