Table of Contents

TestPlan

System Tests

Basic functionality tests:

Gluster server or NFS server tests:

1.1 Bricks or export

Gluster

The brick is the storage filesystem that has been assigned to a volume. To check the bricks from one each servers host you have to ensure there is a device proper mounted in a path with xfs or ext4 filesystem.

<code_bash> 

 df -hT

</code> There shoud be one or more lines like (one each brick to put in gluster cluster): <code_bash> 

 ...
 /dev/mapper/mpathc                 xfs             2.2T  1G  2.2T   1% /brick-nova
 ...

</code>

NFS

The exports is the directory that will be exported to remote host. Check the is a directory mounted and it is inserted in /etc/exports. <code_bash> 

 df -hT

</code> There shoud be one or more lines like (one each brick to put in gluster cluster): <code_bash> 

 ...
 /dev/mapper/mpathc                 xfs             2.2T  1G  2.2T   1% /nfs-path
...

</code>

1.2 Glusterd o NFS status

Gluster

Check the glusterd status also with a ps -ef or ps aux command and ensure the gluster is running. <code_bash> 

 service glusterd status
 or
 /bin/systemctl status  glusterd.service
 
 ps -ef | grep gluster

</code>

NFS

Check all the NFS cluster status also with a ps -ef or ps aux command and ensure the NFS is running. <code_bash> 

 service nfs status
 service pcsd status
 ...
 or
 /bin/systemctl status  nfs.service
 /bin/systemctl status  pcsd.service
 ....

ps -ef | grep nfs 

 ps -ef | grep pcsd
 ps -ef | grep iscsi
 ...

</code>

1.3 Peers or Cluster node status

Gluster

Check the peer status of the gluster servers in the cluster. <code_bash> 

 gluster peer status

</code>

NFS

Check the cluster NFS status of the NFS servers in the cluster.

1.4 Volume or LVM status

Gluster

After the volume creation and after the volume has been started check with gluster command the volume status. <code_bash> 

 gluster volume status <volume_name>

</code>

NFS

After the LVM creation and check with LVM command the volume status. <code_bash> 

 lvm vgdisplay
 lvm lvdisplay

</code>

1.5 Volume or LVM info

Gluster

After the volume creation and after the volume has been started check with gluster command the volume info. <code_bash> 

 gluster volume info <volume_name>

</code>

NFS

After the LVM creation and check with LVM command the volume status. <code_bash> 

 lvm vgdisplay
 lvm lvdisplay

</code>

Tests from clients:

2.1 Mount of a volume

2.1.1 Mount the gluster or NFS volume to a mount point from one of the servers in the gluster cluster.

Gluster

<code_bash> 

 mount -t glusterfs <ip_gluster_server_1>:<volume_name> /mnt

</code>

NFS

<code_bash> 

 mount -t nfd <ip_vip_cluster>:<exports> /mnt

</code>

2.1.2 Umount the volume from client. Stop NFS or glusterd process to server_1 mount the NFS or gluster volume to a mount point using gluster_server_1 or vip

Gluster

<code_bash> 

 mount -t glusterfs <gluster_server_1>:<volume_name> /mnt

</code>

NFS

<code_bash> 

 mount -t nfs <ip_vip_cluster>:<exports> /mnt

</code>

2.1.3 Start glusterd process from gluster_server_1 and stop glusterd process to gluster_server_2 mount the gluster volume to a mount point from gluster_server_2

<code_bash> 

 mount -t glusterfs <gluster_server_2>:<volume_name> /mnt

</code>

Not need in NFS using VIP. Restart all glusterd or NFS process and mount the gluster volume to a path like /mnt

2.2 Write a single file

Gluster

Write a file in the gluster volume just mounted. (the file created will be of 1GB) <code_bash> 

 dd if=/dev/zero of=/mnt/testFile1 bs=1M count=1024 conv=fdatasync | md5sum

</code>

NFS

Write a file in the NFS volume just mounted. (the file created will be of 1GB) <code_bash> 

 dd if=/dev/zero of=/mnt/testFile1 bs=1M count=1024 conv=fdatasync | md5sum

</code>

2.3 Read a single file

Gluster

Read a file in the gluster volume just mounted. <code_bash> 

 head /mnt/testFile1
 md5sum /mnt/testFile1

</code>

NFS

Read a file in the NFS volume just mounted. <code_bash> 

 head /mnt/testFile1
 md5sum /mnt/testFile1

</code>

2.4 Write on a volume concurrently from more clients

Mount the NFS or gluster volume from 2 clients write two file concurrently in the NFS or gluster volume. Change the X value to 1 or 2 in <client_X> in the command below From each clients exec: (the file created will be of 10GB) <code_bash> 

 dd if=/dev/zero of=/mnt/testFile_<client_X>_1 bs=100M count=1024 conv=fdatasync | md5sum

</code>

2.5 Read on a volume concurrently from more clients

Mount the NFS or gluster volume from 2 clients read the same file concurrently in the NFS or gluster volume. From each clients exec: <code_bash> 

 cat /mnt/testFile1_client_1

</code>

2.6 Write and read on a volume concurrently from more clients

Mount the NFS or gluster volume from 2 clients write two file concurrently in the NFS or gluster volume. And read the file while they are writing data. Change the X value to 1 or 2 in <client_X> in the command below From each clients exec: (the file created will be of 10GB) Please open two shell each clients: <code_bash> 

 dd if=/dev/zero of=/mnt/testFile_<client_X>_1 bs=100M count=1024 conv=fdatasync

</code> On the other shell: <code_bash> 

 tail -f /mnt/testFile_<client_X>_1

</code>

2.7 From one client write a big file and stop NFS/pcsd or glusterd process from one server

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data stop NFS/pcsd or glusterd process from one server. Check with ps -ef in the server there are no NFS/pcsd or gluster process (if there are glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server gluster or NFS

While the file copying is running

Gluster

<code_bash> 

service glusterd stop
ps -ef | grep -i gluster

</code>

NFS

<code_bash> 

service nfs stop
service pcsd stop
ps -ef | grep -i nfs
ps -ef | grep -i pcsd

</code>

If there are process NFS/pcsd or gluster running try to:

2.7.1 leave NFS/pcsd or glusterfsd process alive
2.7.2 kill NFS/pcsd or glusterfsd process

2.8 From one client write a big file and stop NFS/pcsd or glusterd process from other server

Same test 2.7 but stopping NFS/pcsd or glusterd process from server_2

2.9 From one client write a big file and kill all NFS/pcsd or glusterd process from one server

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data kill -9 NFS/pcsd or glusterd process from one server.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server gluster or NFS

While the file copying is running

<code_bash> 

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done

</code>

2.10 From one client write a big file and kill all service processes on other server

2.11 From one client write a big file and shutdown one server

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data poweroff server_1.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server NFS or gluster

While the file copying is running <code_bash> poweroff </code>

2.12 From one client write a big file and shutdown other server

Same test 2.11 but poweoff server_2

2.13 From one client write a big file and disable the network interface

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data ifdown interface of server_1 where NFS or gluster are using. Check with ps -ef in the server there are no gluster or NFS process (if there are NFS/pcsd or glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From server NFS or gluster

While the file copying is running. We suppose glusterd is using eth1 <code_bash> ifdown eth1 ps -ef | grep -i gluster ps -ef | grep -i nfs ps -ef | grep -i pcsd </code>

If there are process NFS/pcsd or gluster running try to:

2.13.1 leave NFS/pcsd or glusterfsd process alive
2.13.2 kill NFS/pcsd or glusterfsd process

2.14 From one client write a big file and disable the network interface

Same test 2.11 but put down the eth1 from server_2

2.15 From one client write a big file and stop NFS/pcsd or glusterd process from all servers

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the gluster volume. While the file is writing data stop NFS/pcsd or glusterd process from all server. Check with ps -ef in the server there are no NFS/pcsd or gluster process (if there are NFS/pcsd or glusterfsd process, try to leave the process alive or try in second instance to kill that process)

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From all servers NFS/gluster

While the file copying is running <code_bash> 

service glusterd stop
ps -ef | grep -i gluster
ps -ef | grep -i nfs
ps -ef | grep -i pcsd

</code>

If there are process gluster running try to:

2.15.1 leave NFS/pcsd or glusterfsd process alive
2.15.2 kill NFS/pcsd or glusterfsd process

2.16 From one client write a big file and disable the network interface from each server. After 3 minutes enable network interfaces

Mount the NFS or gluster volume from 1 client write 1 big file (250GB) in the NFS or gluster volume. While the file is writing data put down the network interface from all server. After 3 minutes put up the network interfaces.

From client

create the file in /root or where you have enough space for the big file, check md5 of the file, copy the file to the mount point where gluster volume has been mounted, at the end of the copy check md5. <code_bash> 

 dd if=/dev/zero of=/root/BigFile_1 bs=1G count=254
 md5sum  /root/BigFile_1
 cp /root/BigFile_1 /mnt/
 md5sum /mnt/BigFile1

</code>

From all server

While the file copying is running <code_bash> ifdown eth1 ps -ef | grep -i gluster ps -ef | grep -i pcsd ps -ef | grep -i nfs

sleep 180; ifup eth1 </code> If there are process NFS/pcsd or gluster running leave the process running

ONLY FOR GLUSTER (if needed) All the test above has been done with simple mount. If some tests fail try to mount the volume in back up mode and run again all the tests above: <code_bash> mount -t glusterfs -obackup-volfile-servers=<ip_gluster_server_2> <ip_gluster_server_1>:<volume_name> /mnt </code>

Tests from Openstack dashboard

Prerequisite

from compute

Use only a compute node which has the nova /var/lib/nova/instances path mounted in NFS or gluster volume.

Gluster

<code_bash> mount -t glusterfs -obackup-volfile-servers=<ip_gluster_server_2> <ip_gluster_server_1>:<volume_name> /var/lib/nova/instances </code>

NFS

<code_bash> mount -t nfs <vip_nfs_server> :<volume_nfs> /var/lib/nova/instances </code>

form NFS or gluster server

Ensure the volume has the right permissions for the user nova (usually by default uid 162 and gid 162). From the one server:

Gluster

<code_bash> gluster volume set <volume_name> storage.owner-uid 162 gluster volume set <volume_name> storage.owner-gid 162 </code>

NFS

<code_bash> chown 162.162 <lvm_name_path> </code>

Create 5 virtual hosts in the volume mounted from gluster, and check the virtual hosts are running and filesystem is not in read mode every step in the testplan below:

3.1 From first server stop NFS/pcsd or glusterd process. Check the virtual hosts.

form gluster server 1

<code_bash> 

service glusterd stop

</code>

form NFS server 1

<code_bash> 

service nfs stop
service pcsd stop

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.2 Start NFS/pcsd or glusterd process from first server and stop the NFS/pcsd or glusterd process from second server. Check the virtual hosts.

form server 1

Gluster

<code_bash> 

service glusterd start

</code>

NFS

<code_bash> 

service nfs start
service pcsd start

</code>

form server 2

Gluster

<code_bash> 

service glusterd stop

</code>

NFS

<code_bash> 

service nfs stop
service pcsd stop

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

Start all NFS/pcsd or glusterd on all the servers

3.3 Shutdown first server. Check the virtual hosts.

form NFS or gluster server 1

<code_bash> poweoff </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.4 Power on fist server, ensure Basic Server functionality tests are fine. Shutdown second server. Check the virtual hosts.

form NFS or gluster server 1

power on and check the NFS/pcsd or gluster process are running. Check with the volume status and info the integrity of the volume or with lvm commands.

Gluster

<code_bash> service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> service nfs status service pcsd status lvm lvdisplay </code>

form gluster server 2

<code_bash> poweoff </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.5 Power on second server, ensure Basic Server functionality tests are fine.

form gluster server 2

power on and check the NFS/pcsd or gluster process are running and check with the volume status and info the integrity of the volume.

Gluster

<code_bash> service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> service nfs status service pcsd status lvm lvdisplay </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.6 Kill all NFS/pcsd or gluster process from first server. Check the virtual hosts.

form NFS or gluster server 1 <code_bash> 

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done

</code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.7 Restart all NFS/pcsd or gluster process on server 1 and kill all NFS/pcsd or gluster process from second server.

form NFS or gluster server 1 <code_bash> service glusterd start service nfs start service pcsd start </code>

form NFS or gluster server 2 <code_bash> 

for PIDS in `ps -ef | grep -i -e nfs -e pcsd -e gluster | awk '{ print $1;}'`; do
   kill -9 $PIDS;
 done

</code>

3.8 Disable the network interface NFS or gluster is using from first server. Check the virtual hosts.

form NFS or gluster server 1 (we suppose the eth1 is the interface where NFS or gluster is running

<code_bash> ifdown eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.9 Enalbe the network interface NFS or gluster is using from first server, check NFS/pcsd or gluster service and disable on the network interface on second server. Check the virtual hosts.

form NFS or gluster server 1 (we suppose the eth1 is the interface where gluster is running

Gluster

<code_bash> ifup eth1 service gluster status gluster volume status gluster volume info </code>

NFS

<code_bash> ifup eth1 service nfs status service pcsd status lvm lvdisplay </code>

form gluster server 2 (we suppose the eth1 is the interface where gluster is running

<code_bash> ifdown eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.10 Disable the network interface NFS or gluster is using from each server. Check the virtual hosts. After 2 minutes enable network interfaces. Check the virtual hosts.

form NFS or gluster server 1 (we suppose the eth1 is the interface where NFS or gluster is running)

<code_bash> ifdown eth1 sleep 120; ifup eth1 </code>

form gluster server 2 (we suppose the eth1 is the interface where gluster is running)

<code_bash> ifdown eth1 sleep 120; ifup eth1 </code>

from dashborad or from nova client

Check the vm are running and they works fine.

3.11 Disable the switch port where the network interface of first server is connected.

Find the switch port where the network interface of the fist server, the one NFS or gluster process is using and block the port, simulating a cable cut off.

from dashborad or from nova client

Check the vm are running and they works fine.

3.10 Enable the switch port where the network interface of first server is connected and disable switch port where the network interface of second server is connected.

Find the switch port where the network interface of the second server, the one NFS or gluster process is using and block the port, simulating a cable cut off from second server.

from dashborad or from nova client

Check the vm are running and they works fine.

3.12 Disable the switch ports where the network interfaces of first and second server is connected. Wait 3 minutes and enable again their

from dashborad or from nova client

Check the vm are running and they works fine, before, during and after the port blocking.