Esercitazione 1

Creazione del cluster e configurazione

Verifica che i nodi che apparterranno al cluster possano eseguire ssh senza password

<html> # for i in 1 2 3; do for j in 1 2 3; do ssh c16-serv${i} "ssh c16-serv${j} hostname"; done; done c16-serv1 c16-serv2 c16-serv3 c16-serv1 c16-serv2 c16-serv3 c16-serv1 c16-serv2 c16-serv3 # </html>

Crea il file di input per mmcrcluster

<html> # cat node.lis c16-serv1:quorum-manager: c16-serv2:: c16-serv3:: # </html>

Crea il cluster

<html> # mmcrcluster -N node node node.lis [root@c16-serv1 ~]# mmcrcluster -N node.lis -p c16-serv1 -r /usr/bin/ssh -R /usr/bin/scp -C c16 -A Tue Nov 25 20:57:29 CET 2014: mmcrcluster: Processing node c16-serv1.openlab.infn.it Tue Nov 25 20:57:31 CET 2014: mmcrcluster: Processing node c16-serv2.openlab.infn.it Tue Nov 25 20:57:34 CET 2014: mmcrcluster: Processing node c16-serv3.openlab.infn.it mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations.

  Use the mmchlicense command to designate licenses as needed.<br>

mmcrcluster: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# </html>

Configura le licenze per i nodi del cluster

<html> # mmchlicense server –accept -N c16-serv1,c16-serv2,c16-serv3 The following nodes will be designated as possessing GPFS server licenses:

c16-serv1.openlab.infn.it<br>
c16-serv2.openlab.infn.it<br>
c16-serv3.openlab.infn.it<br>

mmchlicense: Command successfully completed mmchlicense: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# </html>

Visualizza il cluster

<html> # mmlscluster GPFS cluster information ========================

GPFS cluster name:         c16.openlab.infn.it<br>
GPFS cluster id:           7610410449082048296<br>
GPFS UID domain:           c16.openlab.infn.it<br>
Remote shell command:      /usr/bin/ssh<br>
Remote file copy command:  /usr/bin/scp<br>

GPFS cluster configuration servers:

Primary server:    c16-serv1.openlab.infn.it<br>
Secondary server:  (none)<br>

Node Daemon node name IP address Admin node name Designation ——————————————————————————————

 1   c16-serv1.openlab.infn.it  192.168.200.246  c16-serv1.openlab.infn.it  quorum-manager<br>
 2   c16-serv2.openlab.infn.it  192.168.200.247  c16-serv2.openlab.infn.it  <br>
 3   c16-serv3.openlab.infn.it  192.168.200.248  c16-serv3.openlab.infn.it  <br>

# </html>

Fai partire GPFS su tutto il cluster e verifica

<html> # mmstartup -a Tue Nov 25 21:02:29 CET 2014: mmstartup: Starting GPFS … # mmgetstate -a Node number Node name GPFS state ——————————————

     1      c16-serv1        active<br>
     2      c16-serv2        active<br>
     3      c16-serv3        active<br>

# </html>

Modifica il tipo di nodo di c16-serv2 e c16-serv3 in quorum e manager, e verifica

<html> # mmchnode –manager –quorum -N c16-serv2,c16-serv3 Tue Nov 25 21:04:30 CET 2014: mmchnode: Processing node c16-serv2.openlab.infn.it Tue Nov 25 21:04:30 CET 2014: mmchnode: Processing node c16-serv3.openlab.infn.it mmchnode: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# mmlscluster GPFS cluster information ========================

GPFS cluster name:         c16.openlab.infn.it<br>
GPFS cluster id:           7610410449082048296<br>
GPFS UID domain:           c16.openlab.infn.it<br>
Remote shell command:      /usr/bin/ssh<br>
Remote file copy command:  /usr/bin/scp<br>

GPFS cluster configuration servers: ———————————–

Primary server:    c16-serv1.openlab.infn.it<br>
Secondary server:  (none)<br>

Node Daemon node name IP address Admin node name Designation ——————————————————————————————

 1   c16-serv1.openlab.infn.it  192.168.200.246  c16-serv1.openlab.infn.it  quorum-manager<br>
 2   c16-serv2.openlab.infn.it  192.168.200.247  c16-serv2.openlab.infn.it  quorum-manager<br>
 3   c16-serv3.openlab.infn.it  192.168.200.248  c16-serv3.openlab.infn.it  quorum-manager<br>

# </html>

Aggiungi c16-serv2 come secondary configuration server

<html> # mmchcluster -s c16-serv2 mmchcluster: GPFS cluster configuration servers: mmchcluster: Primary server: c16-serv1.openlab.infn.it mmchcluster: Secondary server: c16-serv2.openlab.infn.it mmchcluster: Propagating the new server information to the rest of the nodes. mmchcluster: Command successfully completed # mmlscluster GPFS cluster information ========================

GPFS cluster name:         c16.openlab.infn.it<br>
GPFS cluster id:           7610410449082048296<br>
GPFS UID domain:           c16.openlab.infn.it<br>
Remote shell command:      /usr/bin/ssh<br>
Remote file copy command:  /usr/bin/scp<br>

GPFS cluster configuration servers: ———————————–

Primary server:    c16-serv1.openlab.infn.it<br>
Secondary server:  c16-serv2.openlab.infn.it<br>

Node Daemon node name IP address Admin node name Designation ——————————————————————————————

 1   c16-serv1.openlab.infn.it  192.168.200.246  c16-serv1.openlab.infn.it  quorum-manager<br>
 2   c16-serv2.openlab.infn.it  192.168.200.247  c16-serv2.openlab.infn.it  quorum-manager<br>
 3   c16-serv3.openlab.infn.it  192.168.200.248  c16-serv3.openlab.infn.it  quorum-manager<br>

# </html>

Visualizza la configurazione del cluster

<html> # mmlsconfig Configuration data for cluster c16.openlab.infn.it: ————————————————— myNodeConfigNumber 1 clusterName c16.openlab.infn.it clusterId 7610410449082048296 autoload yes dmapiFileHandleSize 32 minReleaseLevel 3.5.0.11 adminMode central File systems in cluster c16.openlab.infn.it: ——————————————– (none) # </html>

Creazione degli NSD

Visualizza i dischi visti dal nodo

<html> # fdisk -l | grep "^Disk /dev" Disk /dev/vda: 21.5 GB, 21474836480 bytes Disk /dev/vdb: 1073.7 GB, 1073741824000 bytes Disk /dev/vdc: 1073.7 GB, 1073741824000 bytes Disk /dev/vdd: 1073.7 GB, 1073741824000 bytes Disk /dev/vde: 1073.7 GB, 1073741824000 bytes Disk /dev/vdf: 42.9 GB, 42949673984 bytes # </html>

Crea il file di input per mmcrnsd: crea quattro nsd su vdb, vdc, vdd, vde

<html> # cat nsd.lis %nsd: device=/dev/vdb

                nsd=nsd1<br>
                usage=dataAndMetadata<br>
                failureGroup=1<br>
                pool=system<br>

%nsd: device=/dev/vdc

                nsd=nsd2<br>
                usage=dataAndMetadata<br>
                failureGroup=2<br>
                pool=system<br>

%nsd: device=/dev/vdd

                nsd=nsd3<br>
                usage=dataAndMetadata<br>
                failureGroup=3<br>
                pool=system<br>

%nsd: device=/dev/vde

                nsd=nsd4<br>
                usage=dataAndMetadata<br>
                failureGroup=4<br>
                pool=system<br>

# </html>

Crea gli NSD

<html> # mmcrnsd -F nsd.lis mmcrnsd: Processing disk vdb mmcrnsd: Processing disk vdc mmcrnsd: Processing disk vdd mmcrnsd: Processing disk vde mmcrnsd: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# </html>

Verifica la visibilita’ degli NSD sui nodi del cluster

<html> # mmlsnsd -m Disk name NSD volume ID Device Node name Remarks ————————————————————————————— nsd1 C0A8C8F65474E307 /dev/vdb c16-serv1.openlab.infn.it nsd2 C0A8C8F65474E308 /dev/vdc c16-serv1.openlab.infn.it nsd3 C0A8C8F65474E309 /dev/vdd c16-serv1.openlab.infn.it nsd4 C0A8C8F65474E30A /dev/vde c16-serv1.openlab.infn.it # </html>

Modifica gli NSD configurando NSD server per ciascuno, e verifica

<html> # mmchnsd "nsd1:c16-serv1,c16-serv2,c16-serv3:;nsd2:c16-serv2,c16-serv3,c16-serv1:;nsd3:c16-serv3,c16-serv1,c16-serv2:;nsd4:c16-serv1,c16-serv3,c16-serv2:" mmchnsd: Processing disk nsd1 mmchnsd: Processing disk nsd2 mmchnsd: Processing disk nsd3 mmchnsd: Processing disk nsd4 mmchnsd: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# mmlsnsd File system Disk name NSD servers ————————————————————————— (free disk) nsd1 c16-serv1.openlab.infn.it,c16-serv2.openlab.infn.it,c16-serv3.openlab.infn.it (free disk) nsd2 c16-serv2.openlab.infn.it,c16-serv3.openlab.infn.it,c16-serv1.openlab.infn.it (free disk) nsd3 c16-serv3.openlab.infn.it,c16-serv1.openlab.infn.it,c16-serv2.openlab.infn.it (free disk) nsd4 c16-serv1.openlab.infn.it,c16-serv3.openlab.infn.it,c16-serv2.openlab.infn.it # </html>

Creazione del file system

Crea il file system /dev/c16fs, mount point /gpfs/c16fs

<html> # mmcrfs /dev/c16fs -F nsd.lis -A yes -B 1M -j scatter -M 2 -R 2 -T /gpfs/c16fs The following disks of c16fs will be formatted on node c16-serv1:

  nsd1: size 1048576000 KB<br>
  nsd2: size 1048576000 KB<br>
  nsd3: size 1048576000 KB<br>
  nsd4: size 1048576000 KB<br>

Formatting file system … Disks up to size 9.1 TB can be added to storage pool system. Creating Inode File Creating Allocation Maps Creating Log Files Clearing Inode Allocation Map Clearing Block Allocation Map Formatting Allocation Map for storage pool system Completed creation of file system /dev/c16fs. mmcrfs: Propagating the cluster configuration data to all

affected nodes.  This is an asynchronous process.<br>

# </html>

Visualizza NSD (ora appartengono ad un file system), dischi e file system

<html> File system Disk name ————————————————————————— c16fs nsd1 c16fs nsd2 c16fs nsd3 c16fs nsd4 # mmlsdisk /dev/c16fs disk driver sector name type ———— ——– nsd1 nsd 512 nsd2 nsd 512 nsd3 nsd 512 nsd4 nsd 512 # mmlsfs /dev/c16fs flag value ——————- ———————— ———————————– -f 32768 -i 512 -I 32768 -m 1 -M 2 -r 1 -R 2 -j scatter -D nfs4 -k all -n 32 -B 1048576 -Q none style="border:1px dashed; background-color:#D9D9D9;"> NSD servers c16-serv1.openlab.infn.it,c16-serv2.openlab.infn.it,c16-serv3.openlab.infn.it c16-serv2.openlab.infn.it,c16-serv3.openlab.infn.it,c16-serv1.openlab.infn.it c16-serv3.openlab.infn.it,c16-serv1.openlab.infn.it,c16-serv2.openlab.infn.it c16-serv1.openlab.infn.it,c16-serv3.openlab.infn.it,c16-serv2.openlab.infn.it failure holds holds storage size group metadata data status availability pool —— ———– ——– —– ————- ———— ———— 1 Yes Yes ready up system 2 Yes Yes ready up system 3 Yes Yes ready up system 4 Yes Yes ready up system description Minimum fragment size in bytes Inode size in bytes Indirect block size in bytes Default number of metadata replicas Maximum number of metadata replicas Default number of data replicas Maximum number of data replicas Block allocation type File locking semantics in effect ACL semantics in effect Estimated number of nodes that will mount file system Block size Quotas enforced

                  none                     Default quotas enabled<br>

–filesetdf No Fileset df enabled? -V 13.23 (3.5.0.7) File system version –create-time Tue Nov 25 21:24:23 2014 File system creation time -u Yes Support for large LUNs? -z No Is DMAPI enabled? -L 4194304 Logfile size -E Yes Exact mtime mount option -S No Suppress atime mount option -K whenpossible Strict replica allocation option –fastea Yes Fast external attributes enabled? –inode-limit 4098048 Maximum number of inodes -P system Disk storage pools in file system -d nsd1;nsd2;nsd3;nsd4 Disks in file system –perfileset-quota no Per-fileset quota enforcement -A yes Automatic mount option -o none Additional mount options -T /gpfs/c16fs Default mount point –mount-priority 0 Mount priority # </html>

Monta il file system su tutti i nodi

<html> # mmmount /dev/c16fs -a Tue Nov 25 21:26:48 CET 2014: mmmount: Mounting file systems … # mmlsmount /dev/c16fs File system c16fs is mounted on 3 nodes. # mmlsmount /dev/c16fs -L

                                          <br>

File system c16fs is mounted on 3 nodes:

192.168.200.246 c16-serv1                 <br>
192.168.200.248 c16-serv3                 <br>
192.168.200.247 c16-serv2                 <br>

# </html>

Crea un file

<html> # dd if=/dev/zero of=/gpfs/c16fs/datafile bs=1M count=1K 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 5.75137 s, 187 MB/s # </html>

Verifica l’occupazione del file system

<html> # df /gpfs/c16fs/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/c16fs 4194304000 1529856 4192774144 1% /gpfs/c16fs # mmdf /dev/c16fs disk disk size failure holds holds free KB free KB name in KB group metadata data in full blocks in fragments ————— ————- ——– ——– —– ——————– ——————- Disks in storage pool: system (Maximum disk size allowed is 8.3 TB) nsd1 1048576000 1 Yes Yes 1048194048 (100%) 1952 ( 0%) nsd2 1048576000 2 Yes Yes 1048193024 (100%) 992 ( 0%) nsd3 1048576000 3 Yes Yes 1048193024 (100%) 992 ( 0%) nsd4 1048576000 4 Yes Yes 1048194048 (100%) 2976 ( 0%)

———— ——————– ——————-

(pool total) 4194304000 4192774144 (100%) 6912 ( 0%) (total) 4194304000 4192774144 (100%) 6912 ( 0%) Inode Information —————– Number of used inodes: 4039 Number of free inodes: 503865 Number of allocated inodes: 507904 Maximum number of inodes: 4098048 </html>

INFN wiki

Table of Contents

Esercitazione 1

Creazione del cluster e configurazione

Creazione degli NSD

Creazione del file system