This guide explains how to restart an elastic cluster based on htcondor and elastiq services.

The flow to restart an elastic cluster after the shut off has to follow these steps. The cloud administrator has to check if the VM yum-repo-pubb, in the multi-tenant services project, that works as condor and elastiq repository is up and running http://90.147.77.142/repo/. If not, log in on the VM

ssh root@90.147.77.142

and restart it checking the mount of /dev/vdb disk

df
/dev/vdb        15350728  276520  14271392   2% /var/www/html/repo

The root password is the usual one.

In order to restart correctly the elastic cluster after a shut off, follow these steps:

• if possible, delete all the slave nodes via dashboard;
• switch on the master node;
• check if both condor and elastiq services are already running (i.e in centos release they are enabled)

service condor status
service elastiq status

In this case new slaves will be created and will join the cluster in some minutes;

• if only condor service is running but elastiq isn't, please restart elastiq with

service elastiq start

or

elastiqctl restart

and wait for the creation of new slaves that will reach the cluster in some minutes;

• if condor isn't running and some elastiq processes are up and running, kill them with

ps -ef | grep elastiq
kill -9 <n_proc>

and start the condor service with

service condor start

The condor_q should return

condor_q  
-- Schedd:  : <10.64.xx.yyy:zzzzz>
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended

and the condor_status should be empty (no nodes running)

condor_status

then start the elastiq service

service elastiq start

in some minutes the minimum number of nodes should reach the condor cluster and the condor_status should show them i.e.

condor_status
Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
slot1@10-64-22-215 LINUX      X86_64 Unclaimed Idle      0.000 1896  0+00:24:46
slot2@10-64-22-215 LINUX      X86_64 Unclaimed Idle      0.000 1896  0+00:25:05
slot1@10-64-22-217 LINUX      X86_64 Unclaimed Idle      0.000 1896  0+00:24:44
slot2@10-64-22-217 LINUX      X86_64 Unclaimed Idle      0.000 1896  0+00:25:05
slot1@10-64-22-89. LINUX      X86_64 Unclaimed Benchmar  1.000 1896  0+00:00:04
slot2@10-64-22-89. LINUX      X86_64 Unclaimed Idle      0.040 1896  0+00:00:05
                     Machines Owner Claimed Unclaimed Matched Preempting
        X86_64/LINUX        6     0       0         6       0          0
               Total        6     0       0         6       0          0

The log file of elastiq, locate in /var/log/elastiq/elastiq.log, is quite difficult to read if you don't know its structure. When you start the elastiq service, the first part of log file reports the check of cloud user's credentials and of other parameters configured in the elastiq.conf file (i.e. the userdata file for slave nodes)

INFO [__init__.conf] Configuration: ec2.image_id = ami-9f3da3fc (from file)
INFO [__init__.conf] Configuration: ec2.flavour = cldareapd.medium (from file)
INFO [__init__.conf] Configuration: ec2.api_url = https://cloud-areapd.pd.infn.it:8788/services/Cloud (from file)
2INFO [__init__.conf] Configuration: ec2.aws_secret_access_key = <...> (from file)
INFO [__init__.conf] Configuration: ec2.key_name = my_key (from file)
INFO [__init__.conf] Configuration: ec2.user_data_b64 = <...> (from file)
INFO [__init__.conf] Configuration: ec2.aws_access_key_id = <...> (from file)
 
INFO [__init__.conf] Configuration: quota.max_vms = 3.0 (from file)
INFO [__init__.conf] Configuration: quota.min_vms = 1.0 (from file)
NFO [__init__.main] Loaded batch plugin "htcondor"
DEBUG [htcondor.init] HTCondor plugin initialized
DEBUG [__init__.main] EC2 image "ami-9f3da3fc" found

if your credentials are wrong you get an error as

ERROR [__init__.ec2_running_instances] Can't get list of EC2 instances (maybe wrong credentials?)

instead, if you insert a wrong ami_id for the image of slave nodes you get

ERROR [__init__.main] Cannot find EC2 image "ami-00000000"

Elastiq periodically checks all the VMs. If a VM is correctly added to the condor cluster, it logs

DEBUG [__init__.ec2_running_instances] Found IP 10.64.22.188 corresponding to instance

otherwise

WARNING [__init__.ec2_running_instances] Cannot find instance 10.64.22.216 in the list of known IPs
WARNING [__init__.ec2_running_instances] Cannot find instance 10.64.22.182 in the list of known IPs
WARNING [__init__.ec2_running_instances] Cannot find instance 10.64.22.236 in the list of known IPs

When elastiq instantiates a new VM it logs

WARNING [__init__.ec2_scale_up] Quota enabled: requesting 1 (out of desired 1) VMs
INFO [__init__.ec2_scale_up] VM launched OK. Requested: 1/1 | Success: 1 | Failed: 0 | ID: i-f026f340
DEBUG [__init__.save_owned_instances] Saved list of owned instances: i-f026f340

and when elastiq deletes an idle VM it logs

INFO [__init__.check_vms] Host 10-64-22-190.INFN-PD is idle for more than 2400s: requesting shutdown
INFO [__init__.ec2_scale_down] Requesting shutdown of 1 VMs...

In the master node of condor, logs are located in /var/log/condor/ directory and are easy to read and understand:

# ls -l /var/log/condor/
total 76
-rw-r--r--. 1 condor condor 24371 Jan 18 08:42 CollectorLog
-rw-r--r--. 1 root   root     652 Jan 18 08:35 KernelTuning.log
-rw-r--r--. 1 condor condor  2262 Jan 18 08:35 MasterLog
-rw-r--r--. 1 condor condor     0 Jan 18 08:35 MatchLog
-rw-r--r--. 1 condor condor 19126 Jan 18 08:42 NegotiatorLog
-rw-r--r--. 1 root   root   13869 Jan 18 08:42 ProcLog
-rw-r--r--. 1 condor condor   474 Jan 18 08:35 ScheddRestartReport
-rw-r--r--. 1 condor condor  2975 Jan 18 08:40 SchedLog

Generally running processes are:

• for the condor service:

[root@test-centos7-elastiq centos]# ps -ef | grep condor
condor       764       1  0 14:09 ?        00:00:00 /usr/sbin/condor_master -f
root         960     764  0 14:09 ?        00:00:00 condor_procd -A /var/run/condor/procd_pipe -L /var/log/condor/ProcLog -R 1000000 -S 60 -C 996
condor       961     764  0 14:09 ?        00:00:00 condor_collector -f
condor       974     764  0 14:09 ?        00:00:00 condor_negotiator -f
condor       975     764  0 14:09 ?        00:00:00 condor_schedd -f

• for the elastiq service:

[root@test-centos7-elastiq centos]# ps -ef | grep elastiq
elastiq      899       1  0 14:09 ?        00:00:00 SCREEN -dmS __|elastiq|__ /bin/sh -c python /usr/bin/elastiq-real.py --logdir=/var/log/elastiq --config=/etc/elastiq.conf --statefile=/var/lib/elastiq/state 2> /var/log/elastiq/elastiq.err
elastiq      952     899  0 14:09 pts/0    00:00:00 /bin/sh -c python /usr/bin/elastiq-real.py --logdir=/var/log/elastiq --config=/etc/elastiq.conf --statefile=/var/lib/elastiq/state 2> /var/log/elastiq/elastiq.err
elastiq      953     952  0 14:09 pts/0    00:00:01 python /usr/bin/elastiq-real.py --logdir=/var/log/elastiq --config=/etc/elastiq.conf --statefile=/var/lib/elastiq/state

NB: The condor_status information isn't updated as frequently as the check of VM status done by elastiq. It could happen that for some minutes the condor_status shows nodes that have been already removed from the cloud by elastiq.