INFN wiki

The HTCondor-CE must be installed on a HTCondor submit node, that is a machine where a SCHEDD daemon runs:
[root@ce02-htc ~]# condor_config_val DAEMON_LIST
MASTER, SCHEDD, COLLECTOR Furthermore:

• It must also hold a valid X509 host certificate.
• Must be reachable from everywhere to the TCP port 9619.
• The repo for the latest stable release (3.2.1 as of writing) is:
  • https://research.cs.wisc.edu/htcondor/yum/repo.d/htcondor-stable-rhel7.repo
• Needed RPMs are:
  • htcondor-ce-bdii, htcondor-ce-client, htcondor-ce-view, htcondor-ce, htcondor-ce-condor
• To finalize the configuration one may refer to the official documentation here: HTCondor-CE.
• A few supplementary notes for non OSG people are provided below

Official documentation points to lcmaps-voms-authentication which refers to OSG. We could instead refer to the standard configuration of a CREAM-CE, which only needs minor changes:

ln -s /etc/grid-security/voms-grid-mapfile /etc/grid-security/voms-mapfile
cp -a /etc/grid-security/hostcert.pem /etc/grid-security/condorcert.pem
cp -a /etc/grid-security/hostkey.pem /etc/grid-security/condorkey.pem
chown condor.condor /etc/grid-security/condorcert.pem /etc/grid-security/condorkey.pem


HTCondor-CE relies on argus for authorization. A basic configuration example for argus his detailed below.

[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-authz.conf
globus_mapping /usr/lib64/libgsi_pep_callout.so argus_pep_callout
[root@ce02-htc grid-security]#
[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-pep-callout-condor.conf
pep_ssl_server_capath /etc/grid-security/certificates/
pep_ssl_client_cert /etc/grid-security/condorcert.pem
pep_ssl_client_key /etc/grid-security/condorkey.pem
pep_url https:<tuo.argus.server.preferito>:8154/authz
pep_timeout 30 # seconds
xacml_resourceid http:<tuo.argus.server.preferito>/condor-ce



Refer to the official documentation: ARGUS for general help about installation and configuration. We provide below an example set of configurations for the HTCondor-CE:

[root@argus ~]# pap-admin lp

default (local):

resource "http://your.favourite.argus.server/condor-ce" {
obligation "http://glite.org/xacml/obligation/local-environment-map" {
}

action ".*" {
rule permit { vo="ops" }
rule permit { vo="dteam" }
rule permit { vo="alice" }
rule permit { vo="atlas" }
rule permit { vo="ams02.cern.ch" }
rule permit { vo="xenon.biggrid.nl" }
}
}


• You can consider installing the argus service on the HTC-CE host itself. This would probably ease early setup.
• Be aware that Argus need read/write access to the /etc/grid-security/gridmapdir directory, which is usually hosted on a shared filesystem.

To verify that your Argus service is properly configured to work with your HTC-CE:

• Create a valid proxy of a supported VO (i.e.: voms-proxy-init –voms cms)
• install an argus pep client on your HTC-CE (yum install argus-pepcli)
• copy the proxy on the root dir of your HTC-CE as user509.pem
• Execute the following example command (adapt to your case)

[root@htc-ce-02 ~]# pepcli –pepd https://your.argus.server:8154/authz –keyinfo user509.pem –capath /etc/grid-security/certificates –cert /etc/grid-security/hostcert.pem –key /etc/grid-security/hostkey.pem –resourceid "http://your.argus.server/condor-ce" -a pippo

On a working setup you should see an output like:

Resource: http://your.argus.server/condor-ce
Decision: Permit
Obligation: http://glite.org/xacml/obligation/local-environment-map/posix (caller should resolve POSIX account mapping)
Username: cms195
Group: cms
Secondary Groups: cms


As a further check you should see that the empty file /etc/grid-security/gridmapdir/cms195 has a static link whose name is derived from the User DN Subject, which can be identified this way:


[root@argus ~]# ls -li /etc/grid-security/gridmapdir/cms195
383663 -rw-r–r– 2 root root 0 26 giu 2019 /etc/grid-security/gridmapdir/cms195
[root@argus ~]# ls -li /etc/grid-security/gridmapdir/ | grep 383663
383663 -rw-r–r– 2 root root 0 26 giu 2019 %2fdc%3dch%2fdc%3dcern%2fou%3dorganic%20units%2fou%3dusers%2fcn%3dsurname%2fcn%3d606831%2fcn%3dname%20surname:cms:cmsprd:cms 383663 -rw-r–r– 2 root root 0 26 giu 2019 cms195


An entry has to be ADDED to the condor_mapfile to match the certificate DN for the hosts at your site.
These are mapped to the value defined by UID_DOMAIN in HTCondor and HTCondor-CE (in our case: "htc_tier1")


[root@ce02-htc ~]# cat /etc/condor-ce/condor_mapfile
GSI "^\/DC\=com\/DC\=DigiCert-Grid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$"\2@daemon.opensciencegrid.org
GSI "^\/DC\=DigiCert-Grid\/DC\=com\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \2@daemon.opensciencegrid.org
GSI "^\/DC\=org\/DC\=opensciencegrid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \2@daemon.opensciencegrid.org
GSI "^\/DC=ch\/DC=cern\/OU=computers\/CN=?([A-Za-z0-9.\-]*)$" \1@cern.ch
GSI "^\/C\=IT\/O\=INFN\/OU\=Host\/L=CNAF\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \1@htc_tier1
GSI (.*) GSS_ASSIST_GRIDMAP
GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.opensciencegrid.org
CLAIMTOBE .* anonymous@claimtobe
FS (.*) \1



The default configuration path is /etc/condor-ce/config.d/. A tool condor_ce_config_val is provided to inspect configurations, in the same way as condor_config_val.

• Find where and how is defined an identifier (Knob) whose name is exactly known:

[root@ce03-htc ~]# condor_ce_config_val -v HTCONDORCE_VONames
HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo
# at: /etc/condor-ce/config.d/06-ce-bdii.conf, line 14

• See names and value for identifiers matching a part:

[root@ce03-htc ~]# condor_ce_config_val -d HTCONDORCE
# Configuration from machine: ce03-htc.cr.cnaf.infn.it

# Parameters with names that match HTCONDORCE:
HTCONDORCE_BDII_ELECTION = LEADER
HTCONDORCE_BDII_LEADER = ce03-htc.cr.cnaf.infn.it
HTCONDORCE_CORES = 16 # cores per node
[…]

Most of the predefined values already have reasonable values and there should be no reason to alter them; by individually inspecting the configuration files there are comments providing hints about what should be customized.

Worth to mention entries are :

• HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo
• DAEMON_LIST = $(DAEMON_LIST), CEVIEW, GANGLIAD
  

from file /etc/condor-ce/config.d/05-ce-view.conf
This enables a CE monitoring webtool (CEView) which is visible as http://your.ce.fully.qualified.name:80

Atr this point the CE should be able to handle firts job submissions

• From a User Interface having the htcondor-ce-client rpm, after generating a valid proxy, the CE can be tested by
  

condor_ce_trace <fqdn del CE>
use the –debug option to get more details.

the rpm creates two configuration files and python script:
[root@ce02-htc bdii]# rpm -ql htcondor-ce-bdii
/etc/condor/config.d/50-ce-bdii-defaults.conf
/etc/condor/config.d/99-ce-bdii.conf
/var/lib/bdii/gip/provider/htcondor-ce-provider


Note1: the path is under /etc/condor/, thus these are HTCondor configurations, not HTCondor-CE; Note2: i manually had to define one more knob: GLUE2DomainID = $(HTCONDORCE_SiteName)

Otherwise the script htcondor-ce-provider would exit with error. (update: fixed in more recent versions)

To check that the configuration is formally fine just execute /var/lib/bdii/gip/provider/htcondor-ce-provider; a dump of the glue2 schema shoud appear on stdout.

finally, activate the service with systemctl enable bdii
systemctl start bdii