User Tools

Site Tools



HTCondor-CE (old)

Notes for the manual setup of a HTCondor-CE

The HTCondor-CE must be installed on a HTCondor submit node, that is a machine where a SCHEDD daemon runs:
[root@ce02-htc ~]# condor_config_val DAEMON_LIST

  • It must also hold a valid X509 host certificate.
  • Must be reachable from everywhere to the TCP port 9619.
  • The repo for the latest stable release (3.2.1 as of writing) is:
  • Needed RPMs are:
    • htcondor-ce-bdii, htcondor-ce-client, htcondor-ce-view, htcondor-ce, htcondor-ce-condor
  • To finalize the configuration one may refer to the official documentation here: HTCondor-CE.
  • A few supplementary notes for non OSG people are provided below
VOMS e Grid

Official documentation points to lcmaps-voms-authentication which refers to OSG. We could instead refer to the standard configuration of a CREAM-CE, which only needs minor changes:

ln -s /etc/grid-security/voms-grid-mapfile /etc/grid-security/voms-mapfile
cp -a /etc/grid-security/hostcert.pem /etc/grid-security/condorcert.pem
cp -a /etc/grid-security/hostkey.pem /etc/grid-security/condorkey.pem
chown condor.condor /etc/grid-security/condorcert.pem /etc/grid-security/condorkey.pem

GSI and authz

HTCondor-CE relies on argus for authorization. A basic configuration example for argus his detailed below.

[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-authz.conf
globus_mapping /usr/lib64/ argus_pep_callout
[root@ce02-htc grid-security]#
[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-pep-callout-condor.conf
pep_ssl_server_capath /etc/grid-security/certificates/
pep_ssl_client_cert /etc/grid-security/condorcert.pem
pep_ssl_client_key /etc/grid-security/condorkey.pem
pep_url https:<tuo.argus.server.preferito>:8154/authz
pep_timeout 30 # seconds
xacml_resourceid http:

GSI and argus

Refer to the official documentation: ARGUS for general help about installation and configuration. We provide below an example set of configurations for the HTCondor-CE:

[root@argus ~]# pap-admin lp

default (local):

resource "http://your.favourite.argus.server/condor-ce" {
obligation "" {

action ".*" {
rule permit { vo="ops" }
rule permit { vo="dteam" }
rule permit { vo="alice" }
rule permit { vo="atlas" }
rule permit { vo="" }
rule permit { vo="" }

  • You can consider installing the argus service on the HTC-CE host itself. This would probably ease early setup.
  • Be aware that Argus need read/write access to the /etc/grid-security/gridmapdir directory, which is usually hosted on a shared filesystem.

To verify that your Argus service is properly configured to work with your HTC-CE:

  • Create a valid proxy of a supported VO (i.e.: voms-proxy-init –voms cms)
  • install an argus pep client on your HTC-CE (yum install argus-pepcli)
  • copy the proxy on the root dir of your HTC-CE as user509.pem
  • Execute the following example command (adapt to your case)

[root@htc-ce-02 ~]# pepcli –pepd https://your.argus.server:8154/authz –keyinfo user509.pem –capath /etc/grid-security/certificates –cert /etc/grid-security/hostcert.pem –key /etc/grid-security/hostkey.pem –resourceid "http://your.argus.server/condor-ce" -a pippo

On a working setup you should see an output like:

Resource: http://your.argus.server/condor-ce
Decision: Permit
Obligation: (caller should resolve POSIX account mapping)
Username: cms195
Group: cms
Secondary Groups: cms

As a further check you should see that the empty file /etc/grid-security/gridmapdir/cms195 has a static link whose name is derived from the User DN Subject, which can be identified this way:

[root@argus ~]# ls -li /etc/grid-security/gridmapdir/cms195
383663 -rw-r–r– 2 root root 0 26 giu 2019 /etc/grid-security/gridmapdir/cms195
[root@argus ~]# ls -li /etc/grid-security/gridmapdir/ | grep 383663
383663 -rw-r–r– 2 root root 0 26 giu 2019 %2fdc%3dch%2fdc%3dcern%2fou%3dorganic%20units%2fou%3dusers%2fcn%3dsurname%2fcn%3d606831%2fcn%3dname%20surname:cms:cmsprd:cms 383663 -rw-r–r– 2 root root 0 26 giu 2019 cms195


An entry has to be ADDED to the condor_mapfile to match the certificate DN for the hosts at your site.
These are mapped to the value defined by UID_DOMAIN in HTCondor and HTCondor-CE (in our case: "htc_tier1")

[root@ce02-htc ~]# cat /etc/condor-ce/condor_mapfile
GSI "^\/DC\=com\/DC\=DigiCert-Grid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$"\
GSI "^\/DC\=DigiCert-Grid\/DC\=com\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \
GSI "^\/DC\=org\/DC\=opensciencegrid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \
GSI "^\/DC=ch\/DC=cern\/OU=computers\/CN=?([A-Za-z0-9.\-]*)$" \
GSI "^\/C\=IT\/O\=INFN\/OU\=Host\/L=CNAF\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \1@htc_tier1
GSI "(/CN=[-.A-Za-z0-9/= ]+)" \
CLAIMTOBE .* anonymous@claimtobe
FS (.*) \1


The default configuration path is /etc/condor-ce/config.d/. A tool condor_ce_config_val is provided to inspect configurations, in the same way as condor_config_val.

  • Find where and how is defined an identifier (Knob) whose name is exactly known:

[root@ce03-htc ~]# condor_ce_config_val -v HTCONDORCE_VONames
HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo
# at: /etc/condor-ce/config.d/06-ce-bdii.conf, line 14

  • See names and value for identifiers matching a part:

[root@ce03-htc ~]# condor_ce_config_val -d HTCONDORCE
# Configuration from machine:

# Parameters with names that match HTCONDORCE:
HTCONDORCE_CORES = 16 # cores per node

Most of the predefined values already have reasonable values and there should be no reason to alter them; by individually inspecting the configuration files there are comments providing hints about what should be customized.

Worth to mention entries are :

  • HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo

from file /etc/condor-ce/config.d/05-ce-view.conf
This enables a CE monitoring webtool (CEView) which is visible as

Atr this point the CE should be able to handle firts job submissions

Testing the CE

  • From a User Interface having the htcondor-ce-client rpm, after generating a valid proxy, the CE can be tested by

condor_ce_trace <fqdn del CE>
use the –debug option to get more details.


the rpm creates two configuration files and python script:
[root@ce02-htc bdii]# rpm -ql htcondor-ce-bdii

Note1: the path is under /etc/condor/, thus these are HTCondor configurations, not HTCondor-CE; Note2: i manually had to define one more knob: GLUE2DomainID = $(HTCONDORCE_SiteName)

Otherwise the script htcondor-ce-provider would exit with error. (update: fixed in more recent versions)

To check that the configuration is formally fine just execute /var/lib/bdii/gip/provider/htcondor-ce-provider; a dump of the glue2 schema shoud appear on stdout.

finally, activate the service with systemctl enable bdii
systemctl start bdii

progetti/htcondor-tf/htcondor-ce_setup.txt · Last modified: 2020/11/29 12:27 by