INFN wiki

User Tools

Site Tools

Trace:
progetti:htcondor-tf:htcondor-ce_setup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
progetti:htcondor-tf:htcondor-ce_setup [2019/04/18 07:59] – dalpra@infn.itprogetti:htcondor-tf:htcondor-ce_setup [2020/11/29 12:27] (current) – dalpra@infn.it
Line 1: Line 1:
 +==== HTCondor-CE (old) ====
 +
 +== Notes for the manual setup of a HTCondor-CE  ==
 +
 +The HTCondor-CE must be installed on a HTCondor //submit node//, that is a machine where a SCHEDD daemon runs:\\ 
 +''[root@ce02-htc ~]# condor_config_val DAEMON_LIST\\
 +MASTER, SCHEDD, COLLECTOR''
 +Furthermore:
 +
 +   * It must also hold a valid X509 host certificate.
 +   * Must be reachable from everywhere to the TCP port 9619.
 +   * The repo for the latest stable release (3.2.1 as of writing) is:
 +     * [[https://research.cs.wisc.edu/htcondor/yum/repo.d/htcondor-stable-rhel7.repo]]
 +   * Needed RPMs are:
 +     *  htcondor-ce-bdii, htcondor-ce-client, htcondor-ce-view, htcondor-ce, htcondor-ce-condor
 +   * To finalize the configuration one may refer to the official documentation here: [[https://htcondor-ce.readthedocs.io/en/stable/|HTCondor-CE]].
 +   * A few supplementary notes for non OSG people are provided below
 +
 +== VOMS e Grid ==
 +
 +Official documentation points to [[https://opensciencegrid.org/docs/security/lcmaps-voms-authentication/|lcmaps-voms-authentication]] which refers to OSG. We could instead refer to the standard configuration of a CREAM-CE, which only needs minor changes:\\
 +\\
 +''ln -s /etc/grid-security/voms-grid-mapfile /etc/grid-security/voms-mapfile\\
 +cp -a /etc/grid-security/hostcert.pem /etc/grid-security/condorcert.pem\\
 +cp -a /etc/grid-security/hostkey.pem /etc/grid-security/condorkey.pem\\
 +chown condor.condor /etc/grid-security/condorcert.pem /etc/grid-security/condorkey.pem\\
 +''
 +
 +== GSI and authz ==
 +
 +HTCondor-CE relies on argus for authorization. A basic configuration example for argus his detailed below.
 +
 +''
 +[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-authz.conf\\
 +globus_mapping /usr/lib64/libgsi_pep_callout.so argus_pep_callout\\
 +[root@ce02-htc grid-security]#\\
 +[root@ce02-htc grid-security]# cat /etc/grid-security/gsi-pep-callout-condor.conf\\
 +pep_ssl_server_capath /etc/grid-security/certificates/\\
 +pep_ssl_client_cert /etc/grid-security/condorcert.pem\\
 +pep_ssl_client_key /etc/grid-security/condorkey.pem\\
 +pep_url https://<tuo.argus.server.preferito>:8154/authz\\
 +pep_timeout 30 # seconds\\
 +xacml_resourceid http://<tuo.argus.server.preferito>/condor-ce\\
 +\\
 +''
 +
 +== GSI and argus ==
 +
 +Refer to the official documentation: [[https://argus-documentation.readthedocs.io/en/stable/|ARGUS]] for general help about installation and configuration. We provide below an example set of configurations for the HTCondor-CE:
 +
 +''
 +[root@argus ~]# pap-admin lp\\
 +\\
 +default (local):\\
 +\\
 +resource "http://your.favourite.argus.server/condor-ce" {\\
 +    obligation "http://glite.org/xacml/obligation/local-environment-map" {\\
 +    }\\
 +\\
 +    action ".*" {\\
 +        rule permit { vo="ops" }\\
 +        rule permit { vo="dteam" }\\
 +        rule permit { vo="alice" }\\
 +        rule permit { vo="atlas" }\\
 +        rule permit { vo="ams02.cern.ch" }\\
 +        rule permit { vo="xenon.biggrid.nl" }\\
 +    }\\
 +}\\
 +''
 +
 +  * You can consider installing the argus service on the HTC-CE host itself. This would probably ease early setup.
 +  * Be aware that Argus need read/write access to the ''/etc/grid-security/gridmapdir'' directory, which is usually hosted on a shared filesystem.
 +
 +To verify that your Argus service is properly configured to work with your HTC-CE:
 +  * Create a valid proxy of a supported VO (i.e.: ''voms-proxy-init --voms cms'')
 +  * install an argus pep client on your HTC-CE (''yum install argus-pepcli'')
 +  * copy the proxy on the root dir of your HTC-CE as ''user509.pem''
 +  * Execute the following example command (adapt to your case)
 +
 +''
 +[root@htc-ce-02 ~]# pepcli --pepd https://your.argus.server:8154/authz --keyinfo user509.pem --capath /etc/grid-security/certificates --cert /etc/grid-security/hostcert.pem --key /etc/grid-security/hostkey.pem --resourceid "http://your.argus.server/condor-ce" -a pippo
 +''
 +
 +On a working setup you should see an output like:
 +
 +'' 
 +Resource: http://your.argus.server/condor-ce\\
 +Decision: Permit\\
 +Obligation: http://glite.org/xacml/obligation/local-environment-map/posix (caller should resolve POSIX account mapping)\\
 +Username: cms195\\
 +Group: cms\\
 +Secondary Groups: cms\\
 +'' 
 +
 +As a further check you should see that the empty file
 +''/etc/grid-security/gridmapdir/cms195'' has a static link whose name is derived from the User DN Subject, which can be identified this way:
 +
 +''\\
 +[root@argus ~]# ls -li /etc/grid-security/gridmapdir/cms195\\
 +383663 -rw-r--r-- 2 root root 0 26 giu  2019 /etc/grid-security/gridmapdir/cms195\\
 +[root@argus ~]# ls -li /etc/grid-security/gridmapdir/ | grep 383663\\
 + 383663 -rw-r--r-- 2 root root 0 26 giu  2019 %2fdc%3dch%2fdc%3dcern%2fou%3dorganic%20units%2fou%3dusers%2fcn%3dsurname%2fcn%3d606831%2fcn%3dname%20surname:cms:cmsprd:cms
 + 383663 -rw-r--r-- 2 root root 0 26 giu  2019 cms195\\
 +''
 +
 +
 +== condor_mapfile ==
 +
 +An entry has to be ADDED to the condor_mapfile to match the certificate DN for the hosts at your site.\\
 +These are mapped to the value defined by UID_DOMAIN in HTCondor and HTCondor-CE (in our case: "htc_tier1")
 +
 +''\\
 +[root@ce02-htc ~]# cat /etc/condor-ce/condor_mapfile\\
 +GSI "^\/DC\=com\/DC\=DigiCert-Grid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$"\2@daemon.opensciencegrid.org\\
 +GSI "^\/DC\=DigiCert-Grid\/DC\=com\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \2@daemon.opensciencegrid.org\\
 +GSI "^\/DC\=org\/DC\=opensciencegrid\/O=Open Science Grid\/OU\=Services\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \2@daemon.opensciencegrid.org\\
 +GSI "^\/DC=ch\/DC=cern\/OU=computers\/CN=?([A-Za-z0-9.\-]*)$" \1@cern.ch\\
 +GSI "^\/C\=IT\/O\=INFN\/OU\=Host\/L=CNAF\/CN\=(host\/)?([A-Za-z0-9.\-]*)$" \1@htc_tier1\\
 +GSI (.*) GSS_ASSIST_GRIDMAP\\
 +GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.opensciencegrid.org\\
 +CLAIMTOBE .* anonymous@claimtobe\\
 +FS (.*) \1\\
 +\\
 +''
 +
 +=== Configuration ===
 +
 +The default configuration path is ''/etc/condor-ce/config.d/''.
 +A tool ''condor_ce_config_val'' is provided to inspect configurations, in the same way as ''condor_config_val''.
 +
 +== Examples ==
 +  * Find //where// and //how// is defined an identifier (Knob) whose name is exactly known:
 + ''[root@ce03-htc ~]# condor_ce_config_val -v HTCONDORCE_VONames\\
 +HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo\\
 + # at: /etc/condor-ce/config.d/06-ce-bdii.conf, line 14''
 +  * See names and value for identifiers matching a part:
 +''[root@ce03-htc ~]# condor_ce_config_val -d HTCONDORCE\\
 +# Configuration from machine: ce03-htc.cr.cnaf.infn.it\\
 +\\
 +# Parameters with names that match HTCONDORCE:\\
 +HTCONDORCE_BDII_ELECTION = LEADER\\
 +HTCONDORCE_BDII_LEADER = ce03-htc.cr.cnaf.infn.it\\
 +HTCONDORCE_CORES = 16 # cores per node\\
 +[...]
 +''
 +
 +Most of the predefined values already have reasonable values and there should be no reason to alter them; by individually inspecting  the configuration files there are comments providing hints about what should be customized.
 +
 +Worth to mention entries are :
 +  * ''HTCONDORCE_VONames = alice, atlas, cdf, cms, dteam, lhcb, virgo''
 +  * ''DAEMON_LIST = $(DAEMON_LIST), CEVIEW, GANGLIAD''\\
 +from file ''/etc/condor-ce/config.d/05-ce-view.conf''\\
 +This enables a CE monitoring webtool (CEView) which is visible as http://your.ce.fully.qualified.name:80\\
 +
 +Atr this point the CE should be able to handle firts job submissions
 +
 +=== Testing the CE ===
 +
 +  * From a User Interface having the ''htcondor-ce-client'' rpm, after generating a valid proxy, the CE can be tested by \\
 +''condor_ce_trace <fqdn del CE>''\\
 +use the ''--debug'' option to get more details.
 +
 +=== BDII ===
 +
 +the rpm creates two configuration files and python script:
 +''\\
 +[root@ce02-htc bdii]# rpm -ql htcondor-ce-bdii\\
 +/etc/condor/config.d/50-ce-bdii-defaults.conf\\
 +/etc/condor/config.d/99-ce-bdii.conf\\
 +/var/lib/bdii/gip/provider/htcondor-ce-provider\\
 +''
 +
 +**Note1:** the path is under /etc/condor/, thus these are HTCondor configurations, not HTCondor-CE;
 +**Note2:** i manually had to define one more knob:
 +''GLUE2DomainID = $(HTCONDORCE_SiteName)''
 +
 +Otherwise the script ''htcondor-ce-provider'' would exit with error. (update: fixed in more recent versions)
 +
 +To check that the configuration is formally fine just execute /var/lib/bdii/gip/provider/htcondor-ce-provider;
 +a dump of the glue2 schema shoud appear on ''stdout''.
 +
 +finally, activate the service with 
 +''systemctl enable bdii\\
 +systemctl start bdii''
 +
 +
 +
 +
 +
 +
 +
 +
 +
 +
 +
  
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki