progetti:icarus:production-guide
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| progetti:icarus:production-guide [2025/10/15 16:31] – vpia@infn.it | progetti:icarus:production-guide [2025/10/27 15:24] (current) – [Submit the batch of jobs] vpia@infn.it | ||
|---|---|---|---|
| Line 3: | Line 3: | ||
| ====== List of Productions ====== | ====== List of Productions ====== | ||
| ^ Date ^ Type ^ Tag ^ | ^ Date ^ Type ^ Tag ^ | ||
| - | | 03/24 | MC | mc-v09_84_00_01-202403-cnaf-corrsce | | + | | 10/25 | DATA | run3-processing-cnaf-1025-v10_06_00_04p03 |
| - | | 03/24 | DATA | run2-v09_84_00_01-202403-cnaf | + | | 03/24 | MC | mc-v09_84_00_01-202403-cnaf-corrsce |
| - | | 03/24 | MC | mc-v09_84_00_01-202403-cnaf | + | | 03/24 | DATA | run2-v09_84_00_01-202403-cnaf |
| - | | 02/24 | DATA | data_run2-v09_83_01202402-cnaf | + | | 03/24 | MC | mc-v09_84_00_01-202403-cnaf |
| - | | 02/24 | MC | mc_nucosm-v09_83_01202402-cnaf | + | | 02/24 | DATA | data_run2-v09_83_01202402-cnaf |
| - | | 12/23 | DATA | run2-v09_72_00_06-122023-variables | + | | 02/24 | MC | mc_nucosm-v09_83_01202402-cnaf |
| + | | 12/23 | DATA | run2-v09_72_00_06-122023-variables | ||
| ====== General Info ====== | ====== General Info ====== | ||
| Line 17: | Line 18: | ||
| ====== Initial Setup ====== | ====== Initial Setup ====== | ||
| - | The first step is to download and setup all the needed scripts. This must be done **only once** per campaign. All the needed batches of jobs for the current campaign will be submitted with the **same** scripts. Each production request has its own configuration and will be associated to a (//git//) tag (''< | + | The first step is to download and setup all the needed scripts. This must be done **only once** per campaign. All the needed batches of jobs for the current campaign will be submitted with the **same** scripts. Each production request has its own configuration and will be associated to a (//git//) tag (''< |
| - | cd / | + | |
| + | | ||
| mkdir < | mkdir < | ||
| cd < | cd < | ||
| Line 33: | Line 35: | ||
| Now, The initial setup is complete. Here is a complete example: | Now, The initial setup is complete. Here is a complete example: | ||
| - | cd / | + | cd / |
| mkdir run2-v09_72_00_06-122023-variables | mkdir run2-v09_72_00_06-122023-variables | ||
| cd run2-v09_72_00_06-122023-variables | cd run2-v09_72_00_06-122023-variables | ||
| Line 63: | Line 65: | ||
| ===== Configure the next job submission ===== | ===== Configure the next job submission ===== | ||
| - | If the current number of //pending// jobs is smaller than **300**, a new batch of jobs can be submitted. First the shifter has to configure the job submission. To do so, the shifter has to go to the '' | + | If the current number of //pending// jobs is smaller than **300**, a new batch of jobs can be submitted. First the shifter has to configure the job submission. To do so, the shifter has to go to the '' |
| - | cd / | + | cd / |
| Then, the shifter has to configure the job submission editing and modifing the '' | Then, the shifter has to configure the job submission editing and modifing the '' | ||
| Line 71: | Line 73: | ||
| ==== Configure the job submission for real data production ==== | ==== Configure the job submission for real data production ==== | ||
| Here, it's requested to modify the '' | Here, it's requested to modify the '' | ||
| - | |||
| - | ===With a batch.info file=== | ||
| - | The list of runs for each batch is provided in the '' | ||
| - | |||
| - | Here what to do for each batch: | ||
| - | * open (with an editor, i.e. '' | ||
| - | * open (with an editor, i.e. '' | ||
| - | * go to the next batch and copy the corresponding list of runs in the **YOUR_CUSTOM_RUN_LIST** variable | ||
| - | * save // | ||
| - | |||
| - | {{ : | ||
| ===From a Google Sheet document=== | ===From a Google Sheet document=== | ||
| Line 147: | Line 138: | ||
| and submit the production with: | and submit the production with: | ||
| - | ./ | + | ./ |
| The script will automatically submit all the needed jobs. This could take a few minutes during which the shell will look unresponsive (it is not). | The script will automatically submit all the needed jobs. This could take a few minutes during which the shell will look unresponsive (it is not). | ||
| + | |||
| + | After regaining control of the shell, you can check if the jobs were correctly submitted by either check the same grafana page shown previously (it updates with a **4-5 minutes delay** so you'll not be able to see the new jobs immediatly) or by running the **condor_q** command: | ||
| + | |||
| + | $ condor_q | ||
| + | | ||
| + | Schedd: sn01-htc.cr.cnaf.infn.it : < | ||
| + | OWNER BATCH_NAME | ||
| + | valerpia ID: 11228736 | ||
| + | valerpia ID: 11228746 | ||
| + | valerpia 11854_BNBMAJORITY | ||
| + | valerpia 11854_BNBMINBIAS | ||
| + | valerpia 11873_BNBMAJORITY | ||
| + | valerpia 11873_BNBMINBIAS | ||
| + | | ||
| + | Total for query: 1572 jobs; 1445 completed, 0 removed, 0 idle, 127 running, 0 held, 0 suspended | ||
| + | Total for valerpia: 1572 jobs; 1445 completed, 0 removed, 0 idle, 127 running, 0 held, 0 suspended | ||
| + | Total for all users: 89517 jobs; 60905 completed, 0 removed, 18666 idle, 9666 running, 280 held, 0 suspended | ||
| + | |||
| + | The new jobs should be at the bottom of the list, either in the RUN or IDLE stage. | ||
| If the run list was provided with a Google Sheet document, update the **Submitted column** to **yes**. | If the run list was provided with a Google Sheet document, update the **Submitted column** to **yes**. | ||
progetti/icarus/production-guide.1760545895.txt.gz · Last modified: 2025/10/15 16:31 by vpia@infn.it