====== JOB SUBMISSION ======

==== Job submission ====
Instructions on how to submit jobs are [[https://confluence.infn.it/display/TD/HTCondor+jobs|here]]. Be aware if the job needs your certificate (e.g. if the job uses gfal tools) add the following lines to the submitting script:

  x509userproxy           = $(X509_USER_PROXY)
  use_x509userproxy       = true

In a nutshell:
  * Launch job: 

  condor_submit -name sn-02.cr.cnaf.infn.it -spool run.sub

  * Monitor user jobs:

  condor_q -name sn-02.cr.cnaf.infn.it <user>
 
  * Monitor single job:

  condor_q -name sn-02.cr.cnaf.infn.it <cluster id>

  * Deeper monitoring of a job:

  condor_q -better-analyze -name sn-02.cr.cnaf.infn.it <cluster id>
 
  * Get the output:

  condor_transfer_data -name sn-02.cr.cnaf.infn.it <cluster id>
 
==== Submitting simple test production ====
To test everything is working correctly, clone this [[https://baltig.infn.it/icarus/cnaf-test-prod-script/|repository]] in your home and follow the istructions in [[https://baltig.infn.it/icarus/cnaf-test-prod-script/-/tree/master#launch-the-test-production|"Launch the test production"]].

==== Production ====
[[https://baltig.infn.it/icarus/prod-at-cnaf|Here]] is the machinary to submit a production.

==== Adding metadata to CNAF productions ====
Files produced on the CNAF nodes need the injection of additional metadata in order to be declared on samweb.
Production files should be distinguished into two different categories:
  - Stage0 and Stage1 artroot files;
  - Calibration ntuples and CAFs or Flat CAFs.
These two categories require a different treatment in order to be declared on the SAM catalogue.

 === Stage0 and Stage 1 artroot files ===
Stage0 and Stage1 productions on FNAL resources receive the injection of the missing metadata directly on the production nodes. A similar approach is performed on the CNAF machines and it requires requires few different actions.
The 1st action is the creation of a wrapper for the original fcl file, this can be done using an //sbnutil// tool:

  setup sbnutil -q icarus
  sh sbnpoms_wrapperfcl_maker.sh --fclname yourfcl.fcl --wrappername ./wrapper_yourfcl.fcl

The 2nd action is the injection in the missing metadata inside the //wrapper_yourfcl.fcl//, this procedure will instruct the wrapper on the additional metadata to provide to the output file. The injection is performed as following:

  sh sbnpoms_metadata_injector_data_2.sh --inputfclname wrapper_yourfcl.fcl --mdfclname yourfcl.fcl --mdprojectname yourprojectname --mdprojectstage yourstagename --mdprojectversion youricaruscodeversion --mdprojectsoftware icaruscode --mdproductionname yourproductionname  --mdproductiontype official --mdappversion youricaruscodeversion --mdfiletype datatype --mdappfamily art --mdruntype physics --mdgroupname icarus

where "yourprojectname" is the identifier of your production project, "yourstagename" is the stage name (either stage0 or stage1), "youricaruscodeversion" is the release used for this production, "yourproductionname" is the name assigned to your production and "datatype" can be either data or MC depenting on the type of production. The "sbnpoms_metadata_injector_data_2.sh" is a script provided by FNAL experts and already present in the folder of the official CNAF production repository.
By using the //wrapper_yourfcl.fcl// with the injected metadata, the stage0 or stage1 artroot files will have included the needed metadata for the SAM declaration.
Once the artroot files will be retrieved from the working node, an additional step is required to inject an additional metadata and to create the json file for the catalogue declaration:

  streamname=$(samweb -e icarus get-metadata --json rawfilename.root | jq '.data_stream')
  python3 sbnpoms_metadata_extractor_cnaf.py stageoutput.root > stageoutput_temp.json ;
  jq '. += {"data_stream": '${streamname}'}' ${outputStage0}_temp.json > stageout.json

where "streamname" is the name of the data-stream (e.g. BNB Majority, NuMI Majority, OffBeam BNB Majority, etc.) retrieved from the artroot raw file used as the production input, "sbnpoms_metadata_extractor_cnaf.py" is a script provided on the CNAF production repository to copy the metadata and create a temporary json file, stageoutput.root is the artroot file to be declared on SAM, and the last command injects the streamname metadata in the temporary json file, in order to create the final json file.


 === Calibration NTuples and CAFs / Flat CAFs files ===
Calibration NTuples (the Supplemental*.root output of the Stage1 fcl files) and the CAFs / Flat CAFs need to undergo a similar procedure as they are not artroot files, but simple //Root// trees. 
 == Calibration NTuples ==
As the calibration NTuples are a summary product of the Stage1 fcl file, when injecting the metadata in the Stage1 wrapper, an addional requirment is needed:

  sh sbnpoms_metadata_injector_data_2.sh --inputfclname wrapper_yourstage1fcl.fcl --mdfclname yourstage1fcl.fcl --mdprojectname yourprojectname --mdprojectstage yourstagename --mdprojectversion youricaruscodeversion --mdprojectsoftware icaruscode --mdproductionname yourproductionname  --mdproductiontype official --mdappversion youricaruscodeversion --mdfiletype datatype --mdappfamily art --mdruntype physics --mdgroupname icarus --tfilemdjsonname Supplemental--stage1_temp.json

where the last last argument will create a temporary json file which will be later needed for the Calibration NTuple declaration.
The temporary json is created ahead of the Calibration Ntuple file creation, therefore it will be missing relevant metadata as, for example, the file size. When retrieving from the working node the output of the production campaign, the stage1 folder will contain the Supplemental--stage1_temp.json previosuly created. At this step additional metadata need to be injected in the temporary json files:

  fn=yoursupplementalfile.root
  outputSupplemental=$(basename ${fn})
  fs=$(stat --printf=%s ${fn})
  cs=$(ifdh checksum ${fn} )
  jq '. += {"file_name": "'${outputSupplemental}'","file_size":'${fs}',"data_stream": '${streamname}',"checksum":'${cs// }'}' Supplemental--stage1_temp.json  > ${outputSupplemental}.json

where "yoursupplementalfile.root" is the calibration ntuple file to be declared to SAM, and "outputSupplemental" is the final json needed for the SAM declariation.

 == CAFs and Flat CAFs ==

The procedure to create the CAFs / Flat CAFs json files is similar to the Calibration Ntuples, with the exception of two additional argument: 

   sh sbnpoms_metadata_injector_data_2.sh --inputfclname wrapper_yourcaffcl.fcl --mdfclname yourcaffcl.fcl --mdprojectname yourprojectname --mdprojectstage CAF --mdprojectversion youricaruscodeversion --mdprojectsoftware icaruscode --mdproductionname yourproductionname  --mdproductiontype official --mdappversion youricaruscodeversion --mdfiletype datatype --mdappfamily art --mdruntype physics --mdgroupname icarus --tfilemdjsonname CAFmaker_temp.json --tfileformat caf --tdatatier cafana

where "CAFmaker_temp.json" is the temporary CAFs json file and "caf" and "cafana" are respectivily metadata describing the tree format and the data tier.
Similarly to the Calibration NTuples, a json file will be created for each of the outputs of the CAFMaker stage:

   fn=yourcafname.root
   fs=$(stat --printf=%s ${fn})
   cs=$(ifdh checksum ${fn} )
   outputCaf=$(basename ${fn})
   jq '. += {"file_name": "'${outputCaf}'","file_size": '${fs}',"data_stream": '${streamname}',"file_format": "'${fformat}'","checksum":'${cs// }'}' ${cafmakerjson} > ${outputCaf}.json

where yourcafname.root is each of the output CAFs or Flat CAFs of the CAFMaker stage, "fformat" is either: "caf_unlind", "flatcaf_unblind", "caf_blind", "flatcaf_blind", "caf_prescaled" and "flatcaf_prescaled" to be assigned depending on the file currently under declaration.

==== CNAF Production Files declaration ====
Once all the Stage0, Stage1, Calibration NTuples, CAFs and Flat CAFs files have been produced and a corresponding json file is created, according to the instructions in the previous section, the files can be declared to the SAM catalogue. It is important to note that the declaration should be in a hierarchical way, starting from stage0.json, to stage1.json and calibrationntuple.json and finishing with cafs/flatcafs.json as the validation of the metadata will look for the presence of the file parent.
The declaration is perfomed using these commands and having sbnutil setup with a valid FNAL certificate:

   samweb validate-metadata stagefile.json
   samweb declare-file stagefile.json
   samweb add-file-location stagefile.root cnafdisk:/icarusdata/plain/data/yourproductionfolder/path/to/file/

where "validate-metadata" verifies that the json file contains all the relevant metadata, "declare-file" performs the declaration of the file to the SAM catalogue and "add-file-location" assigns the location of the produced file.
Note: following the declaration of the file location, the file itself should not be moved. If moved, the declaration of the file location should be changed as well.

==== Support ====
For any kind of support refer to [[mailto:user-support@cnaf.infn.it|user-support@cnaf.infn.it]]