====== Data handling at CNAF ======

==== Personal Certificate and VO Enrollment ====
To read and write the storage areas using [[https://dmc-docs.web.cern.ch/dmc-docs/gfal2-util.html|gfal tools]] first you need be enrolled in the ICARUS experiment VO: **icarus-exp.org**. The enrollment requires a valid personal certificate. Instructions on how to obtain it are [[https://confluence.infn.it/display/TD/8+-+Digital+Personal+Certificates+and+Proxies+management|here]]. 
Once you have a valid certificate, you have to install it in a browser and point (with that browser) to the Enrollment Url reported in [[https://operations-portal.egi.eu/vo/view/voname/icarus-exp.org| operation portal]] site. Select the valid personal certificate just installed when asked for by the browser. Once you have filled the enrollment form, you have to wait for the VO manager to approve your enrollment.

==== Read/Write storage areas ====
Once enrolled, you need create a proxy. Instructions on how to create it are [[https://confluence.infn.it/display/TD/8+-+Digital+Personal+Certificates+and+Proxies+management|here]]. 

The usefull tools to handle files are:
  * **gfal-ls**: List information about the file
  * **gfal-copy**: Copy file
  * **gfal-rm**: Remove file
  * **gfal-mkdir**: Create a directory
Detailed information about **gfal utils** can be found [[https://confluence.infn.it/display/TD/10+-+Data+Transfers|here]]

==== Create proxy with fermilab VOMS extensions ====
Being //**fnal_user**// your username at FNAL:

  source /cvmfs/icarus.opensciencegrid.org/products/icarus/setup_icarus.sh
  setup cigetcert
  cigetcert -i 'Fermi National Accelerator Laboratory' --myproxyserver=myproxy.cnaf.infn.it --myproxyretrievers='/DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=(fifebatch|(hepcjobsub0(1|2))|jobsubdevgpvm01).fnal.gov' --myproxyhours=24 --hours=672 -u <fnal_user>
  voms-proxy-init -noregen -rfc -voms 'fermilab:/fermilab/icarus/Role=Analysis'

or in alternative:

  kx509
  voms-proxy-init -noregen -rfc -voms 'fermilab:/fermilab/icarus/Role=Analysis'

==== Samweb ====
It is possible to use samweb at CNAF. First create a proxy with fermilab VOMS extensions, then:

  source /cvmfs/icarus.opensciencegrid.org/products/icarus/setup_icarus.sh
  setup fife_utils

In the following links to webpage with usefull info:
  * [[https://cdcvs.fnal.gov/redmine/projects/sam-main/wiki/Sam_web_client_Command_Reference#samweb-base-options-delete-definition-command-options-ltdataset-definitiongt|on how to handle with samweb definitions]]
  * [[https://cdcvs.fnal.gov/redmine/projects/sam-web/wiki/Dimension_Syntax|on dimension syntax]]

==== Locate a file ====
To locate a file which filename is //filename.root//, do:

  filename="filename.root"
  md5=$(echo -n "user.icaruspro:${filename}" | md5sum)
  ls -lrt /storage/gpfs_data/icarus/plain/user/icaruspro/${md5:0:2}/${md5:2:2}/${filename}  

==== Example: how to get the list of the file paths for a samweb selection ====
  * First, you get the list of the files from samweb

  samweb -e icarus list-files "run_number = 8460 and data_stream = bnbmajority and data_tier = raw" > list_filenames.txt
 
  * Then, you have to convert the filename into filepath using the following commands:

  for filename in $(cat list_filenames.txt); 
  do 
    md5=$(echo -n "user.icaruspro:${filename}" | md5sum); 
    filepath="/storage/gpfs_data/icarus/plain/user/icaruspro/${md5:0:2}/${md5:2:2}/${filename}"; 
    if [ -f ${filepath} ]; 
    then 
      echo ${filepath} >> list_filepaths.txt; 
    else 
      echo ${filename} >> list_missing_files.txt; 
    fi;
  done;
 
  * You will have:
    * the list of the path of the existing files in //list_filepaths.txt//
    * the list of the name of the missing files in //list_missing_files.txt//


==== File transfer from FNAL machines ====
To be able to copy files from FNAL to CNAF you need (**only once**):
  * to be enrolled in the VO. See [[https://wiki.infn.it/progetti/icarus/home#personal_certificate_and_vo_enrollment|session]] above.
  * copy ''~/.globes'' folder in your home in the FNAL machines, using the following command on the CNAF machines:

  scp -r ~/.globes "fnal_user"@icarusgpvm01.fnal.gov:~/.

  * to create ''vomses'' folder in your home in the FNAL machines, using the following command on the FNAL machines:

  mkdir -p ~/vomses
 
  * copy the content of /etc/vomses in the CNAF machines in the created folder, using the following command on the CNAF machines:

  scp -r /etc/vomses/icarus-exp.org* "fnal_user"@icarusgpvm01.fnal.gov:~/vomses/.

Then, you do, ignoring possible errors:

  voms-proxy-init --voms icarus-exp.org --valid 72:00 -vomses ~/vomses/icarus-exp.org-vomsigi-na.unina.it
  gfal-ls srm://storm-fe-archive.cr.cnaf.infn.it:8444/srm/managerv2?SFN=/<space token>

where <space token> can be:
  * icarus for tape (i.e. /storage/gpfs_archive/icarus)
  * icarusplain for plain area on disk (i.e. /storage/gpfs_data/icarus/plain) [__**deprecated**__]
  * icarusdata for not plain area on disk (i.e. /storage/gpfs_data/icarus)

==== File transfer\access from CNAF machines ====
  * First, you create a proxy with fermilab VOMS extensions or in alternative a token:

  source /opt/exp_software/icarus/HTGETTOKEN/setup.sh
  export BEARER_TOKEN_FILE=/tmp/bt_u$(id -u)
  htgettoken -a htvaultprod.fnal.gov -i icarus
  export BEARER_TOKEN=$(cat ${BEARER_TOKEN_FILE})

then you should be able to copy files from FNAL:

  xrdcp xroots://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/icarus/archive/rucio/user/icaruspro/00/00/data_dl1_fstrmNUMI_run8009_8_20220406T062939.root .
  xrdcp file-test.root xroots://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/icarus/persistent/users/mtenti/.
  gfal-ls https://fndcadoor.fnal.gov:2880/pnfs/fnal.gov/usr/icarus/persistent/users/mtenti
  gfal-ls gsiftp://fndca1.fnal.gov:2811/pnfs/fnal.gov/usr/icarus/persistent/users/mtenti
  gfal-copy file-test.root https://fndcadoor.fnal.gov:2880/pnfs/fnal.gov/usr/icarus/persistent/users/mtenti/.

  * you can also access remote file:

  $> root -l
  root [0] TFile *_file0 = TFile::Open("xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/icarus/archive/sbn/sbn_fd/data/artroot/reconstructed/bnbmajority/Run1/offline/icaruscode/icarus_decoder_v09_37_02_09/stage1/v09_37_02_09/00/00/84/60/data_dl2_fstrmBNBMAJORITY_run8460_3_20220608T115225_20220805T191625_pot_20220805T214639-stage0_20220812T075544-stage1.root")


==== Support ====
For any kind of support refer to [[mailto:user-support@cnaf.infn.it|user-support@cnaf.infn.it]]