Basic port of the application

This is the bare minimum required to run CSCDriver on the grid: a simple script paired with a jdl (which should be changed to run on different data sets) are used to submit a plain Normal Job that brings the data back with the output sandbox.

Major drawbacks of such approach are that the job handling (re-submitting of failed/aborted jobs and so on) are usually a major annoyance.

This set can be of practical use at least in one occasion: to test the validity of the package with the CSCDisplay code without concerning oneself with the complexity of the further ports.

Here is the jdl used for the submission:

# CSCDriver_basic.jdl
Executable = "CSCDriver_basic.py";
Arguments = "srm://t2-dpm-01.na.infn.it/dpm/na.infn.it/home/(...)/Dataset.aan.root";
StdOutput = "output.txt";
StdError  = "output.txt";
InputSandbox  = {"CSCDriver_basic.py","atlas_voms_cert"};
OutputSandbox = {"code/results.txt","code/summary/summary_gridjob.root","output.txt"};

As can be seen, the jdl requires the jdl for the VO holding the data to be present in the submission directory, with the name atlas_voms_cert, since we are using data from the atlas VO.

An euindia certificate is also required, but in it's standard location, for the submission of the jobs. This is the list of commands to execute to produce the atlas certificate and then create the euindia one:

$ voms-proxy-init -valid 100:0 -voms atlas -out atlas_voms_cert
$ voms-proxy-init -valid 100:0 -voms euindia

during the execution of those lines, the password protecting the certificate will be required twice.

Validity of the atlas cerfificate (as well as it's expiration) can be checked with the voms-proxy-info -file atlas_voms_cert -all command.

Here is the script CSCDriver_basic.py, which will be submitted with the application.

#!/usr/bin/env python
###
### Created by Riccardo Di Meo (International Centre for Theoretical
### Physic - EUIndiaGRID Project) and Stefano Cozzini (Democritos - SISSA)
###
### This work is licensed under the Creative Commons "2.5 Italy
### Attribution-Noncommercial" License.
###
### To view a copy of this license, visit
### "http://creativecommons.org/licenses/by-nc/2.5/it/legalcode"
###

import os, sys, time

# Error codes
DONE        = 0
ARG_ERR     = 1
VOMS_ERROR  = 2
CODE_ERROR1 = 3
CODE_ERROR1 = 4
LCG_GT_ERR1 = 5
LCG_GT_ERR2 = 6
GSIFTP_ERR  = 7
CSCD_ERROR  = 8
UNEXP_EXCP  = 9

# Second parameter for CSCDriver
BLOCKS = 100000
# Location where the tar package with the CSCDriver and library is
# stored on the euindia grid
CODE_LOCATION = "lfn:/grid/euindia/user/CSCDriver_code.tar.bz2"
# BDII for atlas, should be correctly set up before the lcg-gt step
LCG_GFAL_INFOSYS = "lcg-bdii.cern.ch:2170"
# Name of the atlas certificate: must match the jdl input sandbox
DATA_VOMS = "atlas_voms_cert"

def log(msg, error = None):
    print msg
    if error != None:
        sys.exit(error)

def run_CSCDisplay():

    # First and only argument: the SURL to retrieve and work with
    try:
        surl = sys.argv[1]
    except IndexError, err:
        log("surl not passed to the script!", ARG_ERR)


    # Change the permissions to the certificate
    try:
        os.chmod(DATA_VOMS, 0600)
    except OSError,err:
        log("Cannot chmod the data certificate: %s" % err, VOMS_ERROR)

    # Retrieve the program
    start = time.time()
    res = os.system("lcg-cp %s file:`pwd`/code.tar.bz2" % CODE_LOCATION)
    if res != 0:
        log("lcg-cp of the program returned %d" % res, CODE_ERROR1)

    # Unpack it
    res = os.system("tar xjf code.tar.bz2")
    if res != 0:
        log("untar of the program returned %d" % res, CODE_ERROR2)
    stop = time.time()
    log("Program set up in %d\"" % (stop - start))

    # That was the last operation with the euindia certificates:
    # change the environment
    os.putenv("LCG_GFAL_INFOSYS", LCG_GFAL_INFOSYS)
    os.putenv("X509_USER_CERT", os.path.join(os.getcwd(), DATA_VOMS))
    os.putenv("X509_USER_KEY", os.path.join(os.getcwd(), DATA_VOMS))

    # Enter the data directory
    os.chdir("code/Data")

    # Get the turl for the surl
    start = time.time()
    res = os.popen("lcg-gt %s gsiftp" % surl, "r")
    tmp = res.readlines()
    if res.close() != None:
        log("Error executing lcg-gt (%d)!" % res, LCG_GT_ERR1)
    if tmp[0].find("gsiftp") != 0:
        log("Unrecognized string returned by lcg-gt (%s)!" % tmp[0][:-1],
            LCG_GT_ERR2)
    turl = tmp[0][:-1]
    stop = time.time()
    log("The turl was obtained in %d\"" % (stop - start))

    # Download the turl
    start = time.time()
    res = os.system("globus-url-copy %s file:`pwd`/data.root" % turl)
    if res != 0:
        log("Failed to download the data from the SE (%d)" % res,
            GSIFTP_ERR)
    os.chdir("..")
    stop = time.time()
    log("Data downloaded in %d\"" % (stop - start))

    # Create the .def file in sample
    f = file(os.path.join("samples", "gridjob.def"), "w")
    f.write("TITLE: gridjob\nData/data.root\n")
    f.close()

    # Export the LD_LIBRARY_PATH to include the root libs as well
    ldvar = os.getenv("LD_LIBRARY_PATH", None)
    rootpath = os.path.join(os.getcwd(), "lib")
    if ldvar == None:
        os.putenv("LD_LIBRARY_PATH", rootpath)
    else:
        os.putenv("LD_LIBRARY_PATH", ldvar + ":" + rootpath)

    # Run the code
    start = time.time()
    res = os.system("./CSCDriver gridjob %d" % BLOCKS)
    if res != 0:
        log("CSCDriver didn't returned 0 (%d)!" % res, CSCD_ERROR)
    stop = time.time()
    log("Program executed in %d\"" % (stop - start))


# Run the code only if called as a script
if __name__ == "__main__":
    try:
        run_CSCDisplay()
    except SystemExit, code:
        sys.exit(code)
    except Exception, err:
        log("Uncaught exception (%s)!", UNEXP_EXCP)

As you can see, since it is assumed to be constant for each set of data, the second parameter to the CSCDriver program is encoded in the variable BLOCK, with a list of other which can be changed (cum grano salis).