[8.0] Interpret outputPath with LFN: prefix as an absolute one

[atsareg]

This PR allows to define absolute outputPath by specifying it with the LFN: prefix.
It also allows using wild cards in outputData file names specified as LFNs

BEGINRELEASENOTES

*WorkloadManagement
NEW: JobWrapper - interpret outputPath with LFN: prefix as an absolute one
NEW: JobWrapper - allow wild cards in output LFNs

For examples look into release.notes

ENDRELEASENOTES

[arrabito]

Thank you. All seems fine to me.

[fstagni]

@aldbr has added a lot of tests for JobWrapper in https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/JobWrapper/test/Test_JobWrapper.py.

Can you see if (some of) those can be used here? I am always somewhat scared of changing the JobWrapper.

[fstagni]

Shouldn't this block be before the previous 2 lines? For the case when outputPath == "LFN:/some/where/some/thing.xyz"

[atsareg]

Can you see if (some of) those can be used here? I am always somewhat scared of changing the JobWrapper.

Yes, there are many tests to JobWrapper added to the integration/9.0 . In 8.0 there are just few. Will see if I can add something here

[fstagni]

I would like to double check this. What if the user is indeed "not prevented" to upload to such location, but there are no effective policies preventing it? For example, is a simple user prevented from specifying "LFN:/lhcb/user/a/anotheruser" ?

[marianne013]

What does prevent it now, if anything ? We've been getting around this restriction for ever by using dirac-dms-add-file directly.

[fstagni]

I am not trying to fix the whole loose security system of the grid, because as we know "the tokens will solve that" ™️
But at least we can try to fix one such use case server side.

[marianne013]

If you are feeling ambitious ;-). But I have never seen an incident like that.

[fstagni]

My llm sees the following issues:

1. No global deduplication: List.uniqueElements is applied per-LFN, but if multiple LFNs have overlapping glob patterns, globbedLfnList can still contain duplicates.
2. Comparison can fail due to duplicates: globbedLfnList != lfnList might always be True if duplicates inflate the list, even when no useful expansion occurred.
3. Order not preserved: The result order depends on input order + glob expansion order, which may be inconsistent. [we might ignore this]

Better approach:

globbedLfnList = []
for lfn in lfnList:
    lfnPath = os.path.dirname(lfn)
    lfnLocal = os.path.basename(lfn)
    globbedLfnList += [os.path.join(lfnPath, gLfn) for gLfn in getGlobbedFiles(lfnLocal)]

if globbedLfnList:
    globbedLfnList = List.uniqueElements(globbedLfnList)
    if globbedLfnList != lfnList:
        self.log.info("Found a pattern in the output data LFN list, LFNs to upload are:", ", ".join(globbedLfnList))
        lfnList = globbedLfnList