Work Queue supports this functionality. This would be a great use case for the WorkQueueExecutor, but there may be a few tweaks needed in to fully exploit those features. cc @dthain, @btovar, @tjdasso

Cross ref #1326

@annawoodard does the WorkQueueExecutor have the same basic functionality as the htex? That is, is wqex a superset of htex or, if not, what functionality will be given up?

In regards to WorkQueue, each task can specify the number of cores required. However, the WorkQueue executor does not currently specify the number of cores per task, but it shouldn't be a hard feature to implement. With that being said, how will the Parsl app specify the number of cores it needs to run the task, when it is submitted to the WQExecutor?

One possibility is to add it as a keyword argument to the decorator, for example:

@python_app(cores=2)
def foo():
    return 'Hello, world!'

This would be defined here and here for the python and bash app decorators, which would pass it along to the app here. Then at call time before this we could just add it into the kwargs that will be serialized along with the function and passed to the submit method here, at which point WQ could pull it out of the kwargs and use it to specify the cores per task.

as a reminder, we generally have tried to avoid mixing resource info and program/app info. I don't know if there's any way to not do this in the context of this issue, however

If so, I would recommend:

@python_app(resources = {cores=2})
def etc...

as the list of resources may get long, and you may not want to have a super long list of attributes.

@danielskatz I don't think my proposal above is in conflict with our 'write once, run anywhere' aspiration. If you know your task has fixed resource requirements, then I don't see the problem with saying so in the code-- that's not going to change. The thing that does change is where you are running it, and that is still nicely factorized out in the config.

as a reminder, we generally have tried to avoid mixing resource info and program/app info. I don't know if there's any way to not do this in the context of this issue, however

I think one can make the case that 'resources' are really closer to describing the app (kind of like an argument to malloc), rather than the 'resource' where the app will run.

ok, ok ...

as the list of resources may get long, and you may not want to have a super long list of attributes.

@btovar I slightly favor keyword args because in my view it's a bit more natural to document the options, their types and defaults in the docstring:

python_app(..., cores=1, memory=None)
    Decorator function for making python apps.

    Parameters
    ----------
    ...
    cores: int
        Number of cores the task needs. Default is 1.
    memory: float
        Memory to provision per task in MB. Default is ...
    ...

which can be accessed in the interpreter via help(python_app). (Of course, it could also be documented as a single keyword arg, so maybe this isn't a particularly persuasive point.) The other advantage is that it makes it easier to specify and test that the passed type is what is expected in mypy (cc @benclifford).

@btovar I slightly favor keyword args because in my view it's a bit more natural to document the options, their types and defaults in the docstring:

Ah yes, that makes sense.

does the WorkQueueExecutor have the same basic functionality as the htex? That is, is wqex a superset of htex or, if not, what functionality will be given up?

@TomGlanzman The main differences that come to mind are are 1) at the moment WQ is not pip installable, so you would need to do that as a separate step (but my understanding is that it will be very soon), and 2) wqex was added recently so while WQ itself is mature and robust software, there may be a few kinks to iron out with the executor because it is so fresh it hasn't been extensively tested 'in the wild' yet.

Thanks @annawoodard. Is the suggestion that I attempt to migrate to the wqex at some point or that some of its functionality will be incorporated into the htex? (It is not clear to me how wqex might be used at NERSC.)