Parsl / parsl

This commit adds an addition block level to htex worker log paths,
like this:

  .../runinfo/004/worker-nodes/block-0/627785a8965e/manager.log

This has been useful during a run with multiple large blocks to
discern which logs belong to which block.

* Fixed docstrings to reflect that status() returns a list of JobStatus objects rather than strings

* Use JobStatus class for scaling-in in the IPP executor

Coverage 5.0.0 just appeared which appears to not work with pytest
properly. This patch pins coverage to 4.5.4 which is a version
which has worked successfully previously.

Coverage 5.0.0 + pytest fails with:

INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/home/travis/build/Parsl/parsl/.coverage': UNIQUE constraint failed: meta.key

when run when no .coverage file exists. Subsequent runs do not raise
this error.

Some slurm configurations do not use a partition parameter, but instead
take a --qos parameter.

This change is intended to allow the partition parameter to be
suppressed (with the user then supplying a --qos parameter in
scheduler_options.

Fixes #518

…1500)

* Wrap block status in a class with a state enum and an optional message (to propagate errors)

* added type annotation

* added type annotation

* Fix jumbled legends and colors in workflow dags.

Prior to this, colours and labels for jobs in the two DAG graphs were
jumbled up.

The code incorrectly assumed that iterating over a dict would iterate in
the order that the entries appear in a hard-coded literal dict in the
source tree.

For example:

The code assumed that the first element added to the node_traces list, and
thus the element referred to by node_traces[0], would be the element in
groups_list with value (0,...)

This incorrect assumption led to adding DAG nodes to the wrong Scatter
object.

This patch changes the way that the node_traces list is constructed by
populating it with marker places and then using index notation to set the
appropriate entry in the list based on hard-coded index.

Further, that wrong scatter object then had its colour mutated. This patch
sets the colour correctly on initial creation and does not change it again.

For the DAG coloured by state plot, the naming if-statement had no final
else: case, leading to some nodes being incorrectly labelled. This patch
adds an 'Unknown' case which is reached, for example, in jobs which are
never launched due to dependency failure.

* fix flake8

* remove now-unused nodes_traces initialisation

* factor out index,colour lookup because it is no longer mode-sensitive

* remove binding of color that is no longer needed

* adding new general Parsl info to README

* removing 'simply'

Status translations are done by the superclass TorqueProvider, using
the translations in parsl.providers.torque.torque.translate_table

That table is identical to the table being removed.

… certain state before. This check is nondeterministic and depends on the rate at which the job state changes as well as the interval at which the queuing system is polled. It is entirely possible for a job to very quickly move from queued -> running -> failed -> removed. If the check above is kept, such a job will eternally be reported as PENDING. For example: (#1492)

submit():
	status = 'PENDING'
job runs and is removed from queue
status():
	if status in ['RUNNING', 'KILLING', 'EXITING']: # false, since status == 'PENDING'
		status = 'EXITING'
	# status == 'PENDING'

…ders. This may or may not be the right thing, so this is more of a question mark in PR #1420. This is, therefore, more intricately tied with the job states enum from pr #1420 and makes little sense by itself. That said, we should decide whether SUBMITTING is/should be different from PENDING and, if not, merge this PR. (#1491)

* add exception in monitoring db insert/update

* change the order of logger exception and rollback

* add try catch around rollback

* Run CI testing against Python 3.8

* Bring in python 3.8 changes from corresponding source file in ipyparallel

* undo whitespace change from re-import of ipyparallel source

Prior to this commit, the probe will run for the entire timeout
period. This results in slow worker startup if the timeout is
manually changed to something much longer.

Designed to standardize launching tasks from within containers, etc

… mypy 0.750 onwards (#1481)

…#1441)

This makes mypy happier.

It is unnecessarily complicated to use a list in one part and
a set in another part. Either section might produce duplicates
which need removing - for example, multiple interfaces might
have the same address.

This will test that the code is at least able to start up
and execute in a simple setup. It will not test any of the
more interesting functionality of the ad hoc provider.

… launcher failures are reported back to the user. (#1466)

There are two types of errors that this will catch: errors in the assumptions of the launcher script (such as filesystem errors while writing sub-scripts or missing mpirun/srun/parallel/etc.) and application/block errors if the launching command propagates them. Both are useful to be reported. Moreover, from the school of thought of "all errors should be appropriately handled or reported", this is a must.

* Change parsl.auto filenames to component based names

This makes it easier to understand where parsl.auto files
are coming from, and what might be expected in them. This
has proved useful in some debugging situations.

* add torque

* Switching over to Mac safe queue